[U-Boot] [PATCH] Allow for parallel builds and saved output

[Andy Fleming]

On Sun, Nov 20, 2011 at 11:15 PM, Simon Glass <sjg at chromium.org> wrote:
> Hi Mike,
>
> On Sun, Nov 20, 2011 at 1:23 PM, Mike Frysinger <vapier at gentoo.org> wrote:
>> On Thursday 03 November 2011 03:28:29 Andy Fleming wrote:
>>
>> you don't need those semicolons.  also, this is not as good as it should be:
>> if you're running 10 jobs in parallel, you fork 10, and then you wait for all
>> of them to finish before forking another set of 10.
>>
>> what you could do is something like:
>>        JOB_IDX=0
>>        JOB_IDX_FIRST=0
>>        BUILD_NBUILDS=1
>>
>>        ... foreach target ... ; do
>>                build_target &
>>                pids[$(( JOB_IDX++ ))]=$!
>>                if [ $(( JOB_IDX - JOB_IDX_FIRST )) -ge ${BUILD_NBUILDS} ] ; then
>>                        wait ${pids[$(( JOB_IDX_FIRST++ ))]}
>>                fi
>>        done
>>        wait
>>
>> this isn't perfect as it assumes the first job launched will always finish first,
>> but it's a lot closer than the current code.
>
> Since all the jobs are launched at the same time this is not really a
> valid assumption is it? IMO on this point it's good enough as it is
> for now...

Mike's idea led me to a slightly crazier one that (I think) works.
Basically, I have each build_targets invocation create a file named
LOG/build_N when it is done, where N is the value of TOTAL_CNT when it
is invoked. Then I keep track of the oldest uncompleted build #. When
we hit our build limit, the script scans for LOG/build_N, for N from
oldest to TOTAL_CNT, and updates the count of outstanding builds, as
appropriate (and moves the index of the oldest). If the scanning
completes without freeing up room for another build, it waits for a
second, and then re-scans.

I tried this on my 24-core system with BUILD_NBUILDS set to 8 and
BUILD_NCPUS set to 4, and pegged the system at ~24-30 the whole time.

I will note that it only shaved about a minute off the 15 minutes it
took to build all 85xx. I'm still debugging a bit (I want to try 16
builds/2 cores, and clean up a bit), but I figured I'd ask what people
thought of this approach.

My next patch will also address Mike's comments. If I'm feeling
inspired, I might address some of your suggestions (I think I can use
the build-completion files to coalesce some of the debug output).

Andy