[U-Boot] [PATCH] Allow for parallel builds and saved output

[Andy Fleming]

On Mon, Nov 21, 2011 at 1:01 PM, Mike Frysinger <vapier at gentoo.org> wrote:
> On Monday 21 November 2011 01:28:50 Andy Fleming wrote:
>> On Sun, Nov 20, 2011 at 11:15 PM, Simon Glass wrote:
>> > On Sun, Nov 20, 2011 at 1:23 PM, Mike Frysinger wrote:
>> >> On Thursday 03 November 2011 03:28:29 Andy Fleming wrote:
>> >>
>> >> you don't need those semicolons.  also, this is not as good as it should
>> >> be: if you're running 10 jobs in parallel, you fork 10, and then you
>> >> wait for all of them to finish before forking another set of 10.
>> >>
>> >> what you could do is something like:
>> >>        JOB_IDX=0
>> >>        JOB_IDX_FIRST=0
>> >>        BUILD_NBUILDS=1
>> >>
>> >>        ... foreach target ... ; do
>> >>                build_target &
>> >>                pids[$(( JOB_IDX++ ))]=$!
>> >>                if [ $(( JOB_IDX - JOB_IDX_FIRST )) -ge ${BUILD_NBUILDS}
>> >> ] ; then wait ${pids[$(( JOB_IDX_FIRST++ ))]}
>> >>                fi
>> >>        done
>> >>        wait
>> >>
>> >> this isn't perfect as it assumes the first job launched will always
>> >> finish first, but it's a lot closer than the current code.
>> >
>> > Since all the jobs are launched at the same time this is not really a
>> > valid assumption is it? IMO on this point it's good enough as it is
>> > for now...
>>
>> Mike's idea led me to a slightly crazier one that (I think) works.
>> Basically, I have each build_targets invocation create a file named
>> LOG/build_N when it is done, where N is the value of TOTAL_CNT when it
>> is invoked. Then I keep track of the oldest uncompleted build #. When
>> we hit our build limit, the script scans for LOG/build_N, for N from
>> oldest to TOTAL_CNT, and updates the count of outstanding builds, as
>> appropriate (and moves the index of the oldest). If the scanning
>> completes without freeing up room for another build, it waits for a
>> second, and then re-scans.
>
> sounds like it would work, but i'd want to see how much code it takes to
> actually implement this
>
>> I tried this on my 24-core system with BUILD_NBUILDS set to 8 and
>> BUILD_NCPUS set to 4, and pegged the system at ~24-30 the whole time.
>>
>> I will note that it only shaved about a minute off the 15 minutes it
>> took to build all 85xx.
>
> compare to what ?  your original patch, or my tweaked version ?

Compared to my original. However, I think that may be due to my
configuration. 8 builds with -j4 makes was 14, but 16 builds with -j2
resulted in just over 10 minutes.  I'm doing more testing.

Andy