[U-Boot] [PATCH] Allow for parallel builds and saved output

[Simon Glass]

Hi Mike,

On Mon, Nov 21, 2011 at 11:00 AM, Mike Frysinger <vapier at gentoo.org> wrote:
> On Monday 21 November 2011 00:15:21 Simon Glass wrote:
>> On Sun, Nov 20, 2011 at 1:23 PM, Mike Frysinger wrote:
>> > On Thursday 03 November 2011 03:28:29 Andy Fleming wrote:
>> >> +             # Limit number of parallel builds
>> >> +             if [ ${CURRENT_COUNT} -gt ${BUILD_NBUILDS} ] ; then
>> >> +                     CURRENT_COUNT=0;
>> >> +                     wait;
>> >>               fi
>> >
>> > you don't need those semicolons.  also, this is not as good as it should
>> > be: if you're running 10 jobs in parallel, you fork 10, and then you
>> > wait for all of them to finish before forking another set of 10.
>> >
>> > what you could do is something like:
>> >        JOB_IDX=0
>> >        JOB_IDX_FIRST=0
>> >        BUILD_NBUILDS=1
>> >
>> >        ... foreach target ... ; do
>> >                build_target &
>> >                pids[$(( JOB_IDX++ ))]=$!
>> >                if [ $(( JOB_IDX - JOB_IDX_FIRST )) -ge ${BUILD_NBUILDS} ]
>> > ; then wait ${pids[$(( JOB_IDX_FIRST++ ))]}
>> >                fi
>> >        done
>> >        wait
>> >
>> > this isn't perfect as it assumes the first job launched will always
>> > finish first, but it's a lot closer than the current code.
>>
>> Since all the jobs are launched at the same time this is not really a
>> valid assumption is it? IMO on this point it's good enough as it is
>> for now...
>
> my point was, it's about as valid as you can get in bash.  you can't do non-
> blocking wait's in bash, nor can you wait on a list of pids and have it return
> only one at a time.  the launching-jobs-at-the-same-time isn't really the
> worse part: builds can vary greatly in terms of how much code they actually
> compile.  some boards are very small while others are quite fat.
>
> so your choices are:
>        - make a general "good enough" assumption (which i did here)
>        - have the main thread manually poll every child through some IPC mech
> such as looking for files the children touch when they're done, and *then* do
> the wait explicitly on that child

You missed one:

- write it in Python :-)

(which I might try if patman is accepted)

>
> the problem with the latter is that the code tends to get much more
> complicated, and the main thread ends up doing quite a bit of busy work which
> kills CPU cycles.  my original assumption ends up keeping the CPU's fairly
> loaded i think, and is fairly simplistic code.
> -mike
>

It should be easy enough in Python to just maintain 20 threads (or
whatever) and start the next job when a thread becomes idle.

Regards,
Simon