bugmake - Bugs: bug #28134, inconsistent reporting of linux...

 
 

bug #28134: inconsistent reporting of linux kernel builds with -j2

Submitted by:  Phil Carmody <fatphil>
Submitted on:  Mon 30 Nov 2009 06:39:48 PM UTC  
 
Severity: 3 - NormalItem Group: Bug
Status: FixedPrivacy: Public
Assigned to: Paul D. Smith <psmith>Open/Closed: Closed
Component Version: 3.81Operating System: POSIX-Based
Fixed Release: 4.0Triage Status: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 20 Oct 2013 05:52:10 PM UTC, comment #4:

In 4.0 we set stdout and stderr to append mode (for systems where this is supported). So you should not see this problem anymore. Also in 4.0 there is the output-sync feature you can use if you want (but this is not necessary to avoid losing output).

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Fri 15 Jan 2010 06:29:30 AM UTC, comment #3:

As a complete digression, and based on 26 years of doing parallel builds (damn it's been a long time), I just wanted to point out that you can generally use 50% more jobs than your number of CPUs/cores. E.g., for a dual-core machine, use make -j3. Compilation tends to be heavily I/O bound; if you only use as many jobs as you have cores, you will tend to always have at least 30% idle CPU. Bump up the jobs by 50% and you can utilize that extra CPU without overloading the machine. Go beyond that and you tend to lose.

Howard Chu <hyc>
Thu 17 Dec 2009 05:45:52 PM UTC, comment #2:

Well, that may look a little like voodoo, but it seems to work from the few brief tests I gave it (about 4 kernel builds, I think). It even seems to work in the less-well-behaved >>f 2>&1 case that I use.

It's part of an automated build checker, so if there are problems with it, then they will make themselves clear within a week or so.

Thanks!

Phil Carmody <fatphil>
Thu 17 Dec 2009 10:15:41 AM UTC, comment #1:

Try >>/tmp/C_out 2>>/tmp/C_err so that append mode is used so that writes to the files from different processes are atomic and don't overwrite each other.

Ralf Wildenhues <rwild>
Mon 30 Nov 2009 06:39:48 PM UTC, original submission:

This only seems to happen with both parallel builds and with 'sparse' checking (C=1 or C=2). However, it's almost 100% reproducable. The actual build itself seems correct, it's just that stdout and stderr don't tell the whole story about what's being built.

I did make mrproper and a make defconfig to ensure that the system was in an identical test for two builds. (A ls -lR proves that both builds were in an identical clean state.)

Then I used the following command:
make ARCH=arm CROSS_COMPILE=arm-linux- -j2 C=1 V=2 >/tmp/C_out 2>/tmp/C_err

(The V=2 is unnecessary, it just adds more info to the output)
Again I grabbed an ls -lR, cleaned, and made again, capturing stdout and stderr to new files.

I sorted the _out files, and then ran diff:
<pre>
$ diff /tmp/[CD]s_out
872d871
< CC [M] net/mac80211/ibss.o - due to target missing
972d970
< CC net/ipv4/tcp_timer.o - due to target missing
1294d1291
< CHECK drivers/mmc/card/queue.c - due to target missing
1902c1899
< CHECK net/ipv4/tcp_timer.c - due to target missing
---

> CHECK net/ipv4/tcp_timer.c - due to target missing CC net/ipv4/tcp_timer.o - due to target missing

</pre>

As you can see - some output (tcp_timer) is mangled, and some files seem to not be CC'd (ibss.o) or CHECK'ed (queue.c).

However, looking at the final ls -lR, all of the required targets have been build correctly. So for example, ibss.o exists both times.

For my sins, I'm using Ubuntu 9.04 on an Intel Core2Duo system.

Phil Carmody <fatphil>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by psmith (Posted a comment)
  • -unavailable- added by hyc (Posted a comment)
  • -unavailable- added by rwild (Posted a comment)
  • -unavailable- added by fatphil (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 20 Oct 2013 05:52:10 PM UTCpsmithStatusNone=>Fixed
      Assigned toNone=>psmith
      Open/ClosedOpen=>Closed
      Fixed ReleaseNone=>4.0

    Back to the top


    Powered by Savane 3.1-cleanup