bugmake - Bugs: bug #18124, make-3.81 isn't parallel build safe

 
 

bug #18124: make-3.81 isn't parallel build safe

Submitted by:  None
Submitted on:  Thu 26 Oct 2006 06:15:15 PM UTC  
 
Severity: 3 - NormalItem Group: Bug
Status: FixedPrivacy: Public
Assigned to: Paul D. Smith <psmith>Open/Closed: Closed
Component Version: 3.81Operating System: POSIX-Based
Fixed Release: 3.82Triage Status: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 09 Jun 2009 03:14:20 PM UTC, comment #3:

OK, this is a complicated scenario but Petr Machata did a nice job of explaining it in comment #21 in the Red Hat Bugzilla bug entry. However the fix implemented there seems to me a bit too tricky.

My understanding of the bug is that if (a) we are a submake called in a jobserver context (so we have jobserver-fds set), and (b) we were invoked with an explicit value of -jN to override the jobserver settings and make us a new master server, AND (c) we have to re-exec ourselves due to remaking of makefiles, THEN when we re-exec ourselves we have closed our FDs but we still have --jobserver-fds in MAKEFLAGS. This causes the re-exec'd make to re-run the "make us a new master server" code and re-close those FDs... which were already closed before we re-exec'd and so might now be open for different reasons (directory cache for example).

The change Petr did was to simply not close the FDs unless we were the original master (restarts == 1) but this doesn't feel right to me: sub-masters need to clean up as well.

I made changes so that if we're re-execing and we're a jobserver master, we won't leave stray instances of --jobserver-fds in the MAKEFLAGS variable once we've closed the FDs. This should allow the re-exec to get a "clean" jobserver environment (with -jN on the command line, and no --jobserver-fds). This also has the added bonus that the warning "disabling jobserver mode" is only printed once, not every time we re-exec.

Roland McGrath (IIRC) once emailed me and expressed distaste for the whole idea of the --jobserver-fds option in MAKEFLAGS; he may have been right. I'll have to go dig up his email because I don't, offhand, remember the suggested alternative.

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
 Spam posted by an anonymous
Mon 30 Oct 2006 01:19:08 PM UTC, comment #2:

FYI, there's some conversation on this bug over in the Red Hat Bugzilla database. I don't understand the bug and the patch doesn't enlighten me, so I'm asking for some more detail.

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Fri 27 Oct 2006 01:12:41 PM UTC, comment #1:

Here is a new patch to fix. When make re-execs itself, it calls
clean_jobserver which may close job_fds. After it is re-execed,
it reads job_fds from jobserver_fds again and closes them when
jobserver mode is disabled. It closes the same fd twice. The
second time it closes the wrong file. This patch sets
jobserver_fds_invalid_flag after closing job_fds and checks it
before closing job_fds.

Anonymous
Thu 26 Oct 2006 06:15:15 PM UTC, original submission:

From

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=212111

To reproduce it, after glibc build is done, in glibc build directory:

[hjl@gnu-25 build-x86_64-linux]$ rm -rf iconv
[hjl@gnu-25 build-x86_64-linux]$ make -j4 PARALLELMFLAGS=-j4 > make.log
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
mkdir /export/build/gnu/glibc-nptl-local/build-x86_64-linux/iconv
gconv_open.c: In function ‘__gconv_open’:
gconv_open.c:59: warning: ‘ptr’ may be used uninitialized in this function
gconv_open.c: In function ‘__gconv_open’:
gconv_open.c:59: warning: ‘ptr’ may be used uninitialized in this function
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
No rule to make target
`/export/build/gnu/glibc-nptl-local/build-x86_64-linux/iconv/charmap.o', needed
by `others'
make[1]: *** [iconv/others] Aborted (core dumped)
make: *** [all] Error 2
[hjl@gnu-25 build-x86_64-linux]$

he problem is when start_job_command closes job_fds, it doesn't set
them to -1. Then the same fd is returned by opendir. Later it is used
for pipe again. From there, everything goes down hill.

This patch works for me.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #11093:  make-3.81-job-4.patch added by None (5kB - text/x-patch - An updated patch)
file #11083:  make-3.81-job-1.patch added by None (1015B - text/x-patch - A patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by psmith (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 28 Jul 2010 06:19:19 AM UTCpsmithFixed Release4.0=>3.82
    Tue 09 Jun 2009 03:14:20 PM UTCpsmithStatusNone=>Fixed
      Assigned toNone=>psmith
      Open/ClosedOpen=>Closed
      Fixed ReleaseNone=>4.0
    Fri 27 Oct 2006 01:12:41 PM UTCNoneAttached File-=>Added make-3.81-job-4.patch, #11093
    Thu 26 Oct 2006 06:15:15 PM UTCNoneAttached File-=>Added make-3.81-job-1.patch, #11083

    Back to the top


    Powered by Savane 3.1-cleanup