bugmake - Bugs: bug #18124, make-3.81 isn't parallel build safe

 
 

bug #18124: make-3.81 isn't parallel build safe

Submitter:  None
Submitted:  Thu 26 Oct 2006 06:15:15 PM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  3.81 Operating System:  POSIX-Based
Fixed Release:  3.82 Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 09 Jun 2009 03:14:20 PM UTC, comment #4: 

OK, this is a complicated scenario but Petr Machata did a nice job of explaining it in comment #21 in the Red Hat Bugzilla bug entry.  However the fix implemented there seems to me a bit too tricky.

My understanding of the bug is that if (a) we are a submake called in a jobserver context (so we have jobserver-fds set), and (b) we were invoked with an explicit value of -jN to override the jobserver settings and make us a new master server, AND (c) we have to re-exec ourselves due to remaking of makefiles, THEN when we re-exec ourselves we have closed our FDs but we still have --jobserver-fds in MAKEFLAGS.  This causes the re-exec'd make to re-run the "make us a new master server" code and re-close those FDs... which were already closed before we re-exec'd and so might now be open for different reasons (directory cache for example).

The change Petr did was to simply not close the FDs unless we were the original master (restarts == 1) but this doesn't feel right to me: sub-masters need to clean up as well.

I made changes so that if we're re-execing and we're a jobserver master, we won't leave stray instances of --jobserver-fds in the MAKEFLAGS variable once we've closed the FDs.  This should allow the re-exec to get a "clean" jobserver environment (with -jN on the command line, and no --jobserver-fds).  This also has the added bonus that the warning "disabling jobserver mode" is only printed once, not every time we re-exec.

Roland McGrath (IIRC) once emailed me and expressed distaste for the whole idea of the --jobserver-fds option in MAKEFLAGS; he may have been right.  I'll have to go dig up his email because I don't, offhand, remember the suggested alternative.

 Paul D. Smith <psmith>
Group administrator
  Spam posted by anonymous
Mon 30 Oct 2006 01:19:08 PM UTC, comment #2: 

FYI, there's some conversation on this bug over in the Red Hat Bugzilla database.  I don't understand the bug and the patch doesn't enlighten me, so I'm asking for some more detail.

 Paul D. Smith <psmith>
Group administrator
Fri 27 Oct 2006 01:12:41 PM UTC, comment #1: 

Here is a new patch to fix. When make re-execs itself, it calls
clean_jobserver which may close job_fds.  After it is re-execed,
it reads job_fds from jobserver_fds again and closes them when
jobserver mode is disabled.  It closes the same fd twice. The
second time it closes the wrong file. This patch sets
jobserver_fds_invalid_flag after closing job_fds and checks it
before closing job_fds.

 Anonymous
Thu 26 Oct 2006 06:15:15 PM UTC, original submission:  

From

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=212111

To reproduce it, after glibc build is done, in glibc build directory:

[hjl@gnu-25 build-x86_64-linux]$ rm -rf iconv
[hjl@gnu-25 build-x86_64-linux]$ make  -j4 PARALLELMFLAGS=-j4 > make.log
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
mkdir /export/build/gnu/glibc-nptl-local/build-x86_64-linux/iconv
gconv_open.c: In function ‘__gconv_open’:
gconv_open.c:59: warning: ‘ptr’ may be used uninitialized in this function
gconv_open.c: In function ‘__gconv_open’:
gconv_open.c:59: warning: ‘ptr’ may be used uninitialized in this function
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: warning: -jN forced in submake: disabling jobserver mode.
No rule to make target
`/export/build/gnu/glibc-nptl-local/build-x86_64-linux/iconv/charmap.o', needed
by `others'
make[1]: * [iconv/others] Aborted (core dumped)
make: * [all] Error 2
[hjl@gnu-25 build-x86_64-linux]$

he problem is when start_job_command closes job_fds, it doesn't set
them to -1. Then the same fd is returned by opendir. Later it is used
for pipe again. From there, everything goes down hill.

This patch works for me.

 Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #11093:  make-3.81-job-4.patch added by None (5KiB - text/x-patch - An updated patch)
file #11083:  make-3.81-job-1.patch added by None (1015B - text/x-patch - A patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-07-28 psmith Fixed Release4.0 3.82
    2009-06-09 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Fixed ReleaseNone 4.0
    2006-10-27 None Attached File- Added make-3.81-job-4.patch, #11093
    2006-10-26 None Attached File- Added make-3.81-job-1.patch, #11083

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.14-8eb0.
    Corresponding source code