bugmake - Bugs: bug #47392, should document --jobserver-fds...

 
 

bug #47392: should document --jobserver-fds and the jobserver pipe

Submitter:  Manoj Srivastava <srivasta>
Submitted:  Fri 11 Mar 2016 06:13:34 PM UTC
   
 
Severity:  3 - Normal Item Group:  Documentation
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  4.2 Operating System:  None
Fixed Release:  4.2 Triage Status:  Medium Effort
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 02 Apr 2016 08:43:41 PM UTC, comment #2: 

I've looked through the latest master commits for GCC and I don't believe the integration with GNU make's jobserver is actually implemented yet.

However I added a chapter into the GNU make manual discussing integration of GNU make into other build environments and one section of this discusses integration with the jobserver.

Note that I also took this opportunity, before publishing any official documentation, to rename the internal command line option from --jobserver-fds which is specific to POSIX systems, to --jobserver-auth which is more generic.  However the format and use patterns are the same.

These changes will appear in the next version of GNU make.

Paul D. Smith <psmith>
Group administrator
Sat 12 Mar 2016 06:11:44 AM UTC, comment #1: 

I don't think the man page is the correct place for this sort of detailed, internal documentation.  I'll look into adding it to the manual.  My only concern is that by documenting it we'll be standardizing the interface, perhaps in areas we would like to change in the future.

Paul D. Smith <psmith>
Group administrator
Fri 11 Mar 2016 06:13:34 PM UTC, original submission:  

Hi.

   This enhancement request was filed by a Debian user. Please retain a CC to -email is unavailable- to ensure that the Debian BTS retains a copy of your response.

trying to build a program that, like gcc's LTO linker, can
integrate with make's jobserver.  gcc's LTO linker can run multiple
threads in parallel, and can use make's jobserver pipe to manage how
many threads it runs at a time.  If you pass -flto=jobserv to gcc, and
tell make to treat gcc as a sub-make, gcc will respect make's jobserver
pipe when spawning its threads.

That seems like a really good idea for any program used as part of a
build process and capable of running multiple parallel threads for
performance.  And conversely, any other build system capable of spawning
off other programs and running multiple jobs in parallel could follow
the same convention, to interoperate with make.

So, I wanted to learn how to build a program that could interoperate
with make like GCC did.  I figured out most of the details, but the
interface has some subtleties.  For instance, when make spawns a
sub-make (or any other program launched with '+' and acting like a
sub-make), the sub-make can always run one "free" job in addition to one
job per byte read from the jobserver pipe, and should not write back a
byte when that "free" job completes, because the parent make (or some
ancestor make) will have consumed a byte (or used its free job) to run
the sub-make.

This kind of subtlety makes the interface worth documenting.  Some
third-party articles document the jobserver, but make's own manpage
ought to do so as well.


http://make.mad-scientist.net/papers/jobserver-implementation/ provides
full documentation.

For the manpage, I'd suggest a few additions.

First, the documentation for -j should add a sentence "When make invokes
a sub-make, all instances of make will coordinate to run the specified
number of jobs at a time; see PARALLEL MAKE AND THE JOBSERVER for
details."

Second, in the options section right after -j, add a single entry
"--jobserver-fds R,W" "Internal option make uses to pass the jobserver
pipe read and write file descriptor numbers to sub-makes; see PARALLEL
MAKE AND THE JOBSERVER for details."

Finally, the manpage should include that referenced section "PARALLEL
MAKE AND THE JOBSERVER", giving a couple of paragraphs explaining the
jobserver pipe (and interaction with '+', $(MAKE), and $(MAKEFLAGS)).

  Manoj

Manoj Srivastava <srivasta>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #36608:  0001-document-jobserver-fds-Update-the-man-page.patch added by srivasta (3KiB - text/x-patch - Debian's proposed patch for the manual page)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by srivasta (Submitted the item)
  • -email is unavailable- added by srivasta (Debian BTS)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-04-02 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Fixed ReleaseNone 4.2
        Triage StatusNone Medium Effort
    2016-03-11 srivasta Attached File- Added 0001-document-jobserver-fds-Update-the-man-page.patch, #36608
    2016-03-11 srivasta Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code