bugmake - Bugs: bug #17873, .NOTPARALLEL enhancements

 
 

bug #17873: .NOTPARALLEL enhancements

Submitter:  Dave Yost <yost>
Submitted:  Wed 27 Sep 2006 06:25:32 PM UTC
Votes: 131
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  3.81 Operating System:  Any
Fixed Release:  4.4 Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 02 Jan 2023 07:57:29 PM UTC, comment #48: 

I believe that as of GNU make 4.4, with the .WAIT special prerequisite and enhanced .NOTPARALLEL, the requests here are satisfied.

Paul D. Smith <psmith>
Group administrator
Sat 20 Oct 2012 09:19:49 PM UTC, comment #47: 

Another use case for this would be parallel tests where some of the tests are using OpenMP threads. It would be great to be able to say that specific targets take all (or whatever OMP_NUM_THREADS is set to) job slots.

Reinis Danne <abadar>
Fri 06 Jul 2012 03:19:10 PM UTC, comment #46: 

I would very much like to see an enhancement along the lines of patch #5108.  When building GCC, all of the compiler executables end up getting linked at about the same time.  Just linking one is a significant burden on the system, but trying to link compilers for five different languages at the same time makes it impossible to do anything else on the system.

As mentioned in comment #7, trying to serialize these links via dependencies means lying to make and therefore doing unnecessary and undesirable work when you only want to link a single executable.

I note that patch #5108 seems to create a single global mutex, whereas the documentation for the SCO .MUTEX target suggests that each occurrence of .MUTEX creates a separate mutex, so that given

.MUTEX: this that
.MUTEX: yours mine

"this" and "yours" could be built in parallel, but not "this" and "that".  I'm not sure that this is how SCO make works, but it seems useful to have multiple mutexes.  Hard disk load is not the only resource that you might want a mutex to control: comment #8 gives another example, of latex temporary output files.

Jason Merrill <jasonm>
Thu 31 May 2012 04:50:00 AM UTC, comment #45: 

Another scenario wherein sequential processing of dependencies is useful:

all: subdirs $(TARGETS)

subdirs:
        (cd dir1 && make)
        (cd dir2 && make)

and the $(TARGETS) depend upon libraries (.a files) that are
made by dir1 and/or dir2.  If you do this with make -j, then
make immediately evaluates each $(TARGET) before the "make subdirs" gets a chance to touch/update any of the .a
files.  Make therefore decides that nothing is out of date regarding the link steps, and so each link node in the build tree gets retired without doing anything.  Issuing a second "make" command later, however, causes it to re-link the $(TARGETS) because now one or more .a files is newer than the corresponding $(TARGETS).

It is a pain to have to introduce time-stamp-files just to
force the links to wait until after the "subdirs" has
finished.  Phony targets (e.g., install, subdirs, etc.)
often tend to involve recipes that need to be done serially
-- especially when stringing several together as dependencies
of another phony target.  The previous example of:

reinstall: uninstall install

is precisely this sort of "dependency encoded sequential recipe".

The syntax is negotiable.  But so long as -j exists, there will be a need to specify series/parallel constraints within dependency lists.  The alternatives (e.g., time stamp files) are simply too horrible to contemplate.

Some would argue that the antidote to the above scenario is this:

$(TARGETS) : subdirs

But this gets rid of the ability to do "make target1" without forcing make to also do the "subdirs".  In some cases the developer knows that nothing in the huge sub-tree of subdirs has changed and all he/she needs to do is re-link the local executable.  The $(TARGETS) : subdirs kludge effectively eliminates this capability.

David Warme <davidwarme>
Wed 30 Nov 2011 06:54:32 PM UTC, comment #44: 


> I don't see any functionality asked for in this bug, that cannot be emulated with order-only prerequisites. What am I missing?


One situation where I could use such a feature is to implement, e.g., a "reinstall" target like this:

reinstall: uninstall install

This works well with serial make (given suitable uninstall and install targets), but fails with parallel make, of course.

I don't see how to emulate this with order-only prerequisites. If I do:

reinstall: install | uninstall

both are still run in parallel.

If I do:

reinstall: uninstall install
install: | uninstall

then reinstall works as it should, but making just install will now also make uninstall first. That's not how I expected order-only prerequisites to work, but apparently they do. Maybe it's because the targets are phony.

Am I missing/misunderstanding something? Otherwise, I think such an extension would be useful (whatever the syntax).

Frank Heckenbach <frank>
  Spam posted by petividi
  Spam posted by jackbenny
  Spam posted by jackbenny
  Spam posted by jackbenny
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by sub103
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
Mon 23 Jun 2008 07:28:33 PM UTC, comment #27: 


> What am I missing?


The chance to buy cheap viagra?  ;-)


scnr....

Anonymous
Mon 23 Jun 2008 04:54:26 PM UTC, comment #26: 

I don't see any functionality asked for in this bug, that cannot
be emulated with order-only prerequisites.  What am I missing?

Ralf Wildenhues <rwild>
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
Fri 04 Apr 2008 04:34:22 PM UTC, comment #15: 

..oops, went too soon.  I was going to say "only have to enter a basic plaintext captcha to posta anon, so it could be easily defeated either manually or programmatically - I think it's always 421.

  (DaveK again)

Anonymous
Fri 04 Apr 2008 04:32:34 PM UTC, comment #14: 

Nope, not hax0red, it just allows anyone to add comments without logging in.  You only have to enter a

Presumably there haven't been enough serious enough spam attacks yet to make anyone want to do anything about it, so bug maintainers are just deleting the occasional ones that show up as they come across them.

  cheers,
    DaveK


Anonymous
Thu 03 Apr 2008 10:19:55 PM UTC, comment #13: 

Because even the most correct dependencies fail to account for programs/scripts that use conflicting temporary/auxiliary files.  For example, latex and pdflatex both use the same basename.aux files given the same source file (and with different contents) so they should not be run at the same time.  (I frequently build pdf/dvi/ps this way, with .NOTPARALLEL.)

... and what's with these polluted spam URLs, is the system hax0red?

David Fang <fangism>
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by anonymous
  Spam posted by #66134
Sat 01 Dec 2007 09:44:40 PM UTC, comment #8: 

Hm... I haven't this bug on my system. May be this problem have already been solved?
____________
http://mp3front.com
http://mp3location.com

Invalid User ID <#63237>
Thu 12 Oct 2006 01:52:38 PM UTC, comment #7: 

To answer Howard Chu: in very simple makefiles, dependencies could take care of the serial/parallel problem, but for complex makefiles they simply don't suffice.

Consider, for example, a makefile that builds 3 independent executables, exe1, exe2, and exe3. The only way to allow for the object files to be compiled in parallel, while linking the executables serially (linking is IO bound and massive parallel linking can lock up a machine), would be to introduce fake dependencies of this sort:

exe1: exe2
exe2: exe3

This is all good (if you forget for now that these dependencies are completely bogus and thus a hack) as long as you always want to build everything. But since the dependencies are fake, problems arise easily. In this particular case, even if all you want to do is build 'exe1', you are forced to build exe2 and exe3 due to the bogus dependencies. Namely,

make exe1

Will also build exe2 and exe3 for absolutely no good reason. Considering that some debug builds of executables I work on end up being 600MB, you can easily waste close to an hour on this, depending on the number of executables. A .NOTPARALLEL with decent semantics (not the current one) solves this problem the way it is supposed to be solved.

Cesar Crusius <cesarcrusius>
Thu 12 Oct 2006 01:44:30 PM UTC, comment #6: 

I find the proposed solution confusing, in that it uses a dependency to affect how other dependencies on the same list are built. As far as I know that would be the only use of such semantics in make.

I'd prefer a solution on the lines of patch #5108, as it is more consistent with current behavior. It is also consistent with the solutions adopted by other makes (such as SCO make). Unfortunately .NOTPARALLEL is somewhat stillborn as it is a global switch, and some makefiles may be using it, so we'd have to use some other special target.

Cesar Crusius <cesarcrusius>
Thu 12 Oct 2006 01:33:27 PM UTC, comment #5: 

Correction: similar to patch #5108, not bug.

Cesar Crusius <cesarcrusius>
Thu 28 Sep 2006 09:16:51 PM UTC, comment #4: 

Similar to bug #5108

Dave Yost <yost>
Thu 28 Sep 2006 08:12:21 PM UTC, comment #3: 

Haven't been paying a lot of attention, but when is such a lame thing nas .NOTPARALLEL needed? Why aren't people just writing proper dependencies so that parallel/serial is irrelevant?

Howard Chu <hyc>
Thu 28 Sep 2006 08:03:56 PM UTC, comment #2: 

Clarification:

If .NOTPARALLEL is mentioned as a prerequisite .. built serially in the order given, even if ...

Dave Yost <yost>
Thu 28 Sep 2006 07:54:36 PM UTC, comment #1: 

This proposal allows us to use make -j not only without worry but also with as much concurrency as possible, on any of a set of makefiles that have been appropriately annotated.


BTW, the doc section is:
http://www.gnu.org/software/make/manual/make.html#index-g_t_002eNOTPARALLEL-248

Dave Yost <yost>
Wed 27 Sep 2006 06:25:32 PM UTC, original submission:  

Today all a makefile can do to influence parallel execution is to mark the entire makefile as unsafe for -j by mentioning .NOTPARALLEL as a target.

Finer control is needed. The fix should go something like the following.

.NOTPARALLEL
If .NOTPARALLEL is mentioned as a target, then by default the prerequisites of all targets in this makefile will be built serially, even if the `-j' option is given. If .NOTPARALLEL is mentioned as a prerequisite of a target, then subsequent prerequisites of that target will be built serially, even if the '-j' option is given (and even if .PARALLEL was mentioned previously in this prerequisite list). (See also 5.4 Parallel Execution and 5.7.3 Communicating Options to a Sub-make.)

.PARALLEL
If .PARALLEL is mentioned as a prerequisite of a target, then subsequent prerequisites of that target will be built in parallel if the '-j' option is given, even if .NOTPARALLEL is mentioned as a target (or was mentioned previously in this prerequisite list). (See also 5.4 Parallel Execution and 5.7.3 Communicating Options to a Sub-make.)

Dave Yost <yost>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by abadar (Posted a comment)
  • -email is unavailable- added by abadar
  • -email is unavailable- added by jasonm (Posted a comment)
  • -email is unavailable- added by davidwarme (Posted a comment)
  • -email is unavailable- added by frank (Posted a comment)
  • -email is unavailable- added by petividi (Posted a comment)
  • -email is unavailable- added by jackbenny (Posted a comment)
  • -email is unavailable- added by sub103 (Posted a comment)
  • -email is unavailable- added by rwild (Posted a comment)
  • -email is unavailable- added by fangism (Posted a comment)
  • -email is unavailable- added by #66134 (Posted a comment)
  • -email is unavailable- added by #63237 (Posted a comment)
  •  

    There are 131 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-01-02 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Fixed ReleaseNone 4.4
    2012-10-20 abadar Carbon-Copy- Added abadar
    2006-10-12 cesarcrusius Carbon-Copy- Added cesarcrusius
    2006-09-27 yost Carbon-Copy- Added yost

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code