bugmake - Bugs: bug #17873, .NOTPARALLEL enhancements

 
 

bug #17873: .NOTPARALLEL enhancements

Submitted by:  Dave Yost <yost>
Submitted on:  Wed 27 Sep 2006 06:25:32 PM UTC  
Votes:  131  
 
Severity: 3 - NormalItem Group: Enhancement
Status: NonePrivacy: Public
Assigned to: NoneOpen/Closed: Open
Component Version: 3.81Operating System: Any
Fixed Release: NoneTriage Status: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 20 Oct 2012 09:19:49 PM UTC, comment #16:

Another use case for this would be parallel tests where some of the tests are using OpenMP threads. It would be great to be able to say that specific targets take all (or whatever OMP_NUM_THREADS is set to) job slots.

Reinis Danne <abadar>
Fri 06 Jul 2012 03:19:10 PM UTC, comment #15:

I would very much like to see an enhancement along the lines of patch #5108. When building GCC, all of the compiler executables end up getting linked at about the same time. Just linking one is a significant burden on the system, but trying to link compilers for five different languages at the same time makes it impossible to do anything else on the system.

As mentioned in comment #7, trying to serialize these links via dependencies means lying to make and therefore doing unnecessary and undesirable work when you only want to link a single executable.

I note that patch #5108 seems to create a single global mutex, whereas the documentation for the SCO .MUTEX target suggests that each occurrence of .MUTEX creates a separate mutex, so that given

.MUTEX: this that
.MUTEX: yours mine

"this" and "yours" could be built in parallel, but not "this" and "that". I'm not sure that this is how SCO make works, but it seems useful to have multiple mutexes. Hard disk load is not the only resource that you might want a mutex to control: comment #8 gives another example, of latex temporary output files.

Jason Merrill <jasonm>
Thu 31 May 2012 04:50:00 AM UTC, comment #14:

Another scenario wherein sequential processing of dependencies is useful:

all: subdirs $(TARGETS)

subdirs:
(cd dir1 && make)
(cd dir2 && make)

and the $(TARGETS) depend upon libraries (.a files) that are
made by dir1 and/or dir2. If you do this with make -j, then
make immediately evaluates each $(TARGET) before the "make subdirs" gets a chance to touch/update any of the .a
files. Make therefore decides that nothing is out of date regarding the link steps, and so each link node in the build tree gets retired without doing anything. Issuing a second "make" command later, however, causes it to re-link the $(TARGETS) because now one or more .a files is newer than the corresponding $(TARGETS).

It is a pain to have to introduce time-stamp-files just to
force the links to wait until after the "subdirs" has
finished. Phony targets (e.g., install, subdirs, etc.)
often tend to involve recipes that need to be done serially
-- especially when stringing several together as dependencies
of another phony target. The previous example of:

reinstall: uninstall install

is precisely this sort of "dependency encoded sequential recipe".

The syntax is negotiable. But so long as -j exists, there will be a need to specify series/parallel constraints within dependency lists. The alternatives (e.g., time stamp files) are simply too horrible to contemplate.

Some would argue that the antidote to the above scenario is this:

$(TARGETS) : subdirs

But this gets rid of the ability to do "make target1" without forcing make to also do the "subdirs". In some cases the developer knows that nothing in the huge sub-tree of subdirs has changed and all he/she needs to do is re-link the local executable. The $(TARGETS) : subdirs kludge effectively eliminates this capability.

David Warme <davidwarme>
Wed 30 Nov 2011 06:54:32 PM UTC, comment #13:

> I don't see any functionality asked for in this bug, that cannot be emulated with order-only prerequisites. What am I missing?


One situation where I could use such a feature is to implement, e.g., a "reinstall" target like this:

reinstall: uninstall install

This works well with serial make (given suitable uninstall and install targets), but fails with parallel make, of course.

I don't see how to emulate this with order-only prerequisites. If I do:

reinstall: install | uninstall

both are still run in parallel.

If I do:

reinstall: uninstall install
install: | uninstall

then reinstall works as it should, but making just install will now also make uninstall first. That's not how I expected order-only prerequisites to work, but apparently they do. Maybe it's because the targets are phony.

Am I missing/misunderstanding something? Otherwise, I think such an extension would be useful (whatever the syntax).

Frank Heckenbach <frank>
 Spam posted by petividi
 Spam posted by jackbenny
 Spam posted by jackbenny
 Spam posted by jackbenny
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by sub103
 Spam posted by an anonymous
(Why is this post is considered to be spam? Users may have reported it to be spam or, if it has been recently posted, it may just be waiting for spamchecks to be run.)
Spam content:

http://retube.u.yuku.com/ want http://retube.u.yuku.com/ to straddle my hips, and guide this cock of http://retube.u.yuku.com/ into http://retube.u.yuku.com/ hot cunt!”Mandy moved and placed one leg on each side of http://retube.u.yuku.com/ hips.
”She then says, “Thanks, http://urporntube.u.yuku.com/ can come in, and place http://urporntube.u.yuku.com/ on the table next to the tub, if http://urporntube.u.yuku.com/ don’t mind.
”Part four; http://porhub.u.yuku.com/ laid there for a moment, wondering if this was a dream or http://porhub.u.yuku.com/ really did hear Mandy confess http://porhub.u.yuku.com/ love for him.
She pointed out http://pornhubevideo.u.yuku.com/ ’s table to the maitre’d.
‘Then http://megarotic.u.yuku.com/ left http://megarotic.u.yuku.com/ to walk home on my own’.
While http://megaroticcom.u.yuku.com/ is doing that, Mandy continues to clean http://megaroticcom.u.yuku.com/ cock and balls.
Anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
Mon 23 Jun 2008 07:28:33 PM UTC, comment #12:

> What am I missing?


The chance to buy cheap viagra? ;-)

scnr....

Anonymous
Mon 23 Jun 2008 04:54:26 PM UTC, comment #11:

I don't see any functionality asked for in this bug, that cannot
be emulated with order-only prerequisites. What am I missing?

Ralf Wildenhues <rwild>
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
Fri 04 Apr 2008 04:34:22 PM UTC, comment #10:

..oops, went too soon. I was going to say "only have to enter a basic plaintext captcha to posta anon, so it could be easily defeated either manually or programmatically - I think it's always 421.

(DaveK again)

Anonymous
Fri 04 Apr 2008 04:32:34 PM UTC, comment #9:

Nope, not hax0red, it just allows anyone to add comments without logging in. You only have to enter a

Presumably there haven't been enough serious enough spam attacks yet to make anyone want to do anything about it, so bug maintainers are just deleting the occasional ones that show up as they come across them.

cheers,
DaveK

Anonymous
Thu 03 Apr 2008 10:19:55 PM UTC, comment #8:

Because even the most correct dependencies fail to account for programs/scripts that use conflicting temporary/auxiliary files. For example, latex and pdflatex both use the same basename.aux files given the same source file (and with different contents) so they should not be run at the same time. (I frequently build pdf/dvi/ps this way, with .NOTPARALLEL.)

... and what's with these polluted spam URLs, is the system hax0red?

David Fang <fangism>
 Spam posted by an anonymous
 Spam posted by an anonymous
 Spam posted by an anonymous
Thu 12 Oct 2006 01:52:38 PM UTC, comment #7:

To answer Howard Chu: in very simple makefiles, dependencies could take care of the serial/parallel problem, but for complex makefiles they simply don't suffice.

Consider, for example, a makefile that builds 3 independent executables, exe1, exe2, and exe3. The only way to allow for the object files to be compiled in parallel, while linking the executables serially (linking is IO bound and massive parallel linking can lock up a machine), would be to introduce fake dependencies of this sort:

exe1: exe2
exe2: exe3

This is all good (if you forget for now that these dependencies are completely bogus and thus a hack) as long as you always want to build everything. But since the dependencies are fake, problems arise easily. In this particular case, even if all you want to do is build 'exe1', you are forced to build exe2 and exe3 due to the bogus dependencies. Namely,

make exe1

Will also build exe2 and exe3 for absolutely no good reason. Considering that some debug builds of executables I work on end up being 600MB, you can easily waste close to an hour on this, depending on the number of executables. A .NOTPARALLEL with decent semantics (not the current one) solves this problem the way it is supposed to be solved.

Cesar Crusius <cesarcrusius>
Thu 12 Oct 2006 01:44:30 PM UTC, comment #6:

I find the proposed solution confusing, in that it uses a dependency to affect how other dependencies on the same list are built. As far as I know that would be the only use of such semantics in make.

I'd prefer a solution on the lines of patch #5108, as it is more consistent with current behavior. It is also consistent with the solutions adopted by other makes (such as SCO make). Unfortunately .NOTPARALLEL is somewhat stillborn as it is a global switch, and some makefiles may be using it, so we'd have to use some other special target.

Cesar Crusius <cesarcrusius>
Thu 12 Oct 2006 01:33:27 PM UTC, comment #5:

Correction: similar to patch #5108, not bug.

Cesar Crusius <cesarcrusius>
Thu 28 Sep 2006 09:16:51 PM UTC, comment #4:

Similar to bug #5108

Dave Yost <yost>
Thu 28 Sep 2006 08:12:21 PM UTC, comment #3:

Haven't been paying a lot of attention, but when is such a lame thing nas .NOTPARALLEL needed? Why aren't people just writing proper dependencies so that parallel/serial is irrelevant?

Howard Chu <hyc>
Thu 28 Sep 2006 08:03:56 PM UTC, comment #2:

Clarification:

If .NOTPARALLEL is mentioned as a prerequisite .. built serially in the order given, even if ...

Dave Yost <yost>
Thu 28 Sep 2006 07:54:36 PM UTC, comment #1:

This proposal allows us to use make -j not only without worry but also with as much concurrency as possible, on any of a set of makefiles that have been appropriately annotated.

BTW, the doc section is:
http://www.gnu.org/software/make/manual/make.html#index-g_t_002eNOTPARALLEL-248

Dave Yost <yost>
Wed 27 Sep 2006 06:25:32 PM UTC, original submission:

Today all a makefile can do to influence parallel execution is to mark the entire makefile as unsafe for -j by mentioning .NOTPARALLEL as a target.

Finer control is needed. The fix should go something like the following.

.NOTPARALLEL
If .NOTPARALLEL is mentioned as a target, then by default the prerequisites of all targets in this makefile will be built serially, even if the `-j' option is given. If .NOTPARALLEL is mentioned as a prerequisite of a target, then subsequent prerequisites of that target will be built serially, even if the '-j' option is given (and even if .PARALLEL was mentioned previously in this prerequisite list). (See also 5.4 Parallel Execution and 5.7.3 Communicating Options to a Sub-make.)

.PARALLEL
If .PARALLEL is mentioned as a prerequisite of a target, then subsequent prerequisites of that target will be built in parallel if the '-j' option is given, even if .NOTPARALLEL is mentioned as a target (or was mentioned previously in this prerequisite list). (See also 5.4 Parallel Execution and 5.7.3 Communicating Options to a Sub-make.)

Dave Yost <yost>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by abadar (Posted a comment)
  • -unavailable- added by abadar
  • -unavailable- added by jasonm (Posted a comment)
  • -unavailable- added by davidwarme (Posted a comment)
  • -unavailable- added by frank (Posted a comment)
  • -unavailable- added by petividi (Posted a comment)
  • -unavailable- added by jackbenny (Posted a comment)
  • -unavailable- added by sub103 (Posted a comment)
  • -unavailable- added by rwild (Posted a comment)
  • -unavailable- added by fangism (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 131 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 20 Oct 2012 09:19:49 PM UTCabadarCarbon-Copy-=>Added abadar
    Thu 12 Oct 2006 01:53:07 PM UTCcesarcrusiusCarbon-Copy-=>Added cesarcrusius
    Wed 27 Sep 2006 10:55:46 PM UTCyostCarbon-Copy-=>Added yost

    Back to the top


    Powered by Savane 3.1-cleanup