bugGNU Octave - Bugs: bug #44383, parallel make fails to build...


bug #44383: parallel make fails to build sometimes

Submitted by:  Vladimír Čunát <vcunat>
Submitted on:  Fri 27 Feb 2015 12:20:09 PM UTC  
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Build Failure
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  dev Operating System:  Any

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Sat 28 Feb 2015 07:38:03 PM UTC, comment #7: 

Yeah, I don't know if there is a way to inject our own dependency generation without completely overriding what Automake is doing and having to keep that up to date. And there's still first build problem, you have to build once to get the dependencies in the first place.

I pushed an updated version of my patch that works for me, I ran several test builds and a distcheck. I was even able to trigger this bug by inserting a sleep into the build rule for defaults.h and prove to myself that this fixes at least the problem quoted here.


Mike Miller <mtmiller>
Project Administrator
Fri 27 Feb 2015 07:30:32 PM UTC, comment #6: 

Oh, hmm, I recall that back in the dark ages before we started using automake and libtool, we generated dependency files that included the .df files in addition to the .o files.  So we'd have something like

foo.o foo.df : foo.h bar.h some-other.h ...

Now with automake and libtool, we just get the .lo file as the LHS of the dependency rule in the .Plo files.

I think the way to do this with automake is to override the .cc.lo rule.  If we do that, then I think the dependencies would be accurate and kept up to date automatically.  The only drawback is keeping our custom rules working properly for new and/or multiple versions of automake.  It would be nice if we could somehow insert our own filter into the existing rule, but I don't think there is a way to do that.

John W. Eaton <jwe>
Project Administrator
Fri 27 Feb 2015 07:21:10 PM UTC, comment #5: 

Sure, although I won't get to it until later tonight. Feel free to fix it up and commit it if you want it sooner today.

For your suggestion, are you thinking of some kind of script that would pre-parse the .cc -> .df files and produce a dependency list? Because anything other than that sounds like too much work to try to track all of the individual dependencies manually and keep the Makefiles updated as #includes are added or removed.

Lastly, I think this changes means we can get rid of the "Order matters here" comment, and just alpha-sort the BUILT_SOURCES list now, right?

Mike Miller <mtmiller>
Project Administrator
Fri 27 Feb 2015 07:01:18 PM UTC, comment #4: 

Mike, I think that change is fine.  Could you use the new variable in the definition of BUILT_SOURCES to avoid duplication?  If you don't have time, let me know and I'll make this change.

Or, should we be more specific, so that particular .df files depend on only those files that are actually needed instead of just listing all the generated files as dependencies?

John W. Eaton <jwe>
Project Administrator
Fri 27 Feb 2015 06:35:27 PM UTC, comment #3: 

Is this compilation error random in which file it fails to build?  Or is it always failing at the same file?

I see in your configure there are some warnings about missing Bison.  I'm wondering if some of the missing build components might alter the way/order pre-compilation is done.

What versions of gcc, libtool and other Make tools are you using?  (That info doesn't appear in the configure log.)

Dan Sebald <sebald>
Fri 27 Feb 2015 06:22:47 PM UTC, comment #2: 

And I'm thinking something like the attached to fix this. Still no way to reproduce the build failure here, but this change at least still builds and doesn't introduce any extra build loops for me.

(file #33203)

Mike Miller <mtmiller>
Project Administrator
Fri 27 Feb 2015 05:17:49 PM UTC, comment #1: 

Thanks for your bug report. I am going to mark this confirmed by inspection even though I can't reproduce it myself. Here is the missing dependency path in the libinterp directory:

  • $(BUILT_SOURCES) is built before the compiler is ever invoked
  • some of the $(BUILT_SOURCES) depend on *.df, in particular the files builtins.cc and builtin-defun-decls.h
  • the rule to generate the *.df files involves running the preprocessor on the corresponding source, and that is where the header file in $(BUILT_SOURCES) may not have been generated yet

It would be nice if someone with a lot of cores could reproduce this in libinterp. Try deleting libinterp/corefcn/defaults.h and libinterp/octave-value/ov-java.df and see if a make -j16 triggers this, for example.

Fixing this may mean adding more explicit dependency rules for which stages of the build depend on which generated header files. It's all a bit muddled the way it is declared now. And I suppose it works by accident most of the time simply because builtin-defun-decls.h and builtins.cc are the last two files in the list of $(BUILT_SOURCES).

Mike Miller <mtmiller>
Project Administrator
Fri 27 Feb 2015 12:20:09 PM UTC, original submission:  

It can fail when using heavy parallelism to build.
octave-value/ov-fcn-handle.cc:55:22: fatal error: defaults.h: No such file or directory
 #include "defaults.h"

A full build log can be found on http://hydra.nixos.org/build/19917018/nixlog/1/raw

Vladimír Čunát <vcunat>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #33203:  bug44383.diff added by mtmiller (2KiB - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by vcunat (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-02-28 mtmiller StatusConfirmed => Fixed
        Open/ClosedOpen => Closed
    2015-02-27 mtmiller Attached File- => Added bug44383.diff, #33203
    2015-02-27 mtmiller Item GroupInstallation Failure => Build Failure
        StatusNone => Confirmed
        Release3.8.2 => dev
        Operating SystemGNU/Linux => Any

    Back to the top

    Powered by Savane 3.5