bugMIT/GNU Scheme - Bugs: bug #24338, Build mit-scheme in parallel using...

 
 

bug #24338: Build mit-scheme in parallel using make -j2

Submitter:  Ryan Carsten Schmidt <ryandesign>
Submitted:  Sun 21 Sep 2008 08:49:43 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Feature request
Status:  In Progress Privacy:  Public
Assigned to:  None Originator Name: 
Open/Closed:  Open
Keywords: 
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 10 Jun 2020 02:40:56 PM UTC, comment #12: 

This is an issue for Debian, which would like a highly parallel build. In the past, I noticed two race conditions. One was in the documentation, where temporary files were colliding. The other was in building x11-compat where a parallel build can result in it being built twice and the two can interfere.

Barak A. Pearlmutter <barak>
Mon 30 Sep 2013 11:04:33 PM UTC, comment #11: 

This is what I did for the native build already.  See the target `all-native' and its dependencies for details.  I left the parts of the makefile that are for the liarc and svm builds intact because they are a little more complicated and I didn't have time to test or debug them.  Also, they have some additional bootstrapping steps, although we ought to do those for the native build too.

Taylor R. Campbell <riastradh>
Group Member
Mon 30 Sep 2013 10:00:12 PM UTC, comment #10: 

my guess is that the best approach is to write a new Makefile from scratch instead of trying to fix the existing infrastructure - optimally a non-recursive one, but since the build is split into several stages, each stage could use a different makefile with no harm, as long as it does not start recursing itself.

George Skabinsky <gska>
Fri 30 Aug 2013 06:22:30 PM UTC, comment #9: 

Native builds support `make -j' now.  The liarc build will need some more work.  Anyone want to take a stab at this?

Taylor R. Campbell <riastradh>
Group Member
Wed 13 May 2009 05:11:09 AM UTC, comment #8: 

Taking a look at the paper: "Recursive Make considered Harmful"  [1], I feel that I might want to analyse this issue, in light of some of the points put forward there.

By any chance, has that been looked at?



[1].http://aegis.sourceforge.net/auug97.pdf

Amit <cornucopic>
Group Member
Fri 08 May 2009 06:14:11 PM UTC, comment #7: 

Scheme is built in several large-scale stages that have coarse-grained dependencies.  For the C distribution, a bootstrap compiler must be built out of the pregenerated C code; then the bootstrap compiler must be run to compile the Scheme code; then the C code it generated for the final result must be compiled.  These stages must happen in precisely that sequence, because each stage depends on the product of the previous stage, but within the stages involving compiling C code, the order in which the C compiler is invoked for each C file doesn't matter.

Right now, the top-level makefile and its associated scripts do not exactly reflect the dependency graph; instead they reflect a precomputed topological sort of that dependency graph.  Further, they don't pass `-j' options on to subdirectories when making them.  However, just changing each script to write `make -j 2' rather than `make' is the wrong approach.

Taylor R. Campbell <riastradh>
Group Member
Fri 08 May 2009 04:33:42 PM UTC, comment #6: 

Riastradh,

I learn from the top level 'Makefile' that it recurs into the sub-directories where a recursive 'make' in performed in each of them. So, (please correct me if I am wrong), each of these 'Makefiles' all are responsible for their own dependency resolution and the top-level Makefile do not reflect that. So, from your earlier comment, it looks like there has to be changes made into each of these 'Makefiles' to enable a proper parallel make.


Amit <cornucopic>
Group Member
Fri 08 May 2009 03:43:52 PM UTC, comment #5: 

ryandesign, for changes that you may need to do for building with 'make -jN', please refer to http://paste.lisp.org/display/79645

Amit <cornucopic>
Group Member
Thu 07 May 2009 10:07:42 AM UTC, comment #4: 

Sorry for not responding to your Sept 30, 2008 comment sooner.

Yes, a MacPorts port for mit-scheme was created, and I committed it:

http://trac.macports.org/ticket/16605

I don't know anything about scheme so I don't feel well-placed to respond to your request to have separate C and native backends ports. You can file an issue in the MacPorts issue tracker about this if you wish.

I did not know how to "change the invocations of make to be sequential" so I just turned off parallel building.

Ryan Carsten Schmidt <ryandesign>
Tue 05 May 2009 07:12:27 PM UTC, comment #3: 

I spent a little time several weeks ago trying to do this better, but doing it right requires some very careful changes to the way that the top-level makefile works so that the dependency graph, and only the dependency graph, reflects the order in which everything is built.  Currently some of this information is encoded in the commands that each target executes.  Also, it would be nice for whatever changes are made to work with both GNU make and BSD make (and any POSIX make), but this incurs further complications that made me give up at the time.

Taylor R. Campbell <riastradh>
Group Member
Tue 05 May 2009 09:31:35 AM UTC, comment #2: 

I tried the suggestions given by riastradh and made an additional change to 'funcs.sh' and I successfully built the portable C sources on a dual core box. With -j2 the build took less than 50% of the time taken usually.

I want to work towards a complete fix of this. Wondering what would be a 'complete' fix for this bug? Thanks.


Amit <cornucopic>
Group Member
Tue 30 Sep 2008 07:55:10 PM UTC, comment #1: 

This is probably because when etc/make-liarc.sh runs make, it gives three targets which make tries to execute simultaneously because the makefile explicates no dependency relation between them.  If you change the invocations of make to be sequential, does that help?

Actually taking advantage of multiple jobs would take a bit more work.  You will probably also have to change etc/make-in-subdirs.sh to pass -j, and some other invocations of make such as those in the top-level makefile, but I don't know what else will require changes, and there may be other parts of the build that interact poorly with multiple make jobs.

Also, are you preparing a MacPorts portfile for MIT Scheme, or has someone else done that?  It might be nice to provide separate ports for the native and C back ends.  I did this at one point locally, but never submitted it to MacPorts, and don't remember where I placed the results.

Taylor R. Campbell <riastradh>
Group Member
Sun 21 Sep 2008 08:49:43 PM UTC, original submission:  

mit-scheme takes a long time to compile, so it would be nice to be able to use both of the CPU cores my computer has available. I'm building using MacPorts which uses the script src/etc/make-liarc.sh. I tried patching the run_make function in src/etc/functions.sh to call

run_cmd make -j2 "${@}"

to build using two CPU cores, but it encountered errors. The output of that attempt is attached.

I note it says this a few times:

make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.

Then later it fails with:

In file included from bkpt.c:31:
scheme.h:34:20: error: config.h: No such file or directory


Ryan Carsten Schmidt <ryandesign>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #16537:  mit-scheme-j2.txt added by ryandesign (44KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by barak (Posted a comment)
  • -email is unavailable- added by gska (Posted a comment)
  • -email is unavailable- added by cornucopic (Posted a comment)
  • -email is unavailable- added by riastradh (Posted a comment)
  • -email is unavailable- added by ryandesign (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-08-30 riastradh StatusNone In Progress
    2008-09-21 ryandesign Attached File- Added mit-scheme-j2.txt, #16537

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code