bugMIT/GNU Scheme - Bugs: bug #24338, Build mit-scheme in parallel using...

 
 

bug #24338: Build mit-scheme in parallel using make -j2

Submitted by:  Ryan Schmidt <ryandesign>
Submitted on:  Sun 21 Sep 2008 08:49:43 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Feature request
Status: In ProgressPrivacy: Public
Assigned to: NoneOriginator Name: 
Open/Closed: Open
Keywords: 

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Mon 30 Sep 2013 11:04:33 PM UTC, comment #11:

This is what I did for the native build already. See the target `all-native' and its dependencies for details. I left the parts of the makefile that are for the liarc and svm builds intact because they are a little more complicated and I didn't have time to test or debug them. Also, they have some additional bootstrapping steps, although we ought to do those for the native build too.

Taylor R. Campbell <riastradh>
Project Member
Mon 30 Sep 2013 10:00:12 PM UTC, comment #10:

my guess is that the best approach is to write a new Makefile from scratch instead of trying to fix the existing infrastructure - optimally a non-recursive one, but since the build is split into several stages, each stage could use a different makefile with no harm, as long as it does not start recursing itself.

George Skabinsky <gska>
Fri 30 Aug 2013 06:22:30 PM UTC, comment #9:

Native builds support `make -j' now. The liarc build will need some more work. Anyone want to take a stab at this?

Taylor R. Campbell <riastradh>
Project Member
Wed 13 May 2009 05:11:09 AM UTC, comment #8:

Taking a look at the paper: "Recursive Make considered Harmful" [1], I feel that I might want to analyse this issue, in light of some of the points put forward there.

By any chance, has that been looked at?

[1].http://aegis.sourceforge.net/auug97.pdf

Amit <cornucopic>
Project Member
Fri 08 May 2009 06:14:11 PM UTC, comment #7:

Scheme is built in several large-scale stages that have coarse-grained dependencies. For the C distribution, a bootstrap compiler must be built out of the pregenerated C code; then the bootstrap compiler must be run to compile the Scheme code; then the C code it generated for the final result must be compiled. These stages must happen in precisely that sequence, because each stage depends on the product of the previous stage, but within the stages involving compiling C code, the order in which the C compiler is invoked for each C file doesn't matter.

Right now, the top-level makefile and its associated scripts do not exactly reflect the dependency graph; instead they reflect a precomputed topological sort of that dependency graph. Further, they don't pass `-j' options on to subdirectories when making them. However, just changing each script to write `make -j 2' rather than `make' is the wrong approach.

Taylor R. Campbell <riastradh>
Project Member
Fri 08 May 2009 04:33:42 PM UTC, comment #6:

Riastradh,

I learn from the top level 'Makefile' that it recurs into the sub-directories where a recursive 'make' in performed in each of them. So, (please correct me if I am wrong), each of these 'Makefiles' all are responsible for their own dependency resolution and the top-level Makefile do not reflect that. So, from your earlier comment, it looks like there has to be changes made into each of these 'Makefiles' to enable a proper parallel make.

Amit <cornucopic>
Project Member
Fri 08 May 2009 03:43:52 PM UTC, comment #5:

ryandesign, for changes that you may need to do for building with 'make -jN', please refer to http://paste.lisp.org/display/79645

Amit <cornucopic>
Project Member
Thu 07 May 2009 10:07:42 AM UTC, comment #4:

Sorry for not responding to your Sept 30, 2008 comment sooner.

Yes, a MacPorts port for mit-scheme was created, and I committed it:

http://trac.macports.org/ticket/16605

I don't know anything about scheme so I don't feel well-placed to respond to your request to have separate C and native backends ports. You can file an issue in the MacPorts issue tracker about this if you wish.

I did not know how to "change the invocations of make to be sequential" so I just turned off parallel building.

Ryan Schmidt <ryandesign>
Tue 05 May 2009 07:12:27 PM UTC, comment #3:

I spent a little time several weeks ago trying to do this better, but doing it right requires some very careful changes to the way that the top-level makefile works so that the dependency graph, and only the dependency graph, reflects the order in which everything is built. Currently some of this information is encoded in the commands that each target executes. Also, it would be nice for whatever changes are made to work with both GNU make and BSD make (and any POSIX make), but this incurs further complications that made me give up at the time.

Taylor R. Campbell <riastradh>
Project Member
Tue 05 May 2009 09:31:35 AM UTC, comment #2:

I tried the suggestions given by riastradh and made an additional change to 'funcs.sh' and I successfully built the portable C sources on a dual core box. With -j2 the build took less than 50% of the time taken usually.

I want to work towards a complete fix of this. Wondering what would be a 'complete' fix for this bug? Thanks.

Amit <cornucopic>
Project Member
Tue 30 Sep 2008 07:55:10 PM UTC, comment #1:

This is probably because when etc/make-liarc.sh runs make, it gives three targets which make tries to execute simultaneously because the makefile explicates no dependency relation between them. If you change the invocations of make to be sequential, does that help?

Actually taking advantage of multiple jobs would take a bit more work. You will probably also have to change etc/make-in-subdirs.sh to pass -j, and some other invocations of make such as those in the top-level makefile, but I don't know what else will require changes, and there may be other parts of the build that interact poorly with multiple make jobs.

Also, are you preparing a MacPorts portfile for MIT Scheme, or has someone else done that? It might be nice to provide separate ports for the native and C back ends. I did this at one point locally, but never submitted it to MacPorts, and don't remember where I placed the results.

Taylor R. Campbell <riastradh>
Project Member
Sun 21 Sep 2008 08:49:43 PM UTC, original submission:

mit-scheme takes a long time to compile, so it would be nice to be able to use both of the CPU cores my computer has available. I'm building using MacPorts which uses the script src/etc/make-liarc.sh. I tried patching the run_make function in src/etc/functions.sh to call

run_cmd make -j2 "${@}"

to build using two CPU cores, but it encountered errors. The output of that attempt is attached.

I note it says this a few times:

make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.

Then later it fails with:

In file included from bkpt.c:31:
scheme.h:34:20: error: config.h: No such file or directory

Ryan Schmidt <ryandesign>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #16537:  mit-scheme-j2.txt added by ryandesign (44kB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gska (Posted a comment)
  • -unavailable- added by cornucopic (Posted a comment)
  • -unavailable- added by riastradh (Posted a comment)
  • -unavailable- added by ryandesign (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 30 Aug 2013 06:22:30 PM UTCriastradhStatusNone=>In Progress
    Sun 21 Sep 2008 08:49:43 PM UTCryandesignAttached File-=>Added mit-scheme-j2.txt, #16537

    Back to the top


    Powered by Savane 3.1-cleanup