bugGNU Octave - Bugs: bug #38075, 'make' fails with "Argument...

 
 

bug #38075: 'make' fails with "Argument list too long" message

Submitted by:  Carnë Draug <carandraug>
Submitted on:  Sun 13 Jan 2013 08:32:06 PM UTC  
 
Category: Configuration and Build SystemSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Build Failure
Status: FixedAssigned to: None
Originator Name: Sergei SteshenkoOpen/Closed: Closed
Release: 3.6.2Operating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Thu 28 Feb 2013 06:37:04 PM UTC, comment #22:

Thanks for testing. You were able to reach building the documentation which is beyond where this bug would have surfaced. I will go ahead and commit the patch. Hopefully by the next stable release the other errors you encountered will have been resolved.

Rik <rik5>
Project Administrator
Thu 28 Feb 2013 05:45:48 PM UTC, comment #21:

Now it fails this way:

14836 This is pdfTeXk, Version 3.141592-1.40.3 (Web2C 7.5.6)
14837 file:line:error style messages enabled.
14838 entering extended mode
14839 (./octave.texi
14840 (/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/../texinfo.te
14841 x Loading texinfo [version 2009-04-06.11]: pdf, fonts, page headings, tables,
14842 conditionals, indexing, sectioning, toc, environments, defuns, macros,
14843 cross references, insertions, (/usr/share/texmf/tex/generic/epsf/epsf.tex
14844 This is `epsf.tex' v2.7.3 <23 July 2005>
14845 ) localization, formatting, and turning on texinfo input format.) (./octave.aux
14846
14847 ./octave.aux:4743: TeX capacity exceeded, sorry [save size=5000].
14848 @otherbackslash ->@let \=@realbackslash
14849
14850 @xrdef #1#2->{@indexnofonts @turnoffactive
14851 @xdef @safexrefname {#1}}@expanda...
14852 l.4743 @xrdef{doc-uigetdir-snt}{}
14853
14854 No pages of output.
14855 Transcript written on octave.log.
14856 /mnt/sdb8/sergei/AFSWD_debug/20121021/texinfo-4.13/bin/texi2dvi: etex exited with bad status, quitting.
14857 Makefile:2004: *** [octave.dvi] Error 1
14858
14859 #0 octave.dvi at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/Makefile:2004
14860 #1 dvi-am at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/Makefile:2275
14861 #2 dvi at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/Makefile:2273
14862 #3 all-local at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/Makefile:2920
14863 #4 all-am at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/Makefile:2232
14864 #5 all (.PHONY target)
14865 make[3]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter'
14866 Makefile:1774: *** [all-recursive] Error 1
14867
14868 #0 all-recursive at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/Makefile:1774
14869 #1 all (.PHONY target)
14870 make[2]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc'
14871 make[1]: *** [all-recursive] Error 1
14872
14873 #0 all-recursive at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/Makefile:1931
14874 make[1]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+'
14875 Makefile:1820: *** [all] Error 2
14876
14877 #0 all at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/Makefile:1820
14878 make: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+'

Sergei Steshenko <sergstesh>
Thu 28 Feb 2013 02:54:08 PM UTC, comment #20:

I'll try, but present state indicates bugs in 'configure' - FWIW.

I know it's not an official release; this is just a head up from me.

Sergei Steshenko <sergstesh>
Thu 28 Feb 2013 02:25:39 PM UTC, comment #19:

The code I sent is direct from the development branch so there will be unpolished bits. Try adding "--disable-gui --disable-java" to your other configure options so the distracting new bits of code aren't used.

Rik <rik5>
Project Administrator
Thu 28 Feb 2013 10:08:52 AM UTC, comment #18:

I've tried sent by Rik 3.7.2+ snapshot, and the build fails:

"
13408 In file included from ./src/m-editor/file-editor.h:34:0,
13409 from src/main-window.cc:40:
13410 ./src/m-editor/file-editor-tab.h:26:32: fatal error: Qsci/qsciscintilla.h: No such file or directory
13411 compilation terminated.
13412 Makefile:2450: *** [src/src_libgui_src_la-main-window.lo] Error 1
13413
13414 #0 src/src_libgui_src_la-main-window.lo at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/libgui/Makefile:2450
13415 #1 src/libgui-src.la at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/libgui/Makefile:2540
13416 #2 all-am (.PHONY target)
13417 make[3]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/libgui'
13418 make[2]: *** [all] Error 2
13419
13420 #0 all at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/libgui/Makefile:2236
13421 make[2]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/libgui'
13422 make[1]: *** [all-recursive] Error 1
13423
13424 #0 all-recursive at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/Makefile:1931
13425 make[1]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+'
13426 Makefile:1820: *** [all] Error 2
13427
13428 #0 all at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/Makefile:1820
13429 make: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+'
".

Though it's not related to this bug, but why 'scintilla' there ?

Or having an editor is much more important than, say, OO, namespace and packaging issues ?

Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 06:56:44 PM UTC, comment #17:

octave-3.6.4.tar.bz2 is less than 16M long, so splitting it in 2 .. 3 parts using 'split' and then sending it to me as attachments can be a solution.

Maybe Yahoo will accept a single attachment - today's de-facto standard is 20M IIRC.

I do not insist on a URL, but i need a working tarball.

...

If you decide to send to me my Email, add, say, .jpg extension - Yahoo does not accept .tar.bz2, but accepts tar.bz2.jpg

Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 06:46:51 PM UTC, comment #16:

Rik, I saw a notice earlier today about possible downtime for some GNU services due to hardware maintenance.

John W. Eaton <jwe>
Project Administrator
Wed 27 Feb 2013 06:45:16 PM UTC, comment #15:

There is a whole bunch of free file hosting services.

Dropbox ? Yandex ?

Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 06:43:13 PM UTC, comment #14:

If the new release fails, I'll reopen the bug.

It's really strange you can't release a tarball.

In my world project files even didn't have patches, they were always transferred in full from individual developers to the central project tree, so there wasn't even a notion of patch.

Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 06:36:03 PM UTC, comment #13:

Savannah is being oncooperative and I cannot upload even portions of a tarball.

Rik <rik5>
Project Administrator
Wed 27 Feb 2013 06:24:51 PM UTC, comment #12:

Rik, I'd say don't waste your time. If Sergei wants to help out, then he can apply the patch. Or wait to test a future release candidate.

John W. Eaton <jwe>
Project Administrator
Wed 27 Feb 2013 06:18:25 PM UTC, comment #11:

The patch has already been tested by me and it doesn't harm existing builds. I've never encountered the "Argument list too long" message, however, so I wanted to verify it on a system that is experiencing the problem before committing it (In the same vein as proving with log files that the fix works). It's not elegant, but I can try to upload a tarball to the bug report. It will have to be in pieces because there is a limit on the uploaded file size.

Rik <rik5>
Project Administrator
Wed 27 Feb 2013 04:45:08 PM UTC, comment #10:

In my world those who make mistakes (i.e. bugs) first check bug fixes themselves, and only then verified code with _log_ files proving that the fix works are submitted to central project tree.

This is industry adopted practice in VLSI design.

Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 04:40:07 PM UTC, comment #9:

Sergei,

Why don't you exercise your natural freedom to patch the code using Rik's patch?

Jordi Gutiérrez Hermoso <jordigh>
Project Administrator
Wed 27 Feb 2013 04:27:41 PM UTC, comment #8:

If someone makes a standard release tarball named octave-X.Y.Z. tar.bz2 , e.g. octave-3.6.4a.tar.bz2, and the tarball unpacks creating octave-X.Y.Z directory, e.g. octave-3.6.4a , and if that directory has standard 'configure' script, so it is possible to run

./configure ARGS
make

sequence the usual way, I can test it. I.e. the patch should already be applied to the needed files in the tarball.

There used to be

ftp://ftp.octave.org/pub/octave/bleeding-edge

location for such kind of tarballs. If not, I need to know the URL.

Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 06:20:00 AM UTC, comment #7:

I've made a patch that breaks the pattern creation into 3 calls to the shell to work around the argument list length limit. The patterns are now concatenated into a temporary file and sed is called with the pattern file and the input file. I've attached the patch to this bug report. If this fix can be verified by someone else I will commit it.

(file #27537)

Rik <rik5>
Project Administrator
Fri 18 Jan 2013 07:37:35 PM UTC, comment #6:

I backed out the changeset. We will need to look at a different solution. One could use "export VAR_NAME" in common.mk to export the necessary variables into the environment, but that seems tedious as well.

Rik <rik5>
Project Administrator
Fri 18 Jan 2013 06:30:11 PM UTC, comment #5:

Yeah this change isn't working for me, I don't think ENV works here as John asked some comments back. I'm getting empty substitutions in all files affected, for example scripts/plot/gnuplot_binary.m and src/mkoctfile (shell script variant). Wouldn't you have to coerce make to export all needed variables to the subprocess for this perl script to work?

I can finish a build only by exporting GNUPLOT=gnuplot. That fixes gnuplot_binary so the doc build can finish, but I still have a mkoctfile with empty variables.

Mike Miller <mtmiller>
Project Member
Fri 18 Jan 2013 12:11:34 AM UTC, comment #4:

'sed -f pattern_file' won't work because the patterns are not static at configure time, but can be changed at compile time by running 'make myvar=XXX all'. Make is the only one who knows about these changes.

The current approach works because it relies on the shell to do variable expansion. We place the patterns on the command line with 'sed -e "s|pattern|${variable}|' and the shell does a round of variable expansion to make 'sed -e "s|pattern|replacement|"'.

Once I figured that we couldn't use 'sed -f' I just threw it into Perl and I think the code looks easy to maintain. Each pattern and replacement is on it's own line and it's quite straightforward to copy/paste to add a new variable as they come along.

Performance is an issue for this piece of code because it is called 3 times for every invocation of "make all". Even if there is nothing to rebuild, Make has to run this piece of code to verify that it wasn't called with different options which would trigger a compile.

I'll go ahead and commit it.

Rik <rik5>
Project Administrator
Thu 17 Jan 2013 11:45:54 PM UTC, comment #3:

AFAIR, Perl is needed to build Octave anyway (documentation ?).

In the big picture shell invocations under consideration don't take too much time - Octave takes more than an hour to build anyway.

My point is to stick to KISS principle, i.e. the simplest code which reliably resolves the problem.

Sergei Steshenko <sergstesh>
Thu 17 Jan 2013 11:23:26 PM UTC, comment #2:

One sed invocation per substituteion would be bad.

Couldn't this also be a sed script in a file? Does it have to be Perl?

ENV works here? All those values are exported from Make without having to do so explicitly?

Well, if it works and you've already done it, then go ahead and commit the change.

John W. Eaton <jwe>
Project Administrator
Thu 17 Jan 2013 11:07:59 PM UTC, comment #1:

I've attached a changeset which I think is cleaner. It moves the very long list of substitution values off the shell command line and into a file which can therefore grow arbitrarily long. It only makes one callout to the shell (and to Perl) rather than a series of callouts which would each require a new shell invocation and could be slow. I'm adding jwe to the CC list so he can comment.

I agree 100% with you that something should be done to make the build system robust and portable to systems with different shell limitations.

(file #27296)

Rik <rik5>
Project Administrator
Sun 13 Jan 2013 08:32:06 PM UTC, original submission:

Hi

this bug was reported on the mailing list last November and I've kept it marked since. I'm putting it here so it's not forgotten as I'm unlikely to fix it myself.

Building octave-3.6.2 with somewhat longer list of dependencies, 'make' fails:

The failing part in context:

And, in particular, the offending part is 'do_subst_config_vals' which I have attached. As one can see, the macro itself is quite long, and contents of the variables it exapnds (e.g. CPPFLAGS) are also quite long, so it's surprising it hasn't failed earlier.

I performed quick web search regarding "Argument list too long" and came across http://www.in-ulm.de/~mascheck/various/argmax/ . The page has contains a number of useful advices (though it deal with a different situation: command * ) :

How to avoid the limit in a shell:
If command * fails, then you can

  • iterate with the shell:
    • for i in *; do command "$i"; done (simple, completely robust and portable, may be very slow)
  • iterate with find
    • find . -exec command {} \; (simple, completely robust and portable, may be very slow)
    • find . -exec command {} + (optimizes speed)
    • find . -print0|xargs -0 command (optimizes speed, if find doesn't implement "-exec +" but knows "-print0")
    • find . -print|xargs command (if there's no white space in the arguments)

Note: find descends into directories. To avoid that portably,
you can use

    • "find . ! -name . -prune [...]"
  • If the major part of the arguments consists of long, absolute or relative paths, then try to move your actions into the directory:
    • cd /directory/with/long/path; command *
  • And another quick fix may be to match fewer arguments:
    • command [a-e]; command [f-m]; ...

If I understand correctly the offending command, it's a single call to 'sed' with a request to perform multiple substitutions, and those multiple substitutions is the part causing "Argument list too long".

So, the fix seems to be along the

(simple, completely robust and portable, may be very slow)" lines, i.e. multiple calls to 'sed' with single substitution. T

The "fixed" attachment work for me, i.e. the "Argument list too long" has disappeared, Octave builds and 'make check' OK.

The original thread can be found on nabble. I have just made small rewriting of the messages since I have attached the long pieces of code instead of pasting them on the report.

Carnë

Carnë Draug <carandraug>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #27253:  do_subst_config_vals added by carandraug (9kB - application/octet-stream)
file #27254:  do_subst_config_vals_fix added by carandraug (19kB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jordigh (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by sergstesh (Posted a comment)
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by rik5 (Updated the item)
  • -unavailable- added by rik5 (Please review and give a thumbs up or down)
  • -unavailable- added by carandraug (Submitted the item)
  • -unavailable- added by carandraug (original bug reporter)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 16 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 28 Feb 2013 06:37:04 PM UTCrik5StatusReady For Test=>Fixed
      Open/ClosedOpen=>Closed
    Wed 27 Feb 2013 06:19:06 PM UTCrik5Attached File#27296=>Removed
    Wed 27 Feb 2013 06:18:52 PM UTCrik5Attached File#27537=>Removed
    Wed 27 Feb 2013 06:20:00 AM UTCrik5Attached File-=>Added patch.sh_line_length, #27537
      StatusIn Progress=>Ready For Test
    Fri 18 Jan 2013 07:37:35 PM UTCrik5StatusFixed=>In Progress
      Open/ClosedClosed=>Open
    Fri 18 Jan 2013 12:11:34 AM UTCrik5StatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed
    Thu 17 Jan 2013 11:08:21 PM UTCrik5StatusNone=>In Progress
    Thu 17 Jan 2013 11:07:59 PM UTCrik5Attached File-=>Added fix_long_sh_lines.patch, #27296
      Carbon-Copy-=>Added jwe
    Sun 13 Jan 2013 08:32:06 PM UTCcarandraugAttached File-=>Added do_subst_config_vals, #27253
      Attached File-=>Added do_subst_config_vals_fix, #27254
      Carbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup