bugGNU Octave - Bugs: bug #38075, 'make' fails with "Argument...

 
 

bug #38075: 'make' fails with "Argument list too long" message

Submitter:  Carnë Draug <carandraug>
Submitted:  Sun 13 Jan 2013 08:32:06 PM UTC
   
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Build Failure
Status:  Fixed Assigned to:  None
Originator Name:  Sergei Steshenko Open/Closed:  * Closed
Release:  * 3.6.2 Release: 
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 28 Feb 2013 06:37:04 PM UTC, comment #22: 

Thanks for testing.  You were able to reach building the documentation which is beyond where this bug would have surfaced.  I will go ahead and commit the patch.  Hopefully by the next stable release the other errors you encountered will have been resolved.

Rik <rik5>
Group administrator
Thu 28 Feb 2013 05:45:48 PM UTC, comment #21: 

Now it fails this way:

  14836 This is pdfTeXk, Version 3.141592-1.40.3 (Web2C 7.5.6)
  14837  file:line:error style messages enabled.
  14838 entering extended mode
  14839 (./octave.texi
  14840 (/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/../texinfo.te
  14841 x Loading texinfo [version 2009-04-06.11]: pdf, fonts, page headings, tables,
  14842 conditionals, indexing, sectioning, toc, environments, defuns, macros,
  14843 cross references, insertions, (/usr/share/texmf/tex/generic/epsf/epsf.tex
  14844 This is `epsf.tex' v2.7.3 <23 July 2005>
  14845 ) localization, formatting, and turning on texinfo input format.) (./octave.aux
  14846
  14847 ./octave.aux:4743: TeX capacity exceeded, sorry [save size=5000].
  14848 @otherbackslash ->@let \=@realbackslash
  14849
  14850 @xrdef #1#2->{@indexnofonts @turnoffactive
  14851                                            @xdef @safexrefname {#1}}@expanda...
  14852 l.4743 @xrdef{doc-uigetdir-snt}{}
  14853
  14854 No pages of output.
  14855 Transcript written on octave.log.
  14856 /mnt/sdb8/sergei/AFSWD_debug/20121021/texinfo-4.13/bin/texi2dvi: etex exited with bad status, quitting.
  14857 Makefile:2004: * [octave.dvi] Error 1
  14858
  14859 #0  octave.dvi at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/Makefile:2004
  14860 #1  dvi-am at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/Makefile:2275
  14861 #2  dvi at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/Makefile:2273
  14862 #3  all-local at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/Makefile:2920
  14863 #4  all-am at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter/Makefile:2232
  14864 #5  all (.PHONY target)
  14865 make[3]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/interpreter'
  14866 Makefile:1774: * [all-recursive] Error 1
  14867
  14868 #0  all-recursive at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc/Makefile:1774
  14869 #1  all (.PHONY target)
  14870 make[2]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/doc'
  14871 make[1]: * [all-recursive] Error 1
  14872
  14873 #0  all-recursive at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/Makefile:1931
  14874 make[1]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+'
  14875 Makefile:1820: * [all] Error 2
  14876
  14877 #0  all at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/Makefile:1820
  14878 make: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+'

Sergei Steshenko <sergstesh>
Thu 28 Feb 2013 02:54:08 PM UTC, comment #20: 

I'll try, but present state indicates bugs in 'configure' - FWIW.

I know it's not an official release; this is just a head up from me.

Sergei Steshenko <sergstesh>
Thu 28 Feb 2013 02:25:39 PM UTC, comment #19: 

The code I sent is direct from the development branch so there will be unpolished bits.  Try adding "--disable-gui --disable-java" to your other configure options so the distracting new bits of code aren't used.

Rik <rik5>
Group administrator
Thu 28 Feb 2013 10:08:52 AM UTC, comment #18: 

I've tried sent by Rik 3.7.2+ snapshot, and the build fails:

"
  13408 In file included from ./src/m-editor/file-editor.h:34:0,
  13409                  from src/main-window.cc:40:
  13410 ./src/m-editor/file-editor-tab.h:26:32: fatal error: Qsci/qsciscintilla.h: No such file or directory
  13411 compilation terminated.
  13412 Makefile:2450: * [src/src_libgui_src_la-main-window.lo] Error 1
  13413
  13414 #0  src/src_libgui_src_la-main-window.lo at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/libgui/Makefile:2450
  13415 #1  src/libgui-src.la at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/libgui/Makefile:2540
  13416 #2  all-am (.PHONY target)
  13417 make[3]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/libgui'
  13418 make[2]: * [all] Error 2
  13419
  13420 #0  all at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/libgui/Makefile:2236
  13421 make[2]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/libgui'
  13422 make[1]: * [all-recursive] Error 1
  13423
  13424 #0  all-recursive at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/Makefile:1931
  13425 make[1]: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+'
  13426 Makefile:1820: * [all] Error 2
  13427
  13428 #0  all at /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+/Makefile:1820
  13429 make: Leaving directory `/mnt/sdb8/sergei/AFSWD_debug/build/octave-3.7.2+'
".

Though it's not related to this bug, but why 'scintilla' there ?

Or having an editor is much more important than, say, OO, namespace and packaging issues ?

Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 06:56:44 PM UTC, comment #17: 

octave-3.6.4.tar.bz2 is less than 16M long, so splitting it in 2 .. 3 parts using 'split' and then sending it to me as attachments can be a solution.

Maybe Yahoo will accept a single attachment - today's de-facto standard is 20M IIRC.

I do not insist on a URL, but i need a working tarball.

...

If you decide to send to me my Email, add, say, .jpg extension - Yahoo does not accept .tar.bz2, but accepts tar.bz2.jpg

Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 06:46:51 PM UTC, comment #16: 

Rik, I saw a notice earlier today about possible downtime for some GNU services due to hardware maintenance.

John W. Eaton <jwe>
Group administrator
Wed 27 Feb 2013 06:45:16 PM UTC, comment #15: 

There is a whole bunch of free file hosting services.

Dropbox ? Yandex ?

Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 06:43:13 PM UTC, comment #14: 

If the new release fails, I'll reopen the bug.

It's really strange you can't release a tarball.

In my world project files even didn't have patches, they were always transferred in full from individual developers to the central project tree, so there wasn't even a notion of patch.

Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 06:36:03 PM UTC, comment #13: 

Savannah is being oncooperative and I cannot upload even portions of a tarball.

Rik <rik5>
Group administrator
Wed 27 Feb 2013 06:24:51 PM UTC, comment #12: 

Rik, I'd say don't waste your time.  If Sergei wants to help out, then he can apply the patch.  Or wait to test a future release candidate.

John W. Eaton <jwe>
Group administrator
Wed 27 Feb 2013 06:18:25 PM UTC, comment #11: 

The patch has already been tested by me and it doesn't harm existing builds.  I've never encountered the "Argument list too long" message, however, so I wanted to verify it on a system that is experiencing the problem before committing it (In the same vein as proving with log files that the fix works).  It's not elegant, but I can try to upload a tarball to the bug report.  It will have to be in pieces because there is a limit on the uploaded file size.

Rik <rik5>
Group administrator
Wed 27 Feb 2013 04:45:08 PM UTC, comment #10: 

In my world those who make mistakes (i.e. bugs) first check bug fixes themselves, and only then verified code with _log_ files proving that the fix works are submitted to central project tree.

This is industry adopted practice in VLSI design.


Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 04:40:07 PM UTC, comment #9: 

Sergei,

Why don't you exercise your natural freedom to patch the code using Rik's patch?

Jordi Gutiérrez Hermoso <jordigh>
Group Member
Wed 27 Feb 2013 04:27:41 PM UTC, comment #8: 

If someone makes a standard release tarball named octave-X.Y.Z. tar.bz2 , e.g. octave-3.6.4a.tar.bz2, and the tarball unpacks creating octave-X.Y.Z directory, e.g. octave-3.6.4a , and if that directory has standard 'configure' script, so it is possible to run

./configure ARGS
make

sequence the usual way, I can test it. I.e. the patch should already be applied to the needed files in the tarball.

There used to be

ftp://ftp.octave.org/pub/octave/bleeding-edge

location for such kind of tarballs. If not, I need to know the URL.

Sergei Steshenko <sergstesh>
Wed 27 Feb 2013 06:20:00 AM UTC, comment #7: 

I've made a patch that breaks the pattern creation into 3 calls to the shell to work around the argument list length limit.  The patterns are now concatenated into a temporary file and sed is called with the pattern file and the input file.  I've attached the patch to this bug report.  If this fix can be verified by someone else I will commit it.

(file #27537)

Rik <rik5>
Group administrator
Fri 18 Jan 2013 07:37:35 PM UTC, comment #6: 

I backed out the changeset.  We will need to look at a different solution.  One could use "export VAR_NAME" in common.mk to export the necessary variables into the environment, but that seems tedious as well.

Rik <rik5>
Group administrator
Fri 18 Jan 2013 06:30:11 PM UTC, comment #5: 

Yeah this change isn't working for me, I don't think ENV works here as John asked some comments back. I'm getting empty substitutions in all files affected, for example scripts/plot/gnuplot_binary.m and src/mkoctfile (shell script variant). Wouldn't you have to coerce make to export all needed variables to the subprocess for this perl script to work?

I can finish a build only by exporting GNUPLOT=gnuplot. That fixes gnuplot_binary so the doc build can finish, but I still have a mkoctfile with empty variables.

Mike Miller <mtmiller>
Group Member
Fri 18 Jan 2013 12:11:34 AM UTC, comment #4: 

'sed -f pattern_file' won't work because the patterns are not static at configure time, but can be changed at compile time by running 'make myvar=XXX all'.  Make is the only one who knows about these changes.

The current approach works because it relies on the shell to do variable expansion.  We place the patterns on the command line with 'sed -e "s|pattern|${variable}|' and the shell does a round of variable expansion to make 'sed -e "s|pattern|replacement|"'.

Once I figured that we couldn't use 'sed -f' I just threw it into Perl and I think the code looks easy to maintain.  Each pattern and replacement is on it's own line and it's quite straightforward to copy/paste to add a new variable as they come along.

Performance is an issue for this piece of code because it is called 3 times for every invocation of "make all".  Even if there is nothing to rebuild, Make has to run this piece of code to verify that it wasn't called with different options which would trigger a compile.

I'll go ahead and commit it.

Rik <rik5>
Group administrator
Thu 17 Jan 2013 11:45:54 PM UTC, comment #3: 

AFAIR, Perl is needed to build Octave anyway (documentation ?).

In the big picture shell invocations under consideration don't take too much time - Octave takes more than an hour to build anyway.

My point is to stick to KISS principle, i.e. the simplest code which reliably resolves the problem.

Sergei Steshenko <sergstesh>
Thu 17 Jan 2013 11:23:26 PM UTC, comment #2: 

One sed invocation per substituteion would be bad.

Couldn't this also be a sed script in a file?  Does it have to be Perl?

ENV works here?  All those values are exported from Make without having to do so explicitly?

Well, if it works and you've already done it, then go ahead and commit the change.

John W. Eaton <jwe>
Group administrator
Thu 17 Jan 2013 11:07:59 PM UTC, comment #1: 

I've attached a changeset which I think is cleaner.  It moves the very long list of substitution values off the shell command line and into a file which can therefore grow arbitrarily long.  It only makes one callout to the shell (and to Perl) rather than a series of callouts which would each require a new shell invocation and could be slow.  I'm adding jwe to the CC list so he can comment.

I agree 100% with you that something should be done to make the build system robust and portable to systems with different shell limitations.

(file #27296)

Rik <rik5>
Group administrator
Sun 13 Jan 2013 08:32:06 PM UTC, original submission:  

Hi

this bug was reported on the mailing list last November and I've kept it marked since. I'm putting it here so it's not forgotten as I'm unlikely to fix it myself.

Building octave-3.6.2 with somewhat longer list of dependencies, 'make' fails:


   /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.6.2/src/Makefile:9063    File `oct-conf.h' does not exist.
  /mnt/sdb8/sergei/AFSWD_debug/build/octave-3.6.2/src/Makefile:9063     Must remake target `oct-conf.h'.
making oct-conf.h from oct-conf.h.in
make[2]: execvp: /mnt/sdb8/sergei/AFSWD_debug/20121021/bash-4.2/bin/bash: Argument list too long
Makefile:9063: *** [oct-conf.h] Error 127


The failing part in context:


   9063 oct-conf.h: oct-conf.h.in Makefile
   9064         @$(do_subst_config_vals)
   9065


And, in particular, the offending part is 'do_subst_config_vals' which I have attached. As one can see, the macro itself is quite long, and contents of the variables it exapnds (e.g. CPPFLAGS) are also quite long, so it's surprising it hasn't failed earlier.

I performed quick web search regarding "Argument list too long" and came across http://www.in-ulm.de/~mascheck/various/argmax/ . The page has contains a number of useful advices (though it deal with a different situation: command * ) :

How to avoid the limit in a shell:
If command *  fails, then you can

  • iterate with the shell:
    • for i in *; do command "$i"; done     (simple, completely robust and portable, may be very slow)
  • iterate with find
    • find . -exec command {} \;     (simple, completely robust and portable, may be very slow)
    • find . -exec command {} +       (optimizes speed)
    • find . -print0|xargs -0 command      (optimizes speed, if find doesn't implement "-exec +" but knows "-print0")
    • find . -print|xargs command     (if there's no white space in the arguments)


Note: find descends into directories. To avoid that portably,
you can use
** "find . ! -name . -prune [...]"

  • If the major part of the arguments consists of long, absolute or relative paths, then try to move your actions into the directory:
    • cd /directory/with/long/path; command *
  • And another quick fix may be to match fewer arguments:
    • command [a-e]*; command [f-m]*; ...


If I understand correctly the offending command, it's a single call to 'sed' with a request to perform multiple substitutions, and those multiple substitutions is the part causing "Argument list too long".

So, the fix seems to be along the


for i in *; do command "$i"; done


(simple, completely robust and portable, may be very slow)" lines, i.e. multiple calls to 'sed' with single substitution. T

The "fixed" attachment work for me, i.e. the "Argument list too long" has disappeared, Octave builds and 'make check' OK.

The original thread can be found on nabble. I have just made small rewriting of the messages since I have attached the long pieces of code instead of pasting them on the report.

Carnë

Carnë Draug <carandraug>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #27253:  do_subst_config_vals added by carandraug (9KiB - application/octet-stream)
file #27254:  do_subst_config_vals_fix added by carandraug (19KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jordigh (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by sergstesh (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by rik5 (Please review and give a thumbs up or down)
  • -email is unavailable- added by carandraug (Submitted the item)
  • -email is unavailable- added by carandraug (original bug reporter)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 16 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-02-28 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2013-02-27 rik5 Attached File#27296 Removed
    2013-02-27 rik5 Attached File#27537 Removed
    2013-02-27 rik5 Attached File- Added patch.sh_line_length, #27537
        StatusIn Progress Ready For Test
    2013-01-18 rik5 StatusFixed In Progress
        Open/ClosedClosed Open
    2013-01-18 rik5 StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2013-01-17 rik5 StatusNone In Progress
    2013-01-17 rik5 Attached File- Added fix_long_sh_lines.patch, #27296
        Carbon-Copy- Added jwe
    2013-01-13 carandraug Attached File- Added do_subst_config_vals, #27253
        Attached File- Added do_subst_config_vals_fix, #27254
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.15-e6e5.
    Corresponding source code