bugGNU Octave - Bugs: bug #54202, build fails with Bad file...

 
 

bug #54202: build fails with Bad file descriptor with perl 5.28.0 installed

Submitter:  Julian <marrowsuck>
Submitted:  Fri 29 Jun 2018 08:13:21 AM UTC
   
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Build Failure
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 02 Jul 2018 09:22:55 PM UTC, comment #27: 

Also, if anyone needs a temporary workaround to build Octave 4.4.0 with Perl 5.28.0 installed, on GNU/Linux systems the open file limit can be easily set to a much higher number with 'ulimit -n 10000' for example.

Mike Miller <mtmiller>
Group Member
Mon 02 Jul 2018 09:20:30 PM UTC, comment #26: 

Rik: I would say yes to your question in comment #25.

John W. Eaton <jwe>
Group administrator
Mon 02 Jul 2018 09:08:24 PM UTC, comment #25: 

Another question, should this bug fix for Perl be backported to stable so that anyone with a new computer, but attempting to build the 4.4.1 stable release, will not have problems?

Rik <rik5>
Group administrator
Mon 02 Jul 2018 03:09:17 PM UTC, comment #24: 

That is good news.  I'm going to close this report.  We can revert the change after the 5.28.1 bugfix release has been widely circulated.

Rik <rik5>
Group administrator
Mon 02 Jul 2018 06:00:02 AM UTC, comment #23: 

The good news is the Perl maintainers already have a patch and the problem will be fixed for the 5.28.1 bugfix relesae.

Mike Miller <mtmiller>
Group Member
Mon 02 Jul 2018 04:30:45 AM UTC, comment #22: 

I pushed a full Perl script to use in place of the in-place edit mode of Perl to workaround this bug in version 5.28.  See http://hg.savannah.gnu.org/hgweb/octave/rev/d6850dd2a6b4.

Marking as Ready for Test.

Rik <rik5>
Group administrator
Sun 01 Jul 2018 02:12:51 AM UTC, comment #21: 

I have filed a bug report against perl at https://rt.perl.org/Public/Bug/Display.html?id=133314

Mike Miller <mtmiller>
Group Member
Sat 30 Jun 2018 08:15:32 AM UTC, comment #20: 

Agree, the inplace_edit.pl script works for me with Perl 5.28.0 and with Perl's git master branch.

I will try to submit a bug to the Perl bug tracker and provide the link here when it's done.

Mike Miller <mtmiller>
Group Member
Sat 30 Jun 2018 07:41:35 AM UTC, comment #19: 

I'm also using perl-5.28 and deleting the perl does complete the build process.

The perl script in comment #18 executes without any errors using the example in comment #12.

Julian <marrowsuck>
Sat 30 Jun 2018 05:26:18 AM UTC, comment #18: 

@Mike: I've attached a Perl script called inplace_edit.pl.  The syntax for use is


inplace_edit.pl 'PERL CODE FOR EACH LINE' file1 file2 ...


In particular,


perl -i -pe 's/a/b/' directory/*


becomes


perl inplace_edit.pl 's/a/b/' directory/*


Could you test it stand-alone with 5.28 using the example in comment #12?



(file #44461)

Rik <rik5>
Group administrator
Sat 30 Jun 2018 04:27:15 AM UTC, comment #17: 
Dan Sebald <sebald>
Sat 30 Jun 2018 04:20:48 AM UTC, comment #16: 

As a big fan of Perl, this seems like a ghastly regression.  The whole point of Perl is that it will attempt to do anything the programmer desires.  It may fail because it runs out of disk space, memory, or computes, but the limitation should always be external, not because of something within Perl.

@Mike: Can you report this?  I'm not sure where the Perl bug tracker is.

I can rewrite the one-line modification in to a full script (put it somewhere like build-aux) that definitely closes each FD.  This is a PITA, but I don't know when it will get fixed in Perl.


Rik <rik5>
Group administrator
Sat 30 Jun 2018 04:17:01 AM UTC, comment #15: 

It could also be an incidental change.  Just speculating, but it might be that Perl never set any limit on the argument list.  Rather that limit could be determined by the underlying OS methods it is using.  That is, if it is using unlinkat(), renameat() and fchmodat() in some fundamentally different way than in the past, it could be that's where the 1019 file limit is.  Could the Perl script be changed so that it is broken into four regions with regular expression substitution in the file name?  E.g.,


doc/interpreter/octave.html/[A-K]*
doc/interpreter/octave.html/[L-a]*
doc/interpreter/octave.html/[b-k]*
doc/interpreter/octave.html/[l-z]*
doc/interpreter/octave.html/[^A-z]*


Or perhaps write the in-place edit line as a loop through all files in groups indexed by the first letter somehow.

Dan Sebald <sebald>
Sat 30 Jun 2018 02:11:59 AM UTC, comment #14: 

Perl 5.26 is able to do in-place editing on an argument list with 100,000 files with no problem. Much larger than 100,000 and the problem has to do with the shell argument list length, not a Perl problem.

Perl 5.28 seems to only be able to do in-place editing on up to 1019 files (1024, minus three standard file descriptors, minus a couple more for something internal).

This seems like a regression in Perl's in-place editing feature.

Mike Miller <mtmiller>
Group Member
Sat 30 Jun 2018 01:45:18 AM UTC, comment #13: 

Good investigation Mike.  Here are the v 5.28 release notes:

https://lwn.net/Articles/758127/

which leads more specifically to the in-place editing change:

https://metacpan.org/pod/release/XSAWYERX/perl-5.28.0/pod/perldelta.pod#In-place-editing-with-perl-i-is-now-safer

From what I gather, the movement of a file can no longer take place until it is completely and successfully closed.  unlinkat(), renameat() and fchmodat() come into play.  I'll leave the details to someone who knows perl better than I.

Dan Sebald <sebald>
Sat 30 Jun 2018 01:43:32 AM UTC, comment #12: 

And lastly here is a simple non-Octave example that shows this is a problem with the upgrade from Perl 5.26 to 5.28:


$ mkdir empty
$ cd empty
$ for n in $(seq 1 2000); do echo empty > file$n; done
$ perl5.26.2 -pi -e 's/a/b/g' *
$ perl5.28.0 -pi -e 's/a/b/g' *
Cannot complete in-place edit of file1916: failed to rename work file 'XX7tjdpU' to 'file1916': Bad file descriptor.


Mike Miller <mtmiller>
Group Member
Sat 30 Jun 2018 01:17:24 AM UTC, comment #11: 

And I just built Perl 5.28 locally and I get the same error message. So looks like that command is problematic with Perl 5.28 but works with Perl 5.26. Either a Perl bug or a legitimate issue that we need to work around.

With Perl 5.26.2


$ perl -i -pe 's|(?<=</a>): ||g' doc/interpreter/octave.html/*
$ echo $?
0


no problem.

With Perl 5.28.0, the same command on the same directory contents


$ perl -i -pe 's|(?<=</a>): ||g' doc/interpreter/octave.html/*
Cannot complete in-place edit of doc/interpreter/octave.html/XREFdurbinlevinson.html: failed to rename work file 'doc/interpreter/octave.html/XX7UmV6l' to 'doc/interpreter/octave.html/XREFdurbinlevinson.html': Bad file descriptor, <> line 142011.


Mike Miller <mtmiller>
Group Member
Sat 30 Jun 2018 12:57:23 AM UTC, comment #10: 

JFTR, the error message "Cannot complete in-place edit of %s: failed to rename work file '%s' to '%s': %s" was added to Perl six months ago, and first shows up in Perl 5.28.0 which was released only a week ago.

Mike Miller <mtmiller>
Group Member
Fri 29 Jun 2018 10:20:54 PM UTC, comment #9: 

I should have read Mike's responses more closely.  He identified the same issue.  Which version of Perl are you using (perl -v)?


Rik <rik5>
Group administrator
Fri 29 Jun 2018 10:18:54 PM UTC, comment #8: 

When I do a clean build of the html documentation I have this line


if /bin/bash /home/rik/wip/Projects_Mine/octave-dev/build-aux/missing makeinfo --html   \
 -I doc/interpreter -I /home/rik/wip/Projects_Mine/octave-dev/doc/interpreter \
 --css-ref=octave.css \
 -o doc/interpreter/octave.htp `test -f 'doc/interpreter/octave.texi' || echo '/home/rik/wip/Projects_Mine/octave-dev/'`doc/interpreter/octave.texi; \
then \
  perl -i -pe 's|(?<=</a>): ||g' doc/interpreter/octave.htp/* && \
  rm -rf doc/interpreter/octave.html && \
  mv doc/interpreter/octave.htp doc/interpreter/octave.html && \
  touch doc/interpreter/octave.html/.octave-html-stamp; \
else \
  rm -rf doc/interpreter/octave.htp; exit 1; \
fi


That is where the references to octave.htp are coming from.

Mike is correct that it is Perl which is complaining.  The directory has 2,663 files.  This may be too many to place as arguments on the command line.

In order to clear away cruft and start from scratch, try this


rm -rf doc/interpreter/octave.html
make -j1 V=1


Does this build or not?

Rik <rik5>
Group administrator
Fri 29 Jun 2018 09:51:50 PM UTC, comment #7: 

Julian - you could try deleting that one PERL line from the makefile and building again. That line is not critical to building Octave, it was added to make the user manual appearance better but it won't break without it.

https://hg.savannah.gnu.org/hgweb/octave/rev/336c167b1883

Mike Miller <mtmiller>
Group Member
Fri 29 Jun 2018 09:49:49 PM UTC, comment #6: 

My guess is that the error is coming from the specific line


          $(PERL) -i -pe 's|(?<=</a>): ||g' $(OCTAVE_HTML_TMP_DIR)/* && \


since the error message was about Perl being unable to rename a file that was being edited in-place.

I guess it would help to know what operating system you are building on, which version of Perl you have, what your open file descriptor limit is, and how many files are in the octave.htp directory.

My open file descriptor limit is 1024, I have 2661 files under octave.html, but I don't get an error with this command.

Mike Miller <mtmiller>
Group Member
Fri 29 Jun 2018 09:49:33 PM UTC, comment #5: 

Grasping, but could try 'make -j1' so that there is no question of parallel processes fighting during the creation of the documentation.

Rik <rik5>
Group administrator
Fri 29 Jun 2018 09:39:06 PM UTC, comment #4: 

Yes, it is repeatable and disk space shouldn't be an issue.
I see a doc/interpreter/octave.htp but not octave.html dir.

Here is the part of my Makefile. I have enclosed line 31414 with three "*":

doc/interpreter/octave.html: $(OCTAVE_HTML_STAMP)

$(OCTAVE_HTML_STAMP): doc/interpreter/octave.texi $(srcdir)/doc/interpreter/version-octave.texi | doc/interpreter/$(am__dirstamp)
        $(AM_V_MAKEINFO)rm -rf $(OCTAVE_HTML_DIR)
***        $(AM_V_at)if $(MAKEINFOHTML) $(AM_MAKEINFOHTMLFLAGSL)*** $(MAKEINFOFLAGS) \
         -I doc/interpreter -I $(abs_top_srcdir)/doc/interpreter \
         --css-ref=octave.css \
         -o $(OCTAVE_HTML_TMP_DIR) `test -f 'doc/interpreter/octave.texi' || echo '$(abs_top_srcdir)/'`doc/interpreter/octave.texi; \
        then \
          $(PERL) -i -pe 's|(?<=</a>): ||g' $(OCTAVE_HTML_TMP_DIR)/* && \
          rm -rf $(OCTAVE_HTML_DIR) && \
          mv $(OCTAVE_HTML_TMP_DIR) $(OCTAVE_HTML_DIR) && \
          touch $@; \
        else \
          rm -rf $(OCTAVE_HTML_TMP_DIR); exit 1; \
        fi


Julian <marrowsuck>
Fri 29 Jun 2018 06:28:47 PM UTC, comment #3: 

And is this repeatable?  Maybe there was a temporary situation on the build computer where it ran low on disk space or file descriptors.  If so, re-running 'make all' should complete the second time.

Rik <rik5>
Group administrator
Fri 29 Jun 2018 05:17:23 PM UTC, comment #2: 

Do you even have a build sub-directory called

doc/interpreter/octave.htp

?  On my system I'm seeing

doc/interpreter/octave.html/

Notice in the Makefile warning you posted that the latter directory is sought.  In addition, the MAKEINFO is using the ".html" subdirectory as well.  So, it's a bit confusing.

I'm not sure how the "htp" of "octave.htp" was constructed on your system.  Here are the occurrences of "htp" in the build configuration:


$ grep "htp" *
[snip]
Makefile.in:@AMCOND_BUILD_DOCS_FALSE@        $(AM_V_MAKEINFO)rm -rf $(@:.html=.htp)
Makefile.in:@AMCOND_BUILD_DOCS_FALSE@         -o $(@:.html=.htp) `test -f 'doc/interpreter/octave.texi' || echo '$(srcdir)/'`doc/interpreter/octave.texi; \
Makefile.in:@AMCOND_BUILD_DOCS_FALSE@          rm -rf $@ && mv $(@:.html=.htp) $@; \
Makefile.in:@AMCOND_BUILD_DOCS_FALSE@          rm -rf $(@:.html=.htp); exit 1; \
Makefile.in:@AMCOND_BUILD_DOCS_FALSE@        $(AM_V_MAKEINFO)rm -rf $(@:.html=.htp)
Makefile.in:@AMCOND_BUILD_DOCS_FALSE@         -o $(@:.html=.htp) `test -f 'doc/liboctave/liboctave.texi' || echo '$(srcdir)/'`doc/liboctave/liboctave.texi; \
Makefile.in:@AMCOND_BUILD_DOCS_FALSE@          rm -rf $@ && mv $(@:.html=.htp) $@; \
Makefile.in:@AMCOND_BUILD_DOCS_FALSE@          rm -rf $(@:.html=.htp); exit 1; \
Makefile.in:@AMCOND_BUILD_DOCS_TRUE@doc/interpreter/plot-lightproperties.texi: doc/interpreter/genpropdoc.m $(GRAPHICS_PROPS_SRC)
Makefile.in:@AMCOND_BUILD_DOCS_TRUE@        $(AM_V_MAKEINFO)rm -rf $(@:.html=.htp)
Makefile.in:@AMCOND_BUILD_DOCS_TRUE@         -o $(@:.html=.htp) `test -f 'doc/liboctave/liboctave.texi' || echo '$(srcdir)/'`doc/liboctave/liboctave.texi; \
Makefile.in:@AMCOND_BUILD_DOCS_TRUE@          rm -rf $@ && mv $(@:.html=.htp) $@; \
Makefile.in:@AMCOND_BUILD_DOCS_TRUE@          rm -rf $(@:.html=.htp); exit 1; \


Is this .html=.htp some kind of equivalence in autoconfig lingo?

Dan Sebald <sebald>
Fri 29 Jun 2018 02:32:26 PM UTC, comment #1: 

I don't see this problem.

Is this repeatable?

If I understand the message from make,  it looks like the error occurs at line 31414.  My Makefile only has 28851 lines.  What is at line 31414 of your Makefile?

John W. Eaton <jwe>
Group administrator
Fri 29 Jun 2018 08:13:21 AM UTC, original submission:  

Building from dev (changeset 25514:c63f67d87b4a) fails with

  GEN      doc/interpreter/splinefit4.png
  GEN      doc/interpreter/splinefit6.png
  MAKEINFO doc/interpreter/octave.info
  TEXI2DVI doc/interpreter/octave.dvi
  MAKEINFO doc/interpreter/octave.html/.octave-html-stamp
  DVIPS    doc/interpreter/octave.ps
  TEXI2PDF doc/interpreter/octave.pdf
Cannot complete in-place edit of doc/interpreter/octave.htp/XREFerfcinv.html: failed to rename work file 'doc/interpreter/octave.htp/XXyHVcbm' to 'doc/interpreter/octave.htp/XREFerfcinv.html': Bad file descriptor, <> line 138319.
make[2]: *** [Makefile:31414: doc/interpreter/octave.html/.octave-html-stamp] Error 9
make[2]: *** Waiting for unfinished jobs....


Julian <marrowsuck>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44461:  inplace_edit.pl added by rik5 (512B - application/x-perl)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by marrowsuck (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-07-02 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2018-07-02 rik5 StatusConfirmed Ready For Test
    2018-06-30 rik5 Attached File- Added inplace_edit.pl, #44461
    2018-06-30 mtmiller StatusWorks For Me Confirmed
        SummaryBuild fails with Bad file descriptor build fails with Bad file descriptor with perl 5.28.0 installed
    2018-06-29 jwe StatusNone Works For Me

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code