bugGNU Octave - Bugs: bug #54202, build fails with Bad file...

 
 

bug #54202: build fails with Bad file descriptor with perl 5.28.0 installed

Submitted by:  Julian <marrowsuck>
Submitted on:  Fri 29 Jun 2018 08:13:21 AM UTC  
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Build Failure
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  dev Operating System:  GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Mon 02 Jul 2018 09:22:55 PM UTC, comment #27:

Also, if anyone needs a temporary workaround to build Octave 4.4.0 with Perl 5.28.0 installed, on GNU/Linux systems the open file limit can be easily set to a much higher number with 'ulimit -n 10000' for example.

Mike Miller <mtmiller>
Project Administrator
Mon 02 Jul 2018 09:20:30 PM UTC, comment #26:

Rik: I would say yes to your question in comment #25.

John W. Eaton <jwe>
Project Administrator
Mon 02 Jul 2018 09:08:24 PM UTC, comment #25:

Another question, should this bug fix for Perl be backported to stable so that anyone with a new computer, but attempting to build the 4.4.1 stable release, will not have problems?

Rik <rik5>
Project Administrator
Mon 02 Jul 2018 03:09:17 PM UTC, comment #24:

That is good news. I'm going to close this report. We can revert the change after the 5.28.1 bugfix release has been widely circulated.

Rik <rik5>
Project Administrator
Mon 02 Jul 2018 06:00:02 AM UTC, comment #23:

The good news is the Perl maintainers already have a patch and the problem will be fixed for the 5.28.1 bugfix relesae.

Mike Miller <mtmiller>
Project Administrator
Mon 02 Jul 2018 04:30:45 AM UTC, comment #22:

I pushed a full Perl script to use in place of the in-place edit mode of Perl to workaround this bug in version 5.28. See http://hg.savannah.gnu.org/hgweb/octave/rev/d6850dd2a6b4.

Marking as Ready for Test.

Rik <rik5>
Project Administrator
Sun 01 Jul 2018 02:12:51 AM UTC, comment #21:

I have filed a bug report against perl at https://rt.perl.org/Public/Bug/Display.html?id=133314

Mike Miller <mtmiller>
Project Administrator
Sat 30 Jun 2018 08:15:32 AM UTC, comment #20:

Agree, the inplace_edit.pl script works for me with Perl 5.28.0 and with Perl's git master branch.

I will try to submit a bug to the Perl bug tracker and provide the link here when it's done.

Mike Miller <mtmiller>
Project Administrator
Sat 30 Jun 2018 07:41:35 AM UTC, comment #19:

I'm also using perl-5.28 and deleting the perl does complete the build process.

The perl script in comment #18 executes without any errors using the example in comment #12.

Julian <marrowsuck>
Sat 30 Jun 2018 05:26:18 AM UTC, comment #18:

@Mike: I've attached a Perl script called inplace_edit.pl. The syntax for use is

In particular,

becomes

Could you test it stand-alone with 5.28 using the example in comment #12?

(file #44461)

Rik <rik5>
Project Administrator
Sat 30 Jun 2018 04:27:15 AM UTC, comment #17:

I looked at some kind of bug report for those release notes:

https://metacpan.org/pod/release/XSAWYERX/perl-5.28.0/pod/perldelta.pod#In-place-editing-with-perl-i-is-now-safer

https://rt.perl.org/Public/Bug/Display.html?id=127663

Maybe that is the tracker, i.e., "perlbug".

Dan Sebald <sebald>
Sat 30 Jun 2018 04:20:48 AM UTC, comment #16:

As a big fan of Perl, this seems like a ghastly regression. The whole point of Perl is that it will attempt to do anything the programmer desires. It may fail because it runs out of disk space, memory, or computes, but the limitation should always be external, not because of something within Perl.

@Mike: Can you report this? I'm not sure where the Perl bug tracker is.

I can rewrite the one-line modification in to a full script (put it somewhere like build-aux) that definitely closes each FD. This is a PITA, but I don't know when it will get fixed in Perl.

Rik <rik5>
Project Administrator
Sat 30 Jun 2018 04:17:01 AM UTC, comment #15:

It could also be an incidental change. Just speculating, but it might be that Perl never set any limit on the argument list. Rather that limit could be determined by the underlying OS methods it is using. That is, if it is using unlinkat(), renameat() and fchmodat() in some fundamentally different way than in the past, it could be that's where the 1019 file limit is. Could the Perl script be changed so that it is broken into four regions with regular expression substitution in the file name? E.g.,

Or perhaps write the in-place edit line as a loop through all files in groups indexed by the first letter somehow.

Dan Sebald <sebald>
Sat 30 Jun 2018 02:11:59 AM UTC, comment #14:

Perl 5.26 is able to do in-place editing on an argument list with 100,000 files with no problem. Much larger than 100,000 and the problem has to do with the shell argument list length, not a Perl problem.

Perl 5.28 seems to only be able to do in-place editing on up to 1019 files (1024, minus three standard file descriptors, minus a couple more for something internal).

This seems like a regression in Perl's in-place editing feature.

Mike Miller <mtmiller>
Project Administrator
Sat 30 Jun 2018 01:45:18 AM UTC, comment #13:

Good investigation Mike. Here are the v 5.28 release notes:

https://lwn.net/Articles/758127/

which leads more specifically to the in-place editing change:

https://metacpan.org/pod/release/XSAWYERX/perl-5.28.0/pod/perldelta.pod#In-place-editing-with-perl-i-is-now-safer

From what I gather, the movement of a file can no longer take place until it is completely and successfully closed. unlinkat(), renameat() and fchmodat() come into play. I'll leave the details to someone who knows perl better than I.

Dan Sebald <sebald>
Sat 30 Jun 2018 01:43:32 AM UTC, comment #12:

And lastly here is a simple non-Octave example that shows this is a problem with the upgrade from Perl 5.26 to 5.28:

Mike Miller <mtmiller>
Project Administrator
Sat 30 Jun 2018 01:17:24 AM UTC, comment #11:

And I just built Perl 5.28 locally and I get the same error message. So looks like that command is problematic with Perl 5.28 but works with Perl 5.26. Either a Perl bug or a legitimate issue that we need to work around.

With Perl 5.26.2

no problem.

With Perl 5.28.0, the same command on the same directory contents

Mike Miller <mtmiller>
Project Administrator
Sat 30 Jun 2018 12:57:23 AM UTC, comment #10:

JFTR, the error message "Cannot complete in-place edit of %s: failed to rename work file '%s' to '%s': %s" was added to Perl six months ago, and first shows up in Perl 5.28.0 which was released only a week ago.

Mike Miller <mtmiller>
Project Administrator
Fri 29 Jun 2018 10:20:54 PM UTC, comment #9:

I should have read Mike's responses more closely. He identified the same issue. Which version of Perl are you using (perl -v)?

Rik <rik5>
Project Administrator
Fri 29 Jun 2018 10:18:54 PM UTC, comment #8:

When I do a clean build of the html documentation I have this line

That is where the references to octave.htp are coming from.

Mike is correct that it is Perl which is complaining. The directory has 2,663 files. This may be too many to place as arguments on the command line.

In order to clear away cruft and start from scratch, try this

Does this build or not?

Rik <rik5>
Project Administrator
Fri 29 Jun 2018 09:51:50 PM UTC, comment #7:

Julian - you could try deleting that one PERL line from the makefile and building again. That line is not critical to building Octave, it was added to make the user manual appearance better but it won't break without it.

https://hg.savannah.gnu.org/hgweb/octave/rev/336c167b1883

Mike Miller <mtmiller>
Project Administrator
Fri 29 Jun 2018 09:49:49 PM UTC, comment #6:

My guess is that the error is coming from the specific line

since the error message was about Perl being unable to rename a file that was being edited in-place.

I guess it would help to know what operating system you are building on, which version of Perl you have, what your open file descriptor limit is, and how many files are in the octave.htp directory.

My open file descriptor limit is 1024, I have 2661 files under octave.html, but I don't get an error with this command.

Mike Miller <mtmiller>
Project Administrator
Fri 29 Jun 2018 09:49:33 PM UTC, comment #5:

Grasping, but could try 'make -j1' so that there is no question of parallel processes fighting during the creation of the documentation.

Rik <rik5>
Project Administrator
Fri 29 Jun 2018 09:39:06 PM UTC, comment #4:

Yes, it is repeatable and disk space shouldn't be an issue.
I see a doc/interpreter/octave.htp but not octave.html dir.

Here is the part of my Makefile. I have enclosed line 31414 with three "*":

Julian <marrowsuck>
Fri 29 Jun 2018 06:28:47 PM UTC, comment #3:

And is this repeatable? Maybe there was a temporary situation on the build computer where it ran low on disk space or file descriptors. If so, re-running 'make all' should complete the second time.

Rik <rik5>
Project Administrator
Fri 29 Jun 2018 05:17:23 PM UTC, comment #2:

Do you even have a build sub-directory called

doc/interpreter/octave.htp

? On my system I'm seeing

doc/interpreter/octave.html/

Notice in the Makefile warning you posted that the latter directory is sought. In addition, the MAKEINFO is using the ".html" subdirectory as well. So, it's a bit confusing.

I'm not sure how the "htp" of "octave.htp" was constructed on your system. Here are the occurrences of "htp" in the build configuration:

Is this .html=.htp some kind of equivalence in autoconfig lingo?

Dan Sebald <sebald>
Fri 29 Jun 2018 02:32:26 PM UTC, comment #1:

I don't see this problem.

Is this repeatable?

If I understand the message from make, it looks like the error occurs at line 31414. My Makefile only has 28851 lines. What is at line 31414 of your Makefile?

John W. Eaton <jwe>
Project Administrator
Fri 29 Jun 2018 08:13:21 AM UTC, original submission:

Building from dev (changeset 25514:c63f67d87b4a) fails with

Julian <marrowsuck>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44461:  inplace_edit.pl added by rik5 (512B - application/x-perl)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by marrowsuck (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-07-02 rik5 StatusReady For Test => Fixed
        Open/ClosedOpen => Closed
    2018-07-02 rik5 StatusConfirmed => Ready For Test
    2018-06-30 rik5 Attached File- => Added inplace_edit.pl, #44461
    2018-06-30 mtmiller StatusWorks For Me => Confirmed
        SummaryBuild fails with Bad file descriptor => build fails with Bad file descriptor with perl 5.28.0 installed
    2018-06-29 jwe StatusNone => Works For Me

    Back to the top


    Powered by Savane 3.3