bugGNU Octave - Bugs: bug #47712, libmng no longer builds under...

 
 

bug #47712: libmng no longer builds under mxe-octave

Submitter:  Rik <rik5>
Submitted:  Fri 15 Apr 2016 09:22:30 PM UTC
   
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Regression
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 22 Apr 2016 09:38:40 PM UTC, comment #21: 

@JWE:
You are certainly right that the case of "A && B" looks like a bug. 
I merely figured the race conditions I saw with zip/gzip (comment #4) constitute a case of in itself fair user expectations hitting "physical" limitations, with little to be done about it.
It is the observation that these race conditions happen on the same box in Linux and in Windows (actually MSYS, based on *nix) that makes me think that it could be more than a plain bug.

Looking into "A && B" in more detail, all I know and all I can find is that for "B" to start it is formally sufficient that "A" returns with a success status.  I have no references as to what "success status" exactly means in terms of disk I/O.  It's an obvious assumption that in case of an "A && B" where "B" depends on files made by "A", "B" can immediately access those files.  In between buffers flushed & files closed and "B" having access to "A"'s files another step seems to emerge on fast systems.
Note that my own code mentioned in comment #4 invoked zip/gzip executables through the "system" function with (default) "sync" type. IOW, analogously to "A && B".

AFAIK "flushing pending writes" and "file handles closed" are userland concepts; what really happens with the output data being handed through the operating system to the actual storage device and back is opaque, at least to me.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 22 Apr 2016 04:47:30 PM UTC, comment #20: 

We're not talking about two random processes.  We're talking about the two processes in a shell command list "A && B".  For that, the process A must exit before B starts, correct?  And when a process exits, aren't pending writes supposed to be flushed and open files closed?  Shouldn't that all happen before the exit status is returned to the shell?  If so, then how can any file written by A NOT be available to B unless there is a filesystem or OS bug?  If there is no such guarantee, then I'm amazed that any automake-generated Makefile functions properly since they are full of shell command lists just like this.

If there is some delay before a file created by process A is available to process B, then I would expect random failures to be happening all the time.  And if there is no guarantee "A && B", then I see no reason to expect "A; B" to work either.  Or for a series of commands in a Makefile recipe to work reliably, even if each line is executed in a separate subshell.  How could we know when a file might be flushed and ready to read by another process?

If I'm wrong about all of this, then please point me to some docs or code or textbook or something that explains and warns that there is no guarantee.

John W. Eaton <jwe>
Group administrator
Fri 22 Apr 2016 04:08:16 PM UTC, comment #19: 

@Mike comment #17:
I do not think this is an OS or filesystem bug.

It's rather an issue that naturally follows from the way multi-user/multitasking operating systems work: when writing a file followed by reading from that same file a split second later by different processes there simply cannot be any guarantee that the file already exists completely in time for the reading process. The OS doesn't/cannot know that the writing and reading process are spawned from one and the same shell script; so from "its" POV there's no connection at all between the writing and reading process, and as a consequence no need for it to finish the I/O operations in a certain order.

It is obviously different if the file handle is owned by one and the same process or if a pipe (one process) is active.

Thinking of it, would some sort of pipe be a safe solution?

Philip Nienhuis <philipnienhuis>
Group Member
Wed 20 Apr 2016 10:44:54 PM UTC, comment #18: 

I just pushed a second cset that alters the libmng recipe.  There was an extra mkdir command that was unnecessary and we really needed to be giving the --force option to autoreconf.  Part of the issue is that the archive file for libmng was shipping with a copy of ltmain.sh (libtool) based on version 2.4.  'autoreconf' found the file, and for whatever reason, decided that it was new enough and was not replacing it.

Rik <rik5>
Group administrator
Wed 20 Apr 2016 10:17:32 PM UTC, comment #17: 

Seems like a really bad OS or filesystem driver bug, but so be it.

Mike Miller <mtmiller>
Group Member
Wed 20 Apr 2016 10:11:28 PM UTC, comment #16: 

I pushed the cset.  Closing report.

Rik <rik5>
Group administrator
Wed 20 Apr 2016 06:06:19 PM UTC, comment #15: 

@Rik:
Patch from comment #10 works fine, libmng gets built with JOBS=7 (but of course that is overridden in libmng.mk)

Thanks for fixing this.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 20 Apr 2016 04:26:32 PM UTC, comment #14: 

Yes, it's a race issue.  A && B take place in a single spawned subshell and apparently with SSDs the OS is not reliable.  On the other hand, when the recipe includes the commands on different lines then there is the delay of the spawned subshell for command A as it shuts down, and the delay to invoke a new spawned subshell to execute B.  This seems always to be enough judging by the fact that all the other recipes build correctly.

Rik <rik5>
Group administrator
Wed 20 Apr 2016 04:11:29 PM UTC, comment #13: 

Hmm, I would expect that for


cmd_one && cmd_two


the first command must complete so that the exit status can be checked before the second command runs.  How could that not be the case?  Or is it a problem with the OS in which files written by the first command are not properly flushed and available when the second command starts?  I don't know why that would be different for a list of commands vs a series of commands separated by &&.

John W. Eaton <jwe>
Group administrator
Wed 20 Apr 2016 04:05:25 PM UTC, comment #12: 

@John, @Philip: I used grep to look through the other build recipes in the src directory.  libmng was the only one which was using the pattern of


autoreconf --install && ./configure


Every other recipe separated these in to two steps which gets around the race conditions because the first process has to exit before the next one can start.  I made that change to the libmng.mk recipe so it now looks like the code below.


autoreconf --install
./configure


Hopefully, this will also work for Philip and I can push this change.

Rik <rik5>
Group administrator
Wed 20 Apr 2016 09:07:48 AM UTC, comment #11: 

Yep I'll try tonight.

[OT]
Yesterday evening I saw that (default-)octave-4.1.0+ doesn't get cross-built anymore, it chokes on some missing .dt file somewhere in the "make install" stage. Unexpected as only libmng, qt and qscintilla were rebuilt and the rest of the mxe-octave build wasn't touched. That same mxe-octave worked fine 2 days ago.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 20 Apr 2016 12:06:03 AM UTC, comment #10: 

I've attached a patch that works for me.  Could you apply it and then rebuild libmng normally, i.e., with JOBS equal to 7, and see if it works?


# From top-level mxe directory
rm installed-packages/libmng*
patch -p1 < libmng.patch




(file #36967)

Rik <rik5>
Group administrator
Tue 19 Apr 2016 07:35:10 PM UTC, comment #9: 

OK, tried it.

I've deleted installed-packages/libmng a few times and retried, and libmng gets built fine now but only with JOBS=1; so I suppose that has to be included in src/libmng.mk 
(I usually build mxe-octave binaries with JOBS = anywhere between 5 and 7)

It's building qt right now (~10 mins) + perhaps other dependencies I don't know yet but it seems the libmng issue has been fixed - for me at least.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 19 Apr 2016 05:15:42 AM UTC, comment #8: 

Anyone get a chance to try the patch from comment #5 and see if that helps?

John Donoghue <lostbard>
Group Member
Mon 18 Apr 2016 09:57:23 AM UTC, comment #7: 

"Learn to adapt" ...
I'd think the build system should be adapted to let it cope with very fast hardware.  This issue won't go away, au contraire, I expect it to evolve further.

At the end of 2014/early 2015 it wasn't just libmng that hicked up; IIRC there were ~4 packages that needed JOBS=1. libmng is the only package now that still gives (gave?) problems.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 17 Apr 2016 11:47:11 PM UTC, comment #6: 

At least I don't feel so weird now.  It's probably the SSD and I'll just have to learn to adapt.

Rik <rik5>
Group administrator
Sun 17 Apr 2016 04:20:07 PM UTC, comment #5: 

For the scan_file warning, this appears to be a automake problem, reports by [1]. And also from a number of other reporsts of issues.

I have attached a patch here that adds it mxe-octave - on my computer, I am not currently seeing the issue, so cant tell whether the patch fixes the issue or not, so if someone else can give it a try ....

[1]  https://lists.gnu.org/archive/html/bug-automake/2012-05/msg00034.html



(file #36954)

John Donoghue <lostbard>
Group Member
Sat 16 Apr 2016 06:43:25 PM UTC, comment #4: 

I have reported this (or something very similar) already some time ago (see a.o., https://lists.nongnu.org/archive/html/octave-bug-tracker/2015-05/msg00142.html, and in earlier postings that I cannot find right now)

It consistently happens with libmng on one of my systems that has a fairly fast CPU and a fast SSD. Restarting mxe-octave (make nsis-installer) with JOBS=1 (and still needing to retry several times) allows the build to continue in the end. When mxe starts building the next package (IIRC: dbus) I interrupt it and restart with JOBS=5 (or 7, or whatever).

All other packages, incl. octave, build fine with JOBS=5 or 6 or 7 on that system.

I am fairly sure it is a sort of race condition but I suspect it is related to disk I/O.
It appears that SSDs are often simply too fast. For example, when Octave writes files to a zipped/gzipped archive, some delay is required before accessing that archive to avoid errors. It appears that zip/gzip/etc already return once the actual disk I/O has been handed to the low-level stuff, but as that is scheduled using lazy write etc. the actual disk file may come into existence too late for Octave's next steps. Or maybe it is simply the directory tree in memory (user memory or system memory, I do not know) that is updated too late.

Philip Nienhuis <philipnienhuis>
Group Member
Sat 16 Apr 2016 04:31:37 AM UTC, comment #3: 

I think there must be various race conditions in some of the packages.  The default-octave target also failed unless I set the number of JOBS to 1.

Rik <rik5>
Group administrator
Fri 15 Apr 2016 09:33:08 PM UTC, comment #2: 

I got libmng to compile by altering the libmng.mk file to place a delay of 1 second between the autoreconf and configure lines.


    cd '$(1)' && autoreconf --install && sleep 1 && ./configure \


I have an SSD as a hard drive and it is possible that things are quicker than they would otherwise be on a normal HDD.

Rik <rik5>
Group administrator
Fri 15 Apr 2016 09:26:49 PM UTC, comment #1: 

In the log file, I am seeing this early on


main::scan_file() called too early to check prototype at /home/rik/wip/Projects_Mine/mxe2/usr/bin/aclocal line 643.
libtoolize: putting auxiliary files in `.'.


followed later by the eventual error


make[3]: Entering directory '/home/rik/wip/Projects_Mine/mxe2/tmp-libmng/libmng-2.0.3'
/bin/bash ./libtool --tag=CC   --mode=compile i686-w64-mingw32-gcc -DHAVE_CONFIG_H -I.   -I/home/rik/wip/Projects_Mine/mxe2/usr/i686-w64-mingw32/include  -g -O2 -c -o libmng_callback_xs.lo libmng_callback_xs.c
libtool: Version mismatch error.  This is libtool 2.4.2, but the
libtool: definition of this LT_INIT comes from libtool 2.4.
libtool: You should recreate aclocal.m4 with macros from libtool 2.4.2
libtool: and run autoconf again.
Makefile:409: recipe for target 'libmng_callback_xs.lo' failed
make[3]: *** [libmng_callback_xs.lo] Error 63


This is with setting JOBS=1, but maybe there is still some parallelism in the make process.

Rik <rik5>
Group administrator
Fri 15 Apr 2016 09:22:30 PM UTC, original submission:  

I would be happy if this were just my setup.

I used hg clone to get a brand new copy of the MXE directory (id=5ae69991eaa1).  Next I can autoconf to produce a configure file.

Following that, I executed configure script


#!/bin/tcsh -f

unsetenv CFLAGS
unsetenv CXXFLAGS
unsetenv FFLAGS

./configure --disable-jit --disable-java --enable-octave=default |& tee myconfig.log


And after that, I built with make


#!/bin/tcsh

setenv PATH /home/rik/wip/Projects_Mine/mxe-octave/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:.

make V=1 JOBS=4 OCTAVE_TARGET=default-octave nsis-installer


This produces an error that libtool versions are not in sync (2.4.2 versus 2.4).  The log file from the mxe build for libmng is attached.

Rik <rik5>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #36967:  libmng.patch added by rik5 (1KiB - application/x-download)
file #36954:  automake.patch added by lostbard (4KiB - text/x-patch)
file #36945:  libmng.log added by rik5 (10KiB - text/x-log)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by lostbard (Updated the item)
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by rik5 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-04-20 rik5 StatusPatch Submitted Fixed
        Open/ClosedOpen Closed
    2016-04-20 rik5 Attached File- Added libmng.patch, #36967
        StatusConfirmed Patch Submitted
    2016-04-17 lostbard Attached File- Added automake.patch, #36954
    2016-04-17 philipnienhuis Carbon-Copy- Added philipnienhuis
    2016-04-16 philipnienhuis StatusNone Confirmed
    2016-04-15 rik5 Attached File- Added libmng.log, #36945

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code