Fri 22 Apr 2016 09:38:40 PM UTC, comment #21:
@JWE:
You are certainly right that the case of "A && B" looks like a bug.
I merely figured the race conditions I saw with zip/gzip (comment #4) constitute a case of in itself fair user expectations hitting "physical" limitations, with little to be done about it.
It is the observation that these race conditions happen on the same box in Linux and in Windows (actually MSYS, based on *nix) that makes me think that it could be more than a plain bug.
Looking into "A && B" in more detail, all I know and all I can find is that for "B" to start it is formally sufficient that "A" returns with a success status. I have no references as to what "success status" exactly means in terms of disk I/O. It's an obvious assumption that in case of an "A && B" where "B" depends on files made by "A", "B" can immediately access those files. In between buffers flushed & files closed and "B" having access to "A"'s files another step seems to emerge on fast systems.
Note that my own code mentioned in comment #4 invoked zip/gzip executables through the "system" function with (default) "sync" type. IOW, analogously to "A && B".
AFAIK "flushing pending writes" and "file handles closed" are userland concepts; what really happens with the output data being handed through the operating system to the actual storage device and back is opaque, at least to me.
|
Fri 22 Apr 2016 04:47:30 PM UTC, comment #20:
We're not talking about two random processes. We're talking about the two processes in a shell command list "A && B". For that, the process A must exit before B starts, correct? And when a process exits, aren't pending writes supposed to be flushed and open files closed? Shouldn't that all happen before the exit status is returned to the shell? If so, then how can any file written by A NOT be available to B unless there is a filesystem or OS bug? If there is no such guarantee, then I'm amazed that any automake-generated Makefile functions properly since they are full of shell command lists just like this.
If there is some delay before a file created by process A is available to process B, then I would expect random failures to be happening all the time. And if there is no guarantee "A && B", then I see no reason to expect "A; B" to work either. Or for a series of commands in a Makefile recipe to work reliably, even if each line is executed in a separate subshell. How could we know when a file might be flushed and ready to read by another process?
If I'm wrong about all of this, then please point me to some docs or code or textbook or something that explains and warns that there is no guarantee.
|
Fri 22 Apr 2016 04:08:16 PM UTC, comment #19:
@Mike comment #17:
I do not think this is an OS or filesystem bug.
It's rather an issue that naturally follows from the way multi-user/multitasking operating systems work: when writing a file followed by reading from that same file a split second later by different processes there simply cannot be any guarantee that the file already exists completely in time for the reading process. The OS doesn't/cannot know that the writing and reading process are spawned from one and the same shell script; so from "its" POV there's no connection at all between the writing and reading process, and as a consequence no need for it to finish the I/O operations in a certain order.
It is obviously different if the file handle is owned by one and the same process or if a pipe (one process) is active.
Thinking of it, would some sort of pipe be a safe solution?
|
Wed 20 Apr 2016 10:44:54 PM UTC, comment #18:
I just pushed a second cset that alters the libmng recipe. There was an extra mkdir command that was unnecessary and we really needed to be giving the --force option to autoreconf. Part of the issue is that the archive file for libmng was shipping with a copy of ltmain.sh (libtool) based on version 2.4. 'autoreconf' found the file, and for whatever reason, decided that it was new enough and was not replacing it.
|
Wed 20 Apr 2016 10:17:32 PM UTC, comment #17:
Seems like a really bad OS or filesystem driver bug, but so be it.
|
Wed 20 Apr 2016 10:11:28 PM UTC, comment #16:
I pushed the cset. Closing report.
|
Wed 20 Apr 2016 06:06:19 PM UTC, comment #15:
@Rik:
Patch from comment #10 works fine, libmng gets built with JOBS=7 (but of course that is overridden in libmng.mk)
Thanks for fixing this.
|
Wed 20 Apr 2016 04:26:32 PM UTC, comment #14:
Yes, it's a race issue. A && B take place in a single spawned subshell and apparently with SSDs the OS is not reliable. On the other hand, when the recipe includes the commands on different lines then there is the delay of the spawned subshell for command A as it shuts down, and the delay to invoke a new spawned subshell to execute B. This seems always to be enough judging by the fact that all the other recipes build correctly.
|
Wed 20 Apr 2016 04:11:29 PM UTC, comment #13:
Hmm, I would expect that for
the first command must complete so that the exit status can be checked before the second command runs. How could that not be the case? Or is it a problem with the OS in which files written by the first command are not properly flushed and available when the second command starts? I don't know why that would be different for a list of commands vs a series of commands separated by &&.
|
Wed 20 Apr 2016 04:05:25 PM UTC, comment #12:
@John, @Philip: I used grep to look through the other build recipes in the src directory. libmng was the only one which was using the pattern of
Every other recipe separated these in to two steps which gets around the race conditions because the first process has to exit before the next one can start. I made that change to the libmng.mk recipe so it now looks like the code below.
Hopefully, this will also work for Philip and I can push this change.
|
Wed 20 Apr 2016 09:07:48 AM UTC, comment #11:
Yep I'll try tonight.
[OT]
Yesterday evening I saw that (default-)octave-4.1.0+ doesn't get cross-built anymore, it chokes on some missing .dt file somewhere in the "make install" stage. Unexpected as only libmng, qt and qscintilla were rebuilt and the rest of the mxe-octave build wasn't touched. That same mxe-octave worked fine 2 days ago.
|
Wed 20 Apr 2016 12:06:03 AM UTC, comment #10:
I've attached a patch that works for me. Could you apply it and then rebuild libmng normally, i.e., with JOBS equal to 7, and see if it works?
(file #36967)
|
Tue 19 Apr 2016 07:35:10 PM UTC, comment #9:
OK, tried it.
I've deleted installed-packages/libmng a few times and retried, and libmng gets built fine now but only with JOBS=1; so I suppose that has to be included in src/libmng.mk
(I usually build mxe-octave binaries with JOBS = anywhere between 5 and 7)
It's building qt right now (~10 mins) + perhaps other dependencies I don't know yet but it seems the libmng issue has been fixed - for me at least.
|
Tue 19 Apr 2016 05:15:42 AM UTC, comment #8:
Anyone get a chance to try the patch from comment #5 and see if that helps?
|
Mon 18 Apr 2016 09:57:23 AM UTC, comment #7:
"Learn to adapt" ...
I'd think the build system should be adapted to let it cope with very fast hardware. This issue won't go away, au contraire, I expect it to evolve further.
At the end of 2014/early 2015 it wasn't just libmng that hicked up; IIRC there were ~4 packages that needed JOBS=1. libmng is the only package now that still gives (gave?) problems.
|
Sun 17 Apr 2016 11:47:11 PM UTC, comment #6:
At least I don't feel so weird now. It's probably the SSD and I'll just have to learn to adapt.
|
Sun 17 Apr 2016 04:20:07 PM UTC, comment #5:
For the scan_file warning, this appears to be a automake problem, reports by [1]. And also from a number of other reporsts of issues.
I have attached a patch here that adds it mxe-octave - on my computer, I am not currently seeing the issue, so cant tell whether the patch fixes the issue or not, so if someone else can give it a try ....
[1] https://lists.gnu.org/archive/html/bug-automake/2012-05/msg00034.html
(file #36954)
|
Sat 16 Apr 2016 06:43:25 PM UTC, comment #4:
I have reported this (or something very similar) already some time ago (see a.o., https://lists.nongnu.org/archive/html/octave-bug-tracker/2015-05/msg00142.html, and in earlier postings that I cannot find right now)
It consistently happens with libmng on one of my systems that has a fairly fast CPU and a fast SSD. Restarting mxe-octave (make nsis-installer) with JOBS=1 (and still needing to retry several times) allows the build to continue in the end. When mxe starts building the next package (IIRC: dbus) I interrupt it and restart with JOBS=5 (or 7, or whatever).
All other packages, incl. octave, build fine with JOBS=5 or 6 or 7 on that system.
I am fairly sure it is a sort of race condition but I suspect it is related to disk I/O.
It appears that SSDs are often simply too fast. For example, when Octave writes files to a zipped/gzipped archive, some delay is required before accessing that archive to avoid errors. It appears that zip/gzip/etc already return once the actual disk I/O has been handed to the low-level stuff, but as that is scheduled using lazy write etc. the actual disk file may come into existence too late for Octave's next steps. Or maybe it is simply the directory tree in memory (user memory or system memory, I do not know) that is updated too late.
|
Sat 16 Apr 2016 04:31:37 AM UTC, comment #3:
I think there must be various race conditions in some of the packages. The default-octave target also failed unless I set the number of JOBS to 1.
|
Fri 15 Apr 2016 09:33:08 PM UTC, comment #2:
I got libmng to compile by altering the libmng.mk file to place a delay of 1 second between the autoreconf and configure lines.
I have an SSD as a hard drive and it is possible that things are quicker than they would otherwise be on a normal HDD.
|
Fri 15 Apr 2016 09:26:49 PM UTC, comment #1:
In the log file, I am seeing this early on
followed later by the eventual error
This is with setting JOBS=1, but maybe there is still some parallelism in the make process.
|
Fri 15 Apr 2016 09:22:30 PM UTC, original submission:
I would be happy if this were just my setup.
I used hg clone to get a brand new copy of the MXE directory (id=5ae69991eaa1). Next I can autoconf to produce a configure file.
Following that, I executed configure script
And after that, I built with make
This produces an error that libtool versions are not in sync (2.4.2 versus 2.4). The log file from the mxe build for libmng is attached.
|