bugGNU Octave - Bugs: bug #40606, mxe-octave built octave: fails...

 
 

bug #40606: mxe-octave built octave: fails syscalls.cc-tst

Submitter:  John Donoghue <lostbard>
Submitted:  Sat 16 Nov 2013 10:23:20 PM UTC
   
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  John D Open/Closed:  * Closed
Release:  * dev Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 26 Nov 2013 10:57:15 PM UTC, comment #19: 

Checked in a fix here (http://hg.savannah.gnu.org/hgweb/octave/rev/c3cd335bd71b).  Closing report.

Rik <rik5>
Group administrator
Tue 26 Nov 2013 10:22:26 PM UTC, comment #18: 

Alright, I'm going to re-write the test on the stable branch.  I'll check it in later today.

Rik <rik5>
Group administrator
Tue 26 Nov 2013 10:09:02 PM UTC, comment #17: 

This sounds like a sane approach. I just check on Win8 64 bits, and C:\Windows\System32\sort.exe still exists. So it's reasonable to assume it exists in all Windows flavors we target (WinXP -> Win8).

Michael Goffioul <goffioul>
Tue 26 Nov 2013 10:00:05 PM UTC, comment #16: 

I'm not happy with the current %!test which has lots of if/thens based on operating system.  I think it might be cleaner to divide the current test in two with one protected by "if (isunix || ismac)" and the other protected by "if (ispc && ! isunix)".  In the Windows test we could use a direct path to the Microsoft executable "C:\windows\system32\sort.exe" to guarantee the version we want.

Rik <rik5>
Group administrator
Tue 26 Nov 2013 09:49:07 PM UTC, comment #15: 

msys-coreutils gets installed as part of mk-dist in mxe-octave which I believe has sort.exe in it.

John Donoghue <lostbard>
Group Member
Tue 26 Nov 2013 09:45:24 PM UTC, comment #14: 

That behavior is in-line with CreateProcess documentation, it'll search in that order: directory of the executable, current directory, windows system32 directory, windows directory, PATH.

You could avoid all the hassle by using absolute paths instead:


if ispc ()
  sort_bin = file_in_path (getenv ("PATH"), "sort.exe");
else
  sort_bin = "sort";
endif
if ! isempty (sort_bin)
  % check which "sort" flavor you're using
  ...
  % run the test
  ...
endif


Michael Goffioul <goffioul>
Tue 26 Nov 2013 09:33:46 PM UTC, comment #13: 

I think the difference may be that Octave is sneaking in a GNU version of sort through it's bin directory.  When using the MXE environment I'm not building the installer, just the large zip file.  I then unzip that on the Windows side and run octave-long_date_format\bin\octave-cli.exe.  I notice that there is a sort.exe in this bin directory that is yet a third alternative to the other two at C:\Program Files\GnuWin32\bin and C:\Windows\system32.  If I rename the two GNU versions to __sort.exe then I finally seem to pick up the Microsoft one with popen2.  Otherwise, I get the GNU version through popen2() and the Microsoft version through system().

So, in terms of trying to guess which version of sort we will pick up.  If we can't rely on system() or popen(), do we have to basically repeat the entire popen2 test and try once with the GNU version, and if that fails, try with the Microsoft version?  This seems klunky.

Rik <rik5>
Group administrator
Tue 26 Nov 2013 08:23:48 PM UTC, comment #12: 

How did you determine the actual version of sort.exe that you were executing during the popen2 call?

Michael Goffioul <goffioul>
Tue 26 Nov 2013 08:14:39 PM UTC, comment #11: 

Despite the documentation, the behavior is clear.  When a GNU sort.exe is in the PATH it gets used instead of the Microsoft version.  Maybe they have become stricter in more recent versions of Windows about following the search strategy they document?  Also, the tested behavior on Windows XP is that options are case insensitive so "/r" and "/R" both work.  I'm running cmd.exe and nothing fancy like a bash shell.

Rik <rik5>
Group administrator
Tue 26 Nov 2013 06:05:54 PM UTC, comment #10: 

What do you mean "ahead". C:\windows\system32 will be inspected before inspecting PATH. So if you mean ahead in the PATH, that's irrelevant.

OTOH, if you happen to have sort.exe in the same directory as the executable currently running, or if you have sort.exe in the current directory, it will be picked up before C:\windows\system32.

As for your last example, please note that Win32 version of sort use /R as argument, not /r.

Michael Goffioul <goffioul>
Tue 26 Nov 2013 05:08:00 PM UTC, comment #9: 

It is solved for some systems.  It does work on my system where I have a GNU version of sort.exe ahead of the default one in C:\windows\system32.  Still, if I rename the GNU version to set it aside and pick up the Microsoft version then the test fails.  I'm not sure why because I can get the popen2 call to work if I use just the program alone, but no arguments.  For example,


[in, out, pid] = popen2 ("sort");
fprintf (in, "c\nb\na\n")
fclose (in);
fgets (out)
a
fgets (out)
b
fgets (out)
c
fgets (out)
-1


But if I add cmd arguments


[in, out, pid] = popen2 ("sort", "/r");
fprintf (in, "c\nb\na\n")
fclose (in);
fgets (out)
-1


It might be something strange with the the character '/' being auto-translated into '\'.  I tried escaping the forward slash, but it didn't work.

So, I don't know why on my PC popen2() and system() are both picking up the GNU version.  But regardless, there is a problem when the GNU version does not exist.

Rik <rik5>
Group administrator
Tue 26 Nov 2013 02:04:44 AM UTC, comment #8: 

Rik,

Is the problem really solved on MinGW? Because it's not for me on MSVC. The reason is that there's a catch between using "system" and "popen2", how these are translated into Win32 calls and how they interact with PATH.

When you use "system" in octave, which is translated into "system" from the Win32 runtime library, PATH is used first, so you end up finding MSYS version of "sort" (assuming you launched octave from MSYS bash).

When you use "popen2" in octave, this is translated into "CreateProcess" from Win32 API [1], where lpApplicationName is NULL and lpCommandLine looks like "\"sort\" \"-r\"". If you read from the documentation how CreateProcess will lookup for the executable when lpApplicationName  is NULL and the first element of lpCommandLine does not contain a directory, you'll notice that the 32-bits Windows system directory is searched before PATH. And that's precisely where the windows version of "sort" is located. So even if your previous "system" call found the MSYS version, the "popen2" call will find the windows version. And the test will eventually fail.

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx

Michael Goffioul <goffioul>
Mon 18 Nov 2013 11:08:07 PM UTC, comment #7: 

I checked in a changeset that checks for the particular type of sort being used and then modifies the %!test block appropriately (http://hg.savannah.gnu.org/hgweb/octave/rev/c02b8bf0e1f9).  Closing report.

Rik <rik5>
Group administrator
Sun 17 Nov 2013 07:13:47 PM UTC, comment #6: 

I think the %!test code has simply been too naive in assuming that on a PC platform the sort routine will be that provided by Microsoft.  It might, or it might not be, and this could depend entirely on the local setup.  In my case, I have a third variant in that I have manually installed a number of GNU utils using downloadable installers.  So I don't have MinGW installed, and we could remove sort.exe from <OCTAVE_HOME>/bin, and yet I would still be getting a UNIX-like sort program on the command line.

This is pretty easy to test for.  I tried


status = system ("sort.exe --help");


which returns a non-zero value when sort is the Microsoft version.  Alternatively,


status = system ("sort.exe /?");


will return a non-zero value when sort is a UNIX-like version.  It's only a single %!test so it might look kind of ugly to check the sort routine, but there isn't any performance penalty.  What do you think?


Rik <rik5>
Group administrator
Sun 17 Nov 2013 06:33:53 PM UTC, comment #5: 

@comment #3:
Yes, why not? Would make the test easier as well, perhaps we can skip the "if (ispc)" then.

Older Octave binaries for Windows (up until 3.6.4) didn't have sort.exe in <OCTAVE_HOME>/bin, so they relied on the one in %WINDIR%\system32

If we go along this route we need to be sure that Windows builds are guaranteed to have MinGW sort.exe early in the PATH.
That works automagically now, but I simply don't understand why - do you? As long as it isn't clear why, it cannot be ruled out it'll break any moment.

Maybe msys sort.exe is picked up rather than the Windows one as a consequence of the patches JWE implemented to get <OCTAVE_HOME>/bin added to the PATH.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 17 Nov 2013 01:15:32 PM UTC, comment #4: 

On my system, I don't even have have windows in the path:

>> strsplit (getenv("PATH"), pathsep)
ans =
{
  [1,1] = C:\Program Files\TortoiseHg\
  [1,2] = C:\mingw\MiKTeX 2.9\miktex\bin\x64\
  [1,3] = C:\octave-2013-11-16-20-26\bin
  [1,4] = C:\octave-2013-11-16-20-26\notepad++
  [1,5] = C:\octave-2013-11-16-20-26\libexec\octave\3.7.7+\site\exec\i686-pc-mingw32
  [1,6] = C:\octave-2013-11-16-20-26\libexec\octave\api-v48+\site\exec\i686-pc-mingw32
  [1,7] = C:\octave-2013-11-16-20-26\libexec\octave\site\exec\i686-pc-mingw32
  [1,8] = C:\octave-2013-11-16-20-26\libexec\octave\3.7.7+\exec\i686-pc-mingw32
  [1,9] = C:\octave-2013-11-16-20-26\bin
}
>>


John Donoghue <lostbard>
Group Member
Sun 17 Nov 2013 11:57:54 AM UTC, comment #3: 

Why not just use sort from msys?

John Donoghue <lostbard>
Group Member
Sun 17 Nov 2013 10:53:08 AM UTC, comment #2: 

<OCTAVE_HOME>/bin/sort.exe sneaks in because it is part of msys-coreutils that gets built in mxe-octave.

E.g., renaming it to __sort.exe (in the installed binary on Windows) removes the syscalls.cc FAIL.

But somehow this is a bit intriguing.
Am I right to think (based on Windows PATH) that Octave is supposed to pick up sort.exe in C:\WINDOWS\SYSTEM32 (that does accept an "/R" flag)?

>> strsplit (getenv("PATH"), pathsep)
ans =
{
  [1,1] = C:\Programs\Python33\
  [1,2] = C:\WINDOWS\system32
  [1,3] = C:\WINDOWS
  [1,4] = C:\WINDOWS\System32\Wbem
  [1,5] = C:\Programs\TortoiseSVN\bin
  [1,6] = C:\Programs\gs\gs9.04\bin
  [1,7] = C:\Programs\QuickTime\QTSystem\
  [1,8] = C:\Programs\Mercurial
  [1,9] = X:\Octave\octave-2013-11-16-20-38\bin
  [1,10] = X:\Octave\octave-2013-11-16-20-38\notepad++
  [1,11] = X:\Octave\octave-2013-11-16-20-38\libexec\octave\3.7.7+\site\exec\i686-pc-mingw32
  [1,12] = X:\Octave\octave-2013-11-16-20-38\libexec\octave\api-v48+\site\exec\i686-pc-mingw32
  [1,13] = X:\Octave\octave-2013-11-16-20-38\libexec\octave\site\exec\i686-pc-mingw32
  [1,14] = X:\Octave\octave-2013-11-16-20-38\libexec\octave\3.7.7+\exec\i686-pc-mingw32
  [1,15] = X:\Octave\octave-2013-11-16-20-38\bin
}


Apparently Octave prefers the sort.exe in <OCTAVE_HOME>/bin even when it appears last in the system PATH. Note that I have no "." in my Windows PATH.

For mxe-octave builds, the easiest solution /to get this specific test to succeed/ would be to not build sort.exe or remove it from the dist (it is in <mxe-octave>/msys-base/bin/).
But the test is meant for pipes (or actually system utility calls), not for a specific utility program.

Yet does uncover a problem.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 17 Nov 2013 06:16:01 AM UTC, comment #1: 

Microsoft's sort.exe isn't documented to accept GNU-style option syntax ("-r").  It seems probable that you might be picking up a different sort.exe, perhaps installed as part of MinGW.  Do you have the which utility installed?  If so, you might open a cmd window and use 'which sort' to find the executable being run.  You could also try searching the entire hard drive for sort.exe and check if there are multiple versions.

Rik <rik5>
Group administrator
Sat 16 Nov 2013 10:23:20 PM UTC, original submission:  

On my fedora 19 building mxe-octave with current octave-dev 3.3.7+, installing on a Windows 7 machine and running _run_test_suite_:

sys-calls.cc-tst fails.

Running the same test line for line:

failure 1:

[in, out, pid] = popen2 ("sort", "/R");

sort expects -r option as the unix version does.

Changing it /R to -r provides an asset assert as the result doesn't match expected of:

assert (str, {"these\r\n","strings\r\n","some\r\n","are\r\n"});


Removing the \r and it works.


John Donoghue <lostbard>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by lostbard (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-11-26 rik5 Open/ClosedOpen Closed
    2013-11-26 rik5 StatusIn Progress Fixed
    2013-11-26 rik5 StatusFixed In Progress
        Open/ClosedClosed Open
    2013-11-18 rik5 Open/ClosedOpen Closed
    2013-11-18 rik5 StatusIn Progress Fixed
    2013-11-17 rik5 StatusNone In Progress

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code