bugGNU Octave - Bugs: bug #43036, popen2 on Windows fails quietly if...

 
 

bug #43036: popen2 on Windows fails quietly if command writes to stderr

Submitter:  Colin Macdonald <cbm>
Submitted:  Wed 20 Aug 2014 10:13:23 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 3.8.1 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 03 Jul 2016 05:12:34 AM UTC, comment #22: 

Thanks for confirming the fix. Note that bug #48149 still affects 4.0.3 and default.

Mike Miller <mtmiller>
Group Member
Sun 03 Jul 2016 03:37:25 AM UTC, comment #21: 

Octave-4.0.3 (source and windows installer are now available.)
The bug of popen2 on this report are now fixed in official release.


>> popen2_bug2

EAGAIN =  11
EINVAL =  22
in =  10
out =  3
pid =  7116
<waiting> (win32), s=-1, errno=22
hello stderr
Hello 1
Hello 2
Hello 3
NOW WE TALK TO STDERR...
Betcha will not see this!
Hello 4
Hello 5
Hello 6
Goodbye
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
no many waits, breaking out


Tatsuro MATSUOKA <tmacchant>
Mon 23 May 2016 12:54:26 AM UTC, comment #20: 

When I re-installed windows 10 on my PC, I did not install native windows version of python and sympy.  Therefore I cannot test at the moment.

Instead I am building stale branch of octave for windows using mxe-octave.  I will upload it on my personal web site and its url will be sent you via private mail.

Please wait a while.

Tatsuro MATSUOKA <tmacchant>
Sun 22 May 2016 10:32:16 PM UTC, comment #19: 

Tatsuro MATSUOKA: would you be able to test Symbolic with this fix?  You'd have to change one line in "python_ipc_open2.m".  I'll attach a patch, but basically its changing line 85 to be "pyexec = 'python';".

If it works, then I'll patch Symbolic to not use winwrapy.bat for Octave > 4.0.2.

Thanks if you are able to do this!  Alternatively, if you could make your windows binary available, then I could try to find a system to test on.

(file #37230)

Colin Macdonald <cbm>
Sun 22 May 2016 06:44:07 PM UTC, comment #18: 

This is fixed on stable now too (in case there will be another 4.0.x stable release).

Mike Miller <mtmiller>
Group Member
Sat 21 May 2016 05:19:57 AM UTC, comment #17: 

Yes please commit to stable!  This significantly improves Symbolic on Windows, allowing me to drop some messy workarounds.

Colin Macdonald <cbm>
Fri 20 May 2016 06:56:19 PM UTC, comment #16: 

Thanks for testing, I pushed the change on the default branch

http://hg.savannah.gnu.org/hgweb/octave/rev/30d58f5cc7d0

This is simple enough that it probably could be made on stable also if needed.

Mike Miller <mtmiller>
Group Member
Thu 19 May 2016 12:12:25 AM UTC, comment #15: 

I have applied Mike's patch to 4.0.2 source and rebuild octave
using producer [1].

http://octave.1599824.n4.nabble.com/Octave-build-on-windows-using-octave-binary-itself-td4676652.html


>> popen2_bug2
EAGAIN =  11
EINVAL =  22
in =  4
out =  3
pid =  3592
<waiting> (win32), s=-1, errno=22
hello stderr
Hello 1
Hello 2
Hello 3
NOW WE TALK TO STDERR...
Betcha will not see this!
Hello 4
Hello 5
Hello 6
Goodbye
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
no many waits, breaking out


I could see


Betcha will not see this!
Hello 4
Hello 5
Hello 6
Goodbye


thus Mike's patch seems to fix this issue.

Mike. Please apply your patch to the default branch.

Tatsuro MATSUOKA <tmacchant>
Wed 18 May 2016 11:50:40 AM UTC, comment #14: 

Colin Macdonald

Thanks for the reply. I will manage the mxe-octave or windows native build [1] with Mike's patch tomorrow.

[1] http://octave.1599824.n4.nabble.com/Octave-build-on-windows-using-octave-binary-itself-td4676652.html

Tatsuro MATSUOKA <tmacchant>
Wed 18 May 2016 05:24:19 AM UTC, comment #13: 

Its been awhile but I think that's right.  Here's my summary:

1. Looks like I've made some mistakes here ("echo_bug2.m" should be "echo_bug2.bat", the single quote thing you mentioned).

2. The crucial thing is whether Octave ("popen2_bug2.m") gets any messages after the .bat file writes to stderr.  I.e., whether you see "Hello 4", "Hello 5", etc.

3. So I don't think the "<waiting> (somewhere)" is too important.

Appreciate you testing this!

Colin Macdonald <cbm>
Wed 18 May 2016 12:17:49 AM UTC, comment #12: 

Before trying to the patch by Mike, I would like to know what is expected behavior.

I have executed popen2_bug2.m with echo_bug2.bat on windows (4.0.2)


>> popen2_bug2
EAGAIN =  11
EINVAL =  22
in =  4
out =  3
pid =  3040
<waiting> (win32), s=-1, errno=22
Hello 1
Hello 2
Hello 3
NOW WE TALK TO STDERR...
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
<waiting> (somewhere), s=-1, errno=0
no many waits, breaking out



I have executed the similar on octave on Cygwin (4.0.1) which has
posix popen2.

I have generated echo_bug2.sh as

echo Hello 1
echo Hello 2
echo Hello 3

echo NOW WE TALK TO STDERR...
echo hello stderr 1>&2
echo Betcha will not see this!

echo Hello 4
echo Hello 5
echo Hello 6
echo Goodbye

(I have modified "won't" to "will not" to kill a single quotation mark.)

and use


[in, out, pid] = popen2 ('./echo_bug2.sh')


instead of

[in, out, pid] = popen2 ('echo_bug2.bat')


The result of Cygwin


EAGAIN =  11
EINVAL =  22
in =  12
out =  13
pid =  8616
<waiting> (unix), s=-1, errno=11
hello stderr
Hello 1
Hello 2
Hello 3
NOW WE TALK TO STDERR...
Betcha will not see this!
Hello 4
Hello 5
Hello 6
Goodbye
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
<waiting> (unix), s=-1, errno=11
no many waits, breaking out


For octave for windows
Is
<waiting> (win32), s=-1, errno=0
expected but not
<waiting> (somewhere), s=-1, errno=0
?
Am I right?

Tatsuro MATSUOKA <tmacchant>
Tue 17 May 2016 08:27:10 PM UTC, comment #11: 

I know nothing at all about the Windows API either, but can someone test whether this simple patch fixes the error?

(file #37192)

Mike Miller <mtmiller>
Group Member
Tue 05 Apr 2016 03:38:15 PM UTC, comment #10: 

Looking into the code of octave_popen2 in lo-sysdep.cc.
http://octave.org/doxygen/4.1/d9/d15/lo-sysdep_8cc_source.html

The codes seem not to consider stderr. This is natural considering the original motivation of implementation of popen2. The motivation of implementation of popen2 on octave for windows to uses bi-directional pipe to communicate with gnuplot.exe. Therefore the popen2 function only takes in account communicating with stdin and stdout but not with stderr.

I'm not familiar with the win32 api programing. However, there are examples to communicate not only with stdin and stdout but also with stderr. (I have found them written in Japanese so that I do not refer them.)

Perhaps for those who are familiar with win32 api programing, the correction of the code of the octave_popen2 for stderr is not a hard work. As I told, unfortunately I have little experience with win32 api programming. We need help from those who are good experience of win32 api programing.

Tatsuro MATSUOKA <tmacchant>
Sat 02 Apr 2016 12:24:04 PM UTC, comment #9: 

I have quick looked at the gnulib. The gnulib does not have popen2. Therefore we have to improve the codes in lo-sysdep.cc. Unfortunately, the original author has not appear in more than two years in the octave community.

Tatsuro MATSUOKA <tmacchant>
Sat 02 Apr 2016 12:05:22 PM UTC, comment #8: 

The popen2 for windows was implemented before octave started to use the gnulib. Until now,octave uses popen2 in lo-sysdep.cc.In lo-sysdep.cc, popen2 is implemented using win32 API that related to pipe.Perhaps one should improve the codes.

Tatsuro MATSUOKA <tmacchant>
Wed 27 Aug 2014 01:24:59 PM UTC, comment #7: 

Ok, good so you see the same as me now.

If ping writes to stderr, then you'd only get Hello 1, so that is one possibility for the other result.

I also got some strange results when I tried to redir ping's stderr to NUL (add "2> NUL" to the ping calls).  Basically I could see the "hello stderr" msg.  But I could not see the "NOW WE TALK TO STDERR" msg.  Very odd, although its possible that I don't understand batch scripts and I'm redirecting something other than the ping output!  I'll attach that .bat file in case its helpful.

Back to my original report: Do you know whether we are running the "roll-your-own popen2 from lo-sysdep.cc"?  IIRC, non-posix systems don't have pipes so it must be emulated somewhere, the question is, by the MXE stuff or by the implementation in lo-sysdep.cc?


(file #31983)

Colin Macdonald <cbm>
Wed 27 Aug 2014 12:44:26 PM UTC, comment #6: 

Hello 1
Hello 2
Hello 3
NOW WE TALK TO STDERR


It may be something related though that causes it to not read the data.

John Donoghue <lostbard>
Group Member
Wed 27 Aug 2014 11:26:55 AM UTC, comment #5: 

And w/o weights?  popen2_bug2.m

Colin Macdonald <cbm>
Wed 27 Aug 2014 10:23:52 AM UTC, comment #4: 

With the waits I see:
Hello 1


John Donoghue <lostbard>
Group Member
Wed 27 Aug 2014 09:23:57 AM UTC, comment #3: 

The reason I used the ping waits was to demonstrate that after the stderr is written to, fgets returns -1 but errno is still 0 (!).  This is what is indicated by the "<waiting> (somewhere)" message.

Before that, errno is EINVAL (this is like EAGAIN on unix?). 
Perhaps that EINVAL/EAGAIN thing is a bug too, but if so its an aready known as per tests in libinterp/corefcn/syscalls.cc.

Probably the s = -1 and errno = 0 condition represents EOF, but I'm not completely sure.

Anyway, none of that is directly related to this bug.

Colin Macdonald <cbm>
Wed 27 Aug 2014 09:18:12 AM UTC, comment #2: 

Are these the 4 lines you see?

Hello 1
Hello 2
Hello 3
NOW WE TALK TO STDERR


That is indeed the bug.  If not those 4 lines, can you post them and tell me what octave are you using?  Mine is the MXE build.

The ping/wait thing is not really part of the bug, just makes it a bit more like my real IPC case.  Its possible that ping command does not universally work as wait, so that might your first result... (if it gets stuck on the ping command, or for example if ping writes to stderr.)  I'll upload echo_bug2.bat without the ping waits.



(file #31978)

Colin Macdonald <cbm>
Wed 27 Aug 2014 01:18:13 AM UTC, comment #1: 

On my win32 octave 3.8.2, I only ever get the first line that was written to stdout from the bat file.

If I remove the ping 'waits' I get the 4 lines from the bat stdout.

John Donoghue <lostbard>
Group Member
Wed 20 Aug 2014 10:13:23 PM UTC, original submission:  

If a popen2 subprocess every outputs to stderr, octave can't read any more from its stdout.

I'll attach two files.  The .m file does popen2() on the .bat file.  The .bat file echos to stdout several times, then once to stderr.  Subsequent stdout never makes it back to Octave.

I've tested this on MXE.  Its not clear to me whether this runs the "roll-your-own" popen2 from lo-sysdep.cc or something else.

I'm not easily able to test more recent versions (well, probably 3.8.2), although let me know if not reproducible and I'll see what I can do.  (I don't have any admin rights on the windows box, its called something like "Windows 2008 Server" if that's relevant).

On a personal note: I'm rather proud of myself for isolating this.  Working on Windows feels like building lego models in the dark with marshmallows instead of fingers!

Colin Macdonald <cbm>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #37230:  symbolic_2_3_0_skip_winwrapy.patch added by cbm (758B - text/x-patch - for testing)
file #37192:  bug43036.diff added by mtmiller (443B - text/x-diff)
file #31983:  echo_bug_wierd.m added by cbm (592B - text/x-objcsrc)
file #31982:  popen2_bug2.m added by cbm (785B - text/x-objcsrc - same as popen2_bug.m but calls echo_bug2.m (without ping waits))
file #31978:  echo_bug2.m added by cbm (185B - text/x-objcsrc - this version has no ping waits, you can change the .m file to call this on instead)
file #31928:  popen2_bug.m added by cbm (784B - text/x-objcsrc)
file #31929:  echo_bug.bat added by cbm (547B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Updated the item)
  • -email is unavailable- added by tmacchant (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by cbm (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-05-22 cbm Attached File- Added symbolic_2_3_0_skip_winwrapy.patch, #37230
    2016-05-20 mtmiller StatusNone Fixed
        Open/ClosedOpen Closed
    2016-05-17 mtmiller Attached File- Added bug43036.diff, #37192
    2014-08-27 cbm Attached File- Added echo_bug_wierd.m, #31983
    2014-08-27 cbm Attached File- Added popen2_bug2.m, #31982
    2014-08-27 cbm Attached File- Added echo_bug2.m, #31978
    2014-08-20 cbm Attached File- Added popen2_bug.m, #31928
        Attached File- Added echo_bug.bat, #31929

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code