bugGNU Octave - Bugs: bug #41148, [octave forge] (parallel) fails to...

 
 

bug #41148: [octave forge] (parallel) fails to build in windows/mingw

Submitted by:  John Donoghue <lostbard>
Submitted on:  Wed 08 Jan 2014 02:48:02 PM UTC  
 
Category: Octave Forge PackageSeverity: 1 - Wish
Priority: 3 - LowItem Group: Build Failure
Status: PostponedAssigned to: None
Originator Name: John DOpen/Closed: Open
Release: otherOperating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 04 May 2016 08:27:27 PM UTC, comment #32:

Mike, surely it can be installed. But I'd prefer that it can't be installed if it doesn't work.

And the m-code call in parcellfun to Octaves fork() user function is not the only instance of calling the fork() system call. There is also a server function in the package which uses the fork() system call directly. This function will just segfault if this system call is not provided by the OS.

In bug #47843, I've just recommended against trying to implement fork in Octave on MINGW.

And after looking at the code of Cygwin, which somehow implements fork with CreateProcessW, I understand that there is no cleaner solution (as I initially thought possible) to use CreateProcess than to emulate fork() with it. And I think the way to implement fork() with CreateProcess is just terrible. I certainly won't try to copy this effort in the parallel package.

So what I probably do is to check for fork() at configure time, and if lacking, at least disable the server, but probably also abort installation, since installation would be useless.

Olaf Till <i7tiol>
Project Member
Mon 02 May 2016 02:19:34 AM UTC, comment #31:

As for fork on windows, I have posted another thread in Octave ML.
I would like to hear comments from Octave experts.

http://octave.1599824.n4.nabble.com/fork-for-microsoft-windows-native-but-not-Cygwin-td4676646.html

For install problem of parallel packages, the problem may be solved as Mike wrote. Now we can move the discussion how to treat fork() on native windows.

Tatsuro MATSUOKA <tmacchant>
Mon 02 May 2016 01:25:10 AM UTC, comment #30:

I guess we can call this fixed, because the package can now be built and installed successfully in Windows. It doesn't work, but at least it can be installed. The fact that fork doesn't work is a separate bug.

Btw, Olaf, you could use the 2-argument form of fork to get the error message to display the reason why fork failed (I don't know the reason for having an error message return value vs just raising an error).

Mike Miller <mtmiller>
Project Administrator
Sun 01 May 2016 07:33:32 PM UTC, comment #29:

For the implementation fork() on Microsoft windows, I posted to the octave maintainers ML.

http://octave.1599824.n4.nabble.com/fork-for-microsoft-windows-native-but-not-Cygwin-td4676646.html

Tatsuro MATSUOKA <tmacchant>
Sun 01 May 2016 06:06:02 PM UTC, comment #28:

>As for the other issue, it seems that the good news that 'pipe()' (m-code) is now working in Octave at MINGW is not enough -- we also need 'fork()' (m-code) to work ... This applies regardless of whether 'parallel' is configured with or without --disable-cluster.


Native windows C compiler does not have fork(). However, fork () exist in octave. We have to know how the octave implement of fork() on native windows. But this is beyond this topic.

Tatsuro MATSUOKA <tmacchant>
Sun 01 May 2016 05:30:30 PM UTC, comment #27:

> How can I check the install is valid?


The install, configured without --disable-cluster, is not valid on MINGW, since usage of winsock is not yet implemented. What I had liked to be tested was only if the configuration proceeded as expected (see below).

>


As for the other issue, it seems that the good news that 'pipe()' (m-code) is now working in Octave at MINGW is not enough -- we also need 'fork()' (m-code) to work ... This applies regardless of whether 'parallel' is configured with or without --disable-cluster.

And this also means that 'fork()' (C-code) has to be implemented for MINGW in the 'parallel' package also, at least if 'parallel' will be configured without --disable-cluster.

Olaf Till <i7tiol>
Project Member
Sun 01 May 2016 12:08:47 PM UTC, comment #26:
Tatsuro MATSUOKA <tmacchant>
Sun 01 May 2016 11:56:20 AM UTC, comment #25:

http://octave.1599824.n4.nabble.com/optim-struct-parallel-pre-release-check-on-Windows-td4675995.html#a4675999

I executed:

Tatsuro MATSUOKA <tmacchant>
Sun 01 May 2016 11:15:09 AM UTC, comment #24:

I could install the parallel into the octave-4.0.2.

How can I check the install is valid?

Tatsuro MATSUOKA <tmacchant>
Sun 01 May 2016 10:59:21 AM UTC, comment #23:

Yes, this cloning command should get you the correct repository.

Olaf Till <i7tiol>
Project Member
Sun 01 May 2016 10:41:36 AM UTC, comment #22:

Sorry. Cloning command does not appear in the previous comment.

hg clone http://hg.code.sf.net/p/octave/parallel octave-parallel

Tatsuro MATSUOKA <tmacchant>
Sun 01 May 2016 10:39:23 AM UTC, comment #21:

Olaf

This just confirmation.

Is the repository place the below?
https://sourceforge.net/p/octave/parallel/ci/default/tree/

is the command cloning source?

Tatsuro MATSUOKA <tmacchant>
Sun 01 May 2016 10:25:47 AM UTC, comment #20:

As a first step, a changeset is pushed which considers winsock2 at configure time. I can't test it, maybe someone likes to do it. On MINGW, it should substitute HAVE_WINSOCK := yes in src/Makefile and define HAVE_LIBWS2_32 as 1 in config.h.

Olaf Till <i7tiol>
Project Member
Sat 30 Apr 2016 07:51:46 PM UTC, comment #19:

1. Yes, this is intentional. Gnulib is meant to be used internally only, an implementation detail. If a project wants to build against both Octave and gnulib, then it should embed gnulib on its own.

2. Correct, a package must deal with implementation specific stuff itself, using either gnulib or any other approach.

Mike Miller <mtmiller>
Project Administrator
Sat 30 Apr 2016 07:39:00 PM UTC, comment #18:

Copied from duplicated post.

parallel package cannot be install due to the lack of <sys/socket.h> on windows

>> pkg install -verbose parallel-3.1.0.tar.gz

<snip>
checking sys/socket.h usability... no
checking sys/socket.h presence... no
checking for sys/socket.h... no
<snip>
make: Entering directory `/tmp/oct-1B8Dca/parallel-3.1.0/src'
CXX="g++ -std=gnu++11" CXXFLAGS="-g -O2 -IC:/octave/Octave-4.0.2/include -Wno-int-to-pointe
r-cast" C:/octave/Octave-4.0.2/bin/mkoctfile-4.0.2.exe --verbose -c pconnect.cc
g++ -std=gnu++11 -c -I/scratch/jwe/mxe-octave/usr/i686-w64-mingw32/include -IC:\octave\Octa
ve-4.0.2\include\octave-4.0.2\octave\.. -IC:\octave\Octave-4.0.2\include\octave-4.0.2\octave
-IC:\octave\Octave-4.0.2\include -pthread -mieee-fp -g -O2 -IC:/octave/Octave-4.0.2/include
-Wno-int-to-pointer-cast pconnect.cc -o pconnect.o
pconnect.cc:29:24: fatal error: sys/socket.h: No such file or directory
#include <sys/socket.h>
^
compilation terminated.
make: *** [pconnect.o] Error 1
make: Leaving directory `/tmp/oct-1B8Dca/parallel-3.1.0/src'

pkg: error running `make' for the parallel package.
error: called from 'configure_make' in file C:\octave\Octave-4.0.2\share\octave\4.0.2\m\pkg\
private\configure_make.m near line 96, column 9

Apparently configure reports the lack of sys/socket.h.
In octave build tree for 4.0.2 on Ubunu
sys/socket.h
is defined in libgnu directory.
In octave build, sys/socket.h is defined in libgnu and there is no problem.
However, gnulib stuff is completely removed in the binary distribution of octave for windows.
The native windows does not have sys/socket.h.
This causes the error above.

Questions
1. gnulib stuff is completely removed in the binary distribution of octave for windows. Is this intentional?
2. Octave-forge package is build outside of octave build trees.
Removal of the gnulib stuff from the binary distribution suggests that their own implementations of cross-platform development stuff.

Tatsuro MATSUOKA <tmacchant>
Sun 31 May 2015 09:44:21 PM UTC, comment #17:

So it looks like the short-term problem of not being able to install the optim package has been resolved for Windows, but the parallel package does not install. Updating bug status fields to reflect that.

If it's decided that there's no point in making parallel Windows-compatible, this can be updated again later.

Mike Miller <mtmiller>
Project Administrator
Sat 08 Feb 2014 12:34:21 PM UTC, comment #16:

Current solution is still to install the optim that comes packages with the windows octave installer.

John Donoghue <lostbard>
Project Member
Sat 08 Feb 2014 08:45:25 AM UTC, comment #15:

Hi all. What's the newest status on the solution now? I tried to install parallel from forge because of the optim dependency, but the same problem occurred to my windows version of Octave. Please help, thanks!

Alex

Alex Feng <looopass>
Thu 09 Jan 2014 02:21:58 PM UTC, comment #14:

An additional remark: I have the plan to offer TLS in 'parallel', maybe as the default, since the current implementation with sockets can be a security risk. So there will possibly be further difficulties with windows compatibility in the future which I can't solve alone. So even if we reach windows compatibility for now, it may disappear again in the future.

As for 'optim', it occured to me that it should also be possible to perform a runtime check for 'parallel' with a warning if it is not present and parallelity is chosen in the optimization options.
Then we could drop the dependency on 'parallel'.

Olaf Till <i7tiol>
Project Member
Thu 09 Jan 2014 01:57:30 PM UTC, comment #13:

Oh, I didn't remember the poll()s now (that is Hayato Fujiwaras code), sorry. But I think the poll() can be replaced by a suitable select() call, so probable no unsolvalble problem here.

Thanks for the instruction to construct the C stream from the socket descriptor on Windows. Could

osfh = _open_osfhandle((SOCKET)newsockfd, O_RDWR|O_BINARY);

also be used for read(osfh) and write(osfh)? This would probably spare a bit of effort.

According to what you told me it should be feasible to adapt 'parallel' to mingw, but I currently don't know when I will try this. I won't be sad if you should beat me with that (ifdefs for replacing read/write, replace poll() with select(), ifdefs for constructing the C stream in sock-stream.cc). But tell me if you should attempt it, to avoid double work. You could of cause push directly to the repository, I'll still have a look at it.

In the meantime, current 'optim' can only be used on mingw as an adapted package as described below (AFAIK there is currently no means to solve this in the repository for 'optim', since the dependency information is evaluated before configure is run).

Olaf Till <i7tiol>
Project Member
Thu 09 Jan 2014 01:29:42 PM UTC, comment #12:

There is #include <sys/poll.h>, a number of poll calls in pserver.cc, recv.cc, reval.cc, sclose.cc and send.cc.

For the stream this might work:
osfh = _open_osfhandle((SOCKET)newsockfd, O_RDWR|O_BINARY);
file = fdopen( osfh, "rwb" );

Maybe of help:
http://tangentsoft.net/wskfaq/articles/bsd-compatibility.html

John Donoghue <lostbard>
Project Member
Thu 09 Jan 2014 01:07:16 PM UTC, comment #11:

Carnë: The suggestion for disabling the rest of parallel was only meant as a solution for windows for the case that getting the whole package to work there isn't feasible. But according to Johns answer partial disabling will probably no option.

John: poll is not used in parallel, select is used. Good news that it is available. The issue with read/write I could probably work around where it is used directly.

There remains one crucial issue. Can the file descriptor of a socket be converted to a valid C stream with fdopen() (or something equivalent) on Windows? 'parallel' does this to connect an Octave stream to the socket in order to receive/send Octave variables using Octaves own coding.

Olaf Till <i7tiol>
Project Member
Thu 09 Jan 2014 11:59:53 AM UTC, comment #10:

sockets requires -lws2_32, winsocks2.h, call of WSAStartup to initialize winsockets.

the poll function is implemented as WSAPoll, but for some reason was having link issues on my machine, and reportedly is broken.

read, write don't work for sockets.

Select is select in the winsock library.

John Donoghue <lostbard>
Project Member
Thu 09 Jan 2014 11:35:24 AM UTC, comment #9:

> [...]
> I'd say I'll disable (at configure time) everything in 'parallel' except the functions needed for optim.
> [...]


The parallel package is not only a dependency for optim. Please do not turn it into that. What if other packages are to become dependent on other stuff from parallel?

Carnë Draug <carandraug>
Project Member
Thu 09 Jan 2014 09:41:33 AM UTC, comment #8:

Correction: 'select' is also needed for optim. Is this one of the unix functions which would need to be implemented for mingw?

Olaf Till <i7tiol>
Project Member
Thu 09 Jan 2014 09:38:37 AM UTC, comment #7:

The focus should be that parallel doesn't block optim. For optim, the only important functions of parallel are those which send variables over Octave streams. All the socket stuff isn't necessary.

My ability to change something for windows, or mingw, depends on others providing me the necessary information. If the problem can't be solved with adapted include directives and would need implementation of unix functions (which I wouldn't regard as a bug in the parallel package), I'd say I'll disable (at configure time) everything in 'parallel' except the functions needed for optim. But I wouldn't like to test for windows or mingw for doing this, I'd rather test for the presence of the unix function you mentioned. So could you tell me which these are?

This seems to need a bit more time than expected. The next fixes should be tested from the repository without a release.

Another way to make a working package of 'optim' for windows (without changing the code in the repository) could be to remove the dependency on 'parallel' and throw an error ("Parallelity not implemented on Windows.") directly after 'if (parallel_local)' both in inst/private/__dfdp__.m and in inst/jacobs.m. Maybe you'd prefer to do that?

Olaf Till <i7tiol>
Project Member
Wed 08 Jan 2014 09:28:42 PM UTC, comment #6:

A quick look at the code and I think there are several unix functions used that are going to need to be implemented for mingw compilation.

John Donoghue <lostbard>
Project Member
Wed 08 Jan 2014 08:15:30 PM UTC, comment #5:

No more problems with symbolic links however next issue is that mingw doesn't have sys\sockets.h.

A few ifdefs are needed for Winsock (same is done in source forge sockets package) and link with winsock on windows.

If you want, I can get it compiled on mingw and send the changes.

John Donoghue <lostbard>
Project Member
Wed 08 Jan 2014 07:23:00 PM UTC, comment #4:

bug fix released done.

Carnë Draug <carandraug>
Project Member
Wed 08 Jan 2014 03:38:01 PM UTC, comment #3:

Should be fixed now, bugfix release pending.

Olaf Till <i7tiol>
Project Member
Wed 08 Jan 2014 03:18:30 PM UTC, comment #2:

Probably because Windows filesystems have no symlinks. Symlinking was not necessary, I can move autogen.sh to bootstrap. And now that optim depends on parallel we'll probably have to do the next release immediatly. That was fast.

Olaf Till <i7tiol>
Project Member
Wed 08 Jan 2014 02:54:27 PM UTC, comment #1:

Adding maintainer of the parallel package to notification list.

Carnë Draug <carandraug>
Project Member
Wed 08 Jan 2014 02:48:02 PM UTC, original submission:

Running:

John Donoghue <lostbard>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -unavailable- added by jwe (Updated the item)
  • -unavailable- added by tmacchant (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by looopass (Posted a comment)
  • -unavailable- added by i7tiol (Posted a comment)
  • -unavailable- added by carandraug (Olaf Till - maintainer of parallel package)
  • -unavailable- added by lostbard (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 13 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 13 Aug 2017 12:52:50 PM UTCjweSummaryparallel package: fails to build in windows/mingw=>[octave forge] (parallel) fails to build in windows/mingw
    Sat 30 Apr 2016 07:22:45 PM UTCmtmillerDependencies-=>bugs #47810 is dependent
    Sat 30 Apr 2016 07:22:32 PM UTCmtmillerDependencies-=>bugs #42572 is dependent
    Sun 31 May 2015 09:44:21 PM UTCmtmillerSeverity3 - Normal=>1 - Wish
      Priority5 - Normal=>3 - Low
      StatusFixed=>Postponed
      Release3.8.0=>other
      SummaryParallel 2.1.0 fails to build in windows/mingw=>parallel package: fails to build in windows/mingw
    Sat 26 Apr 2014 12:23:14 AM UTClostbardDependencies-=>bugs #42197 is dependent
    Wed 08 Jan 2014 08:15:30 PM UTClostbardOpen/ClosedClosed=>Open
    Wed 08 Jan 2014 07:23:00 PM UTCcarandraugStatusNone=>Fixed
      Open/ClosedOpen=>Closed
    Wed 08 Jan 2014 02:54:27 PM UTCcarandraugCarbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup1