bugGNU Octave - Bugs: bug #47843, fork implementation on native...

 
 

bug #47843: fork implementation on native windows

Submitter:  Tatsuro MATSUOKA <tmacchant>
Submitted:  Wed 04 May 2016 11:21:45 AM UTC
   
 
Category:  Libraries Severity:  1 - Wish
Priority:  5 - Normal Item Group:  Feature Request
Status:  Confirmed Assigned to:  None
Originator Name:  Tatsuro MATSUOKA Open/Closed:  * Open
Release:  * dev Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 05 May 2016 07:57:36 AM UTC, comment #10: 

There are threads, of course. But to use threads for arbitrary user code, there would have to be changes in Octave. You probably know much better than me that these changes would be considerable.

If you want to do parallel computing for arbitrary user code with a package, with the current state of Octave, I think some pendent to fork is necessary: You want to have all user-variables of the parent available in the child, but you don't want all user-variables of the parent over-writable by a child.

Olaf Till <i7tiol>
Group Member
Wed 04 May 2016 08:53:30 PM UTC, comment #9: 

Surely there must be some other way to do parallel computing on Windows systems that does not require fork (which is a Unix concept anyway).

John W. Eaton <jwe>
Group administrator
Wed 04 May 2016 08:36:26 PM UTC, comment #8: 

It's off topic, but for the record:

Contrary to what I thought, reading in the Cygwin source indicates that there is no cleaner possibility to use the process creation model of Windows for the purpose of the parallel package than to emulate fork() with it. And I just mentioned on bug #41148 that I won't try to copy this terrible effort of Cygwin in the parallel package. (And I still recommend not to do it in Octave either.)

Olaf Till <i7tiol>
Group Member
Wed 04 May 2016 06:06:04 PM UTC, comment #7: 

Having read:

https://www.cygwin.com/faq.html#faq.using.fixing-fork-failures

I would now discourage trying to simulate fork for MINGW in Octave.

There may be a clean solution possible within the parallel package, however, by applying the process creation model of Windows, on systems where this is necessary.

(I've not yet made myself familiar with Windows' process creation model, and I don't know how much time I'll have for this...)

Olaf Till <i7tiol>
Group Member
Wed 04 May 2016 05:02:51 PM UTC, comment #6: 

Tatsuro: If the performance of fork() on Cygwin is enough in a practial sense only depends on how much parcellfun has to do. The child processes are generated only once in a call to parcellfun, and even a bad performance of fork() can allow useful calls of parcellfun in computationally extensive tasks.

Olaf Till <i7tiol>
Group Member
Wed 04 May 2016 04:53:48 PM UTC, comment #5: 


> Is there some other method that could be used on Windows
> (or all) systems to make parcellfun work without having to
> use fork?


I can't imagine how the functionality of parcellfun could be achieved with threads.

The complicated hack to start new Octave instances with system() and to use named pipes for data transfer would leave considerable issues with passing of parameters and functions.

So I see no alternative to fork here.

> What would you do in C++ code to make parcellfun work
> on Windows systems without using fork?


If it's decided to leave this problem to the parallel package, I'd probably try to make an oct-function which wraps fork, if possible, and calls something different (CreateProcess?) if fork is not present. I'd probably look at the way Cygwin does the latter.

Mmh... probably this will be difficult, and some say Cygwin has a leaky solution for this:

http://stackoverflow.com/questions/985281/what-is-the-closest-thing-windows-has-to-fork

And now it occurs to me that there may be an additional problem for parcellfun: it nees the Unix

_exit()

function in the child processes, and I don't know a Windows pendant. Maybe Octave could be pursued to exit without calling functions registered with atexit(), then there would be no need for

_exit()

...

So currently I don't know if there will be a feasible solution for parcellfun on MINGW at all. Maybe it is better to try a solution in the parallel package first, including the issue with exit(). If it turns out there is a clean solution for fork, something similar could be done in Octave.

Olaf Till <i7tiol>
Group Member
Wed 04 May 2016 04:36:08 PM UTC, comment #4: 

Perhaps if we may implement fork, its performance at best is similar to that of the Cygwin.  Experiments on the performance can be done on the octave on Cygwin because the parallel package works on Cygwin version of octave.
If the performance of parallel package on Cygwin is enough in the practical sense, it is worth to implement fork function on octave for native windows.
If you feel performance is not enough parallel package on Cygwin, you have to consider other way as JWE said.

Tatsuro MATSUOKA <tmacchant>
Wed 04 May 2016 03:53:15 PM UTC, comment #3: 

Is there some other method that could be used on Windows (or all) systems to make parcellfun work without having to use fork?

What would you do in C++ code to make parcellfun work on Windows systems without using fork?

John W. Eaton <jwe>
Group administrator
Wed 04 May 2016 03:41:28 PM UTC, comment #2: 

JWE: Explicit (as opposed to built-in usage of threads) local parallelism with parcellfun (parallel package) is m-code and uses Octaves fork() (and pipe()).

This also affects servers in parallel computing in clusters with the parallel package, since parcellfun is used as a second level parallelism by some functions for cluster computations.

I'd rather like to see cluster computing using GNU/Linux servers. But explicit local parallel computing should probably work on Windows.

Olaf Till <i7tiol>
Group Member
Wed 04 May 2016 02:54:44 PM UTC, comment #1: 

Tatsuro:  Do you really need these functions to work on Windows systems?

I don't think it is necessary that every Octave function work on every platform.  I initially implemented these system call functions for Unix systems because I could, and I thought it would be interesting to experiment with them from a scripting language like Octave.  If some systems fail to provide the necessary tools to allow them to work, that's fine with me.  I think it is sufficient to just warn in the documentation that they don't work on some systems (only native Windows as far as I know).

John W. Eaton <jwe>
Group administrator
Wed 04 May 2016 11:21:45 AM UTC, original submission:  

This was discussed in

fork() for microsoft windows (native but not Cygwin).

http://octave.1599824.n4.nabble.com/fork-for-microsoft-windows-native-but-not-Cygwin-td4676646.html

and
 
http://savannah.gnu.org/bugs/?41148

fork() is not implemented on native windows due to lack of fork function C and C++ on windows platform.
It is well known Cygwin implements fork using win32 api and have long history to improve implementation.

Perlfork also emulates fork() at the level of the Perl program.
http://docs.activestate.com/activeperl/5.8/lib/pods/perlfork.html

I am not a programmer so that I do not know to implement fork is really needed and reasonable.

So this is my own weak wish. But I register the topic to the tracker to memorize with the tag number.

Tatsuro MATSUOKA <tmacchant>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tmacchant
  • -email is unavailable- added by i7tiol (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mtmiller (Updated the item)
  • -email is unavailable- added by tmacchant (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-05-04 tmacchant Carbon-Copy- Added marco atzeri <marco.atzeri@gmail.com>
    2016-05-04 mtmiller Severity3 - Normal 1 - Wish
        StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code