bugGNU Octave - Bugs: bug #47843, fork implementation on native...

 
 

bug #47843: fork implementation on native windows

Submitted by:  Tatsuro MATSUOKA <tmacchant>
Submitted on:  Wed 04 May 2016 11:21:45 AM UTC  
 
Category: LibrariesSeverity: 1 - Wish
Priority: 5 - NormalItem Group: Feature Request
Status: ConfirmedAssigned to: None
Originator Name: Tatsuro MATSUOKAOpen/Closed: Open
Release: devOperating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Thu 05 May 2016 07:57:36 AM UTC, comment #10:

There are threads, of course. But to use threads for arbitrary user code, there would have to be changes in Octave. You probably know much better than me that these changes would be considerable.

If you want to do parallel computing for arbitrary user code with a package, with the current state of Octave, I think some pendent to fork is necessary: You want to have all user-variables of the parent available in the child, but you don't want all user-variables of the parent over-writable by a child.

Olaf Till <i7tiol>
Project Member
Wed 04 May 2016 08:53:30 PM UTC, comment #9:

Surely there must be some other way to do parallel computing on Windows systems that does not require fork (which is a Unix concept anyway).

John W. Eaton <jwe>
Project Administrator
Wed 04 May 2016 08:36:26 PM UTC, comment #8:

It's off topic, but for the record:

Contrary to what I thought, reading in the Cygwin source indicates that there is no cleaner possibility to use the process creation model of Windows for the purpose of the parallel package than to emulate fork() with it. And I just mentioned on bug #41148 that I won't try to copy this terrible effort of Cygwin in the parallel package. (And I still recommend not to do it in Octave either.)

Olaf Till <i7tiol>
Project Member
Wed 04 May 2016 06:06:04 PM UTC, comment #7:

Having read:

https://www.cygwin.com/faq.html#faq.using.fixing-fork-failures

I would now discourage trying to simulate fork for MINGW in Octave.

There may be a clean solution possible within the parallel package, however, by applying the process creation model of Windows, on systems where this is necessary.

(I've not yet made myself familiar with Windows' process creation model, and I don't know how much time I'll have for this...)

Olaf Till <i7tiol>
Project Member
Wed 04 May 2016 05:02:51 PM UTC, comment #6:

Tatsuro: If the performance of fork() on Cygwin is enough in a practial sense only depends on how much parcellfun has to do. The child processes are generated only once in a call to parcellfun, and even a bad performance of fork() can allow useful calls of parcellfun in computationally extensive tasks.

Olaf Till <i7tiol>
Project Member
Wed 04 May 2016 04:53:48 PM UTC, comment #5:

> Is there some other method that could be used on Windows
> (or all) systems to make parcellfun work without having to
> use fork?


I can't imagine how the functionality of parcellfun could be achieved with threads.

The complicated hack to start new Octave instances with system() and to use named pipes for data transfer would leave considerable issues with passing of parameters and functions.

So I see no alternative to fork here.

> What would you do in C++ code to make parcellfun work
> on Windows systems without using fork?


If it's decided to leave this problem to the parallel package, I'd probably try to make an oct-function which wraps fork, if possible, and calls something different (CreateProcess?) if fork is not present. I'd probably look at the way Cygwin does the latter.

Mmh... probably this will be difficult, and some say Cygwin has a leaky solution for this:

http://stackoverflow.com/questions/985281/what-is-the-closest-thing-windows-has-to-fork

And now it occurs to me that there may be an additional problem for parcellfun: it nees the Unix
function in the child processes, and I don't know a Windows pendant. Maybe Octave could be pursued to exit without calling functions registered with atexit(), then there would be no need for
...

So currently I don't know if there will be a feasible solution for parcellfun on MINGW at all. Maybe it is better to try a solution in the parallel package first, including the issue with exit(). If it turns out there is a clean solution for fork, something similar could be done in Octave.

Olaf Till <i7tiol>
Project Member
Wed 04 May 2016 04:36:08 PM UTC, comment #4:

Perhaps if we may implement fork, its performance at best is similar to that of the Cygwin. Experiments on the performance can be done on the octave on Cygwin because the parallel package works on Cygwin version of octave.
If the performance of parallel package on Cygwin is enough in the practical sense, it is worth to implement fork function on octave for native windows.
If you feel performance is not enough parallel package on Cygwin, you have to consider other way as JWE said.

Tatsuro MATSUOKA <tmacchant>
Wed 04 May 2016 03:53:15 PM UTC, comment #3:

Is there some other method that could be used on Windows (or all) systems to make parcellfun work without having to use fork?

What would you do in C++ code to make parcellfun work on Windows systems without using fork?

John W. Eaton <jwe>
Project Administrator
Wed 04 May 2016 03:41:28 PM UTC, comment #2:

JWE: Explicit (as opposed to built-in usage of threads) local parallelism with parcellfun (parallel package) is m-code and uses Octaves fork() (and pipe()).

This also affects servers in parallel computing in clusters with the parallel package, since parcellfun is used as a second level parallelism by some functions for cluster computations.

I'd rather like to see cluster computing using GNU/Linux servers. But explicit local parallel computing should probably work on Windows.

Olaf Till <i7tiol>
Project Member
Wed 04 May 2016 02:54:44 PM UTC, comment #1:

Tatsuro: Do you really need these functions to work on Windows systems?

I don't think it is necessary that every Octave function work on every platform. I initially implemented these system call functions for Unix systems because I could, and I thought it would be interesting to experiment with them from a scripting language like Octave. If some systems fail to provide the necessary tools to allow them to work, that's fine with me. I think it is sufficient to just warn in the documentation that they don't work on some systems (only native Windows as far as I know).

John W. Eaton <jwe>
Project Administrator
Wed 04 May 2016 11:21:45 AM UTC, original submission:

This was discussed in

fork() for microsoft windows (native but not Cygwin).

http://octave.1599824.n4.nabble.com/fork-for-microsoft-windows-native-but-not-Cygwin-td4676646.html

and

http://savannah.gnu.org/bugs/?41148

fork() is not implemented on native windows due to lack of fork function C and C++ on windows platform.
It is well known Cygwin implements fork using win32 api and have long history to improve implementation.

Perlfork also emulates fork() at the level of the Perl program.
http://docs.activestate.com/activeperl/5.8/lib/pods/perlfork.html

I am not a programmer so that I do not know to implement fork is really needed and reasonable.

So this is my own weak wish. But I register the topic to the tracker to memorize with the tag number.

Tatsuro MATSUOKA <tmacchant>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tmacchant
  • -unavailable- added by i7tiol (Posted a comment)
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by mtmiller (Updated the item)
  • -unavailable- added by tmacchant (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 04 May 2016 04:36:08 PM UTCtmacchantCarbon-Copy-=>Added marco atzeri
    Wed 04 May 2016 02:30:09 PM UTCmtmillerSeverity3 - Normal=>1 - Wish
      StatusNone=>Confirmed

    Back to the top


    Powered by Savane 3.1-cleanup1