bugGNU Parallel - Bugs: bug #43930, Pre-spawn open3


bug #43930: Pre-spawn open3

Submitter:  Ole Tange <tange>
Submitted:  Sun 04 Jan 2015 11:09:38 AM UTC
Category:  None Severity:  1 - Wish
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Sat 12 Oct 2019 10:37:56 PM UTC, comment #6: 

Instead of passing fileno, use exec:

create fifo-stdin, fifo-stderr, fifo-stdout

exec < fifo-stdin > fifo-stdout 2>fifo-stderr
rm fifo-stdin fifo-stdout fifo-stderr
exec $the_shell

This way we do not need to know the output names before spawning.

Instead of fifo-stdout and fifo-stderr these can be the temporary/permanent files. Permanent files should not be rm'ed.

To get started we probably need to pass the fileno of stdin along with the pid to the master process.

By using 'exec $the_shell' big multiline programs can be run. Such as big environment.

Ole Tange <tange>
Group administrator
Fri 18 Aug 2017 09:36:11 PM UTC, comment #5: 


spawn job that reads a perlscript from STDIN. The job's output is pipe-filehandles similar to --linebuffer.

The script will open output file if that is needed and compress if needed.

Status is communicated through tmpfile.

How does the job communicate that it died? Is that SIGCHLD sent top parent if threading?

Ole Tange <tange>
Group administrator
Sun 26 Feb 2017 10:04:48 AM UTC, comment #4: 

--results can put files on a different file system than --tmpdir, so use dir of --results if set.

Ole Tange <tange>
Group administrator
Sun 26 Feb 2017 10:00:26 AM UTC, comment #3: 
Ole Tange <tange>
Group administrator
Wed 31 Aug 2016 08:32:44 AM UTC, comment #2: 


for #of_cpu:
  spawn 1 open3-slave thread

while queue < #of_cpu:
  open3 with compression if needed
  push $pid, $stdin, $stdout, $stderr, $compress_pid, $decomp_pid
  sleep until queue changes

Can we re-link a tmpfile? So we default to anonymous filenames and name them if needed.

Ole Tange <tange>
Group administrator
Mon 05 Jan 2015 07:29:25 AM UTC, comment #1: 


It seems you can share a filehandle between threads by using 'fileno' and "<&=$fileno".

So maintain a queue with preopened open3 filehandles:

share @q;
$exec_first = "exec hexdecode(<STDIN>)";
if @q < 40 :
  my $out = tmpfile(); unlink $out;
  my $err = tmpfile(); unlink $err;
  my $pid = open3(my $in, $out, $err, $exec_first);
  # how to deal with errors?
  push @q, $pid, $in, $out, $err;
sleep until @q < 40

$TMPDIR=--retults or $TMPDIR or /tmp;
fork slave;
sleep until @q >= 4
my ($pid, $in, $out, $err) = pop @q, pop @q, pop @q;
# How to deal with --compress?
# Who gets SIGCHILD?

Ole Tange <tange>
Group administrator
Sun 04 Jan 2015 11:09:38 AM UTC, original submission:  

According to:

  perl -d:NYTProf parallel true ::: {1..1000}
  nytprofcg; kcachegrind nytprof.callgrind

open3 is the major limitation. Can they somehow be pre-spawned on a different core?

One core does open3 of a command that reads from stdin and execs what was read. This waiting process is then handed off to the mother process that gives it the real task.

Maybe something similar can be done with printing? A single task does nothing but printing and reads filehandle-objects from a queue.

Ole Tange <tange>
Group administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.


    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-3cd0.
    Corresponding source code