bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #43930, Pre-spawn open3

 
 

bug #43930: Pre-spawn open3

Submitted by:  Ole Tange <tange>
Submitted on:  Sun 04 Jan 2015 11:09:38 AM UTC  
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 18 Aug 2017 09:36:11 PM UTC, comment #5: 

Idea:

spawn job that reads a perlscript from STDIN. The job's output is pipe-filehandles similar to --linebuffer.

The script will open output file if that is needed and compress if needed.

Status is communicated through tmpfile.

How does the job communicate that it died? Is that SIGCHLD sent top parent if threading?

Ole Tange <tange>
Project Administrator
Sun 26 Feb 2017 10:04:48 AM UTC, comment #4: 

--results can put files on a different file system than --tmpdir, so use dir of --results if set.

Ole Tange <tange>
Project Administrator
Sun 26 Feb 2017 10:00:26 AM UTC, comment #3: 
Ole Tange <tange>
Project Administrator
Wed 31 Aug 2016 08:32:44 AM UTC, comment #2: 

http://perldoc.perl.org/perlthrtut.html#Queues%3a-Passing-Data-Around

for #of_cpu:
  spawn 1 open3-slave thread

slave:
while queue < #of_cpu:
  open3 with compression if needed
  push $pid, $stdin, $stdout, $stderr, $compress_pid, $decomp_pid
  sleep until queue changes

Can we re-link a tmpfile? So we default to anonymous filenames and name them if needed.

Ole Tange <tange>
Project Administrator
Mon 05 Jan 2015 07:29:25 AM UTC, comment #1: 

http://www.perlmonks.org/?node_id=493754

It seems you can share a filehandle between threads by using 'fileno' and "<&=$fileno".

So maintain a queue with preopened open3 filehandles:

slave:
share @q;
$exec_first = "exec hexdecode(<STDIN>)";
if @q < 40 :
  my $out = tmpfile(); unlink $out;
  my $err = tmpfile(); unlink $err;
  my $pid = open3(my $in, $out, $err, $exec_first);
  # how to deal with errors?
  push @q, $pid, $in, $out, $err;
sleep until @q < 40

master:
$TMPDIR=--retults or $TMPDIR or /tmp;
fork slave;
sleep until @q >= 4
my ($pid, $in, $out, $err) = pop @q, pop @q, pop @q;
# How to deal with --compress?
# Who gets SIGCHILD?

Ole Tange <tange>
Project Administrator
Sun 04 Jan 2015 11:09:38 AM UTC, original submission:  

According to:

  perl -d:NYTProf parallel true ::: {1..1000}
  nytprofcg; kcachegrind nytprof.callgrind

open3 is the major limitation. Can they somehow be pre-spawned on a different core?

One core does open3 of a command that reads from stdin and execs what was read. This waiting process is then handed off to the mother process that gives it the real task.

Maybe something similar can be done with printing? A single task does nothing but printing and reads filehandle-objects from a queue.

Ole Tange <tange>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    No changes have been made to this item

    Back to the top


    Powered by Savane 3.5