bugGNU Parallel - Bugs: bug #44041, env_parallel limits

 
 

bug #44041: env_parallel limits

Submitter:  Ole Tange <tange>
Submitted:  Sun 18 Jan 2015 08:48:11 PM UTC
   
 
Category:  None Severity:  2 - Minor
Item Group:  None Status:  Confirmed
Privacy:  Public Assigned to:  tange
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 15 Oct 2018 08:31:36 AM UTC, comment #3: 

This works:

mkfifo fifo
seq 10 | bash fifo &
echo wc > fifo

So from the perl-wrapper we can spawn bash reading a fifo, then save the environment and the command to the fifo.

This way we can still read from stdin.

Anonymous
Sun 14 Oct 2018 10:42:42 PM UTC, comment #2: 

mkfifo envfifo
generateenv > file
(rm file; cat > envfifo; rm envfifo) < file &
parallel --env envfifo

Or if generateenv can be a subshell:

mkfifo envfifo
(generateenv; rm envfifo) > envfifo &
parallel --env envfifo

This will require parallel to start the job as:

cat bigstuff | bash

instead of:

bash -c 'command'

This will not work for --pipe, but env_parallel is incompatible with --pipe anyway.

Ole Tange <tange>
Group administrator
Sun 18 Jan 2015 11:20:57 PM UTC, comment #1: 

-r and not -d
if variable of same name exists: warn: use ./ to distinguish from variable

Ole Tange <tange>
Group administrator
Sun 18 Jan 2015 08:48:11 PM UTC, original submission:  

bash has a 128 kb/variable limit and a 2 MB total limit.

One solution: Use multiple varables (xxx_1, xxx_2) to store each 128 kb.

Another solution: allow --env to pass a filename and read that. Then use <( ) to generate the env info.

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code