bugGNU Parallel - Bugs: bug #44961, perl base64 recipient


bug #44961: perl base64 recipient

Submitted by:  Ole Tange <tange>
Submitted on:  Wed 29 Apr 2015 07:55:39 PM UTC  
Category:  None Severity:  1 - Wish
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Tue 08 Dec 2015 12:15:46 AM UTC, comment #3: 

It does not make sense to compress:

null() { head -c $1 /tmp/seq |  ssh grb.pi.dk 'wc -c'; }
bz() { head -c $1 /tmp/seq | bzip2 | ssh grb.pi.dk 'bzcat|wc -c'; }
export -f null
export -f bz
parallel  -j2 --joblog - {3} {1}0 ::: 100000 300000 1000000 3000000 ::: {1..10} ::: bz null

Only at 30000000 bytes is compression when running on a 10 Mbps line, and it is safe to assume the line is mostly faster.

Ole Tange <tange>
Project Administrator
Wed 29 Apr 2015 08:31:32 PM UTC, comment #2: 

# Input:                                                                                                                     
#   zipper space Base64 16384*space \n                                                                                       

    my @b;
    my $kb8=" "x8192;
    # Read most in blocks of 8 KB                                                                                            
    while(sysread(STDIN,$buf,8192)) {
        if($buf eq $kb8) { last; }
        push @b, $buf;
    ($zip,$base64) = split / /,join "",@b;
    # Read the final partial block byte by byte                                                                              
    # to avoid reading too much from STDIN                                                                                   
    while(sysread(STDIN,$buf,1)) {
        if($buf eq "\n") { last; }
print $zip;
$dec = decode_base64($base64);
if($zip) {
    # filter $dec through zipper                                                                                             
    $dec = $dec;
eval $dec;

Ole Tange <tange>
Project Administrator
Wed 29 Apr 2015 08:05:48 PM UTC, comment #1: 

Problem: <> reads 8kb block.

Ole Tange <tange>
Project Administrator
Wed 29 Apr 2015 07:55:39 PM UTC, original submission:  

Make a perl wrapper that:

reads 1 line of Base64 encoded perl script. Executes the script.

This will make it possible to ignore the csh limitation of word length and the bash limitation of line length.

It should be compatible with --pipe, so the wrapper should pass the rest of STDIN to the --pipe command.

Does it make sense to bzip2 the script when running remote? How long does it take to bzip2/bunzip2 compared to transfer time? Without the bzip2 it will make it easier to simply run the same unpacker locally and remotely. Alternatively the Base64 line can be prepended with 'bzip2' if it needs to be bunzipped.

Ole Tange <tange>
Project Administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-04-29 tange Severity3 - Normal => 1 - Wish

    Back to the top

    Powered by Savane 3.7