bugGNU Parallel - Bugs: bug #51261, Option --results has major...

 
 

bug #51261: Option --results has major performance penalty

Submitted by:  None
Submitted on:  Sun 18 Jun 2017 07:25:25 PM UTC  
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  Need Info
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 23 Jun 2017 10:09:31 AM UTC, comment #1: 

You are directing the output to /dev/null:

parallel ... :::: to_backup.path > /dev/null

This explains what you see.

See https://lists.gnu.org/archive/html/parallel/2017-06/msg00009.html

Let us discuss this on the email list before making any change.

Ole Tange <tange>
Project Administrator
Sun 18 Jun 2017 07:25:25 PM UTC, original submission:  

We're using GNU Parallel as a backup parallelization sulution. Until recently, we've used --return, but that burned through disk write lifetime, which we seek to avoid.

Using --results helps with this but we've noticed that it performed worse than --return despite being nearly identical and added benefit of piping output directly.

After investigating problem, we've noticed huge disk reads on drives which should only be written to during parallel run. It looks like whole written file is being read and fed into /dev/null or similar.

Here's the command we're using (anonimized and simplified):

parallel --slf backup_slf.hosts --results "/storage/path/./{2}_FULL.tgz" --colsep ' ' "GZIP=--fast nice -n 10 tar cpzf - {1} 2>/dev/null" :::: to_backup.path > /dev/null

backup_slf.hosts contains several entries like:
==/
8/root@192.168.128.7
8/root@192.168.128.8
8/root@192.168.128.9
==\

and to_backup.path contains space-separated list of entries like:
==/
/path/to/data/1/ name_1
/path/to/data/2/ name_2
/path/to/data/3/ name_3
/path/to/data/4/ name_4
==\

Rough browsing (I'm nowhere near perl-able person) suggest that problem might be somewhere in sub print_linebuffer but it's hard to be sure for me.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-06-23 tange Severity3 - Normal => 1 - Wish
        StatusNone => Need Info

    Back to the top


    Powered by Savane 3.6