bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #51261, Option --results has major...

 
 

bug #51261: Option --results has major performance penalty

Submitted by:  None
Submitted on:  Sun 18 Jun 2017 07:25:25 PM UTC  
 
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: Need Info
Privacy: PublicAssigned to: None
Open/Closed: Open

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 23 Jun 2017 10:09:31 AM UTC, comment #1:

You are directing the output to /dev/null:

parallel ... :::: to_backup.path > /dev/null

This explains what you see.

See https://lists.gnu.org/archive/html/parallel/2017-06/msg00009.html

Let us discuss this on the email list before making any change.

Ole Tange <tange>
Project Administrator
Sun 18 Jun 2017 07:25:25 PM UTC, original submission:

We're using GNU Parallel as a backup parallelization sulution. Until recently, we've used --return, but that burned through disk write lifetime, which we seek to avoid.

Using --results helps with this but we've noticed that it performed worse than --return despite being nearly identical and added benefit of piping output directly.

After investigating problem, we've noticed huge disk reads on drives which should only be written to during parallel run. It looks like whole written file is being read and fed into /dev/null or similar.

Here's the command we're using (anonimized and simplified):

parallel --slf backup_slf.hosts --results "/storage/path/./{2}_FULL.tgz" --colsep ' ' "GZIP=--fast nice -n 10 tar cpzf - {1} 2>/dev/null" :::: to_backup.path > /dev/null

backup_slf.hosts contains several entries like:
==/
8/root@192.168.128.7
8/root@192.168.128.8
8/root@192.168.128.9
==\

and to_backup.path contains space-separated list of entries like:
==/
/path/to/data/1/ name_1
/path/to/data/2/ name_2
/path/to/data/3/ name_3
/path/to/data/4/ name_4
==\

Rough browsing (I'm nowhere near perl-able person) suggest that problem might be somewhere in sub print_linebuffer but it's hard to be sure for me.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 23 Jun 2017 10:09:31 AM UTCtangeSeverity3 - Normal=>1 - Wish
      StatusNone=>Need Info

    Back to the top


    Powered by Savane 3.1-cleanup1