bugGNU Parallel - Bugs: bug #51261, Option --results has major...


bug #51261: Option --results has major performance penalty

Submitter:  None
Submitted:  Sun 18 Jun 2017 07:25:25 PM UTC
Category:  None Severity:  1 - Wish
Item Group:  None Status:  Need Info
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup

Fri 23 Jun 2017 10:09:31 AM UTC, comment #1: 

You are directing the output to /dev/null:

parallel ... :::: to_backup.path > /dev/null

This explains what you see.

See https://lists.gnu.org/archive/html/parallel/2017-06/msg00009.html

Let us discuss this on the email list before making any change.

Ole Tange <tange>
Group administrator
Sun 18 Jun 2017 07:25:25 PM UTC, original submission:  

We're using GNU Parallel as a backup parallelization sulution. Until recently, we've used --return, but that burned through disk write lifetime, which we seek to avoid.

Using --results helps with this but we've noticed that it performed worse than --return despite being nearly identical and added benefit of piping output directly.

After investigating problem, we've noticed huge disk reads on drives which should only be written to during parallel run. It looks like whole written file is being read and fed into /dev/null or similar.

Here's the command we're using (anonimized and simplified):

parallel --slf backup_slf.hosts --results "/storage/path/./{2}_FULL.tgz" --colsep ' ' "GZIP=--fast nice -n 10 tar cpzf - {1} 2>/dev/null" :::: to_backup.path > /dev/null

backup_slf.hosts contains several entries like:

and to_backup.path contains space-separated list of entries like:
/path/to/data/1/ name_1
/path/to/data/2/ name_2
/path/to/data/3/ name_3
/path/to/data/4/ name_4

Rough browsing (I'm nowhere near perl-able person) suggest that problem might be somewhere in sub print_linebuffer but it's hard to be sure for me.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by tange (Posted a comment)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.


    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-06-23 tange Severity3 - Normal 1 - Wish
        StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-0159.
    Corresponding source code