bugGNU Parallel - Bugs: bug #62512, Speed of --tag --line-buffer

 
 

bug #62512: Speed of --tag --line-buffer

Submitter:  Ole Tange <tange>
Submitted:  Mon 23 May 2022 07:57:36 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 10 Apr 2023 04:38:31 PM UTC, comment #7: 

Prepend filter like:

    #!/usr/bin/perl

    @filter=q(mawk '{print "mytag\t" $0}');

    open(STDOUT, '|-', @filter) || die ("Cannot start @filter");
    while($read=sysread(STDIN,$buf,32767)){syswrite(STDOUT,$buf);}
    close STDOUT;
    exit 8;

It gives 80 MB/s.

Ole Tange <tange>
Group administrator
Sun 08 Jan 2023 09:14:54 PM UTC, comment #6: 

Even faster:

    mawk '{print "abc\t" $0}'

Ole Tange <tange>
Group administrator
Sun 08 Jan 2023 09:02:17 PM UTC, comment #5: 


comment #4:

>     sed -e 's/^/abc\t/'
>
> is 4 times faster than:
>
>     perl -pe 's/^/abc\t/'
>


2x faster than parallel-20170722.

Ole Tange <tange>
Group administrator
Sun 08 Jan 2023 08:59:45 PM UTC, comment #4: 

    sed -e 's/^/abc\t/'

is 4 times faster than:

    perl -pe 's/^/abc\t/'

Ole Tange <tange>
Group administrator
Wed 31 Aug 2022 06:20:42 AM UTC, comment #3: 

Should take < 10s

    time parallel --tag --line-buffer seq  ::: {1000000..1000029} >/dev/null

Ole Tange <tange>
Group administrator
Mon 25 Jul 2022 01:56:24 PM UTC, comment #2: 

Maybe --fasttag can be a wrapper similar to --compress:

    (...) | perl -pe 's/^/tag\t/' > tmpfile

(And similar for stderr).

This way it can run in parallel and not take up CPU-time in the main process.

Anonymous
Mon 25 Jul 2022 12:11:49 PM UTC, comment #1: 

Introduce --fasttag: If --tag does not change use this.

Anonymous
Mon 23 May 2022 07:57:36 PM UTC, original submission:  

What happened between 20170722 and 20170822 and 20210822?

    parallel --tag --line-buffer seq ::: 10000000 | pv > /dev/null

    /usr/local/bin/parallel-20210822 150KB/s
    /usr/local/bin/parallel-20210722 10MB/s
    /usr/local/bin/parallel-20170822 10MB/s
    /usr/local/bin/parallel-20170722 40MB/s

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-8ccc.
    Corresponding source code