bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #42055, --pipe -a bigfile should not...

 
 

bug #42055: --pipe -a bigfile should not require sequential reading of bigfile

Submitted by:  Ole Tange <tange>
Submitted on:  Sat 05 Apr 2014 02:16:57 PM UTC  
 
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 15 Jul 2014 04:03:43 PM UTC, comment #5:

Tests, error, and man page fixed in [1cf5219].

Ole Tange <tange>
Project AdministratorIn charge of this item.
Wed 04 Jun 2014 01:05:26 PM UTC, comment #4:

Regexp implemented.

Needed:

  • tests
  • Give error with incompatible switches: -N -l -L
  • Man page on incompatible switches
Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 05 Apr 2014 02:24:46 PM UTC, comment #3:

I do not see a way of finding splitpoints based on number of lines: When seeking into the file you do not know what line number (or modulo) that you are at, so -N -l -L will not work.

Regexp should be able to work:

($regend)$regstart

$-[0]+length($1) would be the splitpoint.

The minor price you pay is that backreferences in regend (the \1) will be off by one.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 05 Apr 2014 02:20:37 PM UTC, comment #2:

find_split_points for simple strings (not regexp):

open (my $fh, "<", $file) || die;
push(@pos,0);
for(my $pos = $block; $pos < $size; $pos += $block) {
my $buf;
seek($fh, $pos, 0) || die;
while(read($fh,substr($buf,length $buf,0),$dd_block_size)) {
# If match $recend$recstart => Record position
my $i = index($buf,$recendrecstart);
if($i != -1) {
push(@pos,$pos+$i);
# Start looking for next record after this match
$pos += $i;
last;
}
}
}
push(@pos,$size);

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 05 Apr 2014 02:19:04 PM UTC, comment #1:

cat --from splitpoint[n] --to splitpoint[n+1] is:

perl -e 'sysseek(STDIN,shift,0) || die; $left = shift; while($read = sysread(STDIN,$buf, ($left > 32768 ? 32768 : $left))){ $left -= $read; syswrite(STDOUT,$buf);' from to < bigfile

It delivers 3.5 GB/s on my laptop which is much faster than --pipe, so that is great.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 05 Apr 2014 02:16:57 PM UTC, original submission:

The basic idea is:

locate splitpoints (based on --recstart --recend)

Regard the splitpoints as job args.

run: cat --from splitpoint[n] --to splitpoint[n+1] | my_program

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 15 Jul 2014 04:03:43 PM UTCtangeStatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup