bugGNU Parallel - Bugs: bug #63877, --shard on non-numeric columns -...

 
 

bug #63877: --shard on non-numeric columns - but run one job per value

Submitter:  Ole Tange <tange>
Submitted:  Tue 07 Mar 2023 04:10:02 PM UTC
   
 
Category:  None Severity:  1 - Wish
Priority:  5 - Normal Item Group:  None
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
* Mandatory Fields

Post a Comment

Add a New Comment Rich Markup
   

Discussion

Tue 09 May 2023 10:31:40 PM UTC, comment #2: 

You could just sort on the column and use --group-by, but maybe you want to avoid the I/O, maybe the data is streamed.

So something like:

    $shardval = hash($column);
    if(not $job->{shardval}) {
      $job->{shardval} = start_job();
    }
    $job->{shardval}->print_to_job(line)



Ole Tange <tange>
Group administrator
Tue 09 May 2023 10:26:34 PM UTC, comment #1: 

Do not use --bin as the base, but --shard.

If --shard gives a new unique value, spawn another job.

Anonymous
Tue 07 Mar 2023 04:10:02 PM UTC, original submission:  

Assume input:

        a C
        a D
        b A
        b B
        b C
        b D
        c B
        c C
        d B
        d D

You want to bin on col1. You know there are so few different values, you can spawn a process per job, but you do not know the number in advance.

Ole Tange <tange>
Group administrator

 

Attached Files

This item currently has no attached files.

(Note: upload size limit is set to 4.0MiB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Dependencies

This item does not depend on any other items.

No items depend on this one.

 

Mail Notification Carbon-Copy List

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    Votes

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    History

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-05-09 tange Summary--bin on non-numeric columns --shard on non-numeric columns - but run one job per value

    Back to the top

    Powered by Savane 3.16-11ef.
    Corresponding source code