bugGNU Parallel - Bugs: bug #63877, --shard on non-numeric columns -...


bug #63877: --shard on non-numeric columns - but run one job per value

Submitter:  Ole Tange <tange>
Submitted:  Tue 07 Mar 2023 04:10:02 PM UTC
Category:  None Severity:  1 - Wish
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup

Tue 09 May 2023 10:31:40 PM UTC, comment #2: 

You could just sort on the column and use --group-by, but maybe you want to avoid the I/O, maybe the data is streamed.

So something like:

    $shardval = hash($column);
    if(not $job->{shardval}) {
      $job->{shardval} = start_job();

Ole Tange <tange>
Group administrator
Tue 09 May 2023 10:26:34 PM UTC, comment #1: 

Do not use --bin as the base, but --shard.

If --shard gives a new unique value, spawn another job.

Tue 07 Mar 2023 04:10:02 PM UTC, original submission:  

Assume input:

        a C
        a D
        b A
        b B
        b C
        b D
        c B
        c C
        d B
        d D

You want to bin on col1. You know there are so few different values, you can spawn a process per job, but you do not know the number in advance.

Ole Tange <tange>
Group administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.


    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-05-09 tange Summary--bin on non-numeric columns --shard on non-numeric columns - but run one job per value

    Back to the top

    Powered by Savane 3.13-12ab.
    Corresponding source code