bugGNU Parallel - Bugs: bug #66868, Way to wait for batch complete

 
 

bug #66868: Way to wait for batch complete

Submitter:  None
Submitted:  Tue 04 Mar 2025 11:44:20 AM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
* Mandatory Fields

Post a Comment

Add a New Comment Rich Markup
   

Discussion

Jump to the original submission

Thu 06 Mar 2025 08:18:55 PM UTC, comment #6: 

Test case:

  ./src/parallel  --tag  --delay 0.01 --milestone /// 'echo {#} $(date) $(sleep {=1 $_=1+rand()*5=}; date)' ::: a b /// 1 2 3 ::: A B C /// I II

The end time of each batch must be before the start time of the next batch.

Anonymous
Thu 06 Mar 2025 01:10:42 AM UTC, comment #5: 

meant to type --jobs '{= m~///~ ? (skip(),1) : "100%" =}'

Brian R <bowertech>
Thu 06 Mar 2025 01:06:16 AM UTC, comment #4: 

I'm imagining something like --jobs '{= m~///~ ? 1 : "100%" =}' might be able to do the trick as an equivalent to your --milestone example

Brian R <bowertech>
Wed 05 Mar 2025 10:19:57 PM UTC, comment #3: 

Nice.  When you put it that way, I wonder about putting perl code into --filter that manipulates max_slot_number as a side-effect.
Alternately, extend --jobs to take a full perl expression as an argument since it's presumably checked at the start of each job.  That could trigger a pause based on something about the jobs themselves.

Brian R <bowertech>
Tue 04 Mar 2025 01:12:31 PM UTC, comment #2: 

Possibly names:

--batchwait ///
--batchcomplete ///
--milestone ///

parallel --milestone /// dostuff ::: 1 2 3 /// a b /// I II ::: A B /// Charlie Delta Echo

Runs:

1 A
1 B
2 A
2 B
3 A
3 B
[Batch 1 done]
1 Charlie
1 Delta
1 Echo
2 Charlie
2 Delta
2 Echo
3 Charlie
3 Delta
3 Echo
[Batch 2 done]
a A
a B
b A
b B
[Batch 3 done]
a Charlie
a Delta
a Echo
b Charlie
b Delta
b Echo
[Batch 4 done]
I A
I B
II A
II B
[Batch 5 done]
I Charlie
I Delta
I Echo
II Charlie
II Delta
II Echo
[Batch 6 done]


See further explanation in answer 2:
https://x.com/i/g ... F9tLAlY2aNymOWM1V

Anonymous
Tue 04 Mar 2025 12:02:02 PM UTC, comment #1: 

When MAGICVALUE is hit, insert a job that changes $Global::max_slot_number=1. After this insert a job that changes $Global::max_slot_number=old_value.

Challenges: These should be ignored in joblog. The replacement string cannot be evaluated in advance as it changes the running behaviour.

Maybe introduce a job->{'forcewait'} = 1 that activates this behaviour? It will make it easy to ignore in joblog.

Anonymous
Tue 04 Mar 2025 11:44:20 AM UTC, original submission:  

I want to run job 1..10. Job 11 can only be started after 1..10 is complete.

https://unix.stac ... obs/791923#791923


parallel <<_ --delay 1 --line-buffer --colsep , 'true {=1 $Global::max_slot_number=$_; =}; {=2 uq()=}'
8,date +A+%H%M%S; sleep 3; date +A-%H%M%S
8,date +B+%H%M%S; sleep 5; date +B-%H%M%S
8,date +C+%H%M%S; sleep 2; date +C-%H%M%S
8,date +D1+%H%M%S; sleep 0.1; date +D1-%H%M%S
8,date +D2+%H%M%S; sleep 0.1; date +D2-%H%M%S
8,date +D3+%H%M%S; sleep 0.1; date +D3-%H%M%S
8,date +D4+%H%M%S; sleep 0.1; date +D4-%H%M%S
1,echo wait for completion
1,echo wait for completion2
1,echo completion done
8,date +E+%H%M%S; sleep 0.3; date +E-%H%M%S
8,date +F+%H%M%S; sleep 5; date +F-%H%M%S
8,date +G+%H%M%S; sleep 7; date +G-%H%M%S
8,date +H+%H%M%S; sleep 5; date +H-%H%M%S
8,date +I+%H%M%S; sleep 3; date +I-%H%M%S
_


Anonymous

 

Attached Files

This item currently has no attached files.

(Note: upload size limit is set to 4.0MiB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Dependencies

This item does not depend on any other items.

No items depend on this one.

 

Mail Notification Carbon-Copy List

Carbon-Copy List
  • -email is unavailable- added by bowertech (Posted a comment)
  •  

    Votes

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    History

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.16-ed84.
    Corresponding source code