bugGNU Parallel - Bugs: bug #65642, --resume --results

 
 

bug #65642: --resume --results

Submitter:  None
Submitted:  Thu 25 Apr 2024 01:49:45 PM UTC
Votes: 1
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 25 Apr 2024 01:49:45 PM UTC, original submission:  

Run:

    seq 100 | parallel -j10 -t --resume --results res/ 'seq 1024;sleep 1;seq 1024;true' & pid=$!; sleep 3; kill $pid
    seq 100 | parallel -j10 -t --resume --results res/ 'seq 1024;sleep 1;seq 1024;true' & pid=$!; sleep 3; kill $pid
    seq 100 | parallel -j10 -t --resume --results res/ 'seq 1024;sleep 1;seq 1024;true' & pid=$!; sleep 3; kill $pid
    seq 100 | parallel -j10 -t --resume --results res/ 'seq 1024;sleep 1;seq 1024;true' & pid=$!; sleep 3; kill $pid

The size of the output should be the same. But is not:

    4 -rw-rw-r-- 1 ota ota 4013 Apr 25 13:40 res/1/27/stdout
    4 -rw-rw-r-- 1 ota ota 4013 Apr 25 13:40 res/1/28/stdout
    4 -rw-rw-r-- 1 ota ota 4013 Apr 25 13:40 res/1/29/stdout
    4 -rw-rw-r-- 1 ota ota 4013 Apr 25 13:40 res/1/30/stdout
    8 -rw-rw-r-- 1 ota ota 8026 Apr 25 13:40 res/1/10/stdout
    8 -rw-rw-r-- 1 ota ota 8026 Apr 25 13:40 res/1/11/stdout
    8 -rw-rw-r-- 1 ota ota 8026 Apr 25 13:40 res/1/12/stdout
    8 -rw-rw-r-- 1 ota ota 8026 Apr 25 13:40 res/1/13/stdout
    8 -rw-rw-r-- 1 ota ota 8026 Apr 25 13:40 res/1/14/stdout

This is because GNU Parallel mistakenly assumes the job is done when the dir is created. What is needed is some way for GNU Parallel to determine that the job has finished.

One solution would be to add another file called 'joblog'. It could contain the full line from --joblog.

This way we can easily see if the job is incomplete (the file is missing) and also see if the job failed (Exitval != 0).

Thereby --resume, --resume-failed and --retry-failed can be implemented.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by theshank (Voted in favor of this item)
  •  

    There is 1 vote so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-04-25 theshank Carbon-Copy- Added theshank

    Back to the top

    Powered by Savane 3.13-8ccc.
    Corresponding source code