bugGNU Parallel - Bugs: bug #65017, {%} produces bigger slot numbers...

 
 

bug #65017: {%} produces bigger slot numbers than the number of jobs

Submitter:  Miguel M. <miguelusque>
Submitted:  Tue 12 Dec 2023 10:29:07 AM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Need Info
Privacy:  Public Assigned to:  tange
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 13 Dec 2023 05:01:16 PM UTC, comment #5: 

If you need to make many files for your MCVE, you can do something like:

seq 100000 | parallel seq {} '>' {}

Anonymous
Wed 13 Dec 2023 04:56:38 PM UTC, comment #4: 
Anonymous
Wed 13 Dec 2023 02:11:00 PM UTC, comment #3: 

Hi!

Thank you for your message.

The number of jobs is correctly set to 30, and each job deals with 1000 files. The 30 jobs are usually running fine in parallel. My CPU has 96 threads.

I do not understand why this happens. Each job takes around 30 seconds to complete, and I will be running those jobs to process more than 500.000 files.

The job number is bigger that 30 only for a really small amount of time, that is what I think there is something wrong. If I specify that the number of jobs is 30, the slot number should be between 1 and 30, according to the documentation.

Maybe I am wrong. Looking forward to your thoughts.

Miguel M. <miguelusque>
Wed 13 Dec 2023 01:15:12 PM UTC, comment #2: 

If $NUMBER_JOBS is unset, it will evaluate to nothing. Then your example will use '-j nice' which "evaluate the string 'nice' as a perl expressing" which gives 0.

The reason you rarely see GNU Parallel use jobslot > 30 is that it will only use jobslot 31 when 1..30 are all in use. And if your jobs run pretty fast, then job 1 may be done before job 30 is even started.


Anonymous
Wed 13 Dec 2023 01:05:46 PM UTC, comment #1: 

Please see if you can make an MCVE (https://stackoverflow.com/help/minimal-reproducible-example) - preferably on another system.

My guess that for some odd reason $NUMBER_JOBS is evaluated to 0 - not 30. 0 means run as many as possible.

But since I do not have access to your system, please see if you can make an reproducible example on another system that shows the behaviour you experience.

https://www.gnu.org/software/parallel/man.html#reporting-bugs


Anonymous
Tue 12 Dec 2023 10:29:07 AM UTC, original submission:  

Hi!

I am running the following command:

find $INPUT_DIR -type f -name "*.zip" | parallel -N $FILES_JOB --jobs $NUMBER_JOBS nice -n $NICE \
      'python3 my_script.py -i {} -w $(printf "%04d" "{%}")'


$INPUT_DIR is a local folder
$NUMBER_JOBS is equals to 30
$NICE is equals to 0

I am passing {%} to my_script.py, which performs some work on the files passed to it, and produces the output in a folder name, based in the parameter -w which receives the value {%} padded with 0s.

The issue comes because, suddenly and only for a minimum amount of time, the script starts receiving slot numbers bigger than the number of jobs.

Please, find below the folders created by my script, based on the value of the parameter -w.

In can be observerd that it creates folders bigger than the value 30, which should be the maximum slot number, based on the parameters above.

0001
0002
0003
0004
0005
0006
0007
0008
0009
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0040
0042
0043
0044
0045
0048
0049
0055
0058
0059
0061
0062
0064
0065
0066
0068
0069
0071
0073
0074
0075
0081
0083
0086
0088
0089
0090
0092
0094

Hope it helps!

Miguel M. <miguelusque>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Updated the item)
  • -email is unavailable- added by miguelusque (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-12-14 tange StatusNone Need Info
        Assigned toNone tange

    Back to the top

    Powered by Savane 3.13-5884.
    Corresponding source code