bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #32568, Option to use physical cores...

 
 

bug #32568: Option to use physical cores instead of logical

Submitted by:  None
Submitted on:  Tue 22 Feb 2011 05:05:20 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Works For Me
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 26 May 2011 08:57:45 AM UTC, comment #2:

No reply from submitter.

Re-open if you still see this as a problem.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Wed 23 Feb 2011 12:28:59 PM UTC, comment #1:

In the typical situation 1000 CPU limited tasks will run faster when run on 8 hyperthreaded logical cores than on 4 real cores. This is the situation GNU Parallel is optimized for.

Try these to get some numbers:

seq 1 100 | time parallel -j100% head -c 30m /dev/zero '|' bzip2 '>'{};
seq 1 100 | time parallel -j50% head -c 30m /dev/zero '|' bzip2 '>'{};
seq 1 100 | time parallel -j25% head -c 30m /dev/zero '|' bzip2 '>'{};
seq 1 100 | time parallel -j12% head -c 30m /dev/zero '|' bzip2 '>'{};

I do not expect people to be disappointed that the first runs faster than the second - even if it does not run twice as fast. I would, however, expect people to be disappointed, if GNU Parallel chose the second as default.

You write that GNU Parallel oversaturates the CPU. Please show how that can be measured or seen. Also it would be good if you can provide a CPU limited example in which the wall clock time is smaller when using fewer logical CPUs - as I have yet to see a single example of this.

That being said it might make sense to change --use-cpus-instead-of-cores to: --use-physical-cores and --use-logical-cores

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 22 Feb 2011 05:05:20 PM UTC, original submission:

I have a hyperthreaded CPU with four physical cores, each with two logical (hyperthreaded) cores. GNU parallel detects this as eight cores and launches eight jobs at a time, which oversaturates the CPU. It would be nice to have an option similar to --use-cpus-instead-of-cores, but something like --use-physical-cores that would launch four jobs at a time instead of eight.

It might even make sense to make this the default, since (for CPU bound jobs) you typically don't get eight cores of performance out of a four-core-plus-hyperthreading CPU. People might be disappointed since "--number-of-cores says I have eight cores, but it's not going eight times faster!"

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 26 May 2011 08:57:45 AM UTCtangeStatusNeed Info=>Works For Me
      Open/ClosedOpen=>Closed
    Wed 23 Feb 2011 12:28:59 PM UTCtangeStatusNone=>Need Info
      Assigned toNone=>tange

    Back to the top


    Powered by Savane 3.1-cleanup1