bugGNU Octave - Bugs: bug #55643, nproc reports fewer available...

 
 

bug #55643: nproc reports fewer available cores on OpenSUSE (via sched_setaffinity)

Submitter:  Oliver Mössinger <olivpass>
Submitted:  Mon 04 Feb 2019 02:52:57 PM UTC
   
 
Category:  Interpreter Severity:  2 - Minor
Priority:  3 - Low Item Group:  Incorrect Result
Status:  Works For Me Assigned to:  None
Originator Name:  Oliver Mössinger Open/Closed:  * Closed
Release:  * 5.0.91 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 14 Nov 2019 10:24:22 AM UTC, comment #12: 

Yes, adding the following in comment #7 led to gotoblas_affinity_init () from /usr/lib64/libopenblas_pthreads.so.0:


system("gdb --batch --pid %d -ex bt")


On openSUSE Leap 15.0, OpenBLAS was compiled without NO_AFFINITY enabled in version 0.2.20 and this changed in version 0.3.7, see:
https://build.opensuse.org/package/view_file/science/openblas/openblas.changes

I can confirm that everything works back to normal after upgrading to 0.3.7.

Guillaume <gyom>
Wed 13 Nov 2019 09:36:14 PM UTC, comment #11: 

From a brief discussion with gyom on IRC, there is finally some evidence that this problem comes from the OpenBLAS package on openSUSE. The OpenBLAS library appears to be built without NO_AFFINITY enabled, which means it does set CPU affinity when it's initialized, but that may be fixed in version 0.3.7.

Mike Miller <mtmiller>
Group Member
Tue 05 Feb 2019 04:29:03 PM UTC, comment #10: 

I think it is correct that Octave's nproc does not use "all" by default. Should nproc use "overridable" by default to be compatible with the nproc system command? That's what I would expect and prefer.

Other than that, it seems to me that this bug is resolved as working correctly and can be closed.

Just for my own education, where is the sched_setaffinity being done on OpenSUSE? Is this a global default configuration, was this an accident, is this configurable by the user? In case this comes up again it would be good to know how this works.


Mike Miller <mtmiller>
Group Member
Tue 05 Feb 2019 09:36:35 AM UTC, comment #9: 

Okay, I got confused by nproc and I am pretty sure Octave behaves correctly.

Without Octave on my system


$ nproc
4
$ nproc --all
4
$ OMP_NUM_THREADS=30 nproc
30
$ OMP_NUM_THREADS=30 nproc --all
4


Octave 4.2.2 (distro):


$ /usr/bin/octave --eval 'nproc'
ans =  1
$ /usr/bin/octave --eval 'nproc all'
ans =  4
$ /usr/bin/octave --eval 'nproc current'
ans =  1
$ /usr/bin/octave --eval 'nproc overridable'
ans =  1
$ OMP_NUM_THREADS=30 /usr/bin/octave --eval 'nproc'
ans =  1
$ OMP_NUM_THREADS=30 /usr/bin/octave --eval 'nproc all'
ans =  4
$ OMP_NUM_THREADS=30 /usr/bin/octave --eval 'nproc current'
ans =  1
$ OMP_NUM_THREADS=30 /usr/bin/octave --eval 'nproc overridable'
ans =  30


Octave 50.0.90 (self compiled):


$ octave --eval 'nproc'
ans =  4
$ octave --eval 'nproc all'
ans =  4
$ octave --eval 'nproc current'
ans =  4
$ octave --eval 'nproc overridable'
ans =  4
$ OMP_NUM_THREADS=30 octave --eval 'nproc'
ans =  4
$ OMP_NUM_THREADS=30 octave --eval 'nproc all'
ans =  4
$ OMP_NUM_THREADS=30 octave --eval 'nproc current'
ans =  4
$ OMP_NUM_THREADS=30 octave --eval 'nproc overridable'
ans =  30


A solution might be to return "nproc all" by default or a better documentation what is returned: that is the minimum of OMP_NUM_THREADS and the detected number of processors.  The question is, does Octave regard OMP_NUM_THREADS for all of its computations?

File "libgnu/nproc.c":


unsigned long int
num_processors (enum nproc_query query)
{
  unsigned long int omp_env_limit = ULONG_MAX;

  if (query == NPROC_CURRENT_OVERRIDABLE)
    {
      unsigned long int omp_env_threads;
      /* Honor the OpenMP environment variables, recognized also by all
         programs that are based on OpenMP.  */
      omp_env_threads = parse_omp_threads (getenv ("OMP_NUM_THREADS"));
      omp_env_limit = parse_omp_threads (getenv ("OMP_THREAD_LIMIT"));
      if (! omp_env_limit)
        omp_env_limit = ULONG_MAX;

      if (omp_env_threads)
        return MIN (omp_env_threads, omp_env_limit);

      query = NPROC_CURRENT;
    }
  /* Here query is one of NPROC_ALL, NPROC_CURRENT.  */
  {
    unsigned long nprocs = num_processors_ignoring_omp (query);
    return MIN (nprocs, omp_env_limit);
  }
}


Kai Torben Ohlhus <siko1056>
Group Member
Tue 05 Feb 2019 08:42:20 AM UTC, comment #8: 

You are right, also my Octave 5.0.90 installation can be fooled by the environment variable OMP_NUM_THREADS (see bug #51643).  And I can even turn my laptop into a supercomputer :D


OMP_NUM_THREADS=3000000 octave --eval 'system ("nproc"); system ("nproc --all"); nproc'
3000000
4
ans =  4


Kai Torben Ohlhus <siko1056>
Group Member
Tue 05 Feb 2019 08:41:06 AM UTC, comment #7: 

Workaround:


local ms@itms:~> cat /tmp/testoctave.sh
#!/bin/sh

TEMPDIR=$(mktemp -d)
[ -z "$TEMPDIR" ] && exit 1
trap "rm -fr $TEMPDIR" EXIT

cat > "$TEMPDIR/disable_sched_setaffinity.c" <<'EOF' || exit 1
#include <stdio.h>
#include <sched.h>

int sched_setaffinity(pid_t pid, size_t cpusetsize,
                      cpu_set_t *mask)
{
    printf("(ignore: sched_setaffinity(pid=%d, cpusetsize=%zu, mask=%p) -> return success(0))\n", pid, cpusetsize, mask);
    return 0;
}
EOF

gcc -fPIC -O0 -ldl -shared "$TEMPDIR/disable_sched_setaffinity.c" -o "$TEMPDIR/disable_sched_setaffinity.so" || exit 1
LD_PRELOAD="$TEMPDIR/disable_sched_setaffinity.so:$LD_PRELOAD" "$@"

rm -rf "$TEMPDIR"



local ms@itms:~> /tmp/testoctave.sh octave --eval 'nproc'
(ignore: sched_setaffinity(pid=0, cpusetsize=128, mask=0x7ffd0a2807d0) -> return success(0))
(ignore: sched_setaffinity(pid=0, cpusetsize=128, mask=0x7fd2d7f5dae0) -> return success(0))
(ignore: sched_setaffinity(pid=0, cpusetsize=128, mask=0x7fd2d775cae0) -> return success(0))
(ignore: sched_setaffinity(pid=0, cpusetsize=128, mask=0x7fd2d6f5bae0) -> return success(0))
ans =  8


This code deactivates the syscall sched_setaffinity() and this fixes the processor count.

Oliver Mössinger <olivpass>
Tue 05 Feb 2019 08:22:40 AM UTC, comment #6: 

Have a look:

local ms@itms:~> octave --eval 'system ("nproc"); system ("nproc --all"); nproc'
1
8
ans =  1
local ms@itms:~> OMP_NUM_THREADS=6 octave --eval 'system ("nproc"); system ("nproc --all"); nproc'
6
8
ans =  1

Oliver Mössinger <olivpass>
Mon 04 Feb 2019 07:16:34 PM UTC, comment #5: 

You might also try


system ("nproc");
system ("nproc --all");


at the Octave prompt to see if you get different results in Octave's runtime environment for some reason.

If you're getting the same result with your own compiled copy of Octave 6 from the default branch, that's using a pretty up to date version of gnulib.

Mike Miller <mtmiller>
Group Member
Mon 04 Feb 2019 04:54:05 PM UTC, comment #4: 

We use a gnulib function to get the info.  Maybe there was a bug in that function that was fixed after Octave 4.2.2 or 4.4.1 was released?

John W. Eaton <jwe>
Group administrator
Mon 04 Feb 2019 04:44:14 PM UTC, comment #3: 

Strange. On the same version of openSUSE, `nproc` and `nproc --all` both return 16 in a terminal. Then Octave 4.4.1 and Octave 6.0 both return 1 for `nproc` and 16 for `nproc all`

Guillaume <gyom>
Mon 04 Feb 2019 04:41:25 PM UTC, comment #2: 

This works for me with Ubuntu 18.04.

Do you see the same for:

 octave --eval "nproc all"

?

Markus Mützel <mmuetzel>
Group administrator
Mon 04 Feb 2019 04:21:57 PM UTC, comment #1: 

Thank you for your bug report.  I also run openSUSE LEAP 15.0 and can confirm that for the distributed Octave 4.2.2 (and Octave 4.4.1 from the experimental "science" repository) `nproc` reports only 1 processor on my machine.

For the upcoming release (Octave 5.0.90, self-compiled) the reported number is correct.  I cannot point to the fix, but it must have been fixed somewhere along the line.

Thus all we can do for now is to wait for Octave 5.1.0 to be part of openSUSE LEAP, or you try other ways to obtain a more recent version of Octave.

Kai Torben Ohlhus <siko1056>
Group Member
Mon 04 Feb 2019 02:52:57 PM UTC, original submission:  

local ms@itms:~> nproc
8
local ms@itms:~> octave --eval 'nproc'
ans =  1
local ms@itms:~> cat /proc/version
Linux version 4.12.14-lp150.12.45-default (geeko@buildhost) (gcc version 7.3.1 20180323 [gcc-7-branch revision 258812] (SUSE Linux) ) #1 SMP Mon Jan 14 20:29:59 UTC 2019 (7a62739)
local ms@itms:~> cat /etc/os-release
NAME="openSUSE Leap"
VERSION="15.0"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.0"
PRETTY_NAME="openSUSE Leap 15.0"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.0"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"

Oliver Mössinger <olivpass>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by gyom (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by olivpass (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 15 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-02-26 mtmiller Open/ClosedOpen Closed
    2019-02-05 mtmiller Severity3 - Normal 2 - Minor
        Priority5 - Normal 3 - Low
        StatusConfirmed Works For Me
        SummaryFunction &quot;nproc&quot; just reports the environment variable OMP_NUM_THREADS not the &quot;current number of available processors&quot;. nproc reports fewer available cores on OpenSUSE (via sched_setaffinity)
    2019-02-05 siko1056 Severity2 - Minor 3 - Normal
        Priority3 - Low 5 - Normal
        StatusWorks For Me Confirmed
        Release4.2.2 5.0.91
        SummaryMissing some of my processor cores (nproc) Function "nproc" just reports the environment variable OMP_NUM_THREADS not the "current number of available processors".
    2019-02-04 siko1056 CategoryPerformance Interpreter
        Severity3 - Normal 2 - Minor
        Priority5 - Normal 3 - Low
        Item GroupPerformance Incorrect Result
        StatusNone Works For Me

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code