bugGNU Octave - Bugs: bug #59475, Exceptions Running Octave with...

 
 

bug #59475: Exceptions Running Octave with 64-bit Indexes

Submitter:  None
Submitted:  Tue 17 Nov 2020 03:47:58 PM UTC
   
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Invalid / Not an Octave Bug Assigned to:  None
Originator Name:  David Laxer Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 18 Nov 2020 12:37:54 AM UTC, comment #11: 

@David Laxer: I work on building Octave using proper 64-bit for some years.  My knowledge how to get this right is available in two projects:

- https://github.com/gnu-octave/GNU-Octave-enable-64 This projects aims to compile things on your local machine (please update the library versions to your needs).

- https://github.com/gnu-octave/docker More recent attempt to build portable Docker/Singularity images, but the receipts can also be applied to a local machine.  Work in progress, more documentation available in the first project.

Kai Torben Ohlhus <siko1056>
Group Member
Tue 17 Nov 2020 09:35:03 PM UTC, comment #10: 

Sorry, I crossed with your last comment.
That depends on how Octave and the libraries were packaged by the distribution. AFAIK, most distributions don't ship libraries with 64bit Fortran indices.
You'll still be able to use large matrices in Octave. You'll only see error messages if you try to pass matrices that are too large to libraries that can't handle them. Octave takes care to not crash in those cases and display an error message instead (hopefully).

Markus Mützel <mmuetzel>
Group administrator
Tue 17 Nov 2020 09:31:39 PM UTC, comment #9: 

I'm going to close this report as invalid.
But please comment again when you still run into issues with 64bit Fortran indexing with a consistent set of libraries. We'll re-open the report in that case. Or open a new report.

Markus Mützel <mmuetzel>
Group administrator
Tue 17 Nov 2020 09:29:52 PM UTC, comment #8: 

Does the version of octave that apt-get installs support 64-bit FORTRAN indexing?

david laxer <davidlaxer>
Tue 17 Nov 2020 09:24:30 PM UTC, comment #7: 

You'd need to compile all other numeric libraries as well to match 64bit Fortran indexing. At least, SuiteSparse, ARPACK, QRUpdate, SUNDIALS, and possibly others.

Yes, you'd have to set LDFLAGS correspondingly for configure.

You probably won't need to run the bootstrap script again...

Markus Mützel <mmuetzel>
Group administrator
Tue 17 Nov 2020 09:16:55 PM UTC, comment #6: 

OpenBLAS and LAPACK.

--with-blas="-lopenblas" -with-lapack=-llapack

So, I have to set LDFLAGS so ./configure looks in

/home/ubuntu/.local/ for lapack, and
/opt/OpenBLAS/lib/ for OpenBlas


Do I need to run bootstrap again?


david laxer <davidlaxer>
Tue 17 Nov 2020 08:59:43 PM UTC, comment #5: 

IIUC, you'd have to set LDFLAGS for the autotools to detect the libraries at those locations.

Just to make sure, could you please provide a list of all libraries that you built with 64bit indexing?

Markus Mützel <mmuetzel>
Group administrator
Tue 17 Nov 2020 08:53:10 PM UTC, comment #4: 

 ai) ubuntu@ip-10-0-1-82:~/OpenBLAS$ sudo make install
make -j 4 -f Makefile.install install
make[1]: Entering directory '/home/ubuntu/OpenBLAS'
Generating openblas_config.h in /opt/OpenBLAS/include
Generating f77blas.h in /opt/OpenBLAS/include
Generating cblas.h in /opt/OpenBLAS/include
Copying LAPACKE header files to /opt/OpenBLAS/include
Copying the static library to /opt/OpenBLAS/lib
Copying the shared library to /opt/OpenBLAS/lib
Generating openblas.pc in /opt/OpenBLAS/lib/pkgconfig
Generating OpenBLASConfig.cmake in /opt/OpenBLAS/lib/cmake/openblas
Generating OpenBLASConfigVersion.cmake in /opt/OpenBLAS/lib/cmake/openblas
Install OK!
make[1]: Leaving directory '/home/ubuntu/OpenBLAS'
(ai) ubuntu@ip-10-0-1-82:~/OpenBLAS$ ls -l /opt/OpenBLAS/lib/
total 41736
drwxr-xr-x 3 root root     4096 Oct  8  2018 cmake
lrwxrwxrwx 1 root root       33 Nov 17 20:49 libopenblas.a -> libopenblas_haswellp-r0.3.4.dev.a
-rw-r--r-- 1 root root 28498332 Nov 15 15:45 libopenblas_haswellp-r0.3.4.dev.a
-rwxr-xr-x 1 root root 14227976 Nov 15 15:45 libopenblas_haswellp-r0.3.4.dev.so
lrwxrwxrwx 1 root root       34 Nov 17 20:49 libopenblas.so -> libopenblas_haswellp-r0.3.4.dev.so
lrwxrwxrwx 1 root root       34 Nov 17 20:49 libopenblas.so.0 -> libopenblas_haswellp-r0.3.4.dev.so
drwxr-xr-x 2 root root     4096 Oct  8  2018 pkgconfig

(ai) ubuntu@ip-10-0-1-82:~/lapack/build$ sudo make install
[  7%] Built target blas
[100%] Built target lapack
Install the project...
-- Install configuration: "Release"
-- Up-to-date: /home/ubuntu/.local/lapack/cmake/lapack-3.9.0/lapack-targets.cmake
-- Up-to-date: /home/ubuntu/.local/lapack/cmake/lapack-3.9.0/lapack-targets-release.cmake
-- Up-to-date: /home/ubuntu/.local/lapack/pkgconfig/lapack.pc
-- Up-to-date: /home/ubuntu/.local/lapack/cmake/lapack-3.9.0/lapack-config.cmake
-- Up-to-date: /home/ubuntu/.local/lapack/cmake/lapack-3.9.0/lapack-config-version.cmake
-- Up-to-date: /home/ubuntu/.local/lapack/pkgconfig/blas.pc
-- Up-to-date: /home/ubuntu/.local/lapack/libblas.a
-- Up-to-date: /home/ubuntu/.local/lapack/liblapack.a

$ ls -l /home/ubuntu/.local/lapack/
total 12948
drwxr-xr-x 3 root root     4096 Nov 15 15:57 cmake
-rw-r--r-- 1 root root   631862 Nov 15 17:05 libblas.a
-rw-r--r-- 1 root root 12614856 Nov 15 17:06 liblapack.a
drwxr-xr-x 2 root root     4096 Nov 15 15:57 pkgconfig

david laxer <davidlaxer>
Tue 17 Nov 2020 08:46:40 PM UTC, comment #3: 

Are you sure you are linking to your self compiled BLAS and other libraries?
From the snippet of the configure output you showed, it looks like LDFLAGS is empty. Where did you install those libraries?

Markus Mützel <mmuetzel>
Group administrator
Tue 17 Nov 2020 07:42:53 PM UTC, comment #2: 

I did read the doc you alluded to and followed the instructions for building 64 bit libraries.
I download, built and installed OpenBlas, LAPACK, etc. and built them for 64 bit.

E.g.

make BINARY=64

Clearly, something is wrong!

david laxer <davidlaxer>
Tue 17 Nov 2020 03:57:59 PM UTC, comment #1: 

Was your BLAS library actually compiled to use 64-bit indexing?  What about other libraries that also use the BLAS?  They all need to be consistently using 64-bit integers for indexing.

Have you read

https://octave.org/doc/v5.2.0/Compiling-Octave-with-64_002dbit-Indexing.html#Compiling-Octave-with-64_002dbit-Indexing

?

We build Octave for Windows systems with 64-bit BLAS and related libraries, so I'm confident that it is possible to build Octave this way.  But most Linux distributions don't package BLAS and related libraries compiled to use 64-bit integers for indexing.  So if you want to do that, you probably have to correctly configure and build those libraries as well.

I

John W. Eaton <jwe>
Group administrator
Tue 17 Nov 2020 03:47:58 PM UTC, original submission:  

Initially, 64-bit BLAS array dims and indexing:  was 'NO'.


{{{
./../configure --prefix=$HOME/my_octave CPPFLAGS="-I/usr/include/suitesparse" --with-blas="-lopenblas" -with-lapack=-llapack --enable-64
}}}

So, in 'configure' I had to manually set

ax_blas_integer_size=8

'configure' output:
{{{

64-bit BLAS array dims and indexing:  yes


Octave is now configured for x86_64-pc-linux-gnu

  Source directory:              ./..
  Installation prefix:           /home/ubuntu/my_octave
  C compiler:                    gcc  -pthread -fopenmp  -Wall -W -Wshadow -Wformat -Wpointer-arith -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wcast-align -Wcast-qual -g -O2
  C++ compiler:                  g++  -pthread -fopenmp  -Wall -W -Wshadow -Woverloaded-virtual -Wold-style-cast -Wformat -Wpointer-arith -Wwrite-strings -Wcast-align -Wcast-qual -g -O2
  Fortran compiler:              gfortran -g -O2 -std=legacy
  Fortran libraries:              -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/9/../../.. -lgfortran -lm -lquadmath
  Lex libraries:                
  LIBS:                          -lutil  -lm
  LDFLAGS:                      
  Extra LDFLAGS:                

  AMD CPPFLAGS:                 
  AMD LDFLAGS:                  
  AMD libraries:                 -lamd
  ARPACK CPPFLAGS:               -I/usr/include/arpack
  ARPACK LDFLAGS:               
  ARPACK libraries:              -larpack
  BLAS libraries:                -lopenblas
  BZ2 CPPFLAGS:                 
  BZ2 LDFLAGS:                  
  BZ2 libraries:                 -lbz2
  CAMD CPPFLAGS:                
  CAMD LDFLAGS:                 
  CAMD libraries:                -lcamd
  CARBON libraries:             
  CCOLAMD CPPFLAGS:             
  CCOLAMD LDFLAGS:              
  CCOLAMD libraries:             -lccolamd
  CHOLMOD CPPFLAGS:             
  CHOLMOD LDFLAGS:              
  CHOLMOD libraries:             -lcholmod
  COLAMD CPPFLAGS:              
  COLAMD LDFLAGS:               
  COLAMD libraries:              -lcolamd
  CURL CPPFLAGS:                
  CURL LDFLAGS:                 
  CURL libraries:                -lcurl
  CXSPARSE CPPFLAGS:            
  CXSPARSE LDFLAGS:             
  CXSPARSE libraries:            -lcxsparse
  DL libraries:                  -ldl
  FFTW3 CPPFLAGS:               
  FFTW3 LDFLAGS:                
  FFTW3 libraries:               -lfftw3_threads -lfftw3
  FFTW3F CPPFLAGS:              
  FFTW3F LDFLAGS:               
  FFTW3F libraries:              -lfftw3f_threads -lfftw3f
  FLTK CPPFLAGS:                 -I/usr/include/cairo -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/uuid -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/freetype2 -I/usr/include/libpng16
  FLTK LDFLAGS:                 
  FLTK libraries:                -lfltk_gl -lfltk -lX11
  fontconfig CPPFLAGS:           -I/usr/include/uuid -I/usr/include/freetype2 -I/usr/include/libpng16
  fontconfig libraries:          -lfontconfig -lfreetype
  FreeType2 CPPFLAGS:            -I/usr/include/freetype2 -I/usr/include/libpng16
  FreeType2 libraries:           -lfreetype
  GLPK CPPFLAGS:                
  GLPK LDFLAGS:                 
  GLPK libraries:                -lglpk
  HDF5 CPPFLAGS:                 -I/usr/include/hdf5/serial
  HDF5 LDFLAGS:                  -L/usr/lib/x86_64-linux-gnu/hdf5/serial
  HDF5 libraries:                -lhdf5
  Java home:                     /usr/lib/jvm/java-8-oracle
  Java JVM path:                 /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server
  Java CPPFLAGS:                 -I/usr/lib/jvm/java-8-oracle/include -I/usr/lib/jvm/java-8-oracle/include/linux
  Java libraries:               
  KLU CPPFLAGS:                 
  KLU LDFLAGS:                  
  KLU libraries:                 -lklu
  LAPACK libraries:              -llapack
  LLVM CPPFLAGS:                
  LLVM LDFLAGS:                 
  LLVM libraries:               
  Magick++ CPPFLAGS:             -I/usr/include/GraphicsMagick
  Magick++ LDFLAGS:             
  Magick++ libraries:            -lGraphicsMagick++ -lGraphicsMagick
  OpenGL libraries:              -lGL -lGLU
  PCRE CPPFLAGS:                
  PCRE LDFLAGS:                 
  PCRE libraries:                -lpcre
  PortAudio CPPFLAGS:           
  PortAudio LDFLAGS:            
  PortAudio libraries:           -lportaudio -lasound -lm -lpthread
  PTHREAD flags:                 -pthread
  PTHREAD libraries:            
  QHULL CPPFLAGS:               
  QHULL LDFLAGS:                
  QHULL libraries:               -lqhull
  QRUPDATE CPPFLAGS:            
  QRUPDATE LDFLAGS:             
  QRUPDATE libraries:            -lqrupdate
  Qt CPPFLAGS:                   -I/usr/include/x86_64-linux-gnu/qt5/QtNetwork -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtPrintSupport -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtHelp -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtWidgets -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtGui -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtSql -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtXml -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtCore -I/usr/include/x86_64-linux-gnu/qt5
  Qt LDFLAGS:                   
  Qt GUI libraries:              -lQt5Network -lQt5PrintSupport -lQt5Help -lQt5Widgets -lQt5Gui -lQt5Sql -lQt5Xml -lQt5Core -lqscintilla2_qt5
  Qt OpenGL libraries:           -lQt5OpenGL -lQt5Widgets -lQt5Gui -lQt5Core
  Qt moc:                        moc -qt5
  Qt uic:                        uic -qt5
  Qt rcc:                        rcc -qt5
  Qt lrelease:                   lrelease -qt5
  Qt qcollectiongenerator:       qcollectiongenerator -qt5
  Qt qhelpgenerator:             qhelpgenerator -qt5
  READLINE libraries:            -lreadline
  Sndfile CPPFLAGS:             
  Sndfile LDFLAGS:              
  Sndfile libraries:             -lsndfile
  SuiteSparse config libraries:  -lsuitesparseconfig
  SUNDIALS IDA CPPFLAGS:        
  SUNDIALS IDA LDFLAGS:         
  SUNDIALS IDA libraries:        -lsundials_ida
  SUNDIALS NVECTOR CPPFLAGS:    
  SUNDIALS NVECTOR LDFLAGS:     
  SUNDIALS NVECTOR libraries:    -lsundials_nvecserial
  SUNLINSOL KLU CPPFLAGS:       
  SUNLINSOL KLU LDFLAGS:        
  SUNLINSOL KLU libraries:       -lsundials_sunlinsolklu
  TERM libraries:                -lncurses
  UMFPACK CPPFLAGS:             
  UMFPACK LDFLAGS:              
  UMFPACK libraries:             -lumfpack
  X11 include flags:            
  X11 libraries:                 -lX11
  Z CPPFLAGS:                   
  Z LDFLAGS:                    
  Z libraries:                   -lz

  Default pager:                 less
  gnuplot:                       gnuplot

  Build Octave Qt GUI:                  yes (version: 5)
  JIT compiler for loops:               no
  Build Java interface:                 yes
  Build static libraries:               no
  Build shared libraries:               yes
  Dynamic Linking API:                  dlopen
  Include support for GNU readline:     yes
  Use push parser in command line REPL: yes
  64-bit array dims and indexing:       yes
  64-bit BLAS array dims and indexing:  yes
  OpenMP SMP multithreading:            yes
  Truncate intermediate FP results:     yes
  Build cross tools:                    no
  Build docs:                           yes


}}}

I'm not sure why configure was
Initially, 64-bit BLAS array dims and indexing:  was 'NO'.

I ran this test program which exited normally with no errors:

{{{
$ gfortran -o conftest -g -O2   conftest.f  -lopenblas
```
conftest.f:

      program main

      integer*8 two, n
      integer*4 n2(2)
      double precision d, a(1), b(1), ddot
      equivalence (n, n2)

      a(1) = 1.0
      b(1) = 1.0

c Generate 2**32 + 1 in an 8-byte integer.  Whether we have a big
c endian or little endian system, both 4-byte words of this value
c should be 1.

      two = 2
      n = (two ** 32) + 1

c Check that our expectation about the type conversions are correct.

      if (n2(1) .ne. 1 .or. n2(2) .ne. 1) then
        print *, 'invalid assumption about integer type conversion'
        stop 2
      endif

  •     print *, n, n2(1), n2(2)
  •     print *, a(1), b(1)


c DDOT will either see 1 or a large value for N.  Since INCX and INCY
c are both 0, we will never increment the index, so A and B only need to
c have a single element.  If N is interpreted as 1 (BLAS compiled with 4
c byte integers) then the result will be 1.  If N is interpreted as a
c large value (BLAS compiled with 8 byte integers) then the result will
c be the summation a(1)*b(1) 2^32+1 times.  This will also take some
c time to compute, but at least for now it is the unusual case so we are
c much more likely to exit quickly after detecting that the BLAS library
c was compiled with 4-byte integers.

      d = ddot (n, a, 0, b, 0)

  •     print *, a(1), b(1), d


c Success (0 exit status) means we detected BLAS compiled with
c 8-byte integers.

      if (d .eq. 1.0) then
        stop 1
      endif


      end

}}}

Finally, make failed running:  ./.build/doc/interpreter/splinefit4.eps
with index out of range' in ../liboctave/numeric/qr.cc, running
{{{
./scripts/polynomial/splinefit.m
in _splinefit_

doing qr:
[Q,R,dummy] = qr(B2); %#ok
}}}

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by davidlaxer (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-11-17 mmuetzel StatusNone Invalid / Not an Octave Bug
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code