bugGNU Octave - Bugs: bug #45659, version function: options and...

 
 

bug #45659: version function: options and return values supported by Matlab

Submitter:  CH <atcl>
Submitted:  Thu 30 Jul 2015 06:12:57 PM UTC
   
 
Category:  Interpreter Severity:  1 - Wish
Priority:  2 Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 08 Apr 2018 03:42:53 PM UTC, comment #59: 

All features discussed in here seem to be working with the current "version" function. If some other features are missing or there are other issues, this can be re-opened or a new bug report can be filed.

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Fri 09 Feb 2018 07:33:32 PM UTC, comment #58: 

I went ahead and pushed the modified version of the changeset.  Is there anything else that needs to be done for this bug report?

Thanks to everyone for help implementing this feature.  I think it will be useful to be able to easily get this info for debugging purposes.

John W. Eaton <jwe>
Group administrator
Thu 08 Feb 2018 10:21:41 PM UTC, comment #57: 

I'm not sure that fftw version info belongs in _octave_config_info_ either, for the same reason I gave in comment #55.  Maybe it should be handled in the same way as _blas_version_ and _lapack_version_?

John W. Eaton <jwe>
Group administrator
Thu 08 Feb 2018 10:18:10 PM UTC, comment #56: 

I propose the attached change instead.



(file #43209)

John W. Eaton <jwe>
Group administrator
Thu 08 Feb 2018 09:31:29 PM UTC, comment #55: 

Thanks for the patch.

_octave_config_info_ isn't the right place for this.  That is supposed to hold information about the system configuration, i.e., what happens in the config script, so that would be something like --with-blas="-lblas" was used as a configuration option, or the configure script decided to use -lblas.  The actual version of blas or lapack that is being used is not known until run time, so I think this information belongs somewhere else.  Also, I want to avoid adding more static variables if possible.  How much time does it take to execute the discovery functions?  Is there any harm in just providing functions that compute this info when the version function needs it?

John W. Eaton <jwe>
Group administrator
Thu 08 Feb 2018 08:25:55 PM UTC, comment #54: 

Attached is a patch that adds support for the "-blas" and "-lapack" options of the "version" function.
Especially the "-blas" option that checks for linked symbols from a bunch of different BLAS implementations might be useful to identify a bad system configuration.

(file #43208)

Markus Mützel <mmuetzel>
Group administrator
Wed 07 Feb 2018 09:23:27 PM UTC, comment #53: 

I don't think the library configuration issue is a thing that needs fixing in Octave, I think the configure script is doing the right thing as it stands.

I think there are so many ways these libraries can be configured on different systems, and different methods would be needed to try to detect some kind of multiple linkage, I'm not sure how we would address that for a simple warning in the configure script. But if someone does want to dream up a way (not involving ldd for example) to detect that our link options will result in something that links in more than one BLAS implementation, then I think they are welcome to try.

Mike Miller <mtmiller>
Group Member
Wed 07 Feb 2018 09:14:44 PM UTC, comment #52: 

@Mike: That was the issue. After installing libopenblas-dev, I see:

BLAS_LIBS='-lopenblas'


I am now waiting for make to finish.

Maybe it wouldn't be a bad idea to add a warning if we can detect a badly configured system like Martin suggested.

Markus Mützel <mmuetzel>
Group administrator
Wed 07 Feb 2018 09:13:01 PM UTC, comment #51: 

Is this something we can fix?  How can we prevent other libraries from directly linking with other libraries?  Or, how can we detect that linking with -lblas will just do the right thing?  Do we have any way of knowing that there is some kind of packaging magic going on behind the scenes to cause -lblas to really be Atlas or OpenBLAS or whatever and that it will be managed consistently for all dependencies that might use BLAS functions?

John W. Eaton <jwe>
Group administrator
Wed 07 Feb 2018 09:01:55 PM UTC, comment #50: 

Just FYI. Fedora has the same mess with octave linking to multiple blas libraries via other libraries:



 ldd /usr/local/bin/octave-cli | grep blas
        libopenblas.so.0 => /usr/local/lib/libopenblas.so.0 (0x00007ff3b2fbb000)
        libopenblasp.so.0 => /lib64/libopenblasp.so.0 (0x00007ff3a9ec1000)
        libblas.so.3 => /lib64/libblas.so.3 (0x00007ff3a9c6b000)


ldd /usr/local/bin/octave-cli | grep atlas
        libsatlas.so.3 => /usr/lib64/atlas/libsatlas.so.3 (0x00007fbe5517f000)


Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Wed 07 Feb 2018 08:52:38 PM UTC, comment #49: 

If another configure like that still says it is linking against ATLAS, that means you have libatlas-base-dev installed but you do not have libopenblas-dev installed, and you have libopenblas-base configured as the default BLAS library. This is a bad configuration. Try either uninstalling all ATLAS packages or installing libopenblas-dev. Or configure Octave with '--with-blas=blas' to bypass automatic detection of ATLAS and OpenBLAS.

Mike Miller <mtmiller>
Group Member
Wed 07 Feb 2018 08:33:41 PM UTC, comment #48: 

I think I configured with

  $ ../configure CPPFLAGS=-I/usr/include/hdf5/serial LDFLAGS=-L/usr/lib/x86_64-linux-gnu/hdf5/serial


At least this is what config.log says in the beginning. But I was messing around with the settings before. I'll try again after a make maintainer-clean, bootstrap, configure, make cycle.

Markus Mützel <mmuetzel>
Group administrator
Wed 07 Feb 2018 08:15:06 PM UTC, comment #47: 

My guess is that Octave is linking with ATLAS (because of what your config.log says), but other libraries like SuiteSparse are linking with the system-configured BLAS (using update-alternatives), which is OpenBLAS on your system.

Mike Miller <mtmiller>
Group Member
Wed 07 Feb 2018 08:13:47 PM UTC, comment #46: 

With Debian unstable with ATLAS, OpenBLAS, and reference BLAS all installed together, I get


>> check_blas_ver
ans = OpenBLAS (config: NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell)


I think you may have a misconfiguration on your system Markus. Octave will certainly build against any of these options. It looks like you have configured Octave (maybe intentionally, maybe not) to build against ATLAS, but you may have OpenBLAS installed and set up to be the default BLAS implementation.

On my system,


$ ls -l /etc/alternatives/lib{blas,lapack}*
lrwxrwxrwx 1 root root 44 Sep 22 16:36 /etc/alternatives/libblas.a-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas/libblas.a
lrwxrwxrwx 1 root root 47 Sep 22 16:36 /etc/alternatives/libblas.so.3-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
lrwxrwxrwx 1 root root 45 Sep 22 16:36 /etc/alternatives/libblas.so-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas/libblas.so
lrwxrwxrwx 1 root root 46 Sep 22 16:36 /etc/alternatives/liblapack.a-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas/liblapack.a
lrwxrwxrwx 1 root root 49 Sep 22 16:36 /etc/alternatives/liblapack.so.3-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas/liblapack.so.3
lrwxrwxrwx 1 root root 47 Sep 22 16:36 /etc/alternatives/liblapack.so-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas/liblapack.so

$ octave --eval '__octave_config_info__ config_opts'
ans =  '-C' '--prefix=/opt/gnu/octave' '--with-blas=blas' 'JAVA_HOME=/usr/lib/jvm/default-java'


Mike Miller <mtmiller>
Group Member
Wed 07 Feb 2018 08:06:23 PM UTC, comment #45: 

@grisu86: Could you open a new bug report for the issue with linking to different BLAS? I worry I wouldn't be able to describe the issue correctly. We deviated quite a bit from the original scope of this bug report.

Markus Mützel <mmuetzel>
Group administrator
Tue 06 Feb 2018 08:34:52 AM UTC, comment #44: 

@mmuetzel
I installed Octave 4.2.1 using apt on a clean Ubuntu 17.10 system an ran into the same problem. It installs by default OpenBLAS as LAPACK and as BLAS implementation but with a strange behavior to satisfy the update-alternative system. Afterwards I got the following setup:

-> libblas.so.3 is provided by OpenBLAS ( Symlink to /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3)
-> liblapack.so.3 is provides by OpenBLAS ( Symlink to /usr/lib/x86_64-linux-gnu/openblas/liblapack.so.3) but links to  /usr/lib/x86_64-linux-gnu/libopenblas.so.0

This setup is ok as long libblas.so and liblapack.so are provided by the same version of OpenBLAS. If for example libblas.so.3 is changed by

apt install libatlas3-base
update-alternatives --set libblas.so.3-x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3


then the BLAS implementations get mixed and you can find ATLAS and OpenBLAS symbols although Octave is linked correctly against libblas.so and liblapack.so

The characteristic symbols are really a problem.
Some stuff I know:
- OpenBLAS: something from the openblas_get_config
- GotoBLAS: gotoblas_... but not openblas_
- ATLAS: ATL_buildinfo
- Reference: Nothing
- MKL: mkl_...
- ACML: acmlinfo
- FlexiBLAS: flexiblas_get_version and others depending on the loaded backend.



Martin K <grisu86>
Mon 05 Feb 2018 07:59:11 PM UTC, comment #43: 

Martin, thank you for the explanation.
Here is part of the output of ldd on the octave-cli in src/.lib of my build tree:

        liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f5d981d8000)
        libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x00007f5d95b4c000)
        libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f5d8f753000)


Also the version of Octave that is shipped with Ubuntu (apt-get) seems to be linked with BLAS and OpenBLAS:

        liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f5a3d077000)
        libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x00007f5a3a9eb000)
        libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f5a345f2000)


Does this mean we could have trouble?

Strange thing though, I can't find any line in the output of ldd referencing "libatlas". Yet I can call ATL_buildinfo from an .oct file without specifically linking to it and "ATL_sgemm" is found in the loaded symbols. Unless mkoctfile links ATLAS automatically if configure finds it?
In my config.log I see:

  BLAS libraries:                -lcblas -lf77blas -latlas



Wrt checking the linked BLAS implementations: Do you know any specific symbol that can be used to identify reference BLAS (i.e. one that isn't present in the other BLASs)?

Markus Mützel <mmuetzel>
Group administrator
Mon 05 Feb 2018 03:47:50 PM UTC, comment #42: 

I talked to @CH today about this issue because he thinks I know something.

The technical reason what happens, to link a application against OpenBLAS and ATLAS, is described in http://www2.mpi-magdeburg.mpg.de/mpcsc/koehlerm/talks/2018_FIN_Flexiblas.pdf or http://www.netlib.org/lapack/lawnspdf/lawn284.pdf and appears in the case of Octave because the configure script prefers to use libopenblas with not "--with-blas" argument is given. But several depending libraries (like arpack, suitesparse, qrupdate) used from the system are linked against libblas. If the OS is debian/ubuntu this is set by update-alternatives to one of the BLAS libraries installed. In the case of @mmuetzel this is ATLAS. This causes both libraries with the same symbols to be linked against Octave and nobody can say which symbol is called in which situation.  In order to avoid this and make the idea of check_blas_ver.cc properly working, the configure script should search by default for libblas.so and liblapack.so instead of libopenblas.so and during the startup of octave one should search like the check_blas_ver.cc idea for different characteristic symbols from the BLAS libraries and drop a warning if octave is linked against more than one BLAS implementation.

@mmuetzel
Can you check against which BLAS Octave is linked using ldd `which octave-cli` ?

Martin K <grisu86>
Sun 04 Feb 2018 07:11:28 PM UTC, comment #41: 

Thanks Marco. I think your code can come in very useful.

@all: With the recent changes, it is easier to check for functions in loaded or linked-in libraries. With the attached file, I see the following on Ubuntu 17.10 with installed reference BLAS, OpenBLAS and ATLAS:

>> mkoctfile check_blas_ver.cc
>> check_blas_ver
ans = OpenBLAS (config: NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Nehalem)
ATLAS


So it looks like (at least) both OpenBLAS and ATLAS are linked in or loaded into my self-compiled Octave 4.3.0+. Why is this the case? Can it be determined which library is actually used by Octave? Or are some functions used from the ATLAS library and others from OpenBLAS?

FWIW, this is the output of "ATL_buildinfo":

ATLAS version 3.10.3 built by buildd on Tue Sep 19 15:17:33 UTC 2017:
   UNAME    : Linux lcy01-26 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
   INSTFLG  : -1 0 -a 1 -l 1
   ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_x86SSE2 -DATL_CPUMHZ=2600 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664
   F2CDEFS  : -DAdd_ -DF77_INTEGER=int -DStringSunStyle
   CACHEEDGE: 1048576
   F77      : /usr/bin/gfortran, version GNU Fortran (Ubuntu 7.2.0-5ubuntu1) 7.2.0
   F77FLAGS : -g -O2 -fdebug-prefix-map=/build/atlas-jedIp7/atlas-3.10.3=. -fstack-protector-strong -m64 -fPIC
   SMC      : /usr/bin/gcc, version gcc (Ubuntu 7.2.0-5ubuntu1) 7.2.0
   SMCFLAGS : -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/atlas-jedIp7/atlas-3.10.3=. -fstack-protector-strong -m64 -fPIC
   SKC      : /usr/bin/gcc, version gcc (Ubuntu 7.2.0-5ubuntu1) 7.2.0
   SKCFLAGS : -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/atlas-jedIp7/atlas-3.10.3=. -fstack-protector-strong -m64 -fPIC



(file #43171)

Markus Mützel <mmuetzel>
Group administrator
Thu 23 Jun 2016 08:08:16 AM UTC, comment #40: 

Hi, I attach an exercise I made in order to detect LAPACK version (but not if it is from MKL or whatever).

(file #37557)

Marco Caliari <caliari>
Group Member
Wed 13 Apr 2016 08:26:26 PM UTC, comment #39: 

Yeah, I saw ATL_buildinfo, but it prints to the standard output which I deemed useless to us. I don't think we want to open a pipe and fork just so we can get this string from the ATLAS library.

If we can figure out the best way to do this dlsym function detection, and the equivalent for Windows, then maybe we should just detect the presence of each of them until we get a way to actually get a version string from all of them.

Mike Miller <mtmiller>
Group Member
Wed 13 Apr 2016 07:58:36 PM UTC, comment #38: 

For ATLAS, there is ATL_buildinfo [1]. That function prints the version info. Maybe we can parse its output...

[1] http://math-atlas.sourceforge.net/faq.html#probeV

Markus Mützel <mmuetzel>
Group administrator
Wed 13 Apr 2016 07:30:41 PM UTC, comment #37: 

I also don't like that we are giving special treatment to proprietary software here.

How about asking the ATLAS and OpenBLAS maintainers to provide functions that will return version information so that we can use those as well?

John W. Eaton <jwe>
Group administrator
Wed 13 Apr 2016 07:27:50 PM UTC, comment #36: 

In Octave, we prefer C++ style casts instead of C-style casts.

I'd probably use a typedef to define a type for the function and then an assignment with a reinterpret_cast<>:

The blas version function definitely doesn't belong in the oct-shlib file.  Maybe it would be better to add a function to the oct-shlib class to either open the current executable as a shared library, then we could just use the resulting octave_shlib object as we do any other to search for symbols.

But I'm not sure what the best organization is for this yet.  I'll have to thing about this a bit more.

John W. Eaton <jwe>
Group administrator
Wed 13 Apr 2016 05:42:49 PM UTC, comment #35: 

As suggested by Mike, the attached patch checks for functions specific to OpenBLAS or ATLAS and prints that (but not the version numbers). If MKL is loaded, it returns the string returned by mkl_get_version_string.

I put the new function "blas_version" into file "oct-shlib.cc" because it already had some logic for including the correct headers for dlsym. If it should be moved somewhere else please let me know (or feel free to move it).
When compiling with this patch, I get the following warning:

../liboctave/util/oct-shlib.cc: In function 'std::__cxx11::string blas_version()':
../liboctave/util/oct-shlib.cc:534:41: warning: use of old-style cast [-Wold-style-cast]
   *(void **) (&ptr_mkl_getversion_string) = dlsym (NULL, "mkl_get_version_string"
                                         ^

Is this important? I do not know what the current-style cast would be in this case. "dlsym" does not seem to be standard conformant anyway judging from what I could find. Is there any other better way?
Neither can I test whether this works because I do not have access to the MKL.

The output for me with this patch is:

>> version -blas
ans = OpenBLAS


Would a similar approach for LAPACK be OK? Check for ATLAS, give MKL version string, default answer "unknown or reference LAPACK"? Are there any other LAPACK implementations that should be checked? Are these supported at all? I could not find any in ax-lapack.m4.


Another minor change in this patch: Matlab supports any mix of upper and lowercase letters in the options to version. This fixes this difference.

(file #36918)

Markus Mützel <mmuetzel>
Group administrator
Tue 12 Apr 2016 07:29:03 PM UTC, comment #34: 

Just to clarify, both ATLAS and OpenBLAS do have ways to get each of their versions, but only if you include and compile against their header files. Their versions are not encoded in the compiled libraries anywhere, so that is not useful to us the way Octave currently uses those libraries.

Mike Miller <mtmiller>
Group Member
Tue 12 Apr 2016 07:21:14 PM UTC, comment #33: 

Correct, there is no standard way that I know of to get version information, nor is there even a non-standard way to get a version string out of any of the BLAS libraries that I can see, other than Intel's MKL (mkl_get_version_string).

I'd say the best we could possibly do would be to detect that either OpenBLAS or ATLAS are loaded and print that (but not the version numbers), or detect that MKL is loaded and print its version string, or failing any of those shrug and say "unknown or reference BLAS".

Mike Miller <mtmiller>
Group Member
Mon 11 Apr 2016 08:09:15 PM UTC, comment #32: 

I don't see where flexiblas extracts version info from the blas or lapack libraries.  Can you point directly to that info?

In any case, I don't see how that can be done for an arbitrary blas or lapack library given that there is no standard way in the blas and lapack specs to determine version info.

John W. Eaton <jwe>
Group administrator
Mon 11 Apr 2016 07:36:18 PM UTC, comment #31: 

The FlexiBLAS project ( http://www.mpi-magdeburg.mpg.de/projects/flexiblas ) is an abstraction layer to the BLAS library to select a BLAS library during runtime. FlexiBLAS can display vendor string and version info for the currently selected BLAS backend, so may be their solution can also be used here.

Matlab, at least under Linux, can handle changing the BLAS and LAPACK backends via LD_LIBRARY_PATH and the version function returns the correct version strings.

CH <atcl>
Mon 11 Apr 2016 06:04:43 PM UTC, comment #30: 

As for the blas and lapack versions, there is no simple platform-independent way to check for that information.

The info at build time isn't useful because you can build against the reference blas but use OpenBLAS, ATLAS, or some other vendor blas at run time.

Additionaly, there is no standard way for the blas or lapack library to report version info.  I don't think that information is even stored for the reference versions of the lapack or blas libraries.

Matlab can provide this information because they distribute proprietary binaries that are linked with speciic blas and lapack libraries.  As I understand it replacing the Matlab blas or lapack libraries is explicitly unsupported, so if you swap them out yourself and something breaks, it's your fault.

John W. Eaton <jwe>
Group administrator
Mon 11 Apr 2016 05:54:37 PM UTC, comment #29: 

I checked in an additional change to store the id in libgui and the main cli and gui executables and check for consistency at startup.

John W. Eaton <jwe>
Group administrator
Mon 11 Apr 2016 04:33:03 PM UTC, comment #28: 

I checked in some changes to store the hg id separately in liboctave and liboctinterp so we can easily detect mismatches.

A more common problem might be picking up incorrect versions of those libraries from the main octave-gui or octave-cli programs due to some bad LD_LIBRARY_PATH setting or other installation problem.  So maybe we should also store another copy of the hg id in the src directory and check for consistency when Octave starts.  If we do this, should a mismatch be fatal or just produce a warning?

If we require consistency for Octave to start, then I guess we could just pick one and go back to having only "hg_id" as a single field in the _octave_config_info_ struct.

John W. Eaton <jwe>
Group administrator
Mon 11 Apr 2016 03:43:25 PM UTC, comment #27: 

+1 for reproducible builds also. If I clone the same source and build twice, I hope to get the same output. I'd prefer to not have the time that I happen to be building affect the product of that build.

Mike Miller <mtmiller>
Group Member
Mon 11 Apr 2016 03:31:46 PM UTC, comment #26: 

Sorry about the duplicate text in my last comment, I did some cut and paste and failed to remove the copied text.

For Matlab compatibility we just need the release date, correct?  In that case, I think we should not put the build date and time in a source file.  I don't think that matters to users.  Developers who want to know the build time can look at timestamps in their build tree and compare checksums of installed files with the ones in the build tree to know whether they are the same.

I'll make that change.

John W. Eaton <jwe>
Group administrator
Mon 11 Apr 2016 03:21:17 PM UTC, comment #25: 

I checked in your changes to add fftwf_version to _octave_config_info_.  Thanks.

It's not the HG-ID target being tagged with .PHONY that is causing the trouble.  There are move-if-change rules to prevent further action if there are no changes to the build-info.cc source file.

The problem is that we are now putting the build time into the source file.  So that will always change and I see no simple way to fix that.

If you use _TIME_ and _DATE_ as you did before, those could be signifcantly different from the actual last build time if your hg id wasn't changing.  For example, if you were working on some changes and your hg id was always SOMETHING+ (you aren't checking in any changes) but you are repeatedly building.  Then the "build time" would not change because the source file with the _TIME_ and _DATE_ would not change.

If you change %OCTAVE_BUILD_TIME% to 0 in libinterp/build-info.in.cc, then run make a few times you should see what I mean.  HG-ID and build-info.cc will be generated, but build-info.cc shouldn't change so no compiling or linking should happen either.

If we use _TIME_ and _DATE_ as you did before, then the timestamps recorded in the object file could be signifcantly different from the actual last build time if your hg id wasn't changing.  For example, if you were working on some changes and your hg id was always SOMETHING+ (you aren't checking in any changes) but you are repeatedly building.  Then the "build time" would not change because the build-info.cc source file with the _TIME_ and _DATE_ macros would not change, so would not be recompiled.

Do we really want build time and date embedded in a source file?  Is this really a Matlab compatibility issue?  Who cares?  The more important info is the HG ID.

Maybe just record the build time in a text file somewhere and have Octave read it at startup?  Maybe just edit into the version.m file directly?

But I still think that it would be useful to have the hg id stored in liboctave, libinterp and possibly libgui so that we can ensure that the libraries are consistent and the ones that are expected.

John W. Eaton <jwe>
Group administrator
Mon 11 Apr 2016 09:19:51 AM UTC, comment #24: 

The attached patch adds "fftwf_version" to octave_config_info. Under normal circumstances, this should always be the same as "fftw_version".

I did not know about ctime. Maybe it could be added to the "see also" of datestr?

A comment on the make rules for HG-ID and libinterp/build-info.cc: As it stands right now, libinterp/build-info.cc implicitly is a phony target because it depends on the phony target HG-ID. Thus, build-info.cc is created on every run of make. The changed file is then re-compiled and linked in every time. This might lead to an unnecessarily increased compile (and linker!) time when only changes on code for unrelated libraries were done.
My first approach was making only the target that created HG-ID phony and making libinterp/build-info.cc depend on the file instead of on the target.
This however meant that the build time (in the same file) was only updated (automatically) when the hg-id changed. Maybe a small trade-off for local wip builds.
Any of the two approaches has their advantages, imho. Just wanted to re-phrase what I tried to explain in my (not very clear) comment #12.

(file #36891)

Markus Mützel <mmuetzel>
Group administrator
Sun 10 Apr 2016 07:58:28 PM UTC, comment #23: 

I checked in the following changeset:

  http://hg.savannah.gnu.org/hgweb/octave/rev/184b85b31e03

It drops the local date and time.  Worse than there being some small difference between the time the build-info.cc file is generated and compiled is that the _TIME_ and _DATE_ depend on timezone.

You can use the Unix-style time functions that Octave already provides to convert the time to some other form.  See for example, ctime or localtime + asctime or strftime, etc.

Since the fftw version function was defined in oct-fftw.cc, the declaration belongs in oct-fftw.h, not build-info.h.  Should we care about the fftwf version separately from the fftw version?  I suppose they could be different, most likely if there is some installation problem.  Having both could help in diagnosing that but it is probably not likely to see that problem.

John W. Eaton <jwe>
Group administrator
Sun 10 Apr 2016 09:37:34 AM UTC, comment #22: 

The last patch did not work because I forgot to include my local changes to the build-info.h header. This version should compile.
It also adds the new field "build_unixtime" to the __octave_config_info___ structure.

I left the build_time and build_date fields in because they are in a more user readable form. However, they will be a little bit off depending on how fast the system is for compilation. I am OK with removing them if you deem it better.

To get a more user readable date string from the unix time, you can use the following line of code:

>> datestr (__octave_config_info__.build_unixtime/60/60/24 + datenum ("01-Jan-1970"))
ans = 10-Apr-2016 09:05:55

Please, observe that this is UTC+0 independent on your local time zone. The same holds for build_date and build_time (at least when compiling with gcc).

Maybe it would make sence to add small helper functions to convert between unix time and Matlab time (if not already there) when removing build_date and build_time. These could maybe also contain optional conversion to local timezone. But maybe better keep this out of this bug...

(file #36887)

Markus Mützel <mmuetzel>
Group administrator
Sun 10 Apr 2016 01:30:47 AM UTC, comment #21: 

Another small change to the makefile rule for the HG-ID target: The presence of the .hg directory does not necessarily mean that hg is available. It could have been deinstalled or it could be a copy to another PC...

(file #36885)

Markus Mützel <mmuetzel>
Group administrator
Sun 10 Apr 2016 01:10:23 AM UTC, comment #20: 

It seems I was a little late with the last patch. Attached please find the patch rebased to the current tip. Basicly just the part with the fftw version and a change to reflect the new field names in __octave_config_info__.

(file #36884)

Markus Mützel <mmuetzel>
Group administrator
Sun 10 Apr 2016 12:26:37 AM UTC, comment #19: 

With the attached patch the makefile directly inserts the hg id into the template for build-info.cc. Some tests should make sure that hg is present and that we are building from an hg repo. The hg id would be 0 otherwise.

I added FFTW to the version function. Are there any alternative libraries that are supported by Octave that might have their version info available with a different call?
The respective output for me is:

>> version ("-fftw")
ans = fftw-3.3.4-sse2-avx



Regarding BLAS and LAPACK: Does the build system "know" which versions or variants it is using? If yes, can this info be stored somehow?

(file #36883)

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Apr 2016 11:46:11 PM UTC, comment #18: 

Markus: I checked in your changeset then another one to make the changes I was suggesting in comment #17:

  http://hg.savannah.gnu.org/hgweb/octave/rev/fe1447ae68cf
  http://hg.savannah.gnu.org/hgweb/octave/rev/cf552443c104

I tested it by doing "make dist" and then building from the tar.gz file.  Both were out-of-source-tree VPATH builds.

Now I'm wondering whether it would be better to store a universal time stamp (Unix time?) instead of the local time and date.

John W. Eaton <jwe>
Group administrator
Fri 08 Apr 2016 09:54:18 PM UTC, comment #17: 

Markus:

Since the shell script in build-aux just runs a single shell command, I don't think we need it.  The other scripts that do substitutions are there because they substitute configuration variables and it is better to put those variables in a shell script that embed them in the Makefile.  That's also why those shell scripts are generated at configure time.

What happens with this hg id substitution if you are building from a tar file distribution?  In that case "hg id" won't give us a version number and we can't even assume that hg will be available.

John W. Eaton <jwe>
Group administrator
Fri 08 Apr 2016 07:47:35 PM UTC, comment #16: 

I don't know for sure, but I think the BLAS and LAPACK versions may be the ugliest part of this.

It may involve something like checking for specific functions that are only in MKL or ATLAS or OpenBLAS, and then querying each for their particular version strings. I have no idea if netlib BLAS and LAPACK provide any way to query a version.

The undocumentedmatlab link in comment #1 also shows that the version of FFTW can be queried in Matlab.

Mike Miller <mtmiller>
Group Member
Fri 08 Apr 2016 07:41:53 PM UTC, comment #15: 

In this version of the patch, the release date and the build date and time are part of the __octave_config_info__ struct.

Now build-info.cc is the file that is generated by the script and the header is static like you suggested. I agree that this is a much better approach.

Regarding the version functions defined in each library: Would these be versions that only change once in a while (hence not the hg id)? Would these be the same as OCTAVE_VERSION?

With this patch version.m supports all documented flags.
Is it OK to use the try catch statement for the Java functions? Is it reliable to check _octave_config_info_.build_features.JAVA instead?

I don't know how to get the info requested in comment #0. Does anyone have an idea how to check the vendor and version of the linked in BLAS and LAPACK libraries?

The current output of version.m and the new fields of __octave_config_info__ with this patch on my system are:

>> version
ans = 4.1.0+
>> [v,d]=version
v = 4.1.0+
d = 2015-05-26
>> version ("-date")
ans = 2015-05-26
>> version ("-description")
ans =
>> version ("-release")
ans =
>> version ("-java")
ans = Java 1.8.0_66-internal-b17 with Oracle Corporation OpenJDK 64-Bit Server VM mixed mode
>> version ("-blas")
warning: version: option '-blas' not implemented
warning: called from
    version at line 93 column 9
ans =
>> version ("-lapack")
warning: version: option '-lapack' not implemented
warning: called from
    version at line 96 column 9
ans =
>> __octave_config_info__.hgid
ans = e3888aa9a4d1+
>> __octave_config_info__.releasedate
ans = 2015-05-26
>> __octave_config_info__.builddate
ans = Apr  8 2016
>> __octave_config_info__.buildtime
ans = 20:37:04



(file #36869)

Markus Mützel <mmuetzel>
Group administrator
Thu 07 Apr 2016 04:13:38 PM UTC, comment #14: 

Just a heads up, but I'm also looking at something where I need version and config info in liboctave, so some of these things may eventually be moved there.  My changes are not urgent though, so they can wait until the issues in this bug report are resolved.

John W. Eaton <jwe>
Group administrator
Thu 07 Apr 2016 04:06:00 PM UTC, comment #13: 

Thanks for working on this.

Instead of adding a new function to return the release date, wouldn't it be sufficient to just add the release date to the octave_config_info structure?

I'd rather include header files to provide function declarations instead of inserting them by hand (your extern decl for oct_hg_id) in toplev.cc.  So instead of generating the header file, why not make that static and generate the source file instead?  Then toplev.cc could include build-info.h (which won't change with each change to the hg id) and the hg ID can be substituted into the build-info.cc file.

After thinking about it a little more, and also encountering numerous times when bugs are reported and we aren't sure whether the correct versions of libraries are being used, I think it might be useful to have version functions defined in each library (liboctave_version, libinterp_version, and libgui_version).  Then the octave_config_info could contain fields for each of these.  If things are working properly, each of these should return the same result.  But if there is some build problem, we could detect that there is a library version mismatch and at least warn about it.

Also, a small comment about commit messages.  Instead of writing "Add this or that ...", please write things like


* build-info.in.h, build-info.cc: New files.
* mk-build-info-h.in.sh: New script.
* configure.ac, Makefile.am: Update.
* module.mk (update_hg_id, libinterp/build-info.h): New rules.
* defaults.cc (FOCTAVE_RELEASE_DATE): New function.
* version.m: Also return release date.
* toplev.cc (F__octave_config_info__): New field, hgid.


John W. Eaton <jwe>
Group administrator
Thu 07 Apr 2016 08:22:22 AM UTC, comment #12: 

I removed the build date and time from version.m.

The output of version for the attached patch is now:

>> version
ans = 4.1.0+
>> [v,d]=version
v = 4.1.0+
d = 2015-05-26


The date is returned as stored in OCTAVE_RELEASE_DATE. Thus, it is formatted differently from what Matlab returns ("yyyy-mm-dd" instead of "mmmm d, yyyy"). Should I reformat with datestr for maximum compatibility?

The __octave_config_info__ struct now has a new field "hgid" holding the hg id.

>> __octave_config_info__.hgid
ans = 3b9524234d89+


The only file that has to be rebuild when the hg id changes is the new build-info.cc with its generated header.

I left the functions exposing the build date and time and the hg id directly to Octave intact for whom it might interest. My reasoning was that they can reside in the same file that has to be re-build anyway. However, date and time are only updated when build-info.cc is build (e.g. when it has to be re-build to update the hg id).
If it is agreed upon that these are not needed, I am OK with removing them.

Instead of making the header file holding the hg id directly a phony target, I took the following approach to keep the build time as efficient as possible:
- store the current hg id in a text file
- generate the header file that stores the hg id
- keep the function exposing the hg id in a seperate file
On subsequent calls:
- use a phony target that compares the current hg id with the one stored in the file. Only if the hg id has changed:

  • overwrite the text file with the current hg id
  • generate the header file that stores the hg id


Thus, on subsequent builds of the same revision (only local changes), nothing hg id related has to be rebuild. On updates of the hg id, only one additional object has to be rebuild (and linked).

Are you OK with these changes?

(file #36853)

Markus Mützel <mmuetzel>
Group administrator
Wed 06 Apr 2016 04:47:02 AM UTC, comment #11: 

I think it would be fine to have the hg revision (if available) stored somewhere in the binary.  Can it be stored separately from the version info?  Possibly as a separate field in the _octave_config_info_ struct?  To make that fairly efficient for the build, there could be a single file that contains a function that returns the hg version.  Then the only thing that would have to happen at build time is to compile that file (if it has changed) and relink.ee

I don't know whether we want to include the build time and date in the binary.  That would mean recompiling at least one file and relinking every time make runs.  That seems like a bit much to me.  But if others want this feature and someone wants to implement it, it's OK with me as long as it is optional.

John W. Eaton <jwe>
Group administrator
Wed 06 Apr 2016 04:14:37 AM UTC, comment #10: 

Thanks for starting to work on this.

Do we want the build date and time to be updated every time Octave is compiled? Or do we want to just use the OCTAVE_RELEASE_DATE string that's already defined?

Do we want to append the hg revision to the version string like this? Or do we only want it to be queryable with a different option?

I wouldn't be surprised if some people depend on the version return value to be a single word with the dotted version string, appending the hg revision might break that expectation.

And if we do append the hg revision to the version, do we want to suppress that when building an officially tagged release?

Mike Miller <mtmiller>
Group Member
Tue 05 Apr 2016 04:35:53 PM UTC, comment #9: 

Thank you all for your help and support.

The attached patch shows a work in progress without much comments and tests. Please, let me know if this goes in the correct direction.

As already pointed out, the added functions should reside in files which have as few dependencies as possible to cause as little rebuilding as necessary. Thus, I reasoned it would be best to add new files for them right next to where version.h is.

At the moment, it is necessary to touch build_timestamp.in.h for the hg ID, build date and time to update. Maybe it is possible to change the makefiles such that the corresponding files are build each time. They are quite lightweight and it should not cost too much to build them each time. However, I do not know what changes would be necessary.

I did not yet research the -java, -blas and -lapack options. But I am looking forward too hearing feedback on the parts already done.

Here is what Matlab R2012a returns for version:

>> version
ans =
7.14.0.739 (R2012a)
>> version -date
ans =
February 9, 2012
>> version -description
ans =
     ''
>> version -release
ans =
2012a
>> version -java
ans =
Java 1.6.0_17-b04 with Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM mixed mode
>> version -blas
ans =
Intel(R) Math Kernel Library Version 10.3.5 Product Build 20110720 for Intel(R) 64 architecture applications
>> version -lapack
ans =
Intel(R) Math Kernel Library Version 10.3.5 Product Build 20110720 for Intel(R) 64 architecture applications
Linear Algebra PACKage Version 3.3.1
>> [v, d] = version ()
v =
7.14.0.739 (R2012a)
d =
February 9, 2012


Here is what Octave returns for me with the attached patch:

>> version
ans = 4.1.0+ (3b9524234d89+)
>> [v,d,t]=version
v = 4.1.0+ (3b9524234d89+)
d = Apr  5 2016
t = 17:32:05


The plus after the hg id indicates local changes.
Arguments are still ignored.

(file #36841)

Markus Mützel <mmuetzel>
Group administrator
Tue 05 Apr 2016 03:29:13 PM UTC, comment #8: 

See

https://bitbucket.org/snippets/mtmiller/5rb7o/autoconf-version-suffix-from-mercurial

for a script I worked up to generate a mercurial revision string. Note that my goal was for it to be an empty string if building from a revision that is exactly a tagged release (so the version string would not have a suffix in that case). That may not be desired for this depending on how it is used.

Mike Miller <mtmiller>
Group Member
Tue 05 Apr 2016 03:23:46 PM UTC, comment #7: 

Yeah, I took a look at adding the mercurial id to the Octave version but ran into a couple of problems (if this bug is indeed growing to adopt that feature).

First, I tried adding it to configure.ac, but that frankly causes way too much rebuilding.

Second, adding it to the build without needing to reconfigure / rebuild a bunch of things needs a build-aux shell script  to produce the mercurial ID on stdout (easy) that can be $(shell) evaluated into a make variable. This means that make would have to check whether version.h needs to be updated every time make is run. Doable, but unfortunate.

That part aside, I think that adding it to version.h may work. I think there may still be too many cc files including version.h that don't actually need it, if we could reduce that list it would help keep rebuilds fast every time a hg commit is made.

Mike Miller <mtmiller>
Group Member
Tue 05 Apr 2016 01:29:28 PM UTC, comment #6: 

Sorry, it wasn't a bug report, it was a short thread on the maintainers mailing list:

http://lists.gnu.org/archive/html/octave-maintainers/2015-12/msg00229.html

John W. Eaton <jwe>
Group administrator
Tue 05 Apr 2016 01:18:02 PM UTC, comment #5: 

More useful would be the hg id (if building from the mercurial sources).  We already had a discussion about that in the comments for another bug report.

For releases, I don't think there's much we can do that is better than using the version number.  It would be difficult to know whether the build is from modified sources.  Well, I suppose that's true for builds from the mercurial sources as well...

John W. Eaton <jwe>
Group administrator
Tue 05 Apr 2016 11:34:00 AM UTC, comment #4: 

I thought about exposing OCTAVE_RELEASE_DATE for this function. That would be "2015-05-26" for the foreseeable time.

If desired, I could look into how to insert the current date and time at compilation into the code...

Markus Mützel <mmuetzel>
Group administrator
Tue 05 Apr 2016 10:25:48 AM UTC, comment #3: 

I think that this
 [v, d] = version
v = 4.1.0+
d = 2015-05-26

would be very helpful.
For me the date  and maybe the time, would be a way of telling which compile I am running. The version number stays the same for months at at time, but the time will be different each compile.

Doug Stewart <dastew>
Tue 05 Apr 2016 07:14:33 AM UTC, comment #2: 

I would be happy to work on this one if no one else wants to take it first...

Markus Mützel <mmuetzel>
Group administrator
Mon 04 Apr 2016 06:24:02 PM UTC, comment #1: 

Thanks for filing this bug report.

According to

https://www.mathworks.com/help/matlab/ref/version.html

the following should work:


>> version -date
ans = 2015-05-26
>> version -description
ans = something
>> version -release
ans = something
>> version -java
ans = OpenJDK version whatever
>> [v, d] = version
v = 4.1.0+
d = 2015-05-26


And according to this bug report, the versions of BLAS and LAPACK libraries should work with the -blas and -lapack options. These options are not documented at all on Matlab's site, but are demonstrated here:

http://undocumentedmatlab.com/blog/math-libraries-version-info-upgrade

This might be a good small project for a new contributor to work on.

Mike Miller <mtmiller>
Group Member
Thu 30 Jul 2015 06:12:57 PM UTC, original submission:  

Matlab returns the BLAS and LAPACK "vendor string" and version by the version command:


version('-blas')


and


version('-lapack')


CH <atcl>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #43209:  diffs.txt added by jwe (10KiB - text/plain)
file #43208:  bug45659_version_blas_lapack.patch added by mmuetzel (9KiB - application/octet-stream)
file #43171:  check_blas_ver.cc added by mmuetzel (2KiB - application/octet-stream)
file #37557:  lapackver.cc added by caliari (653B - text/x-c++src)
file #36918:  version_blas.patch added by mmuetzel (4KiB - text/x-diff)
file #36891:  version_3.patch added by mmuetzel (2KiB - text/x-diff)
file #36887:  version_2_v3.patch added by mmuetzel (6KiB - text/x-diff)
file #36885:  version_2_v2.patch added by mmuetzel (4KiB - text/x-diff)
file #36884:  version_2.patch added by mmuetzel (3KiB - text/x-diff)
file #36883:  version_beta4.patch added by mmuetzel (11KiB - text/x-diff)
file #36869:  version_beta3.patch added by mmuetzel (12KiB - text/x-diff)
file #36853:  version_beta2.patch added by mmuetzel (11KiB - text/x-diff)
file #36841:  version_beta1.patch added by mmuetzel (8KiB - text/x-diff)

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by grisu86 (Posted a comment)
  • -email is unavailable- added by caliari (Updated the item)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by dastew (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by atcl (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 23 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-02-26 mtmiller Carbon-CopyRemoved 80942 -
    2018-04-08 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2018-02-09 jwe StatusIn Progress Ready For Test
    2018-02-08 jwe Attached File- Added diffs.txt, #43209
    2018-02-08 mmuetzel Attached File- Added bug45659_version_blas_lapack.patch, #43208
    2018-02-04 mmuetzel Attached File- Added check_blas_ver.cc, #43171
    2016-11-10 siko1056 Dependencies- bugs #49575 is dependent
    2016-06-23 caliari Attached File- Added lapackver.cc, #37557
    2016-04-13 mmuetzel Attached File- Added version_blas.patch, #36918
    2016-04-11 mmuetzel Attached File- Added version_3.patch, #36891
    2016-04-10 mmuetzel Attached File- Added version_2_v3.patch, #36887
    2016-04-10 mmuetzel Attached File- Added version_2_v2.patch, #36885
    2016-04-10 mmuetzel Attached File- Added version_2.patch, #36884
    2016-04-10 mmuetzel Attached File- Added version_beta4.patch, #36883
    2016-04-08 mmuetzel Attached File- Added version_beta3.patch, #36869
    2016-04-07 mmuetzel Attached File- Added version_beta2.patch, #36853
    2016-04-06 mtmiller StatusNone In Progress
    2016-04-05 mmuetzel Attached File- Added version_beta1.patch, #36841
    2016-04-04 mtmiller Severity3 - Normal 1 - Wish
        Priority5 - Normal 2
        Release3.8.1 dev
        SummaryBLAS and LAPACK info via version command version function: options and return values supported by Matlab

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code