bugGNU Octave - Bugs: bug #43101, [MXE] Mingw Octave --enable64: ...

 
 

bug #43101: [MXE] Mingw Octave --enable64: HDF5 error mesages on startup

Submitter:  Philip Nienhuis <philipnienhuis>
Submitted:  Thu 28 Aug 2014 02:57:31 PM UTC
   
 
Category:  Libraries Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Other
Status:  Fixed Assigned to:  None
Originator Name:  Philip Nienhuis Open/Closed:  * Closed
Release:  * 3.8.2 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 22 Apr 2018 08:20:32 PM UTC, comment #19: 

This has been fixed, already quite a while ago.
I didn't see it anymore with Octave-4.3.0+ when I started building mxe-octave with --enable-64 (somewhere late 2017)

Closing report.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 05 Oct 2014 02:50:15 PM UTC, comment #18: 

I checked in the following changeset.  It seems to fix the segfault in the sparse matrix tests.

http://hg.savannah.gnu.org/hgweb/octave/rev/3ffb8f53e940

John W. Eaton <jwe>
Group administrator
Thu 04 Sep 2014 05:58:49 PM UTC, comment #17: 

I gave it a try but ran into some problems.

- built hdf5 with --enable-debug=all
- rebuilt octave with --enable64 --enable-bounds-check
as extra configure flags;

- tried to run it in a mingw shell with gdb. But gdb tells me octave-cli.exe is not a valid win32 executable. So I need 64-bit mingw and gdb. I suppose that isn't too hard but it may be a while before I have time to dive in it further.
I did find out that -after re-loading the written temp file in the first hdf5 test in sparse.cc- it is the call to sparse(af) that provokes the crash.

As Octave can be built w/o hdf5 support I think that is a way out for me for the next weeks or so; I think (hope) I can assign more time/priority to this issue later on.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 02 Sep 2014 02:33:47 PM UTC, comment #16: 

If you want to try to debug this problem, then build the gdb target in mxe-octave and then run Octave under gdb, run the test suite, and see where the crash happens.

John W. Eaton <jwe>
Group administrator
Tue 02 Sep 2014 09:44:04 AM UTC, comment #15: 

Good and bad news:

The good news is that the HDF5 error messages have gone. Thanks!

The bad news is that (after commenting out the last 4 tests in eig_cc.tst) _run_test_suite_.m now crashes hard in sparse.tst with a segmentation violation. Pic attached (fntests.log wasn't written out completely due to the crash so there's not much to see in it). (my system: win7 64bit w double xeon CPU).

This week I'll clean up the list of various other issues I found (+ some fixes) with 64-bit Octave and put it on the task tracker.



Philip Nienhuis <philipnienhuis>
Group Member
Mon 01 Sep 2014 09:28:57 PM UTC, comment #14: 

Thanks, fresh clone + rebuilding overnight (will check results tomorrow - if I can find time).

As to HDF5 file errors:
Comment #12 shows HDF5 write errors in sparse.tst. Would these be related to the HDF5 proper (unlikely if only long double was affected) or would you conclude these must be caused by other build issues?

Philip Nienhuis <philipnienhuis>
Group Member
Mon 01 Sep 2014 04:58:55 PM UTC, comment #13: 

I checked in the following changeset that I think fixes the problem:

http://hg.octave.org/mxe-octave/rev/6ac813175f6f

As far as I can tell, the messages were only about long double values and Octave doesn't try to save or load values of those types, so that's probably why there didn't seem to be any real problem other than the noisy warnings.

John W. Eaton <jwe>
Group administrator
Sun 31 Aug 2014 12:03:33 PM UTC, comment #12: 

Combing through fntests.log (made by _run_test_suite_.m) I can verify that HDF5 doesn't work with --enable64 (cross-built).

AFAICS only sparse.tst contains tests for HDF5:


>>>>>>Processing sparse.tst
  ***** testif HAVE_HDF5   # save hdf5
 savefile = tmpnam ();
 as_save = as;
 save ("-hdf5", savefile, "bf", "as_save", "af");
 clear as_save;
 load (savefile, "as_save");
 unlink (savefile);
 assert (as_save, sparse (af));
!!!!! test failed
save: error while writing 'bf' to hdf5 file
  ***** testif HAVE_HDF5   # save hdf5
 savefile = tmpnam ();
 as_save = as;
 save ("-hdf5", savefile, "bf", "as_save", "af");
 clear as_save;
 load (savefile, "as_save");
 unlink (savefile);
 assert (as_save, sparse (af));
!!!!! test failed
save: error while writing 'bf' to hdf5 file


I have categorized the other test FAILures; most of it is tolerance-related (but maybe pointing to wrong casts); about 12 are Java-related but easily fixed. Several are skipped because QHULL isn't included in the build.
Shall I put it into an entry on the Task tracker ?

Philip Nienhuis <philipnienhuis>
Group Member
Sun 31 Aug 2014 08:33:40 AM UTC, comment #11: 

Same with a fresh clone and build: the hdf5 messages still appear.

I have no idea whether the hdf5 functionality itself is affected. run_test_suite_ seems to run fine apart from ov_fcn_handle.cc and sparse.tst - and several tolerance & Java things I'll report separately (partly already done a few weeks ago).

Philip Nienhuis <philipnienhuis>
Group Member
Sat 30 Aug 2014 11:28:38 AM UTC, comment #10: 

Applying the patch didn't help here, still the same behavior.

I deleted installed_packages/stable-octave & -hdf5 to force mxe-octave to rebuild both. Should there be other packages that need to be rebuilt?

Just to be sure I'll do a completely new clone & build with the patch.

Philip Nienhuis <philipnienhuis>
Group Member
Sat 30 Aug 2014 10:41:47 AM UTC, comment #9: 

Hmmm, I didn't see these HDF5 messages when I loaded Matlab-generated .mat files - some of them quite large (the very reason I started using the 64-bit MinGW Octave version).

So H5Fis_hdf5() doesn't seem to be called for each "load" command. But perhaps the format of those .mat files may be immediately clear, so that function H5Fis_hdf5() wouldn't have to be called for them. Does that assumption of me make sense?

Anyway, building now...

Philip Nienhuis <philipnienhuis>
Group Member
Sat 30 Aug 2014 06:10:18 AM UTC, comment #8: 

Yes, loading the package cache file (or any data file loaded with the "load" function) causes Octave to determine the type of file that is being loaded.  Early on in that process, the function H5Fis_hdf5 is called to check whether the file is an HDF5 file.

The HDF5 build system depends on a lot of tests that execute programs.  That doesn't work when cross compiling, so we have to override the tests by setting cache variables on the configure command line.  I remember working on this problem for the --enable-64 build, but maybe I didn't check in my changes?

I just found some old diffs that may have some clues.  I've updated them and made them conditional on mxe-octave's --enable-64 build option.  Do these changes fix the problem for you?  If so, then go ahead and commit them, or let me know and I'll do it.

I don't remember all the details now, but I think I determined the values for these options by doing a native 64-bit build of hdf5 on a mingw system and then comparing the cache values with those used for a 32-bit mingw build.

(file #31999)

John W. Eaton <jwe>
Group administrator
Sat 30 Aug 2014 12:03:43 AM UTC, comment #7: 

I tried compiling the test program from mxe (attached for reference) and compiled as g++ hdf5-test.cpp -lhdf5_hl -lhdf5 -lz.

Compiling and running the program in mxe-octave 32 build - it appears to run ok.

Compiling and running using the gcc/hdf5 compiled with mxe-octave 64 bit, the same warning statements as seen in octave are displayed.

So it is not an octave issue but the build of hdf5 in mxe-octave for 64 bit.




(file #31998)

John Donoghue <lostbard>
Group Member
Fri 29 Aug 2014 10:39:36 PM UTC, comment #6: 

Is it possible that loading a file is triggering HDF5 to spit out these error messages? The main thing that happens when OF packages are installed is the package cache is loaded on startup so that packages marked autoload:yes are loaded.

So does loading an ordinary file (with no format options) saved by Octave in the default save format show these errors also?

Mike Miller <mtmiller>
Group Member
Fri 29 Aug 2014 03:19:13 PM UTC, comment #5: 

The only two tests where I see this are:


  libinterp\octave-value\ov-fcn-handle.cc-tst ............HDF5-DIAG: Error detected in HDF5 (1.8.10)
 thread 0:
  #000: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_p
ath_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #001: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7615 in H
5T__conv_long_ldouble(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #002: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_r
egister(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
  #003: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_p
ath_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
<etc>
:


and lower:


  sparse.tst .............................................HDF5-DIAG: Error detected in HDF5 (1.8.10)
 thread 0:
  #000: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_p
ath_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #001: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7615 in H
5T__conv_long_ldouble(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #002: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_r
egister(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
<etc>
:


Again, the intruiging bit (but perhaps a red herring) is that these errors pop up during start of Octave only when an OF package has bee installed.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 28 Aug 2014 08:35:57 PM UTC, comment #4: 

If you execute _run_test_suite_, it has some tests of the load save code that check the various file formats including Octave's HDF5-based format.

John W. Eaton <jwe>
Group administrator
Thu 28 Aug 2014 05:10:35 PM UTC, comment #3: 

Thanks for the clarification.

How can I test if hdf5 really works in Octave?

Philip Nienhuis <philipnienhuis>
Group Member
Thu 28 Aug 2014 04:15:55 PM UTC, comment #2: 

Octave doesn't know how to read .mat files created by Matlab that use HDF5.  Octave can only read .mat files that use the earlier Matlab format that does not use HDF5.

HDF5 is used in Octave for a separate and incompatible type of files.

John W. Eaton <jwe>
Group administrator
Thu 28 Aug 2014 04:08:43 PM UTC, comment #1: 

I see the same on my 64 build

John Donoghue <lostbard>
Group Member
Thu 28 Aug 2014 02:57:31 PM UTC, original submission:  

While experimenting with an 64-bit MinGW Octave, I found that it generally works well. Most OF packages install & seem to work fine.

But there 's an intruiging issue: Octave starts cleanly as long a there's no OF package installed.  As soon as any OF package is installed (I tried quite a few), I get a long list of HDF5-related error messages:

Octave was configured for "x86_64-w64-mingw32".

Additional information about Octave is available at http://www.octave.org.

Please contribute if you find this software useful.
For more information, visit http://www.octave.org/get-involved.html

Read http://www.octave.org/bugs.html to learn how to submit bug reports.
For information about changes from previous versions, type 'news'.

HDF5-DIAG: Error detected in HDF5 (1.8.10) thread 0:
  #000: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #001: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7615 in H5T__conv_long_ldouble(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #002: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_register(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
  #003: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #004: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7538 in H5T__conv_uint_ldouble(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #005: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_register(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
  #006: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #007: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7461 in H5T__conv_int_ldouble(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #008: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_register(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
  #009: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #010: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7384 in H5T__conv_ushort_ldouble(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #011: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_register(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
  #012: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #013: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7307 in H5T__conv_short_ldouble(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #014: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_register(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
  #015: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #016: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7230 in H5T__conv_uchar_ldouble(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #017: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_register(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
  #018: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #019: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7153 in H5T__conv_schar_ldouble(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #020: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_register(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
  #021: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #022: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7076 in H5T__conv_ldouble_double(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #023: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_register(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
  #024: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #025: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7049 in H5T__conv_ldouble_float(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #026: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_register(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
  #027: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #028: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 7022 in H5T__conv_double_ldouble(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #029: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 2333 in H5T_register(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
  #030: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5T.c line 4484 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #031: /home/philip/devel/octdev/mxe/mxe201408_64/tmp-hdf5/hdf5-1.8.10/src/H5Tconv.c line 6961 in H5T__conv_float_ldouble(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
>>


OF packages need not be loaded at all to provoke these messages. Simply having one or more installed is sufficient.
After uninstalling all OF packages, the error messages disappear.

MXE configure flags: --enable-openblas --enable64

Loading Matlab-generated .mat files works properly; AFAIK hdf5 is used by default on the Matlab side.

Philip Nienhuis <philipnienhuis>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #32015:  sparse_err.png added by philipnienhuis (100KiB - image/png)
file #31999:  hdf5-enable-64-diffs added by jwe (2KiB - application/octet-stream)
file #31998:  hdf5-test.cpp added by lostbard (2KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by philipnienhuis (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-04-22 philipnienhuis StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2014-09-02 philipnienhuis Attached File- Added sparse_err.png, #32015
    2014-08-30 jwe Attached File- Added hdf5-enable-64-diffs, #31999
    2014-08-30 lostbard Attached File- Added hdf5-test.cpp, #31998
    2014-08-28 lostbard StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code