bugGNU Octave - Bugs: bug #43101, [MXE] Mingw Octave --enable64: ...

 
 

bug #43101: [MXE] Mingw Octave --enable64: HDF5 error mesages on startup

Submitted by:  Philip Nienhuis <philipnienhuis>
Submitted on:  Thu 28 Aug 2014 02:57:31 PM UTC  
 
Category: LibrariesSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Other
Status: ConfirmedAssigned to: None
Originator Name: Philip NienhuisOpen/Closed: Open
Release: 3.8.2Operating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun 05 Oct 2014 02:50:15 PM UTC, comment #18:

I checked in the following changeset. It seems to fix the segfault in the sparse matrix tests.

http://hg.savannah.gnu.org/hgweb/octave/rev/3ffb8f53e940

John W. Eaton <jwe>
Project Administrator
Thu 04 Sep 2014 05:58:49 PM UTC, comment #17:

I gave it a try but ran into some problems.

- built hdf5 with --enable-debug=all
- rebuilt octave with --enable64 --enable-bounds-check
as extra configure flags;

- tried to run it in a mingw shell with gdb. But gdb tells me octave-cli.exe is not a valid win32 executable. So I need 64-bit mingw and gdb. I suppose that isn't too hard but it may be a while before I have time to dive in it further.
I did find out that -after re-loading the written temp file in the first hdf5 test in sparse.cc- it is the call to sparse(af) that provokes the crash.

As Octave can be built w/o hdf5 support I think that is a way out for me for the next weeks or so; I think (hope) I can assign more time/priority to this issue later on.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 02 Sep 2014 02:33:47 PM UTC, comment #16:

If you want to try to debug this problem, then build the gdb target in mxe-octave and then run Octave under gdb, run the test suite, and see where the crash happens.

John W. Eaton <jwe>
Project Administrator
Tue 02 Sep 2014 09:44:04 AM UTC, comment #15:

Good and bad news:

The good news is that the HDF5 error messages have gone. Thanks!

The bad news is that (after commenting out the last 4 tests in eig_cc.tst) _run_test_suite_.m now crashes hard in sparse.tst with a segmentation violation. Pic attached (fntests.log wasn't written out completely due to the crash so there's not much to see in it). (my system: win7 64bit w double xeon CPU).

This week I'll clean up the list of various other issues I found (+ some fixes) with 64-bit Octave and put it on the task tracker.

(file #32015)

Philip Nienhuis <philipnienhuis>
Project Member
Mon 01 Sep 2014 09:28:57 PM UTC, comment #14:

Thanks, fresh clone + rebuilding overnight (will check results tomorrow - if I can find time).

As to HDF5 file errors:
Comment #12 shows HDF5 write errors in sparse.tst. Would these be related to the HDF5 proper (unlikely if only long double was affected) or would you conclude these must be caused by other build issues?

Philip Nienhuis <philipnienhuis>
Project Member
Mon 01 Sep 2014 04:58:55 PM UTC, comment #13:

I checked in the following changeset that I think fixes the problem:

http://hg.octave.org/mxe-octave/rev/6ac813175f6f

As far as I can tell, the messages were only about long double values and Octave doesn't try to save or load values of those types, so that's probably why there didn't seem to be any real problem other than the noisy warnings.

John W. Eaton <jwe>
Project Administrator
Sun 31 Aug 2014 12:03:33 PM UTC, comment #12:

Combing through fntests.log (made by _run_test_suite_.m) I can verify that HDF5 doesn't work with --enable64 (cross-built).

AFAICS only sparse.tst contains tests for HDF5:

I have categorized the other test FAILures; most of it is tolerance-related (but maybe pointing to wrong casts); about 12 are Java-related but easily fixed. Several are skipped because QHULL isn't included in the build.
Shall I put it into an entry on the Task tracker ?

Philip Nienhuis <philipnienhuis>
Project Member
Sun 31 Aug 2014 08:33:40 AM UTC, comment #11:

Same with a fresh clone and build: the hdf5 messages still appear.

I have no idea whether the hdf5 functionality itself is affected. run_test_suite_ seems to run fine apart from ov_fcn_handle.cc and sparse.tst - and several tolerance & Java things I'll report separately (partly already done a few weeks ago).

Philip Nienhuis <philipnienhuis>
Project Member
Sat 30 Aug 2014 11:28:38 AM UTC, comment #10:

Applying the patch didn't help here, still the same behavior.

I deleted installed_packages/stable-octave & -hdf5 to force mxe-octave to rebuild both. Should there be other packages that need to be rebuilt?

Just to be sure I'll do a completely new clone & build with the patch.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 30 Aug 2014 10:41:47 AM UTC, comment #9:

Hmmm, I didn't see these HDF5 messages when I loaded Matlab-generated .mat files - some of them quite large (the very reason I started using the 64-bit MinGW Octave version).

So H5Fis_hdf5() doesn't seem to be called for each "load" command. But perhaps the format of those .mat files may be immediately clear, so that function H5Fis_hdf5() wouldn't have to be called for them. Does that assumption of me make sense?

Anyway, building now...

Philip Nienhuis <philipnienhuis>
Project Member
Sat 30 Aug 2014 06:10:18 AM UTC, comment #8:

Yes, loading the package cache file (or any data file loaded with the "load" function) causes Octave to determine the type of file that is being loaded. Early on in that process, the function H5Fis_hdf5 is called to check whether the file is an HDF5 file.

The HDF5 build system depends on a lot of tests that execute programs. That doesn't work when cross compiling, so we have to override the tests by setting cache variables on the configure command line. I remember working on this problem for the --enable-64 build, but maybe I didn't check in my changes?

I just found some old diffs that may have some clues. I've updated them and made them conditional on mxe-octave's --enable-64 build option. Do these changes fix the problem for you? If so, then go ahead and commit them, or let me know and I'll do it.

I don't remember all the details now, but I think I determined the values for these options by doing a native 64-bit build of hdf5 on a mingw system and then comparing the cache values with those used for a 32-bit mingw build.

(file #31999)

John W. Eaton <jwe>
Project Administrator
Sat 30 Aug 2014 12:03:43 AM UTC, comment #7:

I tried compiling the test program from mxe (attached for reference) and compiled as g++ hdf5-test.cpp -lhdf5_hl -lhdf5 -lz.

Compiling and running the program in mxe-octave 32 build - it appears to run ok.

Compiling and running using the gcc/hdf5 compiled with mxe-octave 64 bit, the same warning statements as seen in octave are displayed.

So it is not an octave issue but the build of hdf5 in mxe-octave for 64 bit.

(file #31998)

John Donoghue <lostbard>
Project Member
Fri 29 Aug 2014 10:39:36 PM UTC, comment #6:

Is it possible that loading a file is triggering HDF5 to spit out these error messages? The main thing that happens when OF packages are installed is the package cache is loaded on startup so that packages marked autoload:yes are loaded.

So does loading an ordinary file (with no format options) saved by Octave in the default save format show these errors also?

Mike Miller <mtmiller>
Project Administrator
Fri 29 Aug 2014 03:19:13 PM UTC, comment #5:

The only two tests where I see this are:

and lower:

Again, the intruiging bit (but perhaps a red herring) is that these errors pop up during start of Octave only when an OF package has bee installed.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 28 Aug 2014 08:35:57 PM UTC, comment #4:

If you execute _run_test_suite_, it has some tests of the load save code that check the various file formats including Octave's HDF5-based format.

John W. Eaton <jwe>
Project Administrator
Thu 28 Aug 2014 05:10:35 PM UTC, comment #3:

Thanks for the clarification.

How can I test if hdf5 really works in Octave?

Philip Nienhuis <philipnienhuis>
Project Member
Thu 28 Aug 2014 04:15:55 PM UTC, comment #2:

Octave doesn't know how to read .mat files created by Matlab that use HDF5. Octave can only read .mat files that use the earlier Matlab format that does not use HDF5.

HDF5 is used in Octave for a separate and incompatible type of files.

John W. Eaton <jwe>
Project Administrator
Thu 28 Aug 2014 04:08:43 PM UTC, comment #1:

I see the same on my 64 build

John Donoghue <lostbard>
Project Member
Thu 28 Aug 2014 02:57:31 PM UTC, original submission:

While experimenting with an 64-bit MinGW Octave, I found that it generally works well. Most OF packages install & seem to work fine.

But there 's an intruiging issue: Octave starts cleanly as long a there's no OF package installed. As soon as any OF package is installed (I tried quite a few), I get a long list of HDF5-related error messages:

OF packages need not be loaded at all to provoke these messages. Simply having one or more installed is sufficient.
After uninstalling all OF packages, the error messages disappear.

MXE configure flags: --enable-openblas --enable64

Loading Matlab-generated .mat files works properly; AFAIK hdf5 is used by default on the Matlab side.

Philip Nienhuis <philipnienhuis>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #32015:  sparse_err.png added by philipnienhuis (100KiB - image/png)
file #31999:  hdf5-enable-64-diffs added by jwe (2KiB - application/octet-stream)
file #31998:  hdf5-test.cpp added by lostbard (2KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by lostbard (Posted a comment)
  • -unavailable- added by philipnienhuis (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 02 Sep 2014 09:44:04 AM UTCphilipnienhuisAttached File-=>Added sparse_err.png, #32015
    Sat 30 Aug 2014 06:10:18 AM UTCjweAttached File-=>Added hdf5-enable-64-diffs, #31999
    Sat 30 Aug 2014 12:03:43 AM UTClostbardAttached File-=>Added hdf5-test.cpp, #31998
    Thu 28 Aug 2014 04:08:43 PM UTClostbardStatusNone=>Confirmed

    Back to the top


    Powered by Savane 3.1-cleanup1