patchGNU Octave - Patches: patch #8607, New module h5read to read/write...

 
 

patch #8607: New module h5read to read/write HDF5 files

Submitter:  None
Submitted:  Thu 19 Feb 2015 01:03:15 PM UTC
Votes: 87
 
Category:  Core : new feature Priority:  5 - Normal
Status:  In Progress Privacy:  Public
Assigned to:  None Originator Email:  -email is unavailable-
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 27 May 2024 07:23:17 PM UTC, comment #33: 

apparently the patch is not consistent with Matlab. If I execute their example from https://de.mathworks.com/help/matlab/ref/h5write.html

h5create("myfile.h5","/DS1",[10 20])
mydata = rand(10, 20);
h5write("myfile.h5","/DS1",mydata)

I get

HDF5-DIAG: Error detected in HDF5 (1.14.4-2) thread 0:
  #000: H5S.c line 1310 in H5Screate_simple(): dimensionality is too large
    major: Invalid arguments to routine
    minor: Bad value

If I use a larger dataset

h5create("myfile.h5","/DS1",[10 20 5 1000])
mydata = rand(10, 20, 5, 1000);
h5write("myfile.h5","/DS1",mydata)

I get a segfault.

Muhali <muhali>
Mon 17 Jul 2023 03:06:09 AM UTC, comment #32: 

There is a GitHub repository that aims to provide an interface to the HDF5 library for GNU Octave. Link: https://github.com/pantxo/oct-hdf5 This is built as an Octave package.

Incomplete documentation link: https://pantxo.github.io/oct-hdf5/

I have not gone through the patches. Just informing about similar work.

Also see: https://octave.discourse.group/t/new-feature-hdf5-file-i-o/4323

Vipul Cariappa <vipul_cariappa>
Sun 16 Jul 2023 07:29:03 PM UTC, comment #31: 

I refreshed the patch mostly to get it to compile again in current versions of Octave.
That most importantly includes changes because `error` or `print_usage` no longer return and the `error_state` variable no longer exists.
I also added a new section for these functions to the manual and fixed some syntax errors in the texinfo of the new functions.
I started to adapt the coding style to the current rules. But that's far from finished yet. E.g., error messages don't yet start consistently with the name of the failing function.

I didn't yet move the functions to the core library. Neither did I check how we could share some of the new functions with the existing functions that save in HDF5 format (for `save -hdf5 ...`).
But that and any further cleanup could also be done in follow-up changes.

For the time being, I'm attaching the updated patch.

(file #54932)

Markus Mützel <mmuetzel>
Group administrator
Sun 16 Jul 2023 12:25:22 PM UTC, comment #30: 

It is a pity that this is hanging here without much attention for so long now (about 8 years).

I just had a very short look at the patch. I didn't check if it still applies.
If I understand correctly, it is/was already quite complete when it comes to implementing functions that are likely compatible to Matlab.

However, it adds those HDF5 functions in a new .oct file. Octave core already links to libhdf5, and as jwe mentioned pretty early on, it would be nice if the backbone of this implementation could be shared with the functions that load and save Octave values.
Because of that, it would make sense imho to move these functions to liboctinterp.

Is anyone still working on this?

Markus Mützel <mmuetzel>
Group administrator
Tue 14 Jun 2022 09:00:56 PM UTC, comment #29: 

Yeah, the Octave interface from Nabble really "died". www.archvive,org still has it, see
https://web.archive.org/web/20190925010150/http://octave.1599824.n4.nabble.com/hdf5-hdf5oct-tt4690108.html
... and otherwise search in the lists.gnu.org help-octave archive:
https://lists.gnu.org/archive/html/help-octave/2018-11/index.html

As to hdf5 itself, it seems the discussion went into coma again.
Too bad, hdf5 is a very useful and widely used file format and IMO it should be supported by Octave.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 14 Jun 2022 04:21:56 PM UTC, comment #28: 

Did this just die?  The nabble link is broken for me.

Orion Poplawski <opoplawski>
Sun 18 Nov 2018 09:46:32 AM UTC, comment #27: 
Philip Nienhuis <philipnienhuis>
Group Member
Tue 17 Jul 2018 07:01:23 PM UTC, comment #26: 

Attached is a patch relative to the default branch at July 18, hg id d1216d085280


(file #44573)

Philip Nienhuis <philipnienhuis>
Group Member
Wed 21 Feb 2018 06:03:23 PM UTC, comment #25: 

Sure, but I'm not too fluent in C++ so other devs need to pick up rather than me.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 21 Feb 2018 08:49:16 AM UTC, comment #24: 

I'm more and more tempted to move to python/julia,
that seem better suited for research now.
Since you seem to be interested by this patch, please take it from here.

ederag <ederag>
Tue 20 Feb 2018 09:33:00 PM UTC, comment #23: 

I guess you're right and it should be added to config-module.awk instead, thanks for noting.

Anyway, I have tried to use h5read for loading Matlab 7.3 .mat files. I works to some extent but I think the issue there is that saving/loading classdef stuff still needs adapting code in Octave's internals.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 19 Feb 2018 10:25:09 AM UTC, comment #22: 

At the top of libinterp/dldfcn/module.mk, there is

## DO NOT EDIT -- generated from module-files by config-module.awk


Is libinterp/dldfcn/module.mk really to be edited ?

Probably, since oct-qhull.h is present in module.mk, but not in module-files.

ederag <ederag>
Fri 05 Jan 2018 09:36:20 PM UTC, comment #21: 

Thanks ederag for updating the patch, I'll try it out.

A missing thing is an entry in libinterp/dldfcn/module.mk to include h5read.h in a dist archive. Put it after oct-qhull.h in the top stanza (EXTRA_DIST_FILES IIRC).
This is required to be able to cross-build it, mxe-octave needs a complete dist archive.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 03 Jan 2018 08:53:29 PM UTC, comment #20: 

With the updated patch attached, it seems to work again:

>> m = rand(2, 3)
m =

   0.94236   0.96434   0.71092
   0.23191   0.50169   0.92243

>> h5create("try_w.hdf5", "/m", size(m))
>> h5write("try_w.hdf5", "/m", m)
>> h5read("try_w.hdf5", "/m")
ans =

   0.94236   0.96434   0.71092
   0.23191   0.50169   0.92243




(file #42815)

ederag <ederag>
Wed 03 Jan 2018 01:15:09 PM UTC, comment #19: 

(file #42802) is an updated patch, from Philips one (comment #18).
Most warnings and errors have been removed.
Many octave_value_list have been changed to octave_value,
in order to keep the original return type.
This seemed safer, but to be checked.

On error remain:

In file included from ./config.h:3434:0,
                 from ../libinterp/dldfcn/h5read.h:25,
                 from ../libinterp/dldfcn/h5read.cc:43:
./oct-conf-post.h:76:1: error: redefinition of 'template<class T> void octave_unused_parameter(const T&)'
 octave_unused_parameter (const T&)
 ^~~~~~~~~~~~~~~~~~~~~~~
In file included from ./config.h:3434:0,
                 from ../libinterp/dldfcn/h5read.cc:25:
./oct-conf-post.h:76:1: note: 'template<class T> void octave_unused_parameter(const T&)' previously declared here
 octave_unused_parameter (const T&)
 ^~~~~~~~~~~~~~~~~~~~~~~


ederag <ederag>
Mon 05 Dec 2016 07:40:48 PM UTC, comment #18: 

I've updated the patch so that it applies cleanly to 4.3.0+ bu the build breaks at h5read:

:
  CXX      libinterp/dldfcn/libinterp_dldfcn_h5read_la-h5read.lo
In file included from ./config.h:3541:0,
                 from ../dev_VE/libinterp/dldfcn/h5read.h:25,
                 from ../dev_VE/libinterp/dldfcn/h5read.cc:43:
./oct-conf-post.h:58:1: error: redefinition of 'template<class T> void octave_unused_parameter(const T&)'
 octave_unused_parameter (const T&)
 ^
In file included from ./config.h:3541:0,
                 from ../dev_VE/libinterp/dldfcn/h5read.cc:25:
./oct-conf-post.h:58:1: note: 'template<class T> void octave_unused_parameter(const T&)' previously declared here
 octave_unused_parameter (const T&)
 ^
../dev_VE/libinterp/dldfcn/h5read.cc: In function 'bool any_int_leq_zero(const Matrix&)':
../dev_VE/libinterp/dldfcn/h5read.cc:50:35: warning: 'octave_idx_type Array<T>::length() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:354): use 'numel' instead [-Wdeprecated-declarations]
   for (int i = 0; i < mat.length (); i++)
                                   ^
../dev_VE/libinterp/dldfcn/h5read.cc: In function 'int check_vec(const octave_value&, Matrix&, const char*, bool)':
../dev_VE/libinterp/dldfcn/h5read.cc:77:38: warning: 'octave_idx_type Array<T>::nelem() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:360): use 'numel' instead [-Wdeprecated-declarations]
       for (int i = 0; i < mat.nelem (); i++)
                                      ^
In file included from ../dev_VE/libinterp/corefcn/defun-dld.h:32:0,
                 from ../dev_VE/libinterp/corefcn/oct.h:32,
                 from ../dev_VE/libinterp/dldfcn/h5read.cc:28:
../dev_VE/libinterp/dldfcn/h5read.cc: At global scope:
../dev_VE/libinterp/dldfcn/h5read.cc:200:29: warning: unused parameter 'nargout' [-Wunused-parameter]
 DEFUN_DLD (h5readatt, args, nargout,
                             ^
../dev_VE/libinterp/corefcn/defun-int.h:92:49: note: in definition of macro 'DECLARE_FUNX'
   name (const octave_value_list& args_name, int nargout_name)
                                                 ^
../dev_VE/libinterp/corefcn/defun-dld.h:48:3: note: in expansion of macro 'DECLARE_FUN'
   DECLARE_FUN (name, args_name, nargout_name)
   ^
../dev_VE/libinterp/dldfcn/h5read.cc:200:1: note: in expansion of macro 'DEFUN_DLD'
 DEFUN_DLD (h5readatt, args, nargout,
 ^
../dev_VE/libinterp/dldfcn/h5read.cc: In constructor 'H5File::H5File(const char*, bool)':
../dev_VE/libinterp/dldfcn/h5read.cc:509:25: warning: 'file_stat' is deprecated (declared at ../dev_VE/liboctave/system/file-stat.h:332): use 'octave::sys::file_stat' instead [-Wdeprecated-declarations]
   file_stat fs (filename);
                         ^
../dev_VE/libinterp/dldfcn/h5read.cc: In member function 'hsize_t* H5File::alloc_hsize(const T&, int, bool)':
../dev_VE/libinterp/dldfcn/h5read.cc:575:61: warning: use of old-style cast [-Wold-style-cast]
   hsize_t *hsize = (hsize_t*)malloc (rank * sizeof (hsize_t));
                                                             ^
../dev_VE/libinterp/dldfcn/h5read.cc: In member function 'int H5File::open_dset(const char*)':
../dev_VE/libinterp/dldfcn/h5read.cc:614:53: warning: use of old-style cast [-Wold-style-cast]
   h5_dims = (hsize_t*)malloc (rank * sizeof(hsize_t));
                                                     ^
../dev_VE/libinterp/dldfcn/h5read.cc:615:56: warning: use of old-style cast [-Wold-style-cast]
   h5_maxdims = (hsize_t*)malloc (rank * sizeof(hsize_t));
                                                        ^
../dev_VE/libinterp/dldfcn/h5read.cc: In member function 'octave_value H5File::read_dset_hyperslab(const char*, const Matrix&, const Matrix&, const Matrix&, const Matrix&, int)':
../dev_VE/libinterp/dldfcn/h5read.cc:670:20: warning: 'octave_idx_type Array<T>::nelem() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:360): use 'numel' instead [-Wdeprecated-declarations]
   if (start.nelem () != rank)
                    ^
../dev_VE/libinterp/dldfcn/h5read.cc:675:20: warning: 'octave_idx_type Array<T>::nelem() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:360): use 'numel' instead [-Wdeprecated-declarations]
   if (count.nelem () != rank)
                    ^
../dev_VE/libinterp/dldfcn/h5read.cc:684:22: warning: 'octave_idx_type Array<T>::nelem() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:360): use 'numel' instead [-Wdeprecated-declarations]
   if (_stride.nelem () != rank)
                      ^
../dev_VE/libinterp/dldfcn/h5read.cc:692:21: warning: 'octave_idx_type Array<T>::nelem() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:360): use 'numel' instead [-Wdeprecated-declarations]
   if (_block.nelem () != rank)
                     ^
../dev_VE/libinterp/dldfcn/h5read.cc:709:37: warning: use of old-style cast [-Wold-style-cast]
                  i+1, (int)_stride(i), (int)_block(i));
                                     ^
../dev_VE/libinterp/dldfcn/h5read.cc:709:53: warning: use of old-style cast [-Wold-style-cast]
                  i+1, (int)_stride(i), (int)_block(i));
                                                     ^
../dev_VE/libinterp/dldfcn/h5read.cc:720:29: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       if (h5_dims[rank-i-1] < end)
                             ^
../dev_VE/libinterp/dldfcn/h5read.cc:723:85: warning: use of old-style cast [-Wold-style-cast]
                  " are required for requested hyperslab", i+1, (int)h5_dims[rank-i-1],
                                                                                     ^
../dev_VE/libinterp/dldfcn/h5read.cc: In member function 'octave_value H5File::read_dset()':
../dev_VE/libinterp/dldfcn/h5read.cc:753:8: warning: unused variable 'is_cmplx' [-Wunused-variable]
   bool is_cmplx = false;
        ^
../dev_VE/libinterp/dldfcn/h5read.cc: In member function 'void H5File::write_dset(const char*, octave_value)':
../dev_VE/libinterp/dldfcn/h5read.cc:898:19: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for (int i=1; i < path.length (); i++)
                   ^
../dev_VE/libinterp/dldfcn/h5read.cc: In member function 'void H5File::write_dset_hyperslab(const char*, octave_value, const Matrix&, const Matrix&, const Matrix&, const Matrix&, int)':
../dev_VE/libinterp/dldfcn/h5read.cc:1037:20: warning: 'octave_idx_type Array<T>::nelem() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:360): use 'numel' instead [-Wdeprecated-declarations]
   if (start.nelem () != rank)
                    ^
../dev_VE/libinterp/dldfcn/h5read.cc:1042:20: warning: 'octave_idx_type Array<T>::nelem() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:360): use 'numel' instead [-Wdeprecated-declarations]
   if (count.nelem () != rank)
                    ^
../dev_VE/libinterp/dldfcn/h5read.cc:1050:22: warning: 'octave_idx_type Array<T>::nelem() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:360): use 'numel' instead [-Wdeprecated-declarations]
   if (_stride.nelem () != rank)
                      ^
../dev_VE/libinterp/dldfcn/h5read.cc:1058:21: warning: 'octave_idx_type Array<T>::nelem() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:360): use 'numel' instead [-Wdeprecated-declarations]
   if (_block.nelem () != rank)
                     ^
../dev_VE/libinterp/dldfcn/h5read.cc:1071:37: warning: use of old-style cast [-Wold-style-cast]
                  i+1, (int)_stride(i), (int)_block(i));
                                     ^
../dev_VE/libinterp/dldfcn/h5read.cc:1071:53: warning: use of old-style cast [-Wold-style-cast]
                  i+1, (int)_stride(i), (int)_block(i));
                                                     ^
../dev_VE/libinterp/dldfcn/h5read.cc:1078:32: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       if (h5_maxdims[rank-i-1] < end)
                                ^
../dev_VE/libinterp/dldfcn/h5read.cc:1082:57: warning: use of old-style cast [-Wold-style-cast]
                  i+1, dsetname, (int)h5_maxdims[rank-i-1], end);
                                                         ^
../dev_VE/libinterp/dldfcn/h5read.cc:1089:29: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       if (h5_dims[rank-i-1] < end)
                             ^
../dev_VE/libinterp/dldfcn/h5read.cc: In member function 'octave_value H5File::read_att(const char*, const char*)':
../dev_VE/libinterp/dldfcn/h5read.cc:1188:56: warning: use of old-style cast [-Wold-style-cast]
       char* buf = (char*)calloc (totsize, sizeof (char));
                                                        ^
../dev_VE/libinterp/dldfcn/h5read.cc: In member function 'void H5File::write_att(const char*, const char*, const octave_value&)':
../dev_VE/libinterp/dldfcn/h5read.cc:1330:54: warning: use of old-style cast [-Wold-style-cast]
       buf = (void *) attvalue.string_value ().c_str ();
                                                      ^
../dev_VE/libinterp/dldfcn/h5read.cc:1330:54: warning: cast from type 'const char*' to type 'void*' casts away qualifiers [-Wcast-qual]
../dev_VE/libinterp/dldfcn/h5read.cc:1338:23: warning: use of old-style cast [-Wold-style-cast]
       buf = (void *) &attval_int;
                       ^
../dev_VE/libinterp/dldfcn/h5read.cc:1345:23: warning: use of old-style cast [-Wold-style-cast]
       buf = (void *) &attval_double;
                       ^
../dev_VE/libinterp/dldfcn/h5read.cc:1272:12: warning: unused variable 'dims' [-Wunused-variable]
   hsize_t *dims;
            ^
../dev_VE/libinterp/dldfcn/h5read.cc: In member function 'void H5File::create_dset(const char*, const Matrix&, const char*, const Matrix&)':
../dev_VE/libinterp/dldfcn/h5read.cc:1409:45: warning: 'octave_idx_type Array<T>::nelem() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:360): use 'numel' instead [-Wdeprecated-declarations]
   dspace_id = H5Screate_simple (size.nelem (), dims, maxdims);
                                             ^
../dev_VE/libinterp/dldfcn/h5read.cc:1429:47: warning: 'octave_idx_type Array<T>::nelem() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:360): use 'numel' instead [-Wdeprecated-declarations]
       if (H5Pset_chunk (crp_list, size.nelem (), dims_chunk) < 0)
                                               ^
../dev_VE/libinterp/dldfcn/h5read.cc: In instantiation of 'hsize_t* H5File::alloc_hsize(const T&, int, bool) [with T = Matrix; hsize_t = long long unsigned int]':
../dev_VE/libinterp/dldfcn/h5read.cc:729:66:   required from here
../dev_VE/libinterp/dldfcn/h5read.cc:574:26: warning: 'octave_idx_type Array<T>::length() const [with T = double; octave_idx_type = int]' is deprecated (declared at ../dev_VE/liboctave/array/Array.h:354): use 'numel' instead [-Wdeprecated-declarations]
   int rank = dim.length ();
                          ^
Makefile:18461: recipe for target 'libinterp/dldfcn/libinterp_dldfcn_h5read_la-h5read.lo' failed
make[2]: *** [libinterp/dldfcn/libinterp_dldfcn_h5read_la-h5read.lo] Error 1


Updated patch attached

(file #39154)

Philip Nienhuis <philipnienhuis>
Group Member
Wed 06 Jul 2016 02:09:56 PM UTC, comment #17: 

Carnë, do you think you can review this in time for 4.2, as you suggested in comment #15?

Lachlan Andrew <lachlan>
Fri 20 Nov 2015 04:13:15 AM UTC, comment #16: 

This is a key pain point for scientific and engineering users. HDF5 is a very popular format, growing in popularity.

This hdf5 package should be as high a priority as having imread() working.

-*- <mhirsch>
Mon 01 Jun 2015 06:30:30 PM UTC, comment #15: 

I remember this. We spoke about it on the IRC channel and I said I would review it. But we froze for octave 4.0.0 sooner than I expected and the first release candidate was released on March 5th.

This will have to go into 4.2.

Carnë Draug <carandraug>
Group Member
Mon 01 Jun 2015 02:55:41 PM UTC, comment #14: 

Stefan, thanks for your contribution in providing this patch and helping to test and move this forward. It is definitely appreciated and will hopefully become part of Octave core in a future version.

Just be aware that when this patch was first posted we were weeks away from the first release candidate for version 4.0, and it was therefore highly unlikely to be included in version 4.0, it has more to do with timing than lack of interest.

Mike Miller <mtmiller>
Group Member
Mon 01 Jun 2015 02:02:05 PM UTC, comment #13: 

Over three months ago this patch for a very often demanded feature was posted - is there really nobody who is interested in introducing it into core? I saw that version 4.0 of Octave was released - which is great, but I think it is really a pity that no efforts were made to include this patch for HDF5 support (my first one). This is an important feature if Octave should be of any use for analysis of serious amounts of scientific data (no, we do not write Gigabytes of ascii and binary without metadata). So many codes out there use HDF5 already for years but Octave still cannot read it...

The code is structured, the Octave coding style is adopted, it is Matlab compatible, there are complete help messages, there are no additional dependencies - what else is necessary?

Stefan Großhauser <stegrobt>
Fri 17 Apr 2015 11:36:06 AM UTC, comment #12: 

Hi,

> I haven't been able to read/write hdf5 files yet.

Did it work in the meantime?

>I did hit an issue but I'm not sure it is related:
>With your patch I cannot install the OF communications package >anymore, it complains about undeclared HID_t. That is somewhere >in the hdf5 include tree.


It would be helpful to have more details, if this problem was not just spurious, but is really related.

  • Do you install with `pkg install -forge communications signal` ?
  • does the same problem arise, if compiled without the patch but with the normal hdf5 support?



Stefan Großhauser <stegrobt>
Sat 21 Mar 2015 11:27:09 AM UTC, comment #11: 

I've tried your patch on Windows. It compiles fine, but I haven't been able to read/write hdf5 files yet.

My aim (hope) is to be able to read Matlab's newest .mat file; hopefully this is a way to implement classdef object I/O to .mat files (my ML-addicted colleagues told me those objects are written as some sort of structs)

I did hit an issue but I'm not sure it is related:
With your patch I cannot install the OF communications package anymore, it complains about undeclared HID_t. That is somewhere in the hdf5 include tree.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 04 Mar 2015 02:46:24 PM UTC, comment #10: 

after a few minors the issue reported in comment #9 does not appear any more. I don't know exactly why, but it seems to behave well now.

Hence, I won't change anything in the patch for the next days and weeks, unless someone discovers a bug. Concerning the integration with existing hdf5 routines in octave, my opinion is (because at the moment I am trying to make it work both as a package and as a octave-core file) that one should add the patch first with separate mechanics, and subsequently simplify (because then I could drop the pkg) - but I'm not a maintainer.

cheers

(file #33245)

Stefan Großhauser <stegrobt>
Tue 03 Mar 2015 07:45:04 AM UTC, comment #9: 

% In its present form, reading certain large chunked datasets
% consumes a tremendous amount of memory while it is as fast as
% it should be if the dset is not chunked, or if the load()
% function is used.
% This has to be fixed quickly, but I don't understand it yet,
% I don't see much of a difference in what they both do.
%
% Maybe somebody else can confirm and/or comment?
%
pkg load hdf5oct;

%a = rand(230,40,40,70); % let's call that x)
a = rand(230,43,43,71);  % y)
s = size(a)
s(1)=1
h5create("testchunk.h5","/foo_chunked",size(a),'ChunkSize',s)
h5write("testchunk.h5","/foo_chunked",a)
h5create("testchunk.h5","/foo_unchunked",size(a))
h5write("testchunk.h5","/foo_unchunked",a)

disp("load:")
%str = load("-hdf5","testchunk.h5"); %okay for both x) and y)
%clear str;
disp("read unchunked:")
%b = h5read("testchunk.h5","/foo_unchunked"); %okay for both x) and y)
disp("read chunked:")
b = h5read("testchunk.h5","/foo_chunked"); %okay for x), catastrophic for y)

Stefan Großhauser <stegrobt>
Tue 24 Feb 2015 04:30:41 PM UTC, comment #8: 

Removed a permute statement which consumed a lot of memory and made the behaviour differ from load('-hdf5',..) and matlab.
The current patch also uses two minor utility functions from ls-hdf5 , but no others (yet).

(file #33172)

Stefan Großhauser <stegrobt>
Mon 23 Feb 2015 06:09:13 PM UTC, comment #7: 

I haven't looked at the patch, but if it provides a general way to read and write HDF5 files, then it might be useful to build the Octave load/save functions using the new functions.

John W. Eaton <jwe>
Group administrator
Mon 23 Feb 2015 06:05:29 PM UTC, comment #6: 

Are you saying that we should improve our current ls-hdf5 using the code from this patch? Or that our ls-hdf5 is really meant to be used by our load/save functions so we should leave the code from this patch in a separate file?

Carnë Draug <carandraug>
Group Member
Mon 23 Feb 2015 02:19:17 PM UTC, comment #5: 

The use of HDF5 in Octave's load function is really only intended as a way to be able to read what Octave's save function produces when writing HDF5 files.  It's not intended as a general HDF5 file reader.

It would be better to have a set of general HDF5 file reading functions that could then be used to implement the save/load functionality that we need.

John W. Eaton <jwe>
Group administrator
Mon 23 Feb 2015 01:16:45 PM UTC, comment #4: 

@3: correct me if I am wrong, but the code there implements

load('-hdf5', ...)

which in fact  would profit of certain improvements:

  • It can only read a whole file or whole first level groups, like '/grid', but not '/data/foo/bar' .
  • It crashes if it encounters string datasets, instead of just ignoring them


But I think that this is another story and would require a separate patch. And, one cannot append to existing data or read/write attributes with those octave functions.

This patch here attempts to imitate http://de.mathworks.com/help/matlab/ref/h5write.html and the like.

Or what do you mean with "improvements that this patch requires to read the HDF5 files"?

I think some of the small helper functions could maybe be reused, like hdf5_types_compatible(), that might indeed be good idea.

Anonymous
Mon 23 Feb 2015 11:32:12 AM UTC, comment #3: 

Seems like Octave already has an interface for HDF5 (see ls-hdf5.h at
http://hg.savannah.gnu.org/hgweb/octave/file/290adbbe95bd/libinterp/corefcn/ls-hdf5.h )

Should this file be using it (or part of it)? Also, I will guess that the improvements that this patch requires to read the HDF5 files should be done there as well.

Carnë Draug <carandraug>
Group Member
Fri 20 Feb 2015 04:25:07 PM UTC, comment #2: 

Two fixes made a new patch necessary.

Can somebody compare this to Matlab's behaviour, please? In particular: whether the data order is correct, and especially if vectors come out the same. I don't have a very new version of Matlab.

(file #33140)

Anonymous
Fri 20 Feb 2015 09:15:26 AM UTC, comment #1: 

@License: The original author Tom Mullins approves that his hdf5oct code is copied into Octave under the GPL,
https://github.com/tmullins/hdf5oct/issues/2#issuecomment-75152332

Anonymous
Thu 19 Feb 2015 01:03:15 PM UTC, original submission:  

Hello everyone,

I would like to propose the integration some HDF5 related functions into octave core, maybe even for the new 4.0?

This patch defines the functions h5read, h5readatt, h5write, h5writeatt, h5create, see help strings.
This patch is equivalent to

https://github.com/stegro/hdf5oct/releases/tag/0.4.0

and should already integrate into Octave's build, if I did no mistake.
Finally, I was told to put everything into one rather long file, hope that's the way you want it.

NOTE: The original author of hdf5oct published his h5read function under the LGPL (I have not contacted him). Thorsten Liebig (openEMS) published his h5readatt under the GPL but agrees to any license, if only the stuff makes it into Octave, he says. I changed them both a lot and added new features. Is it necessary to/should one change the hdf5oct license to GPL in order to integrate this patch?

have fun, and thanks again for the hints on #octave

Stefan

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #54932:  patch8607-h5read-dldfcn-20230716.patch added by mmuetzel (47KiB - application/octet-stream)
file #44573:  patch_8607_h5read_20180718.cset added by philipnienhuis (50KiB - application/octet-stream)
file #42815:  patch_8607_h5read_18103ba.patch added by ederag (49KiB - text/x-patch - patch update, works again (against cs 24498:1c96b44feb7a))
file #42802:  patch_8607_h5read_18103aa.patch added by ederag (50KiB - text/x-patch - update of the patch, removing most errors and warnings.)
file #39154:  patch_8607_h5read_20161205.patch added by philipnienhuis (50KiB - application/x-download)
file #33245:  h5read.anothercheck.patch added by stegrobt (50KiB - text/x-patch)
file #33172:  h5read.nopermute.patch added by stegrobt (48KiB - text/x-patch)
file #33140:  h5read.fixnxm.patch added by None (49KiB - text/x-patch - fix for nxm matrices, better check for complex type)
file #33135:  h5read.withouttabs.patch added by None (48KiB - text/x-patch - ok, finally. I better not touch a keybord any more today.)
file #33133:  h5read.withouttabs.patch added by None (49KiB - text/x-patch - some tabs still survived... ^^)
file #33132:  h5read.betterstyle.patch added by None (48KiB - text/x-patch - a quick review produced h5read.betterstyle.patch)
file #33129:  h5read.patch added by None (45KiB - text/x-patch)

 

Depends on the following items: None found

Digest:
   bug dependencies, patch dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by vipul_cariappa (Posted a comment)
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by opoplawski (Posted a comment)
  • -email is unavailable- added by saturn
  • -email is unavailable- added by ederag (Updated the item)
  • -email is unavailable- added by lachlan (Posted a comment)
  • -email is unavailable- added by mhirsch (Posted a comment)
  • -email is unavailable- added by mhirsch (Voted in favor of this item)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by stegrobt (Voted in favor of this item)
  • -email is unavailable- added by ederag (Voted in favor of this item)
  • -email is unavailable- added by stegrobt (Updated the item)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 87 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 21 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-07-17 mmuetzel Dependencies- patch #10323 is dependent
    2023-07-16 philipnienhuis Carbon-Copy- Added philipnienhuis
    2023-07-16 mmuetzel Attached File- Added patch8607-h5read-dldfcn-20230716.patch, #54932
        StatusNone In Progress
    2018-11-18 philipnienhuis CategoryNone Core : new feature
    2018-07-17 philipnienhuis Attached File- Added patch_8607_h5read_20180718.cset, #44573
    2018-05-29 saturn Carbon-Copy- Added saturn
    2018-01-03 ederag Attached File- Added patch_8607_h5read_18103ba.patch, #42815
    2018-01-03 ederag Attached File- Added patch_8607_h5read_18103aa.patch, #42802
    2016-12-05 philipnienhuis Attached File- Added patch_8607_h5read_20161205.patch, #39154
    2016-11-18 mtmiller Dependencies- bugs #44153 is dependent
    2015-11-20 mhirsch Carbon-Copy- Added mhirsch
    2015-06-01 stegrobt Carbon-Copy- Added stegrobt
    2015-03-21 ederag Carbon-Copy- Added ederag
    2015-03-04 stegrobt Attached File- Added h5read.anothercheck.patch, #33245
    2015-02-24 stegrobt Attached File- Added h5read.nopermute.patch, #33172
    2015-02-20 None Attached File- Added h5read.fixnxm.patch, #33140
    2015-02-19 None Attached File- Added h5read.withouttabs.patch, #33135
    2015-02-19 None Attached File- Added h5read.withouttabs.patch, #33133
    2015-02-19 None Attached File- Added h5read.betterstyle.patch, #33132
    2015-02-19 None Attached File- Added h5read.patch, #33129

    Back to the top

    Powered by Savane 3.14-3b9d.
    Corresponding source code