bugGNU Octave - Bugs: bug #45687, fread does not allow different...

 
 

bug #45687: fread does not allow different ARCH when file fopen'd with ARCH

Submitter:  Emmanuel Goua de Baix <egouadeb>
Submitted:  Tue 04 Aug 2015 08:44:34 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 11 Aug 2015 05:16:46 PM UTC, comment #9: 

I added further clarification to the documentation for fread() that the ARCH specified there overrides the ARCH specified with fopen() (http://hg.savannah.gnu.org/hgweb/octave/rev/97edbce9b975).  I think this has been resolved so I'm marking as fixed and closing the report.

Rik <rik5>
Group administrator
Tue 11 Aug 2015 08:24:09 AM UTC, comment #8: 

Rik, your results with the corrected version seem OK for me.
Here are the results when displaying the values (same as yours except that obviously I still get the error for b/ieee-le since I made no change to my Octave):


Octave V4.0.0
fopen ARCH nothing | fread ARCH nothing => OK
fopen ARCH ieee-le | fread ARCH nothing => ans =   2.1787e+005
fopen ARCH ieee-be | fread ARCH nothing => ans =   2.7444e-307
fopen ARCH       b | fread ARCH nothing => ans =   2.7444e-307
fopen ARCH nothing | fread ARCH  native => ans =   2.1787e+005
fopen ARCH ieee-le | fread ARCH  native => ans =   2.1787e+005
fopen ARCH ieee-be | fread ARCH  native => ERROR !
fopen ARCH       b | fread ARCH  native => ERROR !
fopen ARCH nothing | fread ARCH ieee-be => ans =   2.7444e-307
fopen ARCH ieee-le | fread ARCH ieee-be => ans =   2.7444e-307
fopen ARCH ieee-be | fread ARCH ieee-be => ans =   2.7444e-307
fopen ARCH       b | fread ARCH ieee-be => ans =   2.7444e-307
fopen ARCH nothing | fread ARCH ieee-le => ans =   2.1787e+005
fopen ARCH ieee-le | fread ARCH ieee-le => ans =   2.1787e+005
fopen ARCH ieee-be | fread ARCH ieee-le => ERROR !
fopen ARCH       b | fread ARCH ieee-le => ERROR !

ML 2010a (reformatted for better reading)
fopen ARCH nothing | fread ARCH nothing => OK
fopen ARCH ieee-le | fread ARCH nothing => ans =  2.1787e+005
fopen ARCH ieee-be | fread ARCH nothing => ans =  2.7444e-307
fopen ARCH       b | fread ARCH nothing => ans =  2.7444e-307
fopen ARCH nothing | fread ARCH  native => ans =  2.1787e+005
fopen ARCH ieee-le | fread ARCH  native => ans =  2.1787e+005
fopen ARCH ieee-be | fread ARCH  native => ans =  2.1787e+005
fopen ARCH       b | fread ARCH  native => ans =  2.1787e+005
fopen ARCH nothing | fread ARCH ieee-be => ans =  2.7444e-307
fopen ARCH ieee-le | fread ARCH ieee-be => ans =  2.7444e-307
fopen ARCH ieee-be | fread ARCH ieee-be => ans =  2.7444e-307
fopen ARCH       b | fread ARCH ieee-be => ans =  2.7444e-307
fopen ARCH nothing | fread ARCH ieee-le => ans =  2.1787e+005
fopen ARCH ieee-le | fread ARCH ieee-le => ans =  2.1787e+005
fopen ARCH ieee-be | fread ARCH ieee-le => ans =  2.1787e+005
fopen ARCH       b | fread ARCH ieee-le => ans =  2.1787e+005


Now that I think of it, the results b/nothing are unexpected: it behaves like b/ieee-be instead of b/native.
The documentation of Octave about fread() is silent about the default value but the documentation of ML states that 'native' is the default ARCH. Still in fread()'s doc, I think there is no reference about the ARCH used by fopen(). No reference about ARCH used by fread() neither in fopen().

So I feel like the implementation of fread() uses the ARCH of fopen() by default, not necessarly 'native'. The documentation of ML is wrong on this point.

I discovered this issue with a binary file written by a C program where int are big-endian and float are little-endian.
One can also imagine a file where some floats are LE and some others are BE (headaches granted !).
I think that the feature to let the programmer force the ARCH in fread() independantly to the one used by fopen() is good and shall remain IMHO.

Emmanuel Goua de Baix <egouadeb>
Mon 10 Aug 2015 03:20:43 AM UTC, comment #7: 

I changed the documentation on the stable branch to explain the one-letter shortcut codes (http://hg.savannah.gnu.org/hgweb/octave/rev/f02c22f71cdb).

There was a problem in the C++ code that called a non-existent function name which happened to be an enum.  It was sort of miraculous that the compiler didn't flag it as an error.  I fixed that on the default branch here (http://hg.savannah.gnu.org/hgweb/octave/rev/a76e20f2f156).  Octave now runs through to completion for each of the different combinations.  Still, it would be useful to see what the actual value is that should be returned.  Could the original reporter run the newly attached rpt_arch_combinations.m and report the results.  Running it in Octave I get the following.


fopen ARCH nothing | fread ARCH nothing => OK
fopen ARCH ieee-le | fread ARCH nothing => ans =    2.1787e+05
fopen ARCH ieee-be | fread ARCH nothing => ans =   2.7444e-307
fopen ARCH       b | fread ARCH nothing => ans =   2.7444e-307
fopen ARCH nothing | fread ARCH  native => ans =    2.1787e+05
fopen ARCH ieee-le | fread ARCH  native => ans =    2.1787e+05
fopen ARCH ieee-be | fread ARCH  native => ans =    2.1787e+05
fopen ARCH       b | fread ARCH  native => ans =    2.1787e+05
fopen ARCH nothing | fread ARCH ieee-be => ans =   2.7444e-307
fopen ARCH ieee-le | fread ARCH ieee-be => ans =   2.7444e-307
fopen ARCH ieee-be | fread ARCH ieee-be => ans =   2.7444e-307
fopen ARCH       b | fread ARCH ieee-be => ans =   2.7444e-307
fopen ARCH nothing | fread ARCH ieee-le => ans =    2.1787e+05
fopen ARCH ieee-le | fread ARCH ieee-le => ans =    2.1787e+05
fopen ARCH ieee-be | fread ARCH ieee-le => ans =    2.1787e+05
fopen ARCH       b | fread ARCH ieee-le => ans =    2.1787e+05




(file #34619)

Rik <rik5>
Group administrator
Fri 07 Aug 2015 11:01:11 PM UTC, comment #6: 

Ok, I'm able to reproduce on Linux. But I have no idea if this is a valid situation, or whether we just want to have a better error handler for this condition. When the file is opened with ARCH = "native", you can choose the same or different ARCH when doing fread. But if fopen is called with ARCH != "native", then a different ARCH becomes an error in fread.


octave:1> x = pi;
octave:2> fid = fopen ("foo.dat", "w"); fwrite (fid, x, "double"); fclose (fid);
octave:3> fid = fopen ("foo.dat", "r", "b"); y = fread (fid, 1, "double"); fclose (fid);
octave:4> y
y =   3.2074e-192
octave:5> fid = fopen ("foo.dat", "r", "b"); y = fread (fid, 1, "double", 0, "native"); fclose (fid);
error: fread: impossible state reached in file '../liboctave/util/data-conv.cc' at line 702
octave:5> fid = fopen ("foo.dat", "r", "b"); y = fread (fid, 1, "double", 0, "ieee-le"); fclose (fid);
error: fread: impossible state reached in file '../liboctave/util/data-conv.cc' at line 702


The ARCH values "b", "l", and "n" are undocumented synonyms for "ieee-be", "ieee-le", and "native", respectively.

Mike Miller <mtmiller>
Group Member
Thu 06 Aug 2015 11:12:29 AM UTC, comment #5: 

ARRGG ! B***** H*** ! I found my mistake! Sorry, the bug is more subtle than I first thought.

Some explanations: Since several people reported that my example worked on their machine, I tried it again on mine ... and it worked also!
In fact, the example I gave in the original post is a very simplified version of a Matlab script that I am trying to port for Ocatve.
In this original Matlab script (not written by me), the file is opened with: fopen(filename, 'r', 'b') ... not 'rb' !!
Stupidly, I didn't pay attention to this, thinking that it was equivalent!
My first tests to write the example was done with "'r','b'". Then for an unknown reason I written "'rb'" (maybe I was lazy and wanted to save 3 chars ...) in order to test on the 32bits PC. And I didn't try again with "'rb'" on my 64bits PC. I posted the example with this option, thinking it was a particular bug on 64bits machine!
Now, after carefully reading the doc of fopen on both Matlab and Octave, I understand that the third parameter of fopen is an ARCH indicator. 'b' seems to be a short for 'ieee-be' (undocumented on Ocatve).

So to make it short, my issue seems to be that I tell fopen I work in ieee-be then I try to read in ieee-le?

I wrote the attached script to test several combinations of ARCH for fopen and fread.
It prints “OK” when fread succeeds in reading the double in the binary file attached in original post (it does not check the correctness of the value).
It prints “ERROR !“ if an error of fread is caught (assuming it is the same error as reported in original post)

Here are the results:


% OCTAVE V4.0.0 on BOTH Windows7 64bits AND WindowsXP 32bits unlike announced in original post
>> test_arch_combinations()
fopen ARCH nothing | fread ARCH nothing => OK
fopen ARCH ieee-le | fread ARCH nothing => OK
fopen ARCH ieee-be | fread ARCH nothing => OK
fopen ARCH       b | fread ARCH nothing => OK
fopen ARCH nothing | fread ARCH  native => OK
fopen ARCH ieee-le | fread ARCH  native => OK
fopen ARCH ieee-be | fread ARCH  native => ERROR !
fopen ARCH       b | fread ARCH  native => ERROR !
fopen ARCH nothing | fread ARCH ieee-be => OK
fopen ARCH ieee-le | fread ARCH ieee-be => OK
fopen ARCH ieee-be | fread ARCH ieee-be => OK
fopen ARCH       b | fread ARCH ieee-be => OK
fopen ARCH nothing | fread ARCH ieee-le => OK
fopen ARCH ieee-le | fread ARCH ieee-le => OK
fopen ARCH ieee-be | fread ARCH ieee-le => ERROR !
fopen ARCH       b | fread ARCH ieee-le => ERROR !
% MATLAB 2010b on Windows XP 32bits
>> test_arch_combinations()
fopen ARCH nothing | fread ARCH nothing => OK
fopen ARCH ieee-le | fread ARCH nothing => OK
fopen ARCH ieee-be | fread ARCH nothing => OK
fopen ARCH       b | fread ARCH nothing => OK
fopen ARCH nothing | fread ARCH  native => OK
fopen ARCH ieee-le | fread ARCH  native => OK
fopen ARCH ieee-be | fread ARCH  native => OK
fopen ARCH       b | fread ARCH  native => OK
fopen ARCH nothing | fread ARCH ieee-be => OK
fopen ARCH ieee-le | fread ARCH ieee-be => OK
fopen ARCH ieee-be | fread ARCH ieee-be => OK
fopen ARCH       b | fread ARCH ieee-be => OK
fopen ARCH nothing | fread ARCH ieee-le => OK
fopen ARCH ieee-le | fread ARCH ieee-le => OK
fopen ARCH ieee-be | fread ARCH ieee-le => OK
fopen ARCH       b | fread ARCH ieee-le => OK



Thus, it seems that the anomaly appears when the file is opened in ieee-be and then the value is read in ieee-le .

On PC, no ARCH for fopen should be equivalent to ieee-le which doesn’t result in error. Sorry for the mistake in the original post and thank you all for your reactivity.

NOTE: On both my 64 and 32 bits PCs :

 >> native_float_format
ans = ieee-le



(file #34579)

Emmanuel Goua de Baix <egouadeb>
Wed 05 Aug 2015 05:25:38 PM UTC, comment #4: 

This works for me as well on GNU/Linux 64-bit and Windows XP 32-bit.  Someone else with access to a Windows 64-bit installation will have to try and verify this.

The code in question is in liboctave/util/data-conv.cc and is shown below.


void
do_double_format_conversion (void *data, octave_idx_type len,
                             oct_mach_info::float_format from_fmt,
                             oct_mach_info::float_format to_fmt)
{
  switch (to_fmt)
    {
    case oct_mach_info::flt_fmt_ieee_little_endian:
      switch (from_fmt)
        {
        case oct_mach_info::flt_fmt_ieee_little_endian:
          break;

        case oct_mach_info::flt_fmt_ieee_big_endian:
          IEEE_big_double_to_IEEE_little_double (data, len);
          break;

        default:
          gripe_unrecognized_float_fmt ();
          break;
        }
      break;

    case oct_mach_info::flt_fmt_ieee_big_endian:
      switch (from_fmt)
        {
        case oct_mach_info::flt_fmt_ieee_little_endian:
          IEEE_little_double_to_IEEE_big_double (data, len);
          break;

        case oct_mach_info::flt_fmt_ieee_big_endian:
          break;

        default:
          gripe_unrecognized_float_fmt ();
          break;
        }
      break;

    default:
      (*current_liboctave_error_handler)
        ("impossible state reached in file '%s' at line %d",
         __FILE__, __LINE__);
      break;
    }
}


The only way it can get to the default branch of the case statement is if the conversion format is not ieee-le or ieee-be.  The fact that it gets there suggests that something is wrong with the initial determination of the machine's floating point format.  That determination occurs in liboctave/system/mach-info.cc.

Could the original reporter try the following at an Octave prompt?


native_float_format ()


On my Linux Intel machine this returns "ieee-le" as it should.  I suspect that this will return "unknown" for you.

Rik <rik5>
Group administrator
Wed 05 Aug 2015 12:52:31 AM UTC, comment #3: 

Hi, thanks for your bug report. Someone will have to try to reproduce this on Windows 64-bit. It works correctly on a GNU/Linux 64-bit system.

Mike Miller <mtmiller>
Group Member
Tue 04 Aug 2015 10:09:03 AM UTC, comment #2: 

After some few additional tests, I found that the issue is the same with float (single precision)

I specified my workaround solution ( compatible 32/64 bits Matlab/Octave but only on PC (little endian))


a_double_le = swapbytes( double( fread(fid, 1, 'double', 0, 'ieee-be') ) );
a_float_le  = swapbytes( single( fread(fid, 1, 'float' , 0, 'ieee-be') ) );





Emmanuel Goua de Baix <egouadeb>
Tue 04 Aug 2015 08:55:03 AM UTC, comment #1: 

Sorry for the poor text formatting, it is may first post on this site.
I don't know how to edit the orginal post.
The file content given in PS, is not shown correctlly due to tags, the value starts with a zero byte :

00 48 AB 19 8A 98 0A 41

Emmanuel Goua de Baix <egouadeb>
Tue 04 Aug 2015 08:44:34 AM UTC, original submission:  

I found an issue to read a double ieee-le in a binary file :

fid=fopen('test_octave_double_01.hex' , 'rb');
x=fread(fid, 1, 'double', 0, 'ieee-le')

returns this error message :

error: fread: impossible state reached in file '/scratch/jwe/src/mxe-octave/tmp-stable-octave/octave-4.0.0/liboctave/util/data-conv.cc' at line 702


Additonal information :
Error observed on a Windows7 64bits PC with Octave 4.0.0 (installed from octave-4.0.0_0-installer.exe).
On a Windows XP 32bits PC, BOTH Matlab AND Octave 4.0.0 give the expected result :

x =   2.1787e+005

Moreover, on the 64bits PC, the fread command without ARCH option does return a result but not the expected one since bytes need to be swapped (validated by swapping manually the bytes):

x=fread(fid, 1, 'double')
> x =   2.7444e-307

Using the option 'native' results in the same error message than 'ieee-le' (expected since PC is little-endian).
Using the option 'ieee-be' provides the same unwanted value than without option - and no error message !
Reading to type 'uint32' or 'uint64' does not trigger an error for any ARCH.
Not tested with other types (except uint above).
Not tested on Linux nor Mac.
Not tested with other bytes values.
Not tested on other Octave release.

Due to the 64bits dependancy of this issue, this bug might be related to #41672 : fread for "float" not matlab compatible

Workaround :
I will add a swap command in my parsing script and use fread without ARCH ...

PS:
'test_octave_double_01.hex' simply contains the following bytes (can be created/edited by HexEdit) :
00 48 AB 19 8A 98 0A 41

PPS about Item group :
I don't know if it is a Regression, and I don't think it is an Incorrect/Inaccurate Result, and it is more a Failure than a Crash (error message) => Other

Emmanuel Goua de Baix <egouadeb>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #34619:  rpt_arch_combinations.m added by rik5 (2KiB - text/x-objcsrc)
file #34579:  test_arch_combinations.m added by egouadeb (2KiB - application/octet-stream)
file #34573:  test_octave_double_01.hex added by egouadeb (8B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by egouadeb (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-08-11 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2015-08-10 rik5 Attached File- Added rpt_arch_combinations.m, #34619
    2015-08-07 mtmiller StatusNeed Info Confirmed
        Release4.0.0 dev
        Operating SystemMicrosoft Windows Any
        Summaryfread does not support ARCH \'ieee-le\' for double on 64bits PC fread does not allow different ARCH when file fopen'd with ARCH
    2015-08-06 egouadeb Attached File- Added test_arch_combinations.m, #34579
    2015-08-05 mtmiller Item GroupOther Incorrect Result
        StatusNone Need Info
    2015-08-04 egouadeb Attached File- Added test_octave_double_01.hex, #34573

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code