bugGNU Octave - Bugs: bug #63931, Binary fwrite writes incorrect...

 
 

bug #63931: Binary fwrite writes incorrect data to file

Submitter:  None
Submitted:  Wed 15 Mar 2023 06:33:45 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * 8.1.0
Operating System:  * Any Fixed Release:  8.2.0
Planned Release:  8.2.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 08 Apr 2023 08:16:53 AM UTC, comment #65: 

@charprap: Thank you for checking.

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Sat 08 Apr 2023 06:41:14 AM UTC, comment #64: 

Linked to bug https://savannah.gnu.org/bugs/index.php?63930, I also tested this shortly on Windows 10 with the version from buildbot.
It was ok for me.

Charles Praplan <charprap>
Fri 07 Apr 2023 01:56:21 PM UTC, comment #63: 

@hjborsje: If you are on Windows and you'd like to test these changes before Octave 8.2 is released, you could try a nightly build that you can download from here:
https://buildbot.octave.space/#/download

I'd appreciate if you could leave a short message how that fared for you.

Markus Mützel <mmuetzel>
Group administrator
Sun 02 Apr 2023 10:04:36 AM UTC, comment #62: 

Thank you for testing.

This change looks save enough to me to be pushed to the stable branch:
https://hg.savannah.gnu.org/hgweb/octave/rev/c1fc1e821381

Marking as ready for test again.

Markus Mützel <mmuetzel>
Group administrator
Sun 02 Apr 2023 08:04:32 AM UTC, comment #61: 


> A suggested workaround is to set encoding to "utf-8".


I'm not sure if this is a good work-around in general. If users have .m files with non-ASCII characters, they'd need to continue using the same encoding that they used when creating those .m files.
Alternatively, they could transcode all their .m files to UTF-8 and start using that from now on.

Fixing the encoding to "UTF-8" when opening the file stream with `fopen` might be a better workaround for Octave 8.1. But that might require changes in multiple places or in .m files that aren't "owned" by the user.

> I see references to Octave 8.2, but is it available yet as a Windows package?


Nightly builds that include the recent fixes should be available from here when the workers are up again:
https://buildbot.octave.space/#/download

(Wait for a build after March 31 that will default to outputting UTF-8 when no encoding is specified with `fopen`.)

Markus Mützel <mmuetzel>
Group administrator
Sat 01 Apr 2023 05:50:13 PM UTC, comment #60: 


comment #58:

> ...  My encoding is 'system'.


A suggested workaround is to set encoding to "utf-8".

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 01 Apr 2023 05:41:36 PM UTC, comment #59: 

Sorry, I managed to confuse 63931 and 63930.
With correct patch applied it passes.

Dmitri.
--




(file #54543, file #54544, file #54545)

Dmitri A. Sergatskov <dasergatskov>
Sat 01 Apr 2023 05:39:08 PM UTC, comment #58: 

I'm not sure if my comment is still relevant, but I use fwrite for binary files frequently in Octave 8.1 and had it misfire, occasionally writing the same 63 no matter what the intended value was.  My encoding is 'system'. I hope that 8.2 will be available soon, for me this error went unnoticed until the resulting binary file is read by a C++ program.

I see references to Octave 8.2, but is it available yet as a Windows package?

Thanks for any info.

Henk Borsje <hjborsje>
Sat 01 Apr 2023 05:31:40 PM UTC, comment #57: 

I fails for me

$ hg id
ce64ec8fe929+ (stable)
[dima@ryzen clang_gopt]$ grep 'std::ostream '  ../libinterp/corefcn/pr-output.cc
  std::ostream *osp = os.output_stream ();
$ ./run-octave -qf
octave:1> t1
val = 193
cw = 1
cr1 = 0
x = 193
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t1 at line 15 column 5
octave:2> t2
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
------
val = 128
cw = 1
cr1 = 0
x = 128
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t2 at line 15 column 5
octave:3> t10
val = 1
ans = 1
<... deleted ...>
ans = 127
val = 128
ans = 128
ans = 63
error: ASSERT errors for:  assert (a (1),uint8 (b (1)))

  Location  |  Observed  |  Expected  |  Reason
     ()          128           63        Abs err 65 exceeds tol 0 by 6e+01
octave:4> val = hex2dec('a7');
a = ones(2400,1)*val;
f = fopen('a.a','wb', 'n', 'Windows-1252');
fwrite(f,a,'uint8');
fclose(f);

f = fopen('a.a','rb');
b = fread(f,2400,'uint8');
fclose(f);

a(1)
b(1)
ans = 167
ans = 63
octave:14>


t1.m t2.m t10.m are attached
This is on linux

Only patch from this bug report ug63931-do-not-convert-bytes.patch was applied

Dmitri.
--


Dmitri A. Sergatskov <dasergatskov>
Sat 01 Apr 2023 04:40:01 PM UTC, comment #56: 

The attached patch never uses the output stream with the encoding facet when writing bytes even if an encoding (differing from UTF-8) is selected when opening the output stream with `fopen`.

With it, the following modified version of the reproducer from comment #0 writes the correct bytes for me (note the specified encoding for `fopen`):

val = hex2dec('a7');
a = ones(2400,1)*val;
f = fopen('a.a','wb', 'n', 'Windows-1252');
fwrite(f,a,'uint8');
fclose(f);

f = fopen('a.a','rb');
b = fread(f,2400,'uint8');
fclose(f);

a(1)
b(1)


I did only very limited exploratory testing with this change.
Any feedback or test results are appreciated.


(file #54542)

Markus Mützel <mmuetzel>
Group administrator
Sat 01 Apr 2023 11:59:13 AM UTC, comment #55: 

I backed out that change here:
https://hg.savannah.gnu.org/hgweb/octave/rev/d1a846f57379

It is no longer required for me to avoid the issue with the example in comment #0:

>> val = hex2dec('a7');
>> a = ones(2400,1)*val;
>> f = fopen('a.a','wb');
>> fwrite(f,a,'uint8');
>> fclose(f);
>> f = fopen('a.a','rb');
>> b = fread(f,2400,'uint8');
>> fclose(f);
>> a(1)
ans = 167
>> b(1)
ans = 167
>> __mfile_encoding__
ans = windows-1252


Backing out this change restores Matlab compatibility (transcoding is independent on whether a file stream is opened in binary mode or text mode).

Opening as ready for test again.

Markus Mützel <mmuetzel>
Group administrator
Fri 31 Mar 2023 10:14:25 AM UTC, comment #54: 

No complaints in a week. Closing as fixed. This will show up in 8.2.0 when it's released soon.

Arun Giridhar <arungiridhar>
Group Member
Fri 24 Mar 2023 08:13:03 PM UTC, comment #53: 

I pushed a fix to https://hg.savannah.gnu.org/hgweb/octave/rev/fcd97a68e5f7

Marking as ready for test. Please stress-test this fix.

Arun Giridhar <arungiridhar>
Group Member
Fri 24 Mar 2023 02:04:48 PM UTC, comment #52: 
Arun Giridhar <arungiridhar>
Group Member
Mon 20 Mar 2023 07:29:34 PM UTC, comment #51: 

regarding the comment #2 code
"Unrecognized function or variable 'stat'"

Nicholas Jankowski <nrjank>
Group Member
Mon 20 Mar 2023 07:20:51 PM UTC, comment #50: 

It looks like there is no "latin1" encoding in matlab.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 20 Mar 2023 07:19:06 PM UTC, comment #49: 

@nrjank if you're testing on Matlab, could you test the code in https://savannah.gnu.org/bugs/?63415#comment2 pls?

On Octave, it works if the encoding is utf-8. On latin1, it fails the assertion and the file sizes are different.

Arun Giridhar <arungiridhar>
Group Member
Mon 20 Mar 2023 07:18:42 PM UTC, comment #48: 

On a second thought windows cp-1920 is a superset of ISO-8859-1,
so may be it is not a good test case...

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 20 Mar 2023 07:13:57 PM UTC, comment #47: 

Can you get one of the test files from
https://savannah.gnu.org/bugs/?63930

(lat's say fprintf_bug1.m) and try to open in Editor, also
try to run it. Don you get non-english characters (é) and (û)
correctly?

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 20 Mar 2023 07:02:23 PM UTC, comment #46: 

at least on windows, the matlab editor only allows you to specify utf-8 and windows-1252 in the save as dialog. I don't see how to edit command line encoding.  fopen of course will take an encodingIn parameter which supposedly includes ISO-8859-1 but I get an 'invalid machine encoding' error when I try to specify it.

Nicholas Jankowski <nrjank>
Group Member
Mon 20 Mar 2023 06:31:50 PM UTC, comment #45: 

The bug is when you change interpreter encoding from utf-9 to something else e.g. latin1 aka iso-8859-1:


$ octave -qf
octave:1> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 193
------
val = 194
cw = 1
cr1 = 1
x = 194
y = 194
------
val = 195
cw = 1
cr1 = 1
x = 195
y = 195
------
octave:2> __mfile_encoding__ ("latin1")
octave:3> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 63
------
val = 194
cw = 1
cr1 = 0
x = 194
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t1 at line 15 column 5
octave:4>


Not sure wht Matlab's equivalent would be.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 20 Mar 2023 05:53:21 PM UTC, comment #44: 

matlab 2022b:

comment #0 original test script:

val = hex2dec('a7');
a = ones(2400,1)*val;
f = fopen('a.a','wb');
fwrite(f,a,'uint8');
fclose(f);

f = fopen('a.a','rb');
b = fread(f,2400,'uint8');
fclose(f);

a(1)
b(1)
ans =
   167
ans =
   167


 t1 script from comment #25:

>> clear
>> whos
>> t1
val =
  uint8
   193
cw =
     1
cr1 =
     1
x =
  uint8
   193
y =
   193
------
val =
  uint8
   194
cw =
     1
cr1 =
     1
x =
  uint8
   194
y =
   194
------
val =
  uint8
   195
cw =
     1
cr1 =
     1
x =
  uint8
   195
y =
   195
------
>> whos
  Name      Size            Bytes  Class     Attributes

  a         1x1                 1  uint8
  ans       1x1                 8  double
  b         1x1                 8  double
  cr        1x1                 8  double
  cr1       1x1                 8  double
  cw        1x1                 8  double
  f         1x1                 8  double
  ii        1x1                 8  double
  n         1x1                 8  double
  val       1x1                 1  uint8
  x         1x1                 1  uint8
  y         1x1                 8  double


not sure this is capturing the check you were looking for, though.

Nicholas Jankowski <nrjank>
Group Member
Mon 20 Mar 2023 04:16:14 PM UTC, comment #43: 

When you edit file in editor and then press "Run" it saves to utf-8 (tmp) file and sends it to interpreter.
 
If user saves m-file as non utf-8 and then it would run in gui but not in cli; this is also somewhat confusing. So we should tell them to save to utf-8 and if not, explicitly set _mfile_encoding_

Essentially users should work hard to save file as non-utf-8,
so they are fully aware that they are trying to break things.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 20 Mar 2023 03:56:03 PM UTC, comment #42: 


> Also, I am not sure that setting _mfile_encoding_ in the interpreter to the value in the editor is correct approach.

Perhaps interpreter should always be in utf-8 mode and editor
in whatever user chooses, but would convert the script to utf-8 (internally) before sending to interpreter?

I'd guess it would be surprising to a user if a file looks "fine" in the built-in editor, but then causes errors (or unexpected mojibake) when it is executed. So, the editor encoding and the `__mfile_encoding__` should be kept in sync imho.
The editor saves to files. It doesn't "send" code to the interpreter. So, it is unclear to me where the conversion you are proposing should be happening.

Markus Mützel <mmuetzel>
Group administrator
Mon 20 Mar 2023 03:45:55 PM UTC, comment #41: 

As per https://savannah.gnu.org/bugs/?63930 it is not only binary
stream is affected, text too.

Also, I am not sure that setting _mfile_encoding_ in the interpreter to the value in the editor is correct approach.
Perhaps interpreter should always be in utf-8 mode and editor
in whatever user chooses, but would convert the script to utf-8 (internally) before sending to interpreter?

I see here:
 https://www.mathworks.com/matlabcentral/answers/482380-how-to-set-the-encoding-of-matlab-m-files-to-be-utf8

<<<
As of R2020a, the MATLAB Editor will default to saving new  files using UTF-8. When opening existing files, the Editor and other functions like type or fopen automatically determine the current encoding. One can explicitly specify that the file should be saved using UTF-8 using the Save As dialog.

>>>


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 20 Mar 2023 02:39:14 PM UTC, comment #40: 

Can run a Matlab test, but I admit I lost track of the exact detail we’re trying to demonstrate. Would running the t1.m and t2.m test scripts from Dmitri in comment #25 cover it?

Nicholas Jankowski <nrjank>
Group Member
Mon 20 Mar 2023 10:02:54 AM UTC, comment #39: 

Thanks for bisecting.

I'd guess the encoding should not be converted if the output stream was opened in binary mode. I can't tell if that is currently the case (without checking the code again in more detail). I probably won't be able to look into that before in one or two weeks.

In the meantime, could someone please check with Matlab if the encoding is converted when writing to an output stream that was opened in binary mode?

Until this is properly fixed, setting the encoding to "UTF-8" when opening the output stream might be a work-around.

Markus Mützel <mmuetzel>
Group administrator
Mon 20 Mar 2023 01:34:22 AM UTC, comment #38: 


> I can't build it (unless someone has ideas on how to disable symbfact etc)


Try:

../configure --disable-docs --disable-java --without-qrupdate --without-suitesparseconfig --without-amd --without-camd --without-colamd --without-ccolamd --without-cholmod --without-spqr --without-cxsparse --without-umfpack --without-klu --without-sundials_nvecserial --without-sundials_ida --without-sundials_sunlinsolklu --without-arpack --without-opengl --without-qt --without-fltk CFLAGS="-ggdb3 -O0" CXXFLAGS="-ggdb3 -O0" FFLAGS="-ggdb3 -O0"


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 20 Mar 2023 12:45:26 AM UTC, comment #37: 

https://savannah.gnu.org/bugs/index.php?61839

Older bug report which motivated that change.

Arun Giridhar <arungiridhar>
Group Member
Mon 20 Mar 2023 12:43:46 AM UTC, comment #36: 

I rerun bootstrap and configure already. Each time using a new build directory. No luck yet.

Arun Giridhar <arungiridhar>
Group Member
Mon 20 Mar 2023 12:41:20 AM UTC, comment #35: 

Sorry, our comments crossed. Yes, the more I see that changeset, the more plausible it looks that gnulib's uniconv assumed utf-8 and breaks for certain inputs on other encodings.


Arun Giridhar <arungiridhar>
Group Member
Mon 20 Mar 2023 12:37:09 AM UTC, comment #34: 

You may need to re-run ./bootstrap

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 20 Mar 2023 12:36:09 AM UTC, comment #33: 

But yes, you're right. https://hg.savannah.gnu.org/hgweb/octave/rev/0826c503f294 makes changes to utf-8 handling. I can't build it (unless someone has ideas on how to disable symbfact etc) but it looks eminently plausible. It certainly wouldn't be caught on utf-8 encodings.

Arun Giridhar <arungiridhar>
Group Member
Mon 20 Mar 2023 12:32:32 AM UTC, comment #32: 

As per comment 29 -- both of the revision are on 8.0.0
branch, so it must be real this time ?!

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 20 Mar 2023 12:26:10 AM UTC, comment #31: 

I traced the flow from file-io.cc to two places, using Doxygen. One is a bunch of base and instantiated write functions such as that in ov-magic-int.cc. Ultimately those all call the second location anyway, which is where stream::write is defined, as well as several instances of it.

There's nothing that's obviously wrong, but this isn't an obvious bug. There is one type conversion that happens around line 7084, which in turn calls convert_data and in turn convert_chars orconvert_ints.

By eliminating most of the other possibilities, it must be going wrong in lines like;

vt_data[i] = val.value ();

where the value conversion must be getting messed up, and if it's but I have no idea why (except suspecting saturation semantics) nor why it's encoding-dependent.

Re bisection, you probably want "update default" and bound your search between the first bump that starts off Octave 8 (right after 7.1 was made) and the one that starts Octave 9 (right after 8.1 was made). I disabled most of my libraries but couldn't prevent symbfact.cc from building and causing errors. If you have better techniques I'll try them.

Arun Giridhar <arungiridhar>
Group Member
Sun 19 Mar 2023 11:30:56 PM UTC, comment #30: 

You can disable (via configure) most of the libraries; it would also build faster.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 11:27:47 PM UTC, comment #29: 

Branches confuse me. Anyway:
revision 30821 (8.0.0) is good
revision 30822 (8.0.0) is bad.


$  hg log  -r 30822
changeset:   30822:0826c503f294
user:        Markus Mützel <markus.muetzel@gmx.de>
date:        Sat Mar 05 21:20:58 2022 +0100
summary:     Encoding facet based on gnulib uniconv for STL iostreams (bug #61839).


I guess we need Markus on-line.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 11:27:28 PM UTC, comment #28: 

I'm completely unable to bisect. Any version before Jan 15 gives a build error about SuiteSparse -- I have the latest version of that library but earlier versions of Octave used older versions of that library, which had a different API. I used --without-suitesparseconfig, --without-amd, and --without-klu, but it still errors out in symbfact.cc for me.

Giving up on bisection. Will try tracing the flow in fprintf.

Arun Giridhar <arungiridhar>
Group Member
Sun 19 Mar 2023 10:42:20 PM UTC, comment #27: 

That's bizarre. A merge from stable to default should not break default while leaving stable unaffected. That is, if that had been the source of the error, it would have shown up in 7.3 given the June timestamp and the stable branch, not waited until 8.1.

The edits you listed:

https://hg.savannah.gnu.org/hgweb/octave/rev/01e4f3882f54

https://hg.savannah.gnu.org/hgweb/octave/rev/e8238ae72381

Might need a wider range to bisect on, all the way back to the last time default was merged to stable, meaning when 7.1 was released, and it has to be searched on the default branch.

I'll get to this soon as well from my end.

Arun Giridhar <arungiridhar>
Group Member
Sun 19 Mar 2023 08:34:25 PM UTC, comment #26: 

It looks like the last "good" hgid is 01e4f3882f54

hg log -r -800
changeset:   31109:01e4f3882f54
branch:      stable
parent:      31107:bbb59cc6698c
user:        Pantxo Diribarne <pantxo.diribarne@gmail.com>
date:        Fri Jun 24 19:24:22 2022 +0200
summary:     Fix regression with \color[rgb] tex pattern (bug #62668)




octave:2>  __mfile_encoding__("latin1")
octave:3> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 193
------
val = 194
cw = 1
cr1 = 1
x = 194
y = 194
------
val = 195
cw = 1
cr1 = 1
x = 195
y = 195
------
octave:4> t2
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
------
val = 128
cw = 1
cr1 = 1
x = 128
y = 128
------
val = 129
cw = 1
cr1 = 1
x = 129
y = 129
------
octave:5> version -hgid
ans = 01e4f3882f54
octave:6>  __mfile_encoding__
ans = latin1


The next changelog is:

hg log -r -799
changeset:   31110:e8238ae72381
parent:      31108:6ee5bd9ad7cc
parent:      31109:01e4f3882f54
user:        Pantxo Diribarne <pantxo.diribarne@gmail.com>
date:        Tue Jun 28 19:52:36 2022 +0200
summary:     maint: merge stable to default

and it is bad:

octave:2> version -hgid
ans = e8238ae72381
octave:3>  __mfile_encoding__("latin1")
octave:4>  __mfile_encoding__
ans = latin1
octave:5> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 63
------
val = 194
cw = 1
cr1 = 0
x = 194
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t1 at line 15 column 5
octave:6> t2
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
------
val = 128
cw = 1
cr1 = 1
x = 128
y = 63
------
val = 129
cw = 1
cr1 = 1
x = 129
y = 63
------


Dmitri.
--



Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 07:25:44 PM UTC, comment #25: 

And on 7.3.0 this bug does not exist:


octave:1>  __mfile_encoding__("latin1")
octave:2>  __mfile_encoding__
ans = latin1
octave:3> ver
----------------------------------------------------------------------
GNU Octave Version: 7.3.0 (hg id: 2c037ce00450)
GNU Octave License: GNU General Public License
Operating System: Linux 5.14.0-285.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 7 17:32:48 UTC 2023 x86_64
----------------------------------------------------------------------
no packages installed.
octave:4> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 193
------
val = 194
cw = 1
cr1 = 1
x = 194
y = 194
------
val = 195
cw = 1
cr1 = 1
x = 195
y = 195
------
octave:5> t2
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
------
val = 128
cw = 1
cr1 = 1
x = 128
y = 128
------
val = 129
cw = 1
cr1 = 1
x = 129
y = 129
------


(Also attached t1.m and t2.m files to this bug report.)

Dmitri.
--


(file #54511)

Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 07:17:59 PM UTC, comment #24: 

And you do not need gui:


octave:1>  __mfile_encoding__("latin1")
octave:2>  __mfile_encoding__
ans = latin1
octave:3> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 63
------
val = 194
cw = 1
cr1 = 0
x = 194
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t1 at line 15 column 5
octave:4> t2
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
------
val = 128
cw = 1
cr1 = 1
x = 128
y = 63
------
val = 129
cw = 1
cr1 = 1
x = 129
y = 63
------


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 06:35:45 PM UTC, comment #23: 

also fyi:

Bug is also manifests itself when gui run with " --experimental-terminal-widget".

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 06:29:56 PM UTC, comment #22: 

Quite a high-achieving bug. I can see how encoding problems cause the wrong value to be written to file... not at all intuitive in diagnosis but makes sense in retrospect. But I currently have no idea why that causes the wrong number of bytes to be written to file, that too for numeric values that aren't even char.

Arun Giridhar <arungiridhar>
Group Member
Sun 19 Mar 2023 06:22:44 PM UTC, comment #21: 

Yes. It is all the same bug...

>> clear all

fid = fopen ("m_eye.bin", "wb", "ieee-le");
matA = eye (100,150);
n = fwrite (fid, matA, "double");
fclose(fid);

assert (stat ("m_eye.bin").size == numel(matA) * 8)

fid = fopen ("m_rand.bin", "wb", "ieee-le");
matB = rand (100,150);
n = fwrite (fid, matB, "double");
fclose(fid);

assert (stat ("m_rand.bin").size == numel(matB) * 8)

fid = fopen ("m_eye.bin", "rb");
matC = fread (fid, inf, "double");
fclose(fid);

fid = fopen ("m_rand.bin", "rb");
matD = fread (fid, inf, "double");
fclose(fid);

assert (all (matC == matA(:)))
assert (all (matD == matB(:)))

clear ans fid
whos
error: assert (stat ("m_rand.bin").size == numel (matB) * 8) failed
error: called from
    assert at line 107 column 11
error: assert (all (matC == matA (:))) failed
error: called from
    assert at line 107 column 11
error: mx_el_eq: nonconformant arguments (op1 is 14378x1, op2 is 15000x1)
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name        Size                     Bytes  Class
  ====   ====        ====                     =====  =====
         matA      100x150                      800  double
         matB      100x150                   120000  double
         matC    15000x1                     120000  double
         matD    14378x1                     115024  double
         n           1x1                          8  double

Total is 59379 elements using 355832 bytes


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 06:20:30 PM UTC, comment #20: 

@Dmitri: does the encoding also affect bug #63415 behavior for you?

I'll be able to test this in a few hours. Away from PC tight now.

Arun Giridhar <arungiridhar>
Group Member
Sun 19 Mar 2023 04:11:33 PM UTC, comment #19: 

I can reproduce it with flatpak-provided 8.1.0 version.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 02:34:34 PM UTC, comment #18: 

It looks like any encoding that is not utf-8 causing a problem.
Also changing encoding to BOCU-1 brings up error on command prompt:


>> encodingerror: __mfile_encoding__: conversion from encoding 'bocu-1' not supported


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 02:23:47 PM UTC, comment #17: 

OK. If I change encoding (in editor settings in gui) to
iso-8859-1 the problem re-appears. This is with standard optimized compile with the latest 8.1.1 snapshot (d4d3098a15dc)


>> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 63
------
val = 194
cw = 1
cr1 = 0
x = 194
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t1 at line 15 column 5
>> t2
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
------
val = 128
cw = 1
cr1 = 1
x = 128
y = 63
------
val = 129
cw = 1
cr1 = 1
x = 129
y = 63
------


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 16 Mar 2023 03:40:16 PM UTC, comment #16: 

While I was trying to replicate the errors here (unsuccessfully), I found a way to make the GUI disappear on startup (https://octave.discourse.group/t/octave-gui-crash-on-startup/4201)

Note that deleting ~/.config/octave would likely remove the GUI ini file, hence that startup behavior is different from usual.

But it doesn't do anything interesting with the fread and fwrite itself. Those are still passing the assertions in the various tests.

Arun Giridhar <arungiridhar>
Group Member
Thu 16 Mar 2023 02:15:34 PM UTC, comment #15: 

"I removed ~/.config/octave and now I cannot reproduce the problem anymore ?!"

OK, that is definitely strange.  Do you have a copy of the old contents of that directory that you could restore to see whether the problem reappears?

John W. Eaton <jwe>
Group administrator
Thu 16 Mar 2023 01:10:42 PM UTC, comment #14: 

I removed ~/.config/octave and now I cannot reproduce the problem anymore ?!

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 16 Mar 2023 08:55:30 AM UTC, comment #13: 

@jwe: could you compile with "-ggdb3 -O0" as described in bug #63415? Pls also try the test with asserts at https://savannah.gnu.org/bugs/?63415#comment2

Arun Giridhar <arungiridhar>
Group Member
Thu 16 Mar 2023 05:53:28 AM UTC, comment #12: 

I tried compiling with -Os with both clang (14.0.6) and gcc (12.2.0) on a Debian system and I can't reproduce the problem with the t1.m script.  CLI vs. GUI doesn't change anything for me.  I've also compiled with the default configure options (-g -O2) on the same system with the same results.

Or at least I think the t1 script is not showing a problem.  Here is what I see:


>> t1
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
val = 128
cw = 1
cr1 = 1
x = 128
y = 128
val = 129
cw = 1
cr1 = 1
x = 129
y = 129


Is that the correct output?  Could you make the test script use assert or error or something to clearly indicate what is an error instead of having to compare output values visually?

John W. Eaton <jwe>
Group administrator
Thu 16 Mar 2023 04:19:20 AM UTC, comment #11: 

clang compiles seem to reproduce this problem at any optimization level and both 8.1.1 and 9.0.0 version (but also only in gui).


Dmitri A. Sergatskov <dasergatskov>
Thu 16 Mar 2023 02:03:47 AM UTC, comment #10: 

I can also reproduce this with

clang version 15.0.7 (Red Hat 15.0.7-2.el9) (-Os and, presumably, lower). -O2 works fine.

gcc version 12.2.1 20221121 (Red Hat 12.2.1-7) (-Os and, presumably, lower). -O2 works fine.

cli always works fine. Bizarre.

Dmitri,
--


Dmitri A. Sergatskov <dasergatskov>
Thu 16 Mar 2023 01:46:28 AM UTC, comment #9: 

Also changing range of the val to 193 to 195:


>> clear
>> whos
>> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 63
val = 194
cw = 1
cr1 = 0
x = 194
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t1 at line 15 column 5
>>

a.a is an empty file.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 16 Mar 2023 01:38:55 AM UTC, comment #8: 

Here is modified test script:


$ cat t1.m
for ii = 1:3
  val = uint8(126+ii)
  n = 1;
  a = ones(n,1)*val;
  f = fopen('a.a','wb');
  cw = fwrite(f,a,'uint8')
  fclose(f);

  f = fopen('a.a','rb');
  [b, cr] = fread(f,n,'uint8');
  fclose(f);

  cr1 = cr
  x = a(1)
  y = b(1)
endfor


Running it:


>> clear
>> whos
>> t1
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
val = 128
cw = 1
cr1 = 1
x = 128
y = 63
val = 129
cw = 1
cr1 = 1
x = 129
y = 63
>>



od -d a.a
0000000    63
0000001


So some kind of unsigned/signed int mixed up?

Dmitri.
--



Dmitri A. Sergatskov <dasergatskov>
Thu 16 Mar 2023 01:28:18 AM UTC, comment #7: 

Yes it reproduciable with N = 1; but the val value seems to matter.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 16 Mar 2023 01:26:54 AM UTC, comment #6: 

I am working on it.
I can reproduce the problem with "-Os", so may be it is a compiler thing.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 16 Mar 2023 01:19:47 AM UTC, comment #5: 

That's confirmation enough that the two bugs are manifestations of the same deeper problem.

@Dmitri: since I can't replicate it yet, could you check these pls?

  • Reduce 2400 to (1 to 10). If that eliminates the problem, it might be a buffer corruption somewhere.


  • Verify the files independently to see if Octave wrote the wrong value, or if it wrote the correct value but fread is making a mistake.


Also, what compilers are showing the errors?

Arun Giridhar <arungiridhar>
Group Member
Thu 16 Mar 2023 01:00:33 AM UTC, comment #4: 

I can reproduce this bug on comiles where I can reproduce bug
https://savannah.gnu.org/bugs/?63415


>> clear all

fid = fopen ("m_eye.bin", "wb", "ieee-le");
matA = eye (100,150);
n = fwrite (fid, matA, "double");
fclose(fid);

assert (stat ("m_eye.bin").size == numel(matA) * 8)

fid = fopen ("m_rand.bin", "wb", "ieee-le");
matB = rand (100,150);
n = fwrite (fid, matB, "double");
fclose(fid);

assert (stat ("m_rand.bin").size == numel(matB) * 8)

fid = fopen ("m_eye.bin", "rb");
matC = fread (fid, inf, "double");
fclose(fid);

fid = fopen ("m_rand.bin", "rb");
matD = fread (fid, inf, "double");
fclose(fid);

assert (all (matC == matA(:)))
assert (all (matD == matB(:)))

clear ans fid
whos
error: assert (stat ("m_rand.bin").size == numel (matB) * 8) failed
error: called from
    assert at line 107 column 11
error: assert (all (matC == matA (:))) failed
error: called from
    assert at line 107 column 11
error: mx_el_eq: nonconformant arguments (op1 is 14406x1, op2 is 15000x1)
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name        Size                     Bytes  Class
  ====   ====        ====                     =====  =====
         matA      100x150                      800  double
         matB      100x150                   120000  double
         matC    15000x1                     120000  double
         matD    14406x1                     115248  double
         n           1x1                          8  double

Total is 59407 elements using 356056 bytes

>> val = hex2dec('a7');
a = ones(2400,1)*val;
f = fopen('a.a','wb');
fwrite(f,a,'uint8');
fclose(f);

f = fopen('a.a','rb');
b = fread(f,2400,'uint8');
fclose(f);

a(1)
b(1)
ans = 167
ans = 63
>>


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 16 Mar 2023 12:39:52 AM UTC, comment #3: 

@OP: Did you do your test in GUI mode? Could you do it in CLI mode as well?

Arun Giridhar <arungiridhar>
Group Member
Wed 15 Mar 2023 11:46:26 PM UTC, comment #2: 
Arun Giridhar <arungiridhar>
Group Member
Wed 15 Mar 2023 07:00:46 PM UTC, comment #1: 

I cannot reproduce this on Win11 codepage 437 (octave gui)

>> ver
----------------------------------------------------------------------
GNU Octave Version: 8.1.0 (hg id: 75b1efe727d4)
GNU Octave License: GNU General Public License
Operating System: MINGW32_NT-6.2 Windows 6.2  x86_64



>> clear
>> whos
>> val = hex2dec('a7');
>> a = ones(2400,1)*val;
>> f = fopen('a.a','wb');
>> fwrite(f,a,'uint8');
>> fclose(f);
>> f = fopen('a.a','rb');
>> b = fread(f,2400,'uint8');
>> fclose(f);
>> a(1)
ans = 167
>> b(1)
ans = 167
>>


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 15 Mar 2023 06:33:45 PM UTC, original submission:  

If you open a file for binary write and use fwrite write a vector of uint8 values to the file, some of them get changed.  Specifically, writing hex A7 ends up in the file as hex 3F...

Here's a short script that generates incorrect results...


val = hex2dec('a7');
a = ones(2400,1)*val;
f = fopen('a.a','wb');
fwrite(f,a,'uint8');
fclose(f);

f = fopen('a.a','rb');
b = fread(f,2400,'uint8');
fclose(f);

a(1)
b(1)


Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #54543:  t1.m added by dasergatskov (277B - text/x-objcsrc)
file #54544:  t10.m added by dasergatskov (279B - text/x-objcsrc)
file #54545:  t2.m added by dasergatskov (283B - text/x-objcsrc)
file #54542:  bug63931-do-not-convert-bytes.patch added by mmuetzel (2KiB - application/octet-stream)
file #54511:  t1.m added by dasergatskov (262B - text/x-objcsrc)

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by charprap (Posted a comment)
  • -email is unavailable- added by hjborsje (Posted a comment)
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by arungiridhar (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 19 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-04-08 mmuetzel Dependencies- bugs #63968 is dependent
    2023-04-08 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2023-04-02 mmuetzel StatusPatch Submitted Ready For Test
    2023-04-01 dasergatskov Attached File- Added t1.m, #54543
        Attached File- Added t10.m, #54544
        Attached File- Added t2.m, #54545
    2023-04-01 mmuetzel Attached File- Added bug63931-do-not-convert-bytes.patch, #54542
        StatusReady For Test Patch Submitted
    2023-04-01 mmuetzel StatusFixed Ready For Test
        Open/ClosedClosed Open
    2023-03-31 arungiridhar StatusReady For Test Fixed
        Open/ClosedOpen Closed
        Fixed ReleaseNone 8.2.0
    2023-03-24 arungiridhar StatusConfirmed Ready For Test
        Operating SystemMicrosoft Windows Any
        Planned ReleaseNone 8.2.0
    2023-03-19 dasergatskov Attached File- Added t1.m, #54511
    2023-03-16 arungiridhar StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code