@charprap: Thank you for checking.

Closing as fixed.

@hjborsje: If you are on Windows and you'd like to test these changes before Octave 8.2 is released, you could try a nightly build that you can download from here:
https://buildbot.octave.space/#/download

I'd appreciate if you could leave a short message how that fared for you.

> A suggested workaround is to set encoding to "utf-8".

I'm not sure if this is a good work-around in general. If users have .m files with non-ASCII characters, they'd need to continue using the same encoding that they used when creating those .m files.
Alternatively, they could transcode all their .m files to UTF-8 and start using that from now on.

Fixing the encoding to "UTF-8" when opening the file stream with `fopen` might be a better workaround for Octave 8.1. But that might require changes in multiple places or in .m files that aren't "owned" by the user.

> I see references to Octave 8.2, but is it available yet as a Windows package?

Nightly builds that include the recent fixes should be available from here when the workers are up again:
https://buildbot.octave.space/#/download

(Wait for a build after March 31 that will default to outputting UTF-8 when no encoding is specified with `fopen`.)

Sorry, I managed to confuse 63931 and 63930.
With correct patch applied it passes.

Dmitri.
--




(file #54543, file #54544, file #54545)

I fails for me

$ hg id
ce64ec8fe929+ (stable)
[dima@ryzen clang_gopt]$ grep 'std::ostream '  ../libinterp/corefcn/pr-output.cc
  std::ostream *osp = os.output_stream ();
$ ./run-octave -qf
octave:1> t1
val = 193
cw = 1
cr1 = 0
x = 193
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t1 at line 15 column 5
octave:2> t2
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
------
val = 128
cw = 1
cr1 = 0
x = 128
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t2 at line 15 column 5
octave:3> t10
val = 1
ans = 1
<... deleted ...>
ans = 127
val = 128
ans = 128
ans = 63
error: ASSERT errors for:  assert (a (1),uint8 (b (1)))

  Location  |  Observed  |  Expected  |  Reason
     ()          128           63        Abs err 65 exceeds tol 0 by 6e+01
octave:4> val = hex2dec('a7');
a = ones(2400,1)*val;
f = fopen('a.a','wb', 'n', 'Windows-1252');
fwrite(f,a,'uint8');
fclose(f);

f = fopen('a.a','rb');
b = fread(f,2400,'uint8');
fclose(f);

a(1)
b(1)
ans = 167
ans = 63
octave:14>

t1.m t2.m t10.m are attached
This is on linux

Only patch from this bug report ug63931-do-not-convert-bytes.patch was applied

Dmitri.
--

I backed out that change here:
https://hg.savannah.gnu.org/hgweb/octave/rev/d1a846f57379

It is no longer required for me to avoid the issue with the example in comment #0:

>> val = hex2dec('a7');
>> a = ones(2400,1)*val;
>> f = fopen('a.a','wb');
>> fwrite(f,a,'uint8');
>> fclose(f);
>> f = fopen('a.a','rb');
>> b = fread(f,2400,'uint8');
>> fclose(f);
>> a(1)
ans = 167
>> b(1)
ans = 167
>> __mfile_encoding__
ans = windows-1252

Backing out this change restores Matlab compatibility (transcoding is independent on whether a file stream is opened in binary mode or text mode).

Opening as ready for test again.

I pushed a fix to https://hg.savannah.gnu.org/hgweb/octave/rev/fcd97a68e5f7

Marking as ready for test. Please stress-test this fix.

regarding the comment #2 code
"Unrecognized function or variable 'stat'"

@nrjank if you're testing on Matlab, could you test the code in https://savannah.gnu.org/bugs/?63415#comment2 pls?

On Octave, it works if the encoding is utf-8. On latin1, it fails the assertion and the file sizes are different.

Can you get one of the test files from
https://savannah.gnu.org/bugs/?63930

(lat's say fprintf_bug1.m) and try to open in Editor, also
try to run it. Don you get non-english characters (é) and (û)
correctly?

Dmitri.
--

The bug is when you change interpreter encoding from utf-9 to something else e.g. latin1 aka iso-8859-1:

$ octave -qf
octave:1> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 193
------
val = 194
cw = 1
cr1 = 1
x = 194
y = 194
------
val = 195
cw = 1
cr1 = 1
x = 195
y = 195
------
octave:2> __mfile_encoding__ ("latin1")
octave:3> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 63
------
val = 194
cw = 1
cr1 = 0
x = 194
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t1 at line 15 column 5
octave:4>

Not sure wht Matlab's equivalent would be.

Dmitri.
--

When you edit file in editor and then press "Run" it saves to utf-8 (tmp) file and sends it to interpreter.
 
If user saves m-file as non utf-8 and then it would run in gui but not in cli; this is also somewhat confusing. So we should tell them to save to utf-8 and if not, explicitly set _mfile_encoding_

Essentially users should work hard to save file as non-utf-8,
so they are fully aware that they are trying to break things.

Dmitri.
--

As per https://savannah.gnu.org/bugs/?63930 it is not only binary
stream is affected, text too.

Also, I am not sure that setting _mfile_encoding_ in the interpreter to the value in the editor is correct approach.
Perhaps interpreter should always be in utf-8 mode and editor
in whatever user chooses, but would convert the script to utf-8 (internally) before sending to interpreter?

I see here:
 https://www.mathworks.com/matlabcentral/answers/482380-how-to-set-the-encoding-of-matlab-m-files-to-be-utf8

<<<
As of R2020a, the MATLAB Editor will default to saving new  files using UTF-8. When opening existing files, the Editor and other functions like type or fopen automatically determine the current encoding. One can explicitly specify that the file should be saved using UTF-8 using the Save As dialog.

>>>

Dmitri.
--

Thanks for bisecting.

I'd guess the encoding should not be converted if the output stream was opened in binary mode. I can't tell if that is currently the case (without checking the code again in more detail). I probably won't be able to look into that before in one or two weeks.

In the meantime, could someone please check with Matlab if the encoding is converted when writing to an output stream that was opened in binary mode?

Until this is properly fixed, setting the encoding to "UTF-8" when opening the output stream might be a work-around.

https://savannah.gnu.org/bugs/index.php?61839

Older bug report which motivated that change.

Sorry, our comments crossed. Yes, the more I see that changeset, the more plausible it looks that gnulib's uniconv assumed utf-8 and breaks for certain inputs on other encodings.

But yes, you're right. https://hg.savannah.gnu.org/hgweb/octave/rev/0826c503f294 makes changes to utf-8 handling. I can't build it (unless someone has ideas on how to disable symbfact etc) but it looks eminently plausible. It certainly wouldn't be caught on utf-8 encodings.

I traced the flow from file-io.cc to two places, using Doxygen. One is a bunch of base and instantiated write functions such as that in ov-magic-int.cc. Ultimately those all call the second location anyway, which is where stream::write is defined, as well as several instances of it.

There's nothing that's obviously wrong, but this isn't an obvious bug. There is one type conversion that happens around line 7084, which in turn calls convert_data and in turn convert_chars orconvert_ints.

By eliminating most of the other possibilities, it must be going wrong in lines like;

vt_data[i] = val.value ();

where the value conversion must be getting messed up, and if it's but I have no idea why (except suspecting saturation semantics) nor why it's encoding-dependent.

Re bisection, you probably want "update default" and bound your search between the first bump that starts off Octave 8 (right after 7.1 was made) and the one that starts Octave 9 (right after 8.1 was made). I disabled most of my libraries but couldn't prevent symbfact.cc from building and causing errors. If you have better techniques I'll try them.

Branches confuse me. Anyway:
revision 30821 (8.0.0) is good
revision 30822 (8.0.0) is bad.

$  hg log  -r 30822
changeset:   30822:0826c503f294
user:        Markus Mützel <markus.muetzel@gmx.de>
date:        Sat Mar 05 21:20:58 2022 +0100
summary:     Encoding facet based on gnulib uniconv for STL iostreams (bug #61839).

I guess we need Markus on-line.

Dmitri.
--

That's bizarre. A merge from stable to default should not break default while leaving stable unaffected. That is, if that had been the source of the error, it would have shown up in 7.3 given the June timestamp and the stable branch, not waited until 8.1.

The edits you listed:

https://hg.savannah.gnu.org/hgweb/octave/rev/01e4f3882f54

https://hg.savannah.gnu.org/hgweb/octave/rev/e8238ae72381

Might need a wider range to bisect on, all the way back to the last time default was merged to stable, meaning when 7.1 was released, and it has to be searched on the default branch.

I'll get to this soon as well from my end.

And on 7.3.0 this bug does not exist:

octave:1>  __mfile_encoding__("latin1")
octave:2>  __mfile_encoding__
ans = latin1
octave:3> ver
----------------------------------------------------------------------
GNU Octave Version: 7.3.0 (hg id: 2c037ce00450)
GNU Octave License: GNU General Public License
Operating System: Linux 5.14.0-285.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 7 17:32:48 UTC 2023 x86_64
----------------------------------------------------------------------
no packages installed.
octave:4> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 193
------
val = 194
cw = 1
cr1 = 1
x = 194
y = 194
------
val = 195
cw = 1
cr1 = 1
x = 195
y = 195
------
octave:5> t2
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
------
val = 128
cw = 1
cr1 = 1
x = 128
y = 128
------
val = 129
cw = 1
cr1 = 1
x = 129
y = 129
------

(Also attached t1.m and t2.m files to this bug report.)

Dmitri.
--


(file #54511)

also fyi:

Bug is also manifests itself when gui run with " --experimental-terminal-widget".

Dmitri.
--

Yes. It is all the same bug...

>> clear all

fid = fopen ("m_eye.bin", "wb", "ieee-le");
matA = eye (100,150);
n = fwrite (fid, matA, "double");
fclose(fid);

assert (stat ("m_eye.bin").size == numel(matA) * 8)

fid = fopen ("m_rand.bin", "wb", "ieee-le");
matB = rand (100,150);
n = fwrite (fid, matB, "double");
fclose(fid);

assert (stat ("m_rand.bin").size == numel(matB) * 8)

fid = fopen ("m_eye.bin", "rb");
matC = fread (fid, inf, "double");
fclose(fid);

fid = fopen ("m_rand.bin", "rb");
matD = fread (fid, inf, "double");
fclose(fid);

assert (all (matC == matA(:)))
assert (all (matD == matB(:)))

clear ans fid
whos
error: assert (stat ("m_rand.bin").size == numel (matB) * 8) failed
error: called from
    assert at line 107 column 11
error: assert (all (matC == matA (:))) failed
error: called from
    assert at line 107 column 11
error: mx_el_eq: nonconformant arguments (op1 is 14378x1, op2 is 15000x1)
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name        Size                     Bytes  Class
  ====   ====        ====                     =====  =====
         matA      100x150                      800  double
         matB      100x150                   120000  double
         matC    15000x1                     120000  double
         matD    14378x1                     115024  double
         n           1x1                          8  double

Total is 59379 elements using 355832 bytes

Dmitri.
--

I can reproduce it with flatpak-provided 8.1.0 version.

Dmitri.
--

OK. If I change encoding (in editor settings in gui) to
iso-8859-1 the problem re-appears. This is with standard optimized compile with the latest 8.1.1 snapshot (d4d3098a15dc)

>> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 63
------
val = 194
cw = 1
cr1 = 0
x = 194
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t1 at line 15 column 5
>> t2
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
------
val = 128
cw = 1
cr1 = 1
x = 128
y = 63
------
val = 129
cw = 1
cr1 = 1
x = 129
y = 63
------

Dmitri.
--

"I removed ~/.config/octave and now I cannot reproduce the problem anymore ?!"

OK, that is definitely strange.  Do you have a copy of the old contents of that directory that you could restore to see whether the problem reappears?

@jwe: could you compile with "-ggdb3 -O0" as described in bug #63415? Pls also try the test with asserts at https://savannah.gnu.org/bugs/?63415#comment2

clang compiles seem to reproduce this problem at any optimization level and both 8.1.1 and 9.0.0 version (but also only in gui).

Also changing range of the val to 193 to 195:

>> clear
>> whos
>> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 63
val = 194
cw = 1
cr1 = 0
x = 194
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t1 at line 15 column 5
>>

a.a is an empty file.

Dmitri.
--

Yes it reproduciable with N = 1; but the val value seems to matter.

Dmitri.
--

That's confirmation enough that the two bugs are manifestations of the same deeper problem.

@Dmitri: since I can't replicate it yet, could you check these pls?

• Reduce 2400 to (1 to 10). If that eliminates the problem, it might be a buffer corruption somewhere.

• Verify the files independently to see if Octave wrote the wrong value, or if it wrote the correct value but fread is making a mistake.

Also, what compilers are showing the errors?

@OP: Did you do your test in GUI mode? Could you do it in CLI mode as well?

I cannot reproduce this on Win11 codepage 437 (octave gui)

>> ver
----------------------------------------------------------------------
GNU Octave Version: 8.1.0 (hg id: 75b1efe727d4)
GNU Octave License: GNU General Public License
Operating System: MINGW32_NT-6.2 Windows 6.2  x86_64

>> clear
>> whos
>> val = hex2dec('a7');
>> a = ones(2400,1)*val;
>> f = fopen('a.a','wb');
>> fwrite(f,a,'uint8');
>> fclose(f);
>> f = fopen('a.a','rb');
>> b = fread(f,2400,'uint8');
>> fclose(f);
>> a(1)
ans = 167
>> b(1)
ans = 167
>>

Dmitri.
--