bugGNU Octave - Bugs: bug #63930, fprintf writes incorrect...

 
 

bug #63930: fprintf writes incorrect characters when converting the encoding

Submitter:  Charles Praplan <charprap>
Submitted:  Wed 15 Mar 2023 05:09:22 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Charles Praplan Open/Closed:  * Closed
Release:  * 8.1.0 Operating System:  * Any
Fixed Release:  8.2.0 Planned Release:  8.2.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 04 May 2023 02:51:06 PM UTC, comment #75: 

The last changes for this report landed about a month ago. And Octave 8.2.0 has been released in the meantime.

Closing report as fixed.

Markus Mützel <mmuetzel>
Group administrator
Sat 08 Apr 2023 07:58:13 AM UTC, comment #74: 

I pushed two changes to the stable branch that implement the `HAVE_LLVM_LIBCXX` feature test for configure and makes the encoding conversion for streams opened with `fopen` conditional on it:
https://hg.savannah.gnu.org/hgweb/octave/rev/86eb373a6c64
https://hg.savannah.gnu.org/hgweb/octave/rev/ee187a104701

Let's see how that place out on the macOS runners.
If the corresponding tests pass, I think this is about all we should do to fix this on the stable branch.

If we want this to work independent of the used STL, we'd probably need a re-design of how the encoding conversion is currently implemented keeping the resolution for the defect report mentioned in comment #63 in mind.
But it'd probably be better to track that in a separate report.

Markus Mützel <mmuetzel>
Group administrator
Fri 07 Apr 2023 02:41:04 PM UTC, comment #73: 

Never mind. We could probably equivalently write that as `%!testif ; ! _have_feature_ ('XXX')`.

Markus Mützel <mmuetzel>
Group administrator
Fri 07 Apr 2023 02:37:26 PM UTC, comment #72: 

Is it possible to skip a test if a feature is present? I.e., if we were to add HAVE_LLVM_LIBCXX, would it be possible to skip certain tests if it is defined (like we currently to with `%!testif ; ! ismac ()`)?

I only find conditional tests that start with `%!testif HAVE_XXX`. But none like `%!testif ! HAVE_XXX` or whatever the syntax would be.

Markus Mützel <mmuetzel>
Group administrator
Thu 06 Apr 2023 06:07:52 PM UTC, comment #71: 

I'm not sure if it is worth distinguishing between Apple libc++ and LLVM libc++. But if you prefer, we could use HAVE_LLVM_LIBCXX for both. IIUC, Apple is trailing a bit behind when it comes to adopting newer standards. But other than that, both versions are pretty close afaict. (I could be wrong.)

I agree that the additional preprocessor macros with more specific names can help locating code segments that are affected by the same deficiency.
But for the %!testif conditions, those are probably not accessible. We'd need to use HAVE_LLVM_LIBCXX there afaict. But we can always add comments...

Markus Mützel <mmuetzel>
Group administrator
Thu 06 Apr 2023 05:27:48 PM UTC, comment #70: 

I think it is OK to add that check but maybe rename the macro to HAVE_APPLE_LIBCXX or HAVE_LLVM_LIBCXX?

Also, instead of using this macro directly in the Octave sources, I propose that we add something like the following to the oct-conf-post-private.in.h file:


#if defined (HAVE_LLVM_LIBCXX)
#  define HAVE_BROKEN_SOMETHING_OR_OTHER 1
...
#endif


and then use the more specific macro in the source files.  That way, we can add individual tests later without having to hunt down the places to change in the source files.  And maybe the intent of the preprocessor conditional is also somewhat clearer when reading the code.

We could do the same for any other instances where we are currently doing version checks or checks for non-specific macros like "#if defined (_APPLE_)", etc.

John W. Eaton <jwe>
Group administrator
Thu 06 Apr 2023 07:46:55 AM UTC, comment #69: 

The attached patch adds a configure check that would record with which STL Octave was built. That would allow testing for that on build time and on run time.

It would probably be better to use more specific tests for each feature we'd like to gate with this. But using the result from the proposed test would probably still be better than the "ismac()" checks that we are currently using.

Is this acceptable?


(file #54565)

Markus Mützel <mmuetzel>
Group administrator
Wed 05 Apr 2023 06:08:22 PM UTC, comment #68: 


comment #65:

> We'd probably also need a `HAVE_*` macro to be able to skip the corresponding test on that platform.
>


We already skipping some tests by using ismac() (which are
in reality should be "islibc++"). Perhaps this is just another one like that?

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 05 Apr 2023 05:28:00 PM UTC, comment #67: 

Thanks for testing. I'm happy this is working for you now.

This report is already pretty long and rather convoluted as it is. Please, open separate reports with feature requests (if there aren't already existing reports for them).


I pushed another patch that will hopefully avoid the random crashes with `libc++`. It won't fix the issue with empty files though:
https://hg.savannah.gnu.org/hgweb/octave/rev/701fbdfb3bc0

@jwe: Writing a feature test for this would probably be pretty intricate. Would it be ok to simply check if `_LIBCPP_VERSION` is defined instead of actually testing if an encoding facet on a file stream can convert from UTF-8 to a single byte encoding?

Markus Mützel <mmuetzel>
Group administrator
Wed 05 Apr 2023 04:38:20 PM UTC, comment #66: 

I made different tests with the latest version from buildbot.
In particular, I could write files with UTF-8 and Windows-1252 encoding without any problem.
Thank you !

The icing on the cake would be to have:
- an automatic detection of m-files encoding when loading them into the editor
-the possibility to change the encoding in the "save as" dialog box of the editor

Charles Praplan <charprap>
Wed 05 Apr 2023 08:38:09 AM UTC, comment #65: 

@dasergatskov: Thank you. Checking if `_LIBCPP_VERSION` is defined and emitting an error if `fopen` is used with an encoding (other than "UTF-8") would probably help to avoid unexpected behavior.
We'd probably also need a `HAVE_*` macro to be able to skip the corresponding test on that platform.

@charprap: Current nightlies are available from https://buildbot.octave.space/#/download now. The Windows versions shouldn't be affected by the remaining issues with `libc++`. They should be working correctly.

Markus Mützel <mmuetzel>
Group administrator
Tue 04 Apr 2023 06:49:23 PM UTC, comment #64: 
Dmitri A. Sergatskov <dasergatskov>
Tue 04 Apr 2023 06:42:32 PM UTC, comment #63: 

This looks like something isn't quite right in libc++:
https://github.com/llvm/llvm-project/blob/5c950a3127da7c4121da75df9751208ba2aa9cad/libcxx/include/locale#L4110

            do
            {
                const char_type* __e;
                __r = __cv_->out(__st_, this->pbase(), this->pptr(), __e,
                                        __extbuf_, __extbuf_ + __ebs_, __extbe);
                if (__e == this->pbase())
                    return traits_type::eof();
[...]
            } while (__r == codecvt_base::partial);


A this point, `this->pptr()` seems to point only one ahead of `this->pbase()` (inspecting with gdb - not sure why that is though). That means that we got an incomplete UTF-8 character. We need to reset `__e` to `this->pbase()` to restart with more characters in the buffer.
The next expression terminated the conversion.

We currently don't do the part about resetting `__e` correctly. That's the reason for the random crashes. With `libstdc++`, it is initialized reasonably and we don't need to touch it when not converting anything. But the standard doesn't seem to make any guarantees about that.

This might have worked before because we didn't reverse the `from_next` pointer ever. But that was wrong (because it could lead to incorrect conversions from partial multi-byte UTF-8 surrogates).

Looking at the following defect report and its resolution, libc++ might be working according to the standard:
https://cplusplus.github.io/LWG/issue76

However, `libstdc++` seems to do just fine with that situation.

I don't know what the best solution is now. 🤷‍♂️
We probably need to rethink the entire transcoding. It wouldn't even help if we switched to UTF-16 internally because the current approach would still not be standard compliant for characters outside the BMP. (UTF-32 might work.)

As a short-term workaround, it might make sense to disable the transcoding with libc++.
Is there a way to detect on compile time that we will be linking to libc++? Are there configure checks for that?

And we should try to not crash even with libc++. I can probably look at that part some time during this week.

Markus Mützel <mmuetzel>
Group administrator
Tue 04 Apr 2023 06:41:30 PM UTC, comment #62: 


comment #54:

> If I compile w/ "-ggdb3 -O0" it does not crash.
>
> If file is open with non-utf encoding, fprintf does not write anything into it:
>


>
> octave:48> fid = fopen('05.txt', 'w', 'n', 'Windows-1252')
> fid = 22
> octave:49> fprintf(fid, 'ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû')
> octave:50> f_error = fclose(fid)
> f_error = 0
>


>


The same with stock clang-15.0.6/libc++ (compiled with -O0)
But if I fprintf("fid", "1") first, it writes bot 1 and "latin1" string.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 04 Apr 2023 05:57:54 PM UTC, comment #61: 


comment #55:

> Thanks for testing.
> Is this a recent regression? Is it different with the code from a couple of weeks ago (or 8.1.0)?
>


With hg id
8e71a9fc470f (stable)

it writes OK:

05.txt: ISO-8859 text, with no line terminators


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 04 Apr 2023 05:38:21 PM UTC, comment #60: 

Here is with stock Fedora's clang 15.0.6:


Thread 8 "QThread" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffce3bf6c0 (LWP 271806)]
__memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:265
265             VMOVU   (%rsi), %VMM(0)
(gdb) thread applt all bt
Invalid thread ID: applt all bt
(gdb) thread apply all bt

Thread 9 (Thread 0x7fffcd3916c0 (LWP 271807) "QThread"):
#0  0x00007ffff2a5b838 in __GI___sigtimedwait (set=set@entry=0x7ffff4f0a488 <async_signals>, info=info@entry=0x7fffcd390220, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:31
#1  0x00007ffff2a5af04 in __GI___sigwait (set=0x7ffff4f0a488 <async_signals>, sig=0x7fffcd3902ec) at ../sysdeps/unix/sysv/linux/sigwait.c:28
#2  0x00007ffff4c98e3e in signal_watcher(void*) (arg=0x7ffff769dd36 <octave::generic_sig_handler(int)>) at ../liboctave/wrappers/cxx-signal-helpers.cc:179
#3  0x00007ffff2aa9c57 in start_thread (arg=<optimized out>) at pthread_create.c:444
#4  0x00007ffff2b2fa70 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7fffce3bf6c0 (LWP 271806) "QThread"):
#0  __memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:265
#1  0x00007ffff4c9a189 in octave_u8_conv_to_encoding_intern (tocode=0x7fffb48480a1 "windows-1252", handler=iconveh_question_mark, src=0x63 <error: Cannot access memory at address 0x63>, srclen=140736222075389, offsets=0x0, lengthp=0x7fffce3bb310) at ../liboctave/wrappers/uniconv-wrappers.c:63
#2  0x00007ffff4c8f414 in octave::string::codecvt_u8::do_out(__mbstate_t&, char const*, char const*, char const*&, char*, char*, char*&) const (this=<optimized out>, from=0x63 <error: Cannot access memory at address 0x63>, from_end=<optimized out>, from_next=<error reading variable: Cannot access memory at address 0xffffffff>, to=0x7fffb47f7050 "0\a", to_end=0x7fffb47f8050 "\020\020", to_next=@0x7fffce3bb388: 0x7fffb47f7050 "0\a") at ../liboctave/util/oct-string.cc:660
#3  0x00007ffff7657f35 in std::__1::codecvt<char, char, __mbstate_t>::out[abi:v15007](__mbstate_t&, char const*, char const*, char const*&, char*, char*, char*&) const (this=0x0, __st=..., __frm=0x7fffb48619fd "ind_protect_cleanup\n     fclose (id);\n   end_unwind_protect\n endif\nendfunction\n", __frm_end=0x0, __frm_nxt=<optimized out>, __to=0x0, __to_nxt=<optimized out>, __to_end=<optimized out>) at /usr/bin/../include/c++/v1/__locale:975
#4  std::__1::wbuffer_convert<octave::string::deletable_facet<octave::string::codecvt_u8>, char, std::__1::char_traits<char> >::overflow(int) (this=0x7fffb4842a80, __c=195) at /usr/bin/../include/c++/v1/locale:4125
#5  0x00007ffff2d5fb2e in std::__1::basic_streambuf<char, std::__1::char_traits<char> >::xsputn(char const*, long) (this=0x7fffb4842a80, __s=0x7fffb4842d10 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __n=128) at include/c++/v1/streambuf:467
#6  0x0000000000402c04 in std::__1::basic_streambuf<char, std::__1::char_traits<char> >::sputn[abi:v15007](char const*, long) (this=0x7fffb4842a80, __s=0x7fffb4842d10 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __n=128) at /usr/bin/../include/c++/v1/streambuf:232
#7  std::__1::__pad_and_output<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char) (__s=..., __s@entry=..., __ob=__ob@entry=0x7fffb4842d10 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __op=0x7fffb4842d10 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __oe=__oe@entry=0x7fffb4842d90 "", __iob=..., __fl=32 ' ') at /usr/bin/../include/c++/v1/locale:1451
#8  0x0000000000402a62 in std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) (__os=..., __str=0x7fffb4842d10 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __len=<optimized out>) at /usr/bin/../include/c++/v1/ostream:763
--Type <RET> for more, q to quit, c to continue without paging--c
#9  0x00007ffff762c714 in std::__1::operator<< <char, std::__1::char_traits<char>, std::__1::allocator<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (__os=..., __str=...) at /usr/bin/../include/c++/v1/ostream:1086
#10 octave::base_stream::do_printf(octave::printf_format_list&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=this@entry=0x7fffb46a41a0, fmt_list=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:6009
#11 0x00007ffff762ca80 in octave::base_stream::printf(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=0x7fffb46a41a0, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:6072
#12 0x00007ffff764d428 in octave::stream::printf(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=this@entry=0x7fffce3bb810, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:7238
#13 0x00007ffff764d4c5 in octave::stream::printf(octave_value const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=0x7fffce3bb810, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:7256
#14 0x00007ffff745976e in octave::printf_internal(octave::interpreter&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, int) (interp=..., who=..., args=..., nargout=nargout@entry=0) at ../libinterp/corefcn/file-io.cc:813
#15 0x00007ffff74594d5 in octave::Ffprintf(octave::interpreter&, octave_value_list const&, int) (interp=..., args=..., nargout=0) at ../libinterp/corefcn/file-io.cc:843
#16 0x00007ffff736df90 in octave::tree_evaluator::execute_builtin_function(octave_builtin&, int, octave_value_list const&) (this=0x7fffb4004510, builtin_function=..., nargout=0, args=...) at ../libinterp/parse-tree/pt-eval.cc:3444
#17 0x00007ffff726666a in octave_builtin::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-builtin.cc:49
#18 0x00007ffff72a4a10 in octave_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb40b8a60, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-fcn.cc:57
#19 0x00007ffff737799f in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb4646d70, tw=..., nargout=0) at ../libinterp/parse-tree/pt-idx.cc:427
#20 0x00007ffff737978b in octave::tree_index_expression::evaluate(octave::tree_evaluator&, int) (this=0x63, tw=..., nargout=0) at ../libinterp/parse-tree/pt-idx.h:108
#21 0x00007ffff736496f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#22 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#23 0x00007ffff73650f0 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x63, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#24 octave::tree_evaluator::visit_unwind_protect_command(octave::tree_unwind_protect_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4286
#25 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#26 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#27 0x00007ffff737384f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb48499a0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#28 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb4004510, rng=..., line=line@entry=14, ult=..., loop_body=loop_body@entry=0x7fffb48499a0) at ../libinterp/parse-tree/pt-eval.cc:3049
#29 0x00007ffff736395a in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#30 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#31 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#32 0x00007ffff737384f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb4818930, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#33 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb4004510, rng=..., line=line@entry=13, ult=..., loop_body=loop_body@entry=0x7fffb4818930) at ../libinterp/parse-tree/pt-eval.cc:3049
#34 0x00007ffff736395a in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#35 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#36 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#37 0x00007ffff736e82f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb4810c70, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#38 octave::tree_evaluator::execute_user_function(octave_user_function&, int, octave_value_list const&) (this=0x7fffb4004510, user_function=..., nargout=0, xargs=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3665
#39 0x00007ffff72e04a0 in octave_user_function::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:495
#40 0x00007ffff72e044e in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb481e660, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:488
#41 0x00007ffff737799f in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb483e650, tw=..., nargout=0) at ../libinterp/parse-tree/pt-idx.cc:427
#42 0x00007ffff7367e32 in octave::tree_evaluator::eval_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, int&, int) (this=0x7fffb4004510, eval_str=<optimized out>, silent=false, parse_status=@0x7fffce3bc87c: 0, nargout=0) at ../libinterp/parse-tree/pt-eval.cc:1053
#43 0x00007ffff736850c in octave::tree_evaluator::eval(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (this=0x63, try_code=..., nargout=0) at ../libinterp/parse-tree/pt-eval.cc:1112
#44 0x00007ffff75b90e4 in octave::interpreter::eval(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (this=<optimized out>, try_code=..., nargout=0) at ../libinterp/corefcn/interpreter.cc:1512
#45 0x00007ffff73550ca in octave::Feval(octave::interpreter&, octave_value_list const&, int) (interp=..., args=..., nargout=0) at ../libinterp/parse-tree/oct-parse.yy:6476
#46 0x00007ffff736df90 in octave::tree_evaluator::execute_builtin_function(octave_builtin&, int, octave_value_list const&) (this=0x7fffb4004510, builtin_function=..., nargout=0, args=...) at ../libinterp/parse-tree/pt-eval.cc:3444
#47 0x00007ffff726666a in octave_builtin::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-builtin.cc:49
#48 0x00007ffff72a4a10 in octave_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb4011af0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-fcn.cc:57
#49 0x00007ffff737799f in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb463f4b0, tw=..., nargout=0) at ../libinterp/parse-tree/pt-idx.cc:427
#50 0x00007ffff737978b in octave::tree_index_expression::evaluate(octave::tree_evaluator&, int) (this=0x63, tw=..., nargout=0) at ../libinterp/parse-tree/pt-idx.h:108
#51 0x00007ffff736496f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#52 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#53 0x00007ffff7364d80 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb463e510, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#54 octave::tree_evaluator::visit_try_catch_command(octave::tree_try_catch_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4141
#55 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#56 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#57 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#58 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#59 0x00007ffff73650f0 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x63, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#60 octave::tree_evaluator::visit_unwind_protect_command(octave::tree_unwind_protect_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4286
#61 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#62 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#63 0x00007ffff737384f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb45e8290, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#64 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb4004510, rng=..., line=line@entry=330, ult=..., loop_body=loop_body@entry=0x7fffb45e8290) at ../libinterp/parse-tree/pt-eval.cc:3049
#65 0x00007ffff736395a in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#66 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#67 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#68 0x00007ffff736e82f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb4339eb0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#69 octave::tree_evaluator::execute_user_function(octave_user_function&, int, octave_value_list const&) (this=0x7fffb4004510, user_function=..., nargout=2, xargs=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3665
#70 0x00007ffff72e04a0 in octave_user_function::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:495
#71 0x00007ffff72e044e in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb4558a70, tw=..., nargout=2, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:488
#72 0x00007ffff737799f in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb431afb0, tw=..., nargout=2) at ../libinterp/parse-tree/pt-idx.cc:427
#73 0x00007ffff735e98d in octave::tree_multi_assignment::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb45aa500, tw=...) at ../libinterp/parse-tree/pt-assign.cc:201
#74 0x00007ffff735f559 in octave::tree_multi_assignment::evaluate(octave::tree_evaluator&, int) (this=0x63, tw=..., nargout=0) at ../libinterp/parse-tree/pt-assign.h:156
#75 0x00007ffff736496f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#76 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#77 0x00007ffff7366de0 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x63, this@entry=0x7fffce3bdac8, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#78 octave::tree_evaluator::eval(std::__1::shared_ptr<octave::tree_statement_list>&, bool) (this=0x0, this@entry=0x7fffb4004510, stmt_list=..., interactive=true) at ../libinterp/parse-tree/pt-eval.cc:985
#79 0x00007ffff736759e in octave::tree_evaluator::repl() (this=0x7fffb4004510) at ../libinterp/parse-tree/pt-eval.cc:804
#80 0x00007ffff75b73a5 in octave::interpreter::main_loop() (this=0x7fffb4004370) at ../libinterp/corefcn/interpreter.cc:1350
#81 octave::interpreter::execute() (this=0x7fffb4004370) at ../libinterp/corefcn/interpreter.cc:916
#82 0x00007ffff7bf7d04 in octave::interpreter_qobject::execute() (this=0x6f6f00) at ../libgui/src/interpreter-qobject.cc:89
#83 0x00007ffff52df5db in QObject::event(QEvent*) (this=0x6f6f00, e=0x756270) at kernel/qobject.cpp:1347
#84 0x00007ffff5faeca5 in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=<optimized out>, receiver=0x6f6f00, e=0x756270) at kernel/qapplication.cpp:3640
#85 0x00007ffff7c3a01c in octave::octave_qapplication::notify(QObject*, QEvent*) (this=0x0, receiver=0x63, ev=0x7fffb48619fd) at ../libgui/src/octave-qobject.cc:147
#86 0x00007ffff52b3bd8 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0x6f6f00, event=0x756270) at kernel/qcoreapplication.cpp:1064
#87 0x00007ffff52b3df2 in QCoreApplication::sendEvent(QObject*, QEvent*) (receiver=<optimized out>, event=<optimized out>) at kernel/qcoreapplication.cpp:1462
#88 0x00007ffff52b7085 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) (receiver=0x0, event_type=0, data=0x53d280) at kernel/qcoreapplication.cpp:1821
#89 0x00007ffff52b733d in QCoreApplication::sendPostedEvents(QObject*, int) (receiver=<optimized out>, event_type=<optimized out>) at kernel/qcoreapplication.cpp:1680
#90 0x00007ffff5306cdf in postEventSourceDispatch(GSource*, GSourceFunc, gpointer) (s=0x7fffb4018cf0) at kernel/qeventdispatcher_glib.cpp:277
#91 0x00007ffff1e2af58 in g_main_dispatch (context=0x7fffb40174f0) at ../glib/gmain.c:3460
#92 g_main_context_dispatch (context=0x7fffb40174f0) at ../glib/gmain.c:4200
#93 0x00007ffff1e8acd8 in g_main_context_iterate.isra.0 (context=0x7fffb40174f0, block=1, dispatch=1, self=<optimized out>) at ../glib/gmain.c:4276
#94 0x00007ffff1e2c233 in g_main_context_iteration (context=0x7fffb40174f0, may_block=1) at ../glib/gmain.c:4343
#95 0x00007ffff53067c9 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) (this=0x7fffb40687c0, flags=...) at kernel/qeventdispatcher_glib.cpp:423
#96 0x00007ffff52b25ab in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) (this=this@entry=0x7fffce3be270, flags=..., flags@entry=...) at ../../include/QtCore/../../src/corelib/global/qflags.h:69
#97 0x00007ffff50f4110 in QThread::exec() (this=<optimized out>) at ../../include/QtCore/../../src/corelib/global/qflags.h:121
#98 0x00007ffff50f548d in QThreadPrivate::start(void*) (arg=0x756f60) at thread/qthread_unix.cpp:330
#99 0x00007ffff2aa9c57 in start_thread (arg=<optimized out>) at pthread_create.c:444
#100 0x00007ffff2b2fa70 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81


Dmitri.
--


Dmitri A. Sergatskov <dasergatskov>
Tue 04 Apr 2023 04:39:48 PM UTC, comment #59: 

Oops. Should have written "maximum of 128" (64 two-byte UTF-8 characters).

Markus Mützel <mmuetzel>
Group administrator
Tue 04 Apr 2023 04:38:23 PM UTC, comment #58: 

`srclen=140255252715919` doesn't look right. That looks more like it is a pointer address in the range of the other pointers appearing in the backtrace (0x7F8FB884198F).

It is calculated like this in `octave::string::codecvt_u8::do_out` in oct-string.cc:653:

  std::size_t srclen = (from_end-from-pop_end) * sizeof (InternT);


`pop_end` should be 0. `sizeof (InternT)` should be 1 (InternT is char).
So, it should be the difference between the addresses pointed to by `from_end` and `from`. Theoretically that should be a maximum of 64 in that example...

Markus Mützel <mmuetzel>
Group administrator
Tue 04 Apr 2023 04:12:01 PM UTC, comment #57: 

I do not compile against libc++ regularly. I learned that it is not really ABI compatible with libstdc++, only tries to be.
That is why I cannot link e.g. GraphicsMagick or cholmod.

I can try some earlier hg revisions.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 04 Apr 2023 04:05:35 PM UTC, comment #56: 

Compiling with "-ggdb3 -Os" crashes:

Thread 8 "QThread" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffcb7fe640 (LWP 1299975)]
0x00007ffff0ab9641 in __memmove_avx_unaligned () from /usr/lib64/libc.so.6

Thread 8 (Thread 0x7fffcb7fe640 (LWP 1299975) "QThread"):
#0  0x00007ffff0ab9641 in __memmove_avx_unaligned () at /usr/lib64/libc.so.6
#1  0x00007ffff4cf186b in octave_u8_conv_to_encoding_intern (tocode=0x7fffb87c5451 "windows-1252", handler=iconveh_question_mark, src=0x7000000101 <error: Cannot access memory at address 0x7000000101>, srclen=140255252715919, offsets=0x0, lengthp=0x7fffcb7fa340) at ../liboctave/wrappers/uniconv-wrappers.c:63
#2  0x00007ffff4ce6702 in octave::string::codecvt_u8::do_out(__mbstate_t&, char const*, char const*, char const*&, char*, char*, char*&) const (this=<optimized out>, from=0x7000000101 <error: Cannot access memory at address 0x7000000101>, from_end=<optimized out>, from_next=<error reading variable>, to=0x7fffb8840a80 "\200\a", to_end=0x7fffb8841a80 "\020\020", to_next=@0x7fffcb7fa3b8: 0x7fffb8840a80 "\200\a") at ../liboctave/util/oct-string.cc:660
#3  0x00007ffff76f2b9f in std::__1::codecvt<char, char, __mbstate_t>::out(__mbstate_t&, char const*, char const*, char const*&, char*, char*, char*&) const (this=0x0, __st=..., __frm=0x7f8fb884198f <error: Cannot access memory at address 0x7f8fb884198f>, __frm_end=0x7ffff0b47fe7 <mmap64+23> "H=", __frm_nxt=<optimized out>, __to=0x0, __to_nxt=<optimized out>, __to_end=<optimized out>) at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/__locale:975
#4  std::__1::wbuffer_convert<octave::string::deletable_facet<octave::string::codecvt_u8>, char, std::__1::char_traits<char> >::overflow(int) (this=0x7fffb87ed840, __c=195) at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/locale:4102
#5  0x00007ffff0fa1cae in std::__1::basic_streambuf<char, std::__1::char_traits<char> >::xsputn(char const*, long) () at /opt/AMD/aocc-compiler-4.0.0/lib/libc++.so.1
#6  0x00000000002034ad in std::__1::basic_streambuf<char, std::__1::char_traits<char> >::sputn(char const*, long) (this=0x7fffb87ed840, __s=0x7fffb8861250 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __n=128) at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/streambuf:231
#7  std::__1::__pad_and_output<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char) (__s=..., __s@entry=..., __ob=__ob@entry=0x7fffb8861250 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __op=0x7fffb8861250 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __oe=__oe@entry=0x7fffb88612d0 "", __iob=..., __fl=32 ' ') at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/locale:1440
#8  0x000000000020330b in std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) (__os=..., __str=0x7fffb8861250 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __len=<optimized out>) at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/ostream:726
#9  0x00007ffff76c7673 in std::__1::operator<< <char, std::__1::char_traits<char>, std::__1::allocator<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (__os=..., __str=...) at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/ostream:1049
#10 octave::base_stream::do_printf(octave::printf_format_list&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=this@entry=0x7fffb8844010, fmt_list=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:6009
#11 0x00007ffff76c7a20 in octave::base_stream::printf(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=0x7fffb8844010, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:6072
#12 0x00007ffff76e837a in octave::stream::printf(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=this@entry=0x7fffcb7fa830, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:7238
#13 0x00007ffff76e8417 in octave::stream::printf(octave_value const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=0x7fffcb7fa830, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:7256
#14 0x00007ffff74e9892 in octave::printf_internal(octave::interpreter&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, int) (interp=..., who=..., args=..., nargout=nargout@entry=0) at ../libinterp/corefcn/file-io.cc:813
#15 0x00007ffff74e95ee in octave::Ffprintf(octave::interpreter&, octave_value_list const&, int) (interp=..., args=..., nargout=0) at ../libinterp/corefcn/file-io.cc:843
#16 0x00007ffff73fbfac in octave::tree_evaluator::execute_builtin_function(octave_builtin&, int, octave_value_list const&) (this=0x7fffb8056520, builtin_function=..., nargout=0, args=...) at ../libinterp/parse-tree/pt-eval.cc:3444
#17 0x00007ffff72f1862 in octave_builtin::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=-256606233, args=...) at ../libinterp/octave-value/ov-builtin.cc:49
#18 0x00007ffff732ffc8 in octave_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb80bdb20, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-fcn.cc:57
#19 0x00007ffff7405bc7 in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb883b680, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.cc:427
#20 0x00007ffff740793d in octave::tree_index_expression::evaluate(octave::tree_evaluator&, int) (this=0x7000000101, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.h:108
#21 0x00007ffff73f2899 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#22 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#23 0x00007ffff73f2ffa in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7000000101, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#24 octave::tree_evaluator::visit_unwind_protect_command(octave::tree_unwind_protect_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4286
#25 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#26 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#27 0x00007ffff7401917 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb88189a0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
--Type <RET> for more, q to quit, c to continue without paging--c
#28 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb8056520, rng=..., line=line@entry=14, ult=..., loop_body=loop_body@entry=0x7fffb88189a0) at ../libinterp/parse-tree/pt-eval.cc:3049
#29 0x00007ffff73f1876 in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#30 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#31 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#32 0x00007ffff7401917 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb87494d0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#33 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb8056520, rng=..., line=line@entry=13, ult=..., loop_body=loop_body@entry=0x7fffb87494d0) at ../libinterp/parse-tree/pt-eval.cc:3049
#34 0x00007ffff73f1876 in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#35 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#36 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#37 0x00007ffff73fc759 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb88053d0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#38 octave::tree_evaluator::execute_user_function(octave_user_function&, int, octave_value_list const&) (this=0x7fffb8056520, user_function=..., nargout=0, xargs=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3665
#39 0x00007ffff736c408 in octave_user_function::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=-256606233, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:495
#40 0x00007ffff736c3b6 in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb881c7f0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:488
#41 0x00007ffff7405bc7 in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb86e9600, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.cc:427
#42 0x00007ffff73f5c8a in octave::tree_evaluator::eval_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, int&, int) (this=0x7fffb8056520, eval_str=<optimized out>, silent=false, parse_status=@0x7fffcb7fb87c: 0, nargout=0) at ../libinterp/parse-tree/pt-eval.cc:1053
#43 0x00007ffff73f62ea in octave::tree_evaluator::eval(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (this=0x7000000101, try_code=..., nargout=0) at ../libinterp/parse-tree/pt-eval.cc:1112
#44 0x00007ffff764b5bc in octave::interpreter::eval(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (this=<optimized out>, try_code=..., nargout=-256606233) at ../libinterp/corefcn/interpreter.cc:1512
#45 0x00007ffff73e2b75 in octave::Feval(octave::interpreter&, octave_value_list const&, int) (interp=..., args=..., nargout=-256606233) at ../libinterp/parse-tree/oct-parse.yy:6476
#46 0x00007ffff73fbfac in octave::tree_evaluator::execute_builtin_function(octave_builtin&, int, octave_value_list const&) (this=0x7fffb8056520, builtin_function=..., nargout=0, args=...) at ../libinterp/parse-tree/pt-eval.cc:3444
#47 0x00007ffff72f1862 in octave_builtin::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=-256606233, args=...) at ../libinterp/octave-value/ov-builtin.cc:49
#48 0x00007ffff732ffc8 in octave_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb8070870, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-fcn.cc:57
#49 0x00007ffff7405bc7 in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb863b7e0, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.cc:427
#50 0x00007ffff740793d in octave::tree_index_expression::evaluate(octave::tree_evaluator&, int) (this=0x7000000101, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.h:108
#51 0x00007ffff73f2899 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#52 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#53 0x00007ffff73f2c9f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb863a840, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#54 octave::tree_evaluator::visit_try_catch_command(octave::tree_try_catch_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4141
#55 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#56 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#57 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#58 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#59 0x00007ffff73f2ffa in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7000000101, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#60 octave::tree_evaluator::visit_unwind_protect_command(octave::tree_unwind_protect_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4286
#61 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#62 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#63 0x00007ffff7401917 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb85e44e0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#64 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb8056520, rng=..., line=line@entry=330, ult=..., loop_body=loop_body@entry=0x7fffb85e44e0) at ../libinterp/parse-tree/pt-eval.cc:3049
#65 0x00007ffff73f1876 in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#66 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#67 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#68 0x00007ffff73fc759 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb834c1b0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#69 octave::tree_evaluator::execute_user_function(octave_user_function&, int, octave_value_list const&) (this=0x7fffb8056520, user_function=..., nargout=2, xargs=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3665
#70 0x00007ffff736c408 in octave_user_function::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=-256606233, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:495
#71 0x00007ffff736c3b6 in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb8530ed0, tw=..., nargout=2, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:488
#72 0x00007ffff7405bc7 in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb8384770, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.cc:427
#73 0x00007ffff73ec791 in octave::tree_multi_assignment::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb8576580, tw=...) at ../libinterp/parse-tree/pt-assign.cc:201
#74 0x00007ffff73ed393 in octave::tree_multi_assignment::evaluate(octave::tree_evaluator&, int) (this=0x7000000101, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-assign.h:156
#75 0x00007ffff73f2899 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#76 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#77 0x00007ffff73f4cb8 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7000000101, this@entry=0x7fffcb7fcaa8, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#78 octave::tree_evaluator::eval(std::__1::shared_ptr<octave::tree_statement_list>&, bool) (this=0x0, this@entry=0x7fffb8056520, stmt_list=..., interactive=true) at ../libinterp/parse-tree/pt-eval.cc:985
#79 0x00007ffff73f5454 in octave::tree_evaluator::repl() (this=0x7fffb8056520) at ../libinterp/parse-tree/pt-eval.cc:804
#80 0x00007ffff7649877 in octave::interpreter::main_loop() (this=0x7fffb8056380) at ../libinterp/corefcn/interpreter.cc:1350
#81 octave::interpreter::execute() (this=0x7fffb8056380) at ../libinterp/corefcn/interpreter.cc:916
#82 0x00007ffff7d0baec in octave::interpreter_qobject::execute() (this=0x57c530) at ../libgui/src/interpreter-qobject.cc:89
#83 0x00007ffff50d1a99 in QObject::event(QEvent*) (this=0x57c530, e=0x7b0c40) at kernel/qobject.cpp:1314
#84 0x00007ffff5dae423 in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=<optimized out>, receiver=0x57c530, e=0x7b0c40) at kernel/qapplication.cpp:3637
#85 0x00007ffff7d6f204 in octave::octave_qapplication::notify(QObject*, QEvent*) (this=0x0, receiver=0x7000000101, ev=0x7f8fb884198f) at ../libgui/src/octave-qobject.cc:147
#86 0x00007ffff50a7358 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0x57c530, event=0x7b0c40) at kernel/qcoreapplication.cpp:1064
#87 0x00007ffff50aa8b6 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) (receiver=0x0, event_type=0, data=0x636400) at kernel/qcoreapplication.cpp:1821
#88 0x00007ffff50f8fa7 in postEventSourceDispatch(GSource*, GSourceFunc, gpointer) (s=0x7fffb80121f0) at kernel/qeventdispatcher_glib.cpp:277
#89 0x00007ffff051ae2f in g_main_context_dispatch () at /usr/lib64/libglib-2.0.so.0
#90 0x00007ffff056f508 in g_main_context_iterate.constprop () at /usr/lib64/libglib-2.0.so.0
#91 0x00007ffff05185f3 in g_main_context_iteration () at /usr/lib64/libglib-2.0.so.0
#92 0x00007ffff50f8a48 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) (this=0x7fffb806cd30, flags=...) at kernel/qeventdispatcher_glib.cpp:423
#93 0x00007ffff50a5d62 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) (this=this@entry=0x7fffcb7fd210, flags=..., flags@entry=...) at ../../include/QtCore/../../src/corelib/global/qflags.h:69
#94 0x00007ffff4ee89ca in QThread::exec() (this=<optimized out>) at ../../include/QtCore/../../src/corelib/global/qflags.h:121
#95 0x00007ffff4ee9bf9 in QThreadPrivate::start(void*) (arg=0x5137c0) at thread/qthread_unix.cpp:331
#96 0x00007ffff0a9f832 in start_thread () at /usr/lib64/libc.so.6
#97 0x00007ffff0a3f450 in clone3 () at /usr/lib64/libc.so.6


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 04 Apr 2023 04:01:33 PM UTC, comment #55: 

Thanks for testing.
Is this a recent regression? Is it different with the code from a couple of weeks ago (or 8.1.0)?

Markus Mützel <mmuetzel>
Group administrator
Tue 04 Apr 2023 03:43:35 PM UTC, comment #54: 

If I compile w/ "-ggdb3 -O0" it does not crash.

If file is open with non-utf encoding, fprintf does not write anything into it:



octave:48> fid = fopen('05.txt', 'w', 'n', 'Windows-1252')
fid = 22
octave:49> fprintf(fid, 'ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû')
octave:50> f_error = fclose(fid)
f_error = 0


and file 05.txt. (I tried few other encodings with the same result). If I omit encoding, it would write utf-8 string:


octave:54> fid = fopen('06.txt', 'w', 'n')
fid = 22
octave:55> fprintf(fid, 'ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû')
octave:56> f_error = fclose(fid)
f_error = 0



$ file 06.txt
06.txt: UTF-8 Unicode text, with no line terminators
$ cat 06.txt
ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû


This is with AMD clang/libc

$ clang -v
AMD clang version 14.0.6 (CLANG: AOCC_4.0.0-Build#434 2022_10_28) (based on LLVM Mirror.Version.14.0.6)



Dmitri.
--


Dmitri A. Sergatskov <dasergatskov>
Tue 04 Apr 2023 02:48:17 PM UTC, comment #53: 

I get this crash on linux if I compile against libc++.
I will try to get some backtrace.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 04 Apr 2023 11:45:28 AM UTC, comment #52: 

The previous comment was from the logs of a macOS 11 runner.
It looks like that test crashed on a macOS 12 runner:

fatal: caught signal Segmentation fault: 11 -- stopping myself...
/bin/sh: line 1: 81520 Segmentation fault: 11  ( /bin/sh ../run-octave --no-init-file --silent --no-history -p /Users/runner/work/octave/octave/.build/test/mex /Users/runner/work/octave/octave/.build/../test/fntests.m /Users/runner/work/octave/octave/.build/../test )
make[4]: *** [check-local] Error 139
  io.tst .........................................................
make[3]: *** [check-am] Error 2
make[2]: *** [check] Error 2
make[1]: *** [check-recursive] Error 1
make: *** [check] Error 2


Ooof.

Is there something related that is specifically different on macOS compared to other platforms?

Markus Mützel <mmuetzel>
Group administrator
Tue 04 Apr 2023 11:34:25 AM UTC, comment #51: 

The new test failed on macOS:
https://github.com/gnu-octave/octave/actions/runs/4599036971/jobs/8123896137#step:13:90687

***** test <*63930>
 w_modes = {"wb", "wt"};
 # 64 non-ASCII characters that can be represented in 'windows-1252'
 f_texts{1} = repmat ('ÀÂÈÊÌàäéèêìîöòùû', 1, 4);
 # prepend space to misalign surrogate border from a multiple of 2.
 f_texts{2} = [' ', f_texts{1}];
 # byte values of character sequence in 'windows-1252'
 native_bytes{1} = repmat (...
   [192 194 200 202 204 224 228 233 232 234 236 238 246 242 249 251], ...
   1, 4).';
 native_bytes{2} = [double(' '); native_bytes{1}];
 for i_mode = 1:numel (w_modes)
   for i_text = 1:numel (f_texts)
     fname = tempname ();
     fid = fopen (fname, w_modes{i_mode}, 'n', 'windows-1252');
     unwind_protect
       fprintf (fid, f_texts{i_text});
       fclose (fid);
       # open without encoding facet and read bytes
       fid = fopen (fname, 'rb');
       buf = fread (fid);
       assert (buf, native_bytes{i_text});
     unwind_protect_cleanup
       fclose (fid);
       unlink (fname);
     end_unwind_protect
   endfor
 endfor
!!!!! regression: https://octave.org/testfailure/?63930
ASSERT errors for:  assert (buf,native_bytes {i_text})

  Location  |  Observed  |  Expected  |  Reason
     .          O(0x1)      E(64x1)      Dimensions don't match


That is the "baseline" test (similar to `fprintf_bug4.m` from comment #4). That was already working correctly on Windows and Linux before the recent changes.

Did that ever work on macOS? Or is this a regression from the changes here?
Is someone able to run that test with Octave 8.1.0 on macOS?

Markus Mützel <mmuetzel>
Group administrator
Tue 04 Apr 2023 09:04:10 AM UTC, comment #50: 

The octave.space builder is back online and is currently part way through a build of revision 63038dcbd648.

John W. Eaton <jwe>
Group administrator
Mon 03 Apr 2023 04:26:53 PM UTC, comment #49: 

I moved the workaround for the gnulib/iconv behavior a level down to the wrapper split it into two patches and added test cases for both changes:
https://hg.savannah.gnu.org/hgweb/octave/rev/470134b3fc28
https://hg.savannah.gnu.org/hgweb/octave/rev/63038dcbd648

I hope I didn't introduce unexpected regressions.

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Mon 03 Apr 2023 08:55:54 AM UTC, comment #48: 

The change that defaults to UTF-8 encoded output files (Matlab compatible) already landed on the stable branch (see comment #40).

If you are on Windows, you could wait for a nightly build from after 31 Mar 2023:
https://buildbot.octave.space/#/download

Unfortunately, the worker that usually builds those nightlies is currently down due to a power outage with no ETA when it will come back online:
https://octave.discourse.group/t/buildbot-tempo-question/4176/9

Alternatively, you could use the MSYS2 build artifact from the GitHub-CI like described here:
https://wiki.octave.org/Continuous_Build#GitHub_build_artifacts


For the patch from comment #46, you'd need to build Octave yourself.

Markus Mützel <mmuetzel>
Group administrator
Sun 02 Apr 2023 03:48:38 PM UTC, comment #47: 

Is there a way to easily test this for a non SW-developer (i.e. without having to rebuild Octave)?

Charles Praplan <charprap>
Sun 02 Apr 2023 01:23:28 PM UTC, comment #46: 

The attached patch handles invalid UTF-8 better than the previous one and also fixes a couple of thinkos.

Trying to get this correct for all cases isn't trivial. I removed one FIXME and added five new ones. I guess that is another proof for the 80-20 rule: It takes about 20% of the time to get something 80% right. But 80% of the time to get the last 20%.

Anyway, I still think this patch is an improvement upon what we currently have because the remaining issues are in more niche cases compared to the more general issue before.

Comments and testing is still appreciated.


(file #54550)

Markus Mützel <mmuetzel>
Group administrator
Sun 02 Apr 2023 11:27:50 AM UTC, comment #45: 

There were still some issues with the latest patch. But the issue I am currently stuck with seems to be with the conversion functions from gnulib.
Save the following code to, e.g., `test-u8-conv.c`:

#include "config.h"

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include "uniconv.h"

int main (void)
{
  char *tocode = "windows-1252";
  char src[3] = {230, 231, 10};
  size_t lengthp = 0;

  char *out_str = u8_conv_to_encoding (tocode, iconveh_question_mark,
                                       src, 3, NULL, NULL, &lengthp);

  printf ("lengthp = %d\n", lengthp);
  printf ("out_str = {");
  for (size_t i_out = 0; i_out < lengthp; i_out++)
      printf ("%d, ", out_str[i_out]);
  printf ("}\n");

  src[1] = 10;
  out_str = u8_conv_to_encoding (tocode, iconveh_question_mark,
                                 src, 2, NULL, NULL, &lengthp);

  printf ("lengthp = %d\n", lengthp);
  printf ("out_str = {");
  for (size_t i_out = 0; i_out < lengthp; i_out++)
      printf ("%d, ", out_str[i_out]);
  printf ("}\n");

  return 0;
}


And compile it in the Octave build tree with:

gcc test-u8-conv.c -otest_u8.exe -Ilibgnu -I../gnulib/lib -Llibgnu/.libs -lgnu -liconv


The output for me (on Windows 11 with MinGW64 64-bit):

$ ./test_u8.exe
lengthp = 3
out_str = {63, 63, 10, }
lengthp = 0
out_str = {}


The first conversion replaced the invalid UTF-8 characters with 63 (i.e., "?").
I'd have expected that the second conversion resulted in {63, 10} (i.e., "?\n"). But it returns an empty string.

We could try to implement a workaround. But should this be fixed in gnulib instead?

Markus Mützel <mmuetzel>
Group administrator
Sun 02 Apr 2023 09:44:04 AM UTC, comment #44: 

There are still issues with the latest patch if the string to be written contains invalid UTF-8 at the end of the 32 byte buffer chunks.
I'm looking into it currently.

Markus Mützel <mmuetzel>
Group administrator
Sun 02 Apr 2023 08:10:04 AM UTC, comment #43: 

Could you please elaborate on which parts of the character encoding conversion are still buggy. It is an intrinsic property of conversions between single byte encodings and variable byte encodings (like UTF-8) that the number of bytes might change that are used to encode a given character. That alone is not a bug afaict.

Markus Mützel <mmuetzel>
Group administrator
Sat 01 Apr 2023 11:51:30 PM UTC, comment #42: 

This solves the reported problem and is probably good enough for 8.x release. But the whole encoding issue remains buggy
(e.g. saving and loading a char array that contains non-utf-8
chars results in conversion of this array to utf-8 and thus increasing it size). So we should advise people os trying to stick to UTF-8 encoding everywhere.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 01 Apr 2023 03:02:12 PM UTC, comment #41: 

The attached patch checks if the buffer ends with a complete UTF-8 surrogate when transcoding.
With it, a modified version of `fprintf_bug5.m` outputs a file that looks correct to me.
Here, the modified version (note the specified encoding in the `fopen` command):

close all
clear
clc

fid = fopen('05.txt', 'w', 'n', 'Windows-1252');

fprintf(fid, 'ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû');
fprintf(fid, 'ÈÊÌàäîöòùûÀÂÈÊÌàäéèêìîöòéèêìùûÀÂ');
fprintf(fid, 'àäéèêìîöûÀÂÈÊÌàäéèòùûÀÂîöòùêìÈÊÌ');
fprintf(fid, 'ìîéèÌàäéê îöòùûÀÂöòùûÀÂÈÊÌàäÈÊèê');
fprintf(fid, 'öòùûÀÂàäéÊèÈÊêìîöòùûÀÂÈÌàäéèÌêìî');
fprintf(fid, 'ÊÌàäùûÀÂÈéèÌàöòùûÀäéêìîÂÈÊèêìîöò');

fprintf(fid, 'ÌàäùûÀÂÈÊÌàÀÂéèêìêìîöòùîöòÈÊäéèû');
fprintf(fid, 'äéèÀÂÈÊÌàäéÈÊêìîöîöòùûÀòùûÌàèêìÂ');
fprintf(fid, 'êìîÊÌàäéèêìàäöòùûùûÀÂÈÊÀÂÈéèîöòÌ');
fprintf(fid, 'ùûÀèÌàäéêìîìîÂÈÊÌûÀÂÈÊèàäéöòöòùê');
fprintf(fid, 'ÀÂÈàäéêìîöòöòÊÌàäÂÈÊèêìéèÌùûùûÀî');
fprintf(fid, 'ÈÊÌéêìîöòùûùûàäéèÊèêìîöÌàäÀÂÀÂÈò');

f_error = fclose(fid);


I've only run some preliminary tests though. E.g., I haven't checked with invalid UTF-8 sequences somewhere in or at the end of the output string.
Any feedback or help in testing is appreciated.


(file #54541)

Markus Mützel <mmuetzel>
Group administrator
Fri 31 Mar 2023 04:13:32 PM UTC, comment #40: 

I pushed a change to the stable branch that defaults the encoding to UTF-8 when opening streams with `fopen`:
https://hg.savannah.gnu.org/hgweb/octave/rev/0ed6d1f01625

That fixes the issue with the scripts from comment #0 for me. The files are successfully written (in UTF-8).
That is compatible to newer versions of Matlab which also write in UTF-8 if no differing encoding is specified when opening the stream with `fopen`.

That will mitigate this regression for users that don't specify an encoding when opening a stream with `fopen`. I'd guess that is the large majority of cases.

With that in place, we could probably revert https://hg.savannah.gnu.org/hgweb/octave/rev/fcd97a68e5f7 to restore Matlab compatibility without re-introducing the regression for the large majority of cases.

Re-titling the report to reflect the remaining issue.

Markus Mützel <mmuetzel>
Group administrator
Fri 31 Mar 2023 01:26:39 PM UTC, comment #39: 


> The fact that "t" changed encodings in octave is manifestation of other bugs.


Did it change something before https://hg.savannah.gnu.org/hgweb/octave/rev/fcd97a68e5f7 ?

Given that it looks like the output stream is converted in chunks of 32 bytes (see comment #8), a single byte "\n" in contrast to two bytes "\r\n" might have pushed the issue to a position where it didn't matter...

Markus Mützel <mmuetzel>
Group administrator
Fri 31 Mar 2023 01:21:49 PM UTC, comment #38: 

As I mentioned in comment #28 https://savannah.gnu.org/bugs/?63930#comment28 all the "t" mode does is changes (or not) line endings \n <--> \r\n.
From: https://www.mathworks.com/help/matlab/ref/fopen.html

<<<
On Windows® systems, in text mode:

Read operations that encounter a carriage return followed by a newline character ('\r\n') remove the carriage return from the input.

Write operations insert a carriage return before any newline character in the output.

Open or create a new file in text mode if you want to write to it in MATLAB and then open it in Microsoft® Notepad, or any text editor that does not recognize '\n' as a newline sequence. When writing to the file, end each line with '\r\n'. For an example, see fprintf. Otherwise, open files in binary mode for better performance.

>>>


The fact that "t" changed encodings in octave is manifestation of other bugs.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 31 Mar 2023 01:12:43 PM UTC, comment #37: 

I saved the following to `Test_encoding.m`:

disp('text mode - UTF-8')
fid = fopen('textmode_utf8.tst', 'wt', 'n', 'UTF-8');
fwrite(fid, 'ä');
fclose(fid);
fid = fopen('textmode_utf8.tst', 'rb');
fread(fid)
fclose(fid);

disp('binary mode - UTF-8')
fid = fopen('binarymode_utf8.tst', 'wb', 'n', 'UTF-8');
fwrite(fid, 'ä');
fclose(fid);
fid = fopen('binarymode_utf8.tst', 'rb');
fread(fid)
fclose(fid);

disp('text mode - Latin1')
fid = fopen('textmode_latin1.tst', 'wt', 'n', 'Latin1');
fwrite(fid, 'ä');
fclose(fid);
fid = fopen('textmode_latin1.tst', 'rb');
fread(fid)
fclose(fid);

disp('binary mode - Latin1')
fid = fopen('binarymode_latin1.tst', 'wb', 'n', 'Latin1');
fwrite(fid, 'ä');
fclose(fid);
fid = fopen('binarymode_latin1.tst', 'rb');
fread(fid)
fclose(fid);


Running it in Matlab R2022a resulted in the following:

>> Test_encoding
text mode - UTF-8

ans =

   195
   164

binary mode - UTF-8

ans =

   195
   164

text mode - Latin1

ans =

   228

binary mode - Latin1

ans =

   228

>>


So, Matlab seems to transcode strings independent on whether a file stream was opened in text mode or in binary mode.

Should we revert https://hg.savannah.gnu.org/hgweb/octave/rev/fcd97a68e5f7 in favor of a different change? At least, on the default branch?
We could keep that change to mitigate the effects of this issue on the stable branch.

Markus Mützel <mmuetzel>
Group administrator
Tue 28 Mar 2023 03:10:35 PM UTC, comment #36: 

I'm assuming that you mean "char arrays" when you write "strings".

That'd be a major change. We can't do that to fix the issue in a dot-release. But maybe you could bring that up in the next online meeting?

As a "low-hanging" short-term solution, defaulting to no conversion again (i.e., UTF-8 output independent of the value set for `__mfile_encoding__`) might help to reduce the number of users that are affected.

Markus Mützel <mmuetzel>
Group administrator
Tue 28 Mar 2023 02:57:58 PM UTC, comment #35: 

I meant converting Octave to use UTF-16 as internal encoding for the strings.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 28 Mar 2023 02:55:28 PM UTC, comment #34: 

I might be missing the point. But why are you expecting less issues when converting a UTF-8 stream "on-the-fly" to UTF-16 compared to any other target encoding? The "end of buffer" issue would be exactly the same afaict.
Or are you voting for changing the internal representation of char arrays in Octave to UTF-16?

Markus Mützel <mmuetzel>
Group administrator
Tue 28 Mar 2023 02:14:56 PM UTC, comment #33: 

I would vote to convert strings to UTF-16 first.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 28 Mar 2023 01:46:22 PM UTC, comment #32: 

Sorry for the long text. But this is kind of complicated and we should try to give it some structure if we like to fix this.

Re-encoding the .m files doesn't necessarily mean that the output encoding changes. We should really separate this into "independent" steps in our minds:

1. The parser reads the code from the file.
In this step, the input encoding matters. That is what `__mfile_encoding__` was originally meant to affect.
In older versions of Matlab and Octave this was always the encoding matching the system locale. In newer versions of Matlab, it seems to use heuristics to parse files either in the encoding matching the system locale or UTF-8. In "new-ish" versions of Octave, the encoding for the parser can be set using the `__mfile_encoding__` function.

2. Content is written to an output stream with the `f*` family of functions.
In this step, the output encoding matters.
In Matlab, this always happened (and still does) using the encoding matching the system locale afaict. In older versions of Octave, `char` arrays were actually treated as byte arrays. So, they were just written as is. But that caused issues when we actually needed to know which string a given byte array represented. So, since some time `char` arrays are UTF-8 encoded strings in Octave. Since before Octave 8 (or 7?), the `f*` family of functions didn't convert anything. So, they either wrote whatever bytes the `char` array contained (in old-Octave) - or UTF-8 (in newer Octave).
However, the latter caused incompatibilities with Matlab (which writes in the system locale by default). (Or at least, it used to do that at some point.)
To be closer to Matlab's behavior, Octave was changed to use the encoding selected by `__mfile_encoding__` (i.e., the input encoding used by the parser) also for writing to files by default.
Since Octave now also implements the option to specify the encoding when opening a stream, it is also possible to specify an encoding differing from the encoding used by the parser.

There is also a third case where encoding matters: the encoding the built-in editor uses when saving files. Theoretically, that could be yet another differing encoding. But to avoid surprises, we decided to keep the encoding that the parser uses and the encoding the editor uses in sync.

Additionally, there is a difference between using the GUI and the CLI: The interpreter itself doesn't have options that persist between sessions. The GUI does. That means that if you change the encoding used by the built-in editor (and hence the parser), that setting will persist when you re-start Octave. In contrast, if you change the encoding used by the parser with `__mfile_encoding__` in the CLI, that setting won't persist (unless you do that in one of the startup files).

As a side note: The reason why this seems to be less of an issue for Matlab might be that they are using UTF-16 encoded strings for the internal representation of their char arrays. Most strings consist only (or mainly) of characters from within the BMP. So, they are represented by a single code unit in UTF-16 (compared to surrogate pairs for characters outside the BMP).
Octave on the other hand chose to use UTF-8 for which only ASCII characters can be represented with a single code unit (a byte). All other characters need 2 to 4 bytes to be represented. So, for Octave it is "much more likely" that a buffer might "happen to end" in the middle of a multi-byte sequence.


Skipping the encoding for byte streams is probably a step in the correct direction. But it might still make sense to check what Matlab is doing. E.g., does writing "ä" (or any other multi-byte character) to a byte stream with encoding set to "UTF-8" differ from a file that is written as a text stream with the same encoding?

For the remaining issue of how to correctly convert the encoding on-the-fly when writing to a stream, the following "solutions" come to mind:
1. Change the internal representation of character arrays in Octave to UTF-16 (from currently UTF-8). That would make it less likely for the internal buffer to end "in the middle of a code point". And it would also make Octave more similar to Matlab in that respect.
A couple of years back, we decided against that because we were trying to avoid character conversions on too many interfaces. But now, the conversion facilities in Octave are ready (and we are converting on many interfaces anyway). That won't be something we can even consider to do for a minor release.

2. Try to improve our `codecvt_u8` that manages the conversion from or to UTF-8. I haven't looked into that (and probably won't be able to do so in the nearest future).
But the changes would probably need to happen close to this FIXME note:
https://hg.savannah.gnu.org/hgweb/octave/file/fcd97a68e5f7/liboctave/util/oct-string.cc#l635

3. As a short term "solution", we could change the default for output streams back to UTF-8. If users need to write in a different encoding, using `fflush` on the stream often might be a (unreliable) work-around.

Markus Mützel <mmuetzel>
Group administrator
Sat 25 Mar 2023 01:03:09 AM UTC, comment #31: 

When you re-encoded m-file you just actually replaced chars in the strings that being printed out. Of course that will change output files encodings as well.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 25 Mar 2023 12:25:55 AM UTC, comment #30: 

I rolled all the different fprintf tests in this thread into one file called fprintf_test.m. Initially that file was ISO-8859 encoded.


$ octave --eval "__mfile_encoding__ ('latin1'), __mfile_encoding__, fprintf_test"
ans = latin1

$ file *
01_D.txt:       ISO-8859 text
02_D.txt:       ISO-8859 text
04.txt:         ISO-8859 text, with very long lines (384), with no line terminators
05.txt:         ISO-8859 text, with very long lines (384), with no line terminators
06.txt:         ISO-8859 text, with very long lines (384), with no line terminators
07.txt:         ISO-8859 text, with very long lines (384), with no line terminators
fprintf_test.m: ISO-8859 text, with CRLF line terminators


Then I opened fprintf_test.m in a text editor and saved it as UTF-8 with Unix line endings.


$ gedit fprintf_test.m   # saved as UTF-8 with Unix line endings

$ file *
01_D.txt:       ISO-8859 text
02_D.txt:       ISO-8859 text
04.txt:         ISO-8859 text, with very long lines (384), with no line terminators
05.txt:         ISO-8859 text, with very long lines (384), with no line terminators
06.txt:         ISO-8859 text, with very long lines (384), with no line terminators
07.txt:         ISO-8859 text, with very long lines (384), with no line terminators
fprintf_test.m: Unicode text, UTF-8 text


Now running the same command again:

$ octave --eval "__mfile_encoding__ ('latin1'), __mfile_encoding__, fprintf_test"
ans = latin1

$ file *txt
01_D.txt: ISO-8859 text
02_D.txt: Unicode text, UTF-8 text
04.txt:   Unicode text, UTF-8 text, with very long lines (384), with no line terminators
05.txt:   Non-ISO extended-ASCII text, with very long lines (767), with no line terminators
06.txt:   Non-ISO extended-ASCII text, with very long lines (766), with no line terminators
07.txt:   Non-ISO extended-ASCII text, with very long lines (765), with no line terminators


Evidently some functions tacitly pick up on the m-file's encoding and some don't, some give UTF-8, and some give "extended ASCII" that's not ISO or Unicode.

Are we supposed to carry over the m-file's encoding for file operations?

Could someone test on Windows what happens to the text files when the source file changes encoding?

Arun Giridhar <arungiridhar>
Group Member
Fri 24 Mar 2023 11:01:50 PM UTC, comment #29: 
Arun Giridhar <arungiridhar>
Group Member
Fri 24 Mar 2023 09:56:39 PM UTC, comment #28: 

It passes for me too. I can also see that with "t" the file in
latin1 encoding (1-byte per char) and w/o it is in utf-8
(2-bytes per char).


$ od -c /tmp/oct-p1si8a
0000000   a   b   c   , 344 366 374  \n
0000010
$ od -c /tmp/oct-egUaDK
0000000   a   b   c   , 303 244 303 266 303 274  \n
0000013


May be it is good enough for now.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 24 Mar 2023 09:45:16 PM UTC, comment #27: 

When I change "w+" to "wt+", the BIST passes for me. Is that the right thing to do or will it break something else?

Arun Giridhar <arungiridhar>
Group Member
Fri 24 Mar 2023 09:44:51 PM UTC, comment #26: 

https://www.mathworks.com/help/matlab/ref/fwrite.html

<<<
The MATLAB®char type is not a fixed size, and the number of bytes depends on the encoding scheme associated with the file. Set encoding with fopen.

>>>


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 24 Mar 2023 09:42:53 PM UTC, comment #25: 

One of related problems is that strings are not matlab compatible.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 24 Mar 2023 09:41:04 PM UTC, comment #24: 

That is the only one.
As far as I know "t" mode means converting \n to \r\n . 

As always it is up to matlab compatibility.

If it were up to me it would apply only to char/string
formats in fscanf/fprintf.

It is a mess.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 24 Mar 2023 09:30:38 PM UTC, comment #23: 

It broke only one BIST for me, which after working through the checks manually, I am thinking the BIST needs to be updated. This is the BIST:


## file stream with encoding
%!test
%! f = tempname ();
%! fid = fopen (f, "w+", "n", "iso-8859-1");
%! unwind_protect
%!   fprintf (fid, "abc,äöü\n");
%!   fflush (fid);
%!   fseek (fid, 0, "bof");
%!   obs = textscan (fid, "%s", "delimiter", ",");
%!   fclose (fid);
%!   assert (obs, { {"abc"; "äöü"} });
%! unwind_protect_cleanup
%!   unlink (f);
%! end_unwind_protect


The behavior hinges on what exactly is mode "w+" for opening a file. The help text says that's supposed to be read and write, discarding old contents if any, but *doesn't* say if it's binary mode or not. Which is it supposed to be? The code itself does this:


  else if (mode == "w+b" || mode == "wb+" || mode == "w+")
    retval = (std::ios::in | std::ios::out | std::ios::trunc
              | std::ios::binary);


So that code opens it in binary mode read/write/trunc, and the BIST explicitly sets the encoding to ISO-8859-1. IS a binary mode file operation supposed to even have a text encoding? Should that BIST be changed to say "wt+" instead of "w+", because "wt+" does not use binary mode:

  else if (mode == "w+t" || mode == "wt+")
    retval = std::ios::in | std::ios::out | std::ios::trunc;


From the context of the BIST, it's to test text encoding, so "wt+" makes more sense than "w+". Thoughts?

Do you have other failing tests?

Arun Giridhar <arungiridhar>
Group Member
Fri 24 Mar 2023 08:51:07 PM UTC, comment #22: 

It broke some BISTs in
file-io.cc-tst

(see buildbots e.g.)

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 24 Mar 2023 08:17:39 PM UTC, comment #21: 

This bug is still open but I pushed a patch for fwrite to https://hg.savannah.gnu.org/hgweb/octave/rev/fcd97a68e5f7 so that at least that is getting tested and iterated while this is being fixed.

The side effect of that fix on fprintf is that it will change conversion of the output file to UTF-8 as seen in the comments below. So that's the next thing to fix. I'm thinking that there's no single-point solution that works for both fwrite and fprintf.

Arun Giridhar <arungiridhar>
Group Member
Fri 24 Mar 2023 04:40:00 PM UTC, comment #20: 

Thank you for working on this problem.

Just recall that I have additionally some random behaviour (Window 10).

In file #54500, only one special character is wrong (on last line). All others are ok.

Comment #8 describes some systematical behaviour for which each 16th character is wrong.



Charles Praplan <charprap>
Fri 24 Mar 2023 04:39:55 PM UTC, comment #19: 

Also if you run (either patched or unpatched) it w/o setting
latin1 encoding (i.e. with utf-8 encoding) you get a some strange
characters instead of é and û -->  <357> <277> <275>.

So the fact that editor now is quietly setting _mfile_encoding_
violates principle of least surprises: one day you develop
your scripts in gui and they run fine, another day you run them in
batch mode and they produce some garbage.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 24 Mar 2023 04:07:09 PM UTC, comment #18: 

Right... I see the same:

$ file *txt
01_D_patched.txt:   Unicode text, UTF-8 text
01_D_unpatched.txt: ISO-8859 text
02_D_patched.txt:   Unicode text, UTF-8 text
02_D_unpatched.txt: ISO-8859 text


I tried tweaking the patch like this:

diff -r c735cce18c7a libinterp/corefcn/oct-stream.h
--- a/libinterp/corefcn/oct-stream.h    Thu Mar 23 16:22:59 2023 -0400
+++ b/libinterp/corefcn/oct-stream.h    Fri Mar 24 11:57:24 2023 -0400
@@ -127,7 +127,7 @@ public:

     // wrap the output stream with encoding conversion facet
     std::ostream *os = output_stream ();
-    if (os && *os)
+    if (os && *os && ! (m_mode & std::ios::binary))
       {
         convfacet_u8 *facet = new convfacet_u8 (m_encoding);
         std::wbuffer_convert<convfacet_u8, char> *converter


But it gave the same results as the previous patch. It fixes fwrite but fprintf needs some exception?

Arun Giridhar <arungiridhar>
Group Member
Fri 24 Mar 2023 03:26:57 PM UTC, comment #17: 

Using fprintf_bug1 as a test script.
01_D_p.txt is the output of the patched version, 01_D.txt is unpatched version:




$ diff 01_D_p.txt 01_D.txt
1c1
< Décompte                             2022
---
> D�compte                             2022
25,34c25,34
< Coûts                             2022 :   238.70  [CHF]
< Coûts                             2022 :   238.70  [CHF]
< Coûts                             2022 :   238.70  [CHF]
< Coûts                             2022 :   238.70  [CHF]
< Coûts                             2022 :   238.70  [CHF]
< Coûts                             2022 :   238.70  [CHF]
< Coûts                             2022 :   238.70  [CHF]
< Coûts                             2022 :   238.70  [CHF]
< Coûts                             2022 :   238.70  [CHF]
< Coûts                             2022 :   238.70  [CHF]
---
> Co�ts                             2022 :   238.70  [CHF]
> Co�ts                             2022 :   238.70  [CHF]
> Co�ts                             2022 :   238.70  [CHF]
> Co�ts                             2022 :   238.70  [CHF]
> Co�ts                             2022 :   238.70  [CHF]
> Co�ts                             2022 :   238.70  [CHF]
> Co�ts                             2022 :   238.70  [CHF]
> Co�ts                             2022 :   238.70  [CHF]
> Co�ts                             2022 :   238.70  [CHF]
> Co�ts                             2022 :   238.70  [CHF]


or


$ od -c 01_D.txt | head -2
0000000   D 351   c   o   m   p   t   e
0000020
$ od -c 01_D_p.txt | head -2
0000000   D 303 251   c   o   m   p   t   e
0000020


$ od -c 01_D.txt | tail -6
0001500   2       :               2   3   8   .   7   0           [   C
0001520   H   F   ]  \n   C   o 373   t   s
0001540
0001560                           2   0   2   2       :               2
0001600   3   8   .   7   0           [   C   H   F   ]  \n
0001615
$ od -c 01_D_p.txt | tail -6
0001520   2   3   8   .   7   0           [   C   H   F   ]  \n   C   o
0001540 303 273   t   s
0001560
0001600       2   0   2   2       :               2   3   8   .   7   0
0001620           [   C   H   F   ]  \n
0001630


So the output file is utf-8 encoded now (while the source is latin1). Not sure if this is a desired outcome.

Dmitri.
--


 



Dmitri A. Sergatskov <dasergatskov>
Fri 24 Mar 2023 02:01:13 PM UTC, comment #16: 
Arun Giridhar <arungiridhar>
Group Member
Mon 20 Mar 2023 10:20:41 PM UTC, comment #15: 

@Charles, I see that you linked bug #63931. There's also a related bug #63415. Most of the discussion is happening there if you're interested in tracking this problem.

A good summary of what went wrong is here:
https://savannah.gnu.org/bugs/?63415#comment33


Arun Giridhar <arungiridhar>
Group Member
Sun 19 Mar 2023 05:49:53 PM UTC, comment #14: 

Yes, this is a serious bug and switching to utf-8 is a temp workaround.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 05:44:27 PM UTC, comment #13: 

Thank you for your answer,

I agree that changing the encoding seems to solve the problem even for binary files.

But I assume that your proposal is only a workaround and that the problem will be corrected soon. Right?

What's the point of having the best computation software, if the results cannot be output reliably?

And regarding binary files, from my point of view, write operations should not be dependent on the encoding. The following simple code should always output the same file, but the result depends on the encoding.

file = 'file_251.bin';
fid = fopen(file, 'w');
fwrite(fid, 251);
f_error = fclose(fid);



Charles Praplan <charprap>
Sun 19 Mar 2023 02:26:30 PM UTC, comment #12: 

Change your editor encodings to "UTF-8"

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 02:19:50 PM UTC, comment #11: 

It has everything to do with encoding. If I change my encoding
(in editor setting) to iso-8859-1 the problem appears again:



>> t1
val = 193
cw = 1
cr1 = 1
x = 193
y = 63
------
val = 194
cw = 1
cr1 = 0
x = 194
error: b(1): out of bound 0 (dimensions are 0x0)
error: called from
    t1 at line 15 column 5
>> t2
val = 127
cw = 1
cr1 = 1
x = 127
y = 127
------
val = 128
cw = 1
cr1 = 1
x = 128
y = 63
------
val = 129
cw = 1
cr1 = 1
x = 129
y = 63
------


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 19 Mar 2023 02:08:08 PM UTC, comment #10: 

Bugs https://savannah.gnu.org/bugs/?63930 and https://savannah.gnu.org/bugs/?63931 are probably related.

When writing bytes with large value to a file (either with fprintf for text files or with fwrite for binary files) the desired value is sometimes replaced by 0x3F.

Thus, this has probably nothing to do with code pages and encoding...

Charles Praplan <charprap>
Sat 18 Mar 2023 10:06:26 AM UTC, comment #9: 

I guess I didn't mention my settings in the octave editor:

Text encoding ... : SYSTEM (CP1252)

Charles Praplan <charprap>
Sat 18 Mar 2023 09:59:23 AM UTC, comment #8: 

I found some systematical behaviour of the problem. This is described below:

Test cases and comparisons are appended (fprintf_bug.tgz). I also join pictures of the
comparisons because I'm not sure that you see the same thing than me.

1)
If the m-file only writes special characters to the file, the proble does not appear.

Test case : fprintf_bug4.m => 04.txt

2)
Once a standard character is outputted (incl \n), each 16th character is wrong (replaced by char 3F).

Test case : fprintf_bug5.m   =>  05.txt


3)
If a second standard character is outputted, the problem disappear for the rest of the file.

Test case : fprintf_bug6.m   =>  06.txt

4)
if a third standard character is outputted, the problem appears again for the rest of the file.

Test case : fprintf_bug7.m   =>  07.txt


(file #54508)

Charles Praplan <charprap>
Thu 16 Mar 2023 03:56:23 PM UTC, comment #7: 

The 2 text files are now appended.
Thank you.

(file #54499, file #54500)

Charles Praplan <charprap>
Thu 16 Mar 2023 01:33:03 PM UTC, comment #6: 

Thanks.

The encoding of the strings is actually converted twice.
1. When parsing the .m file, the entire file is converted from codepage 1252 to UTF-8. (That is done so that Octave can potentially feed those strings to other libraries, like for regex matching or printing text in figures.)
2. When writing to the file, each character array is converted from UTF-8 to the target encoding. That is either the one specified when opening the file (with `fopen`), or, if none is specified, the encoding that is used for parsing .m files (the encoding returned by `__mfile_encoding__`). That was done to be compatible with Matlab.

I'm guessing that something might be going wrong in the second conversion if the output is buffered, and the buffer is flushed when the end of it happens to be "between" bytes of a multi-byte UTF-8 character...

Markus Mützel <mmuetzel>
Group administrator
Thu 16 Mar 2023 12:08:45 PM UTC, comment #5: 

No, the 2 text files are created on the same PC.
But I tried another PC to be more convinced that the problem is not in my PC.
I will attach the generate files later.

Running the script creates 2 files which should have only one difference.
But I get one second difference.

Effectively the problem occurs with
&#251; | \373 | =FB | u ^ | (û) | Small u, circumflex accent

Running fprintf_bug1.m, octave outputs this character most of the time correctly but sometimes not: char 0x3F is sometimes outputted instead of 0xFB.

I do not want to use UTF-8 encoding. I want to generate pure text files.

Charles Praplan <charprap>
Thu 16 Mar 2023 07:27:22 AM UTC, comment #4: 

Do I understand correctly that the two text files were created on two different PCs with different locale?

Instead of images of text, could you please attach the actual generated text files?

Markus Mützel <mmuetzel>
Group administrator
Wed 15 Mar 2023 11:30:01 PM UTC, comment #3: 

Yes the scripts apparently are using "latin1" encoding


&#233; | \351 | =E9 | e ' | (é) | Small e, acute accent

and

&#251; | \373 | =FB | u ^ | (û) | Small u, circumflex accent


I think octave expects UTF-8 encoding, so it gets confused.
If I change those to UTF-8 characters, the output looks OK.
Converted .m files are attached.

Dmitri.
--





(file #54495, file #54496)

Dmitri A. Sergatskov <dasergatskov>
Wed 15 Mar 2023 09:25:41 PM UTC, comment #2: 

It seems that there might be an issue with the encoding of the text file being written by the fprintf function in Octave 8.1.0. The character on line 29 that is being printed incorrectly might be due to a mismatch in the encoding being used by the function and the encoding of the file.

One possible solution to this issue is to explicitly specify the encoding being used by the fprintf function. For example, if the text file is in UTF-8 encoding, the fprintf function can be called with the '-encoding' option set to 'UTF-8'

PRIYANSHU SHUKLA <priyanshu>
Wed 15 Mar 2023 06:55:00 PM UTC, comment #1: 

I cannot reproduce this (or I misunderstood the bug report) on
Win11 octave gui codepage 437


GNU Octave Version: 8.1.0 (hg id: 75b1efe727d4)
GNU Octave License: GNU General Public License
Operating System: MINGW32_NT-6.2 Windows 6.2  x86_64



$ diff -b 01_D.txt 02_D.txt
4c4
< A           no 1 : aaaaa
---
> A           no 1 : aaaaab


Files are attached.

Dmitri.
--


(file #54491, file #54492)

Dmitri A. Sergatskov <dasergatskov>
Wed 15 Mar 2023 05:09:22 PM UTC, original submission:  

fprintf_bug1.m illustrates a strange behavior of the fprintf function which appeared with the new version of Octave (8.1.0).

The script writes 2 text files with the same content, except on line 5, but the difference is small.
One of the two files is written correctly, while the other one contains an error on another line (29) containing a special character.

The comparison between the 2 text file is plotted in the appended file File_comparison_1.png.

The character which is printed where a ? in the comparison is 0x3F instead of 0xFB.

Any small change in the m-file can modify the result. Therefore additional test files with the corresponding comparison.

I tried this on 2 different PCs with different codepages:
PS C:\> chcp
Active code page: 850

PS C:\> chcp
Active code page: 437

Charles Praplan <charprap>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #54565:  bug63930-check-for-LLVM-STL.patch added by mmuetzel (2KiB - application/octet-stream)
file #54550:  bug63930-incomplete-utf-8-v2.patch added by mmuetzel (5KiB - application/octet-stream)
file #54541:  bug63930-incomplete-utf-8.patch added by mmuetzel (2KiB - application/octet-stream)
file #54508:  fprintf_bug.tgz added by charprap (147KiB - application/x-compressed)
file #54499:  01_D.txt added by charprap (907B - text/plain)
file #54500:  02_D.txt added by charprap (908B - text/plain)
file #54495:  fprintf_bug1c.m added by dasergatskov (2KiB - text/x-objcsrc)
file #54496:  fprintf_bug2c.m added by dasergatskov (2KiB - text/x-objcsrc)
file #54491:  01_D.txt.gz added by dasergatskov (177B - application/x-gzip)
file #54492:  02_D.txt.gz added by dasergatskov (178B - application/x-gzip)
file #54487:  fprintf_bug1.m added by charprap (2KiB - text/plain)
file #54488:  fprintf_bug2.m added by charprap (2KiB - text/plain)
file #54489:  File_comparison_1.png added by charprap (27KiB - image/png)
file #54490:  File_comparison_2.png added by charprap (26KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by arungiridhar (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by priyanshu (Posted a comment)
  • -email is unavailable- added by dasergatskov (Updated the item)
  • -email is unavailable- added by charprap (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 25 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-05-04 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2023-04-07 mmuetzel StatusIn Progress Ready For Test
    2023-04-06 mmuetzel Attached File- Added bug63930-check-for-LLVM-STL.patch, #54565
        StatusReady For Test In Progress
    2023-04-03 mmuetzel StatusPatch Submitted Ready For Test
        Fixed ReleaseNone 8.2.0
    2023-04-02 mmuetzel Attached File- Added bug63930-incomplete-utf-8-v2.patch, #54550
    2023-04-01 mmuetzel Attached File- Added bug63930-incomplete-utf-8.patch, #54541
        StatusIn Progress Patch Submitted
    2023-03-31 mmuetzel Summaryfprintf problem fprintf writes incorrect characters when converting the encoding
    2023-03-24 arungiridhar StatusNeed Info In Progress
    2023-03-24 arungiridhar Operating SystemMicrosoft Windows Any
        Planned ReleaseNone 8.2.0
    2023-03-18 charprap Attached File- Added fprintf_bug.tgz, #54508
    2023-03-16 charprap Attached File- Added 01_D.txt, #54499
        Attached File- Added 02_D.txt, #54500
    2023-03-16 mmuetzel StatusNone Need Info
    2023-03-15 dasergatskov Attached File- Added fprintf_bug1c.m, #54495
        Attached File- Added fprintf_bug2c.m, #54496
    2023-03-15 dasergatskov Attached File- Added 01_D.txt.gz, #54491
        Attached File- Added 02_D.txt.gz, #54492
    2023-03-15 charprap Attached File- Added fprintf_bug1.m, #54487
        Attached File- Added fprintf_bug2.m, #54488
        Attached File- Added File_comparison_1.png, #54489

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code