I pushed two changes to the stable branch that implement the `HAVE_LLVM_LIBCXX` feature test for configure and makes the encoding conversion for streams opened with `fopen` conditional on it:
https://hg.savannah.gnu.org/hgweb/octave/rev/86eb373a6c64
https://hg.savannah.gnu.org/hgweb/octave/rev/ee187a104701

Let's see how that place out on the macOS runners.
If the corresponding tests pass, I think this is about all we should do to fix this on the stable branch.

If we want this to work independent of the used STL, we'd probably need a re-design of how the encoding conversion is currently implemented keeping the resolution for the defect report mentioned in comment #63 in mind.
But it'd probably be better to track that in a separate report.

Is it possible to skip a test if a feature is present? I.e., if we were to add HAVE_LLVM_LIBCXX, would it be possible to skip certain tests if it is defined (like we currently to with `%!testif ; ! ismac ()`)?

I only find conditional tests that start with `%!testif HAVE_XXX`. But none like `%!testif ! HAVE_XXX` or whatever the syntax would be.

I think it is OK to add that check but maybe rename the macro to HAVE_APPLE_LIBCXX or HAVE_LLVM_LIBCXX?

Also, instead of using this macro directly in the Octave sources, I propose that we add something like the following to the oct-conf-post-private.in.h file:

#if defined (HAVE_LLVM_LIBCXX)
#  define HAVE_BROKEN_SOMETHING_OR_OTHER 1
...
#endif

and then use the more specific macro in the source files.  That way, we can add individual tests later without having to hunt down the places to change in the source files.  And maybe the intent of the preprocessor conditional is also somewhat clearer when reading the code.

We could do the same for any other instances where we are currently doing version checks or checks for non-specific macros like "#if defined (_APPLE_)", etc.

comment #65:

> We'd probably also need a `HAVE_*` macro to be able to skip the corresponding test on that platform.
>

We already skipping some tests by using ismac() (which are
in reality should be "islibc++"). Perhaps this is just another one like that?

Dmitri.
--

I made different tests with the latest version from buildbot.
In particular, I could write files with UTF-8 and Windows-1252 encoding without any problem.
Thank you !

The icing on the cake would be to have:
- an automatic detection of m-files encoding when loading them into the editor
-the possibility to change the encoding in the "save as" dialog box of the editor

https://stackoverflow.com/questions/31657499/how-to-detect-stdlib-libc-in-the-preprocessor

?

comment #54:

> If I compile w/ "-ggdb3 -O0" it does not crash.
>
> If file is open with non-utf encoding, fprintf does not write anything into it:
>

>
> octave:48> fid = fopen('05.txt', 'w', 'n', 'Windows-1252')
> fid = 22
> octave:49> fprintf(fid, 'ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû')
> octave:50> f_error = fclose(fid)
> f_error = 0
>

>

The same with stock clang-15.0.6/libc++ (compiled with -O0)
But if I fprintf("fid", "1") first, it writes bot 1 and "latin1" string.

Dmitri.
--

Here is with stock Fedora's clang 15.0.6:

Thread 8 "QThread" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffce3bf6c0 (LWP 271806)]
__memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:265
265             VMOVU   (%rsi), %VMM(0)
(gdb) thread applt all bt
Invalid thread ID: applt all bt
(gdb) thread apply all bt

Thread 9 (Thread 0x7fffcd3916c0 (LWP 271807) "QThread"):
#0  0x00007ffff2a5b838 in __GI___sigtimedwait (set=set@entry=0x7ffff4f0a488 <async_signals>, info=info@entry=0x7fffcd390220, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:31
#1  0x00007ffff2a5af04 in __GI___sigwait (set=0x7ffff4f0a488 <async_signals>, sig=0x7fffcd3902ec) at ../sysdeps/unix/sysv/linux/sigwait.c:28
#2  0x00007ffff4c98e3e in signal_watcher(void*) (arg=0x7ffff769dd36 <octave::generic_sig_handler(int)>) at ../liboctave/wrappers/cxx-signal-helpers.cc:179
#3  0x00007ffff2aa9c57 in start_thread (arg=<optimized out>) at pthread_create.c:444
#4  0x00007ffff2b2fa70 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7fffce3bf6c0 (LWP 271806) "QThread"):
#0  __memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:265
#1  0x00007ffff4c9a189 in octave_u8_conv_to_encoding_intern (tocode=0x7fffb48480a1 "windows-1252", handler=iconveh_question_mark, src=0x63 <error: Cannot access memory at address 0x63>, srclen=140736222075389, offsets=0x0, lengthp=0x7fffce3bb310) at ../liboctave/wrappers/uniconv-wrappers.c:63
#2  0x00007ffff4c8f414 in octave::string::codecvt_u8::do_out(__mbstate_t&, char const*, char const*, char const*&, char*, char*, char*&) const (this=<optimized out>, from=0x63 <error: Cannot access memory at address 0x63>, from_end=<optimized out>, from_next=<error reading variable: Cannot access memory at address 0xffffffff>, to=0x7fffb47f7050 "0\a", to_end=0x7fffb47f8050 "\020\020", to_next=@0x7fffce3bb388: 0x7fffb47f7050 "0\a") at ../liboctave/util/oct-string.cc:660
#3  0x00007ffff7657f35 in std::__1::codecvt<char, char, __mbstate_t>::out[abi:v15007](__mbstate_t&, char const*, char const*, char const*&, char*, char*, char*&) const (this=0x0, __st=..., __frm=0x7fffb48619fd "ind_protect_cleanup\n     fclose (id);\n   end_unwind_protect\n endif\nendfunction\n", __frm_end=0x0, __frm_nxt=<optimized out>, __to=0x0, __to_nxt=<optimized out>, __to_end=<optimized out>) at /usr/bin/../include/c++/v1/__locale:975
#4  std::__1::wbuffer_convert<octave::string::deletable_facet<octave::string::codecvt_u8>, char, std::__1::char_traits<char> >::overflow(int) (this=0x7fffb4842a80, __c=195) at /usr/bin/../include/c++/v1/locale:4125
#5  0x00007ffff2d5fb2e in std::__1::basic_streambuf<char, std::__1::char_traits<char> >::xsputn(char const*, long) (this=0x7fffb4842a80, __s=0x7fffb4842d10 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __n=128) at include/c++/v1/streambuf:467
#6  0x0000000000402c04 in std::__1::basic_streambuf<char, std::__1::char_traits<char> >::sputn[abi:v15007](char const*, long) (this=0x7fffb4842a80, __s=0x7fffb4842d10 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __n=128) at /usr/bin/../include/c++/v1/streambuf:232
#7  std::__1::__pad_and_output<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char) (__s=..., __s@entry=..., __ob=__ob@entry=0x7fffb4842d10 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __op=0x7fffb4842d10 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __oe=__oe@entry=0x7fffb4842d90 "", __iob=..., __fl=32 ' ') at /usr/bin/../include/c++/v1/locale:1451
#8  0x0000000000402a62 in std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) (__os=..., __str=0x7fffb4842d10 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __len=<optimized out>) at /usr/bin/../include/c++/v1/ostream:763
--Type <RET> for more, q to quit, c to continue without paging--c
#9  0x00007ffff762c714 in std::__1::operator<< <char, std::__1::char_traits<char>, std::__1::allocator<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (__os=..., __str=...) at /usr/bin/../include/c++/v1/ostream:1086
#10 octave::base_stream::do_printf(octave::printf_format_list&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=this@entry=0x7fffb46a41a0, fmt_list=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:6009
#11 0x00007ffff762ca80 in octave::base_stream::printf(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=0x7fffb46a41a0, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:6072
#12 0x00007ffff764d428 in octave::stream::printf(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=this@entry=0x7fffce3bb810, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:7238
#13 0x00007ffff764d4c5 in octave::stream::printf(octave_value const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=0x7fffce3bb810, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:7256
#14 0x00007ffff745976e in octave::printf_internal(octave::interpreter&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, int) (interp=..., who=..., args=..., nargout=nargout@entry=0) at ../libinterp/corefcn/file-io.cc:813
#15 0x00007ffff74594d5 in octave::Ffprintf(octave::interpreter&, octave_value_list const&, int) (interp=..., args=..., nargout=0) at ../libinterp/corefcn/file-io.cc:843
#16 0x00007ffff736df90 in octave::tree_evaluator::execute_builtin_function(octave_builtin&, int, octave_value_list const&) (this=0x7fffb4004510, builtin_function=..., nargout=0, args=...) at ../libinterp/parse-tree/pt-eval.cc:3444
#17 0x00007ffff726666a in octave_builtin::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-builtin.cc:49
#18 0x00007ffff72a4a10 in octave_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb40b8a60, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-fcn.cc:57
#19 0x00007ffff737799f in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb4646d70, tw=..., nargout=0) at ../libinterp/parse-tree/pt-idx.cc:427
#20 0x00007ffff737978b in octave::tree_index_expression::evaluate(octave::tree_evaluator&, int) (this=0x63, tw=..., nargout=0) at ../libinterp/parse-tree/pt-idx.h:108
#21 0x00007ffff736496f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#22 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#23 0x00007ffff73650f0 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x63, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#24 octave::tree_evaluator::visit_unwind_protect_command(octave::tree_unwind_protect_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4286
#25 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#26 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#27 0x00007ffff737384f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb48499a0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#28 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb4004510, rng=..., line=line@entry=14, ult=..., loop_body=loop_body@entry=0x7fffb48499a0) at ../libinterp/parse-tree/pt-eval.cc:3049
#29 0x00007ffff736395a in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#30 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#31 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#32 0x00007ffff737384f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb4818930, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#33 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb4004510, rng=..., line=line@entry=13, ult=..., loop_body=loop_body@entry=0x7fffb4818930) at ../libinterp/parse-tree/pt-eval.cc:3049
#34 0x00007ffff736395a in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#35 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#36 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#37 0x00007ffff736e82f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb4810c70, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#38 octave::tree_evaluator::execute_user_function(octave_user_function&, int, octave_value_list const&) (this=0x7fffb4004510, user_function=..., nargout=0, xargs=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3665
#39 0x00007ffff72e04a0 in octave_user_function::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:495
#40 0x00007ffff72e044e in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb481e660, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:488
#41 0x00007ffff737799f in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb483e650, tw=..., nargout=0) at ../libinterp/parse-tree/pt-idx.cc:427
#42 0x00007ffff7367e32 in octave::tree_evaluator::eval_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, int&, int) (this=0x7fffb4004510, eval_str=<optimized out>, silent=false, parse_status=@0x7fffce3bc87c: 0, nargout=0) at ../libinterp/parse-tree/pt-eval.cc:1053
#43 0x00007ffff736850c in octave::tree_evaluator::eval(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (this=0x63, try_code=..., nargout=0) at ../libinterp/parse-tree/pt-eval.cc:1112
#44 0x00007ffff75b90e4 in octave::interpreter::eval(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (this=<optimized out>, try_code=..., nargout=0) at ../libinterp/corefcn/interpreter.cc:1512
#45 0x00007ffff73550ca in octave::Feval(octave::interpreter&, octave_value_list const&, int) (interp=..., args=..., nargout=0) at ../libinterp/parse-tree/oct-parse.yy:6476
#46 0x00007ffff736df90 in octave::tree_evaluator::execute_builtin_function(octave_builtin&, int, octave_value_list const&) (this=0x7fffb4004510, builtin_function=..., nargout=0, args=...) at ../libinterp/parse-tree/pt-eval.cc:3444
#47 0x00007ffff726666a in octave_builtin::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-builtin.cc:49
#48 0x00007ffff72a4a10 in octave_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb4011af0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-fcn.cc:57
#49 0x00007ffff737799f in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb463f4b0, tw=..., nargout=0) at ../libinterp/parse-tree/pt-idx.cc:427
#50 0x00007ffff737978b in octave::tree_index_expression::evaluate(octave::tree_evaluator&, int) (this=0x63, tw=..., nargout=0) at ../libinterp/parse-tree/pt-idx.h:108
#51 0x00007ffff736496f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#52 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#53 0x00007ffff7364d80 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb463e510, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#54 octave::tree_evaluator::visit_try_catch_command(octave::tree_try_catch_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4141
#55 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#56 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#57 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#58 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#59 0x00007ffff73650f0 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x63, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#60 octave::tree_evaluator::visit_unwind_protect_command(octave::tree_unwind_protect_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4286
#61 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#62 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#63 0x00007ffff737384f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb45e8290, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#64 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb4004510, rng=..., line=line@entry=330, ult=..., loop_body=loop_body@entry=0x7fffb45e8290) at ../libinterp/parse-tree/pt-eval.cc:3049
#65 0x00007ffff736395a in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb4004510, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#66 0x00007ffff73648f5 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#67 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#68 0x00007ffff736e82f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb4339eb0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#69 octave::tree_evaluator::execute_user_function(octave_user_function&, int, octave_value_list const&) (this=0x7fffb4004510, user_function=..., nargout=2, xargs=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3665
#70 0x00007ffff72e04a0 in octave_user_function::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:495
#71 0x00007ffff72e044e in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb4558a70, tw=..., nargout=2, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:488
#72 0x00007ffff737799f in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb431afb0, tw=..., nargout=2) at ../libinterp/parse-tree/pt-idx.cc:427
#73 0x00007ffff735e98d in octave::tree_multi_assignment::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb45aa500, tw=...) at ../libinterp/parse-tree/pt-assign.cc:201
#74 0x00007ffff735f559 in octave::tree_multi_assignment::evaluate(octave::tree_evaluator&, int) (this=0x63, tw=..., nargout=0) at ../libinterp/parse-tree/pt-assign.h:156
#75 0x00007ffff736496f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb4004510, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#76 0x00007ffff7364ca0 in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb4004510, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#77 0x00007ffff7366de0 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x63, this@entry=0x7fffce3bdac8, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#78 octave::tree_evaluator::eval(std::__1::shared_ptr<octave::tree_statement_list>&, bool) (this=0x0, this@entry=0x7fffb4004510, stmt_list=..., interactive=true) at ../libinterp/parse-tree/pt-eval.cc:985
#79 0x00007ffff736759e in octave::tree_evaluator::repl() (this=0x7fffb4004510) at ../libinterp/parse-tree/pt-eval.cc:804
#80 0x00007ffff75b73a5 in octave::interpreter::main_loop() (this=0x7fffb4004370) at ../libinterp/corefcn/interpreter.cc:1350
#81 octave::interpreter::execute() (this=0x7fffb4004370) at ../libinterp/corefcn/interpreter.cc:916
#82 0x00007ffff7bf7d04 in octave::interpreter_qobject::execute() (this=0x6f6f00) at ../libgui/src/interpreter-qobject.cc:89
#83 0x00007ffff52df5db in QObject::event(QEvent*) (this=0x6f6f00, e=0x756270) at kernel/qobject.cpp:1347
#84 0x00007ffff5faeca5 in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=<optimized out>, receiver=0x6f6f00, e=0x756270) at kernel/qapplication.cpp:3640
#85 0x00007ffff7c3a01c in octave::octave_qapplication::notify(QObject*, QEvent*) (this=0x0, receiver=0x63, ev=0x7fffb48619fd) at ../libgui/src/octave-qobject.cc:147
#86 0x00007ffff52b3bd8 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0x6f6f00, event=0x756270) at kernel/qcoreapplication.cpp:1064
#87 0x00007ffff52b3df2 in QCoreApplication::sendEvent(QObject*, QEvent*) (receiver=<optimized out>, event=<optimized out>) at kernel/qcoreapplication.cpp:1462
#88 0x00007ffff52b7085 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) (receiver=0x0, event_type=0, data=0x53d280) at kernel/qcoreapplication.cpp:1821
#89 0x00007ffff52b733d in QCoreApplication::sendPostedEvents(QObject*, int) (receiver=<optimized out>, event_type=<optimized out>) at kernel/qcoreapplication.cpp:1680
#90 0x00007ffff5306cdf in postEventSourceDispatch(GSource*, GSourceFunc, gpointer) (s=0x7fffb4018cf0) at kernel/qeventdispatcher_glib.cpp:277
#91 0x00007ffff1e2af58 in g_main_dispatch (context=0x7fffb40174f0) at ../glib/gmain.c:3460
#92 g_main_context_dispatch (context=0x7fffb40174f0) at ../glib/gmain.c:4200
#93 0x00007ffff1e8acd8 in g_main_context_iterate.isra.0 (context=0x7fffb40174f0, block=1, dispatch=1, self=<optimized out>) at ../glib/gmain.c:4276
#94 0x00007ffff1e2c233 in g_main_context_iteration (context=0x7fffb40174f0, may_block=1) at ../glib/gmain.c:4343
#95 0x00007ffff53067c9 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) (this=0x7fffb40687c0, flags=...) at kernel/qeventdispatcher_glib.cpp:423
#96 0x00007ffff52b25ab in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) (this=this@entry=0x7fffce3be270, flags=..., flags@entry=...) at ../../include/QtCore/../../src/corelib/global/qflags.h:69
#97 0x00007ffff50f4110 in QThread::exec() (this=<optimized out>) at ../../include/QtCore/../../src/corelib/global/qflags.h:121
#98 0x00007ffff50f548d in QThreadPrivate::start(void*) (arg=0x756f60) at thread/qthread_unix.cpp:330
#99 0x00007ffff2aa9c57 in start_thread (arg=<optimized out>) at pthread_create.c:444
#100 0x00007ffff2b2fa70 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Dmitri.
--

`srclen=140255252715919` doesn't look right. That looks more like it is a pointer address in the range of the other pointers appearing in the backtrace (0x7F8FB884198F).

It is calculated like this in `octave::string::codecvt_u8::do_out` in oct-string.cc:653:

  std::size_t srclen = (from_end-from-pop_end) * sizeof (InternT);

`pop_end` should be 0. `sizeof (InternT)` should be 1 (InternT is char).
So, it should be the difference between the addresses pointed to by `from_end` and `from`. Theoretically that should be a maximum of 64 in that example...

Compiling with "-ggdb3 -Os" crashes:

Thread 8 "QThread" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffcb7fe640 (LWP 1299975)]
0x00007ffff0ab9641 in __memmove_avx_unaligned () from /usr/lib64/libc.so.6

Thread 8 (Thread 0x7fffcb7fe640 (LWP 1299975) "QThread"):
#0  0x00007ffff0ab9641 in __memmove_avx_unaligned () at /usr/lib64/libc.so.6
#1  0x00007ffff4cf186b in octave_u8_conv_to_encoding_intern (tocode=0x7fffb87c5451 "windows-1252", handler=iconveh_question_mark, src=0x7000000101 <error: Cannot access memory at address 0x7000000101>, srclen=140255252715919, offsets=0x0, lengthp=0x7fffcb7fa340) at ../liboctave/wrappers/uniconv-wrappers.c:63
#2  0x00007ffff4ce6702 in octave::string::codecvt_u8::do_out(__mbstate_t&, char const*, char const*, char const*&, char*, char*, char*&) const (this=<optimized out>, from=0x7000000101 <error: Cannot access memory at address 0x7000000101>, from_end=<optimized out>, from_next=<error reading variable>, to=0x7fffb8840a80 "\200\a", to_end=0x7fffb8841a80 "\020\020", to_next=@0x7fffcb7fa3b8: 0x7fffb8840a80 "\200\a") at ../liboctave/util/oct-string.cc:660
#3  0x00007ffff76f2b9f in std::__1::codecvt<char, char, __mbstate_t>::out(__mbstate_t&, char const*, char const*, char const*&, char*, char*, char*&) const (this=0x0, __st=..., __frm=0x7f8fb884198f <error: Cannot access memory at address 0x7f8fb884198f>, __frm_end=0x7ffff0b47fe7 <mmap64+23> "H=", __frm_nxt=<optimized out>, __to=0x0, __to_nxt=<optimized out>, __to_end=<optimized out>) at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/__locale:975
#4  std::__1::wbuffer_convert<octave::string::deletable_facet<octave::string::codecvt_u8>, char, std::__1::char_traits<char> >::overflow(int) (this=0x7fffb87ed840, __c=195) at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/locale:4102
#5  0x00007ffff0fa1cae in std::__1::basic_streambuf<char, std::__1::char_traits<char> >::xsputn(char const*, long) () at /opt/AMD/aocc-compiler-4.0.0/lib/libc++.so.1
#6  0x00000000002034ad in std::__1::basic_streambuf<char, std::__1::char_traits<char> >::sputn(char const*, long) (this=0x7fffb87ed840, __s=0x7fffb8861250 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __n=128) at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/streambuf:231
#7  std::__1::__pad_and_output<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char) (__s=..., __s@entry=..., __ob=__ob@entry=0x7fffb8861250 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __op=0x7fffb8861250 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __oe=__oe@entry=0x7fffb88612d0 "", __iob=..., __fl=32 ' ') at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/locale:1440
#8  0x000000000020330b in std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) (__os=..., __str=0x7fffb8861250 "ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû", __len=<optimized out>) at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/ostream:726
#9  0x00007ffff76c7673 in std::__1::operator<< <char, std::__1::char_traits<char>, std::__1::allocator<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (__os=..., __str=...) at /opt/AMD/aocc-compiler-4.0.0/bin/../include/c++/v1/ostream:1049
#10 octave::base_stream::do_printf(octave::printf_format_list&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=this@entry=0x7fffb8844010, fmt_list=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:6009
#11 0x00007ffff76c7a20 in octave::base_stream::printf(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=0x7fffb8844010, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:6072
#12 0x00007ffff76e837a in octave::stream::printf(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=this@entry=0x7fffcb7fa830, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:7238
#13 0x00007ffff76e8417 in octave::stream::printf(octave_value const&, octave_value_list const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (this=0x7fffcb7fa830, fmt=..., args=..., who=...) at ../libinterp/corefcn/oct-stream.cc:7256
#14 0x00007ffff74e9892 in octave::printf_internal(octave::interpreter&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, octave_value_list const&, int) (interp=..., who=..., args=..., nargout=nargout@entry=0) at ../libinterp/corefcn/file-io.cc:813
#15 0x00007ffff74e95ee in octave::Ffprintf(octave::interpreter&, octave_value_list const&, int) (interp=..., args=..., nargout=0) at ../libinterp/corefcn/file-io.cc:843
#16 0x00007ffff73fbfac in octave::tree_evaluator::execute_builtin_function(octave_builtin&, int, octave_value_list const&) (this=0x7fffb8056520, builtin_function=..., nargout=0, args=...) at ../libinterp/parse-tree/pt-eval.cc:3444
#17 0x00007ffff72f1862 in octave_builtin::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=-256606233, args=...) at ../libinterp/octave-value/ov-builtin.cc:49
#18 0x00007ffff732ffc8 in octave_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb80bdb20, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-fcn.cc:57
#19 0x00007ffff7405bc7 in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb883b680, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.cc:427
#20 0x00007ffff740793d in octave::tree_index_expression::evaluate(octave::tree_evaluator&, int) (this=0x7000000101, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.h:108
#21 0x00007ffff73f2899 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#22 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#23 0x00007ffff73f2ffa in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7000000101, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#24 octave::tree_evaluator::visit_unwind_protect_command(octave::tree_unwind_protect_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4286
#25 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#26 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#27 0x00007ffff7401917 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb88189a0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
--Type <RET> for more, q to quit, c to continue without paging--c
#28 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb8056520, rng=..., line=line@entry=14, ult=..., loop_body=loop_body@entry=0x7fffb88189a0) at ../libinterp/parse-tree/pt-eval.cc:3049
#29 0x00007ffff73f1876 in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#30 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#31 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#32 0x00007ffff7401917 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb87494d0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#33 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb8056520, rng=..., line=line@entry=13, ult=..., loop_body=loop_body@entry=0x7fffb87494d0) at ../libinterp/parse-tree/pt-eval.cc:3049
#34 0x00007ffff73f1876 in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#35 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#36 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#37 0x00007ffff73fc759 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb88053d0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#38 octave::tree_evaluator::execute_user_function(octave_user_function&, int, octave_value_list const&) (this=0x7fffb8056520, user_function=..., nargout=0, xargs=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3665
#39 0x00007ffff736c408 in octave_user_function::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=-256606233, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:495
#40 0x00007ffff736c3b6 in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb881c7f0, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:488
#41 0x00007ffff7405bc7 in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb86e9600, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.cc:427
#42 0x00007ffff73f5c8a in octave::tree_evaluator::eval_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, int&, int) (this=0x7fffb8056520, eval_str=<optimized out>, silent=false, parse_status=@0x7fffcb7fb87c: 0, nargout=0) at ../libinterp/parse-tree/pt-eval.cc:1053
#43 0x00007ffff73f62ea in octave::tree_evaluator::eval(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (this=0x7000000101, try_code=..., nargout=0) at ../libinterp/parse-tree/pt-eval.cc:1112
#44 0x00007ffff764b5bc in octave::interpreter::eval(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) (this=<optimized out>, try_code=..., nargout=-256606233) at ../libinterp/corefcn/interpreter.cc:1512
#45 0x00007ffff73e2b75 in octave::Feval(octave::interpreter&, octave_value_list const&, int) (interp=..., args=..., nargout=-256606233) at ../libinterp/parse-tree/oct-parse.yy:6476
#46 0x00007ffff73fbfac in octave::tree_evaluator::execute_builtin_function(octave_builtin&, int, octave_value_list const&) (this=0x7fffb8056520, builtin_function=..., nargout=0, args=...) at ../libinterp/parse-tree/pt-eval.cc:3444
#47 0x00007ffff72f1862 in octave_builtin::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=-256606233, args=...) at ../libinterp/octave-value/ov-builtin.cc:49
#48 0x00007ffff732ffc8 in octave_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb8070870, tw=..., nargout=0, args=...) at ../libinterp/octave-value/ov-fcn.cc:57
#49 0x00007ffff7405bc7 in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb863b7e0, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.cc:427
#50 0x00007ffff740793d in octave::tree_index_expression::evaluate(octave::tree_evaluator&, int) (this=0x7000000101, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.h:108
#51 0x00007ffff73f2899 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#52 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#53 0x00007ffff73f2c9f in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb863a840, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#54 octave::tree_evaluator::visit_try_catch_command(octave::tree_try_catch_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4141
#55 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#56 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#57 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#58 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#59 0x00007ffff73f2ffa in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7000000101, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#60 octave::tree_evaluator::visit_unwind_protect_command(octave::tree_unwind_protect_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:4286
#61 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#62 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#63 0x00007ffff7401917 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb85e44e0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#64 octave::tree_evaluator::execute_range_loop<double>(octave::range<double, void> const&, int, octave::octave_lvalue&, octave::tree_statement_list*) (this=this@entry=0x7fffb8056520, rng=..., line=line@entry=330, ult=..., loop_body=loop_body@entry=0x7fffb85e44e0) at ../libinterp/parse-tree/pt-eval.cc:3049
#65 0x00007ffff73f1876 in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) (this=0x7fffb8056520, cmd=...) at ../libinterp/parse-tree/pt-eval.cc:3096
#66 0x00007ffff73f281f in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=...) at ../libinterp/parse-tree/pt-eval.cc:3916
#67 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#68 0x00007ffff73fc759 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7fffb834c1b0, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#69 octave::tree_evaluator::execute_user_function(octave_user_function&, int, octave_value_list const&) (this=0x7fffb8056520, user_function=..., nargout=2, xargs=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3665
#70 0x00007ffff736c408 in octave_user_function::execute(octave::tree_evaluator&, int, octave_value_list const&) (this=0x0, tw=..., nargout=-256606233, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:495
#71 0x00007ffff736c3b6 in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) (this=0x7fffb8530ed0, tw=..., nargout=2, args=...) at ../libinterp/octave-value/ov-usr-fcn.cc:488
#72 0x00007ffff7405bc7 in octave::tree_index_expression::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb8384770, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-idx.cc:427
#73 0x00007ffff73ec791 in octave::tree_multi_assignment::evaluate_n(octave::tree_evaluator&, int) (this=0x7fffb8576580, tw=...) at ../libinterp/parse-tree/pt-assign.cc:201
#74 0x00007ffff73ed393 in octave::tree_multi_assignment::evaluate(octave::tree_evaluator&, int) (this=0x7000000101, tw=..., nargout=-256606233) at ../libinterp/parse-tree/pt-assign.h:156
#75 0x00007ffff73f2899 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fffb8056520, stmt=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:3941
#76 0x00007ffff73f2bca in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fffb8056520, lst=<optimized out>) at ../libinterp/parse-tree/pt-eval.cc:4026
#77 0x00007ffff73f4cb8 in octave::tree_statement_list::accept(octave::tree_walker&) (this=0x7000000101, this@entry=0x7fffcb7fcaa8, tw=...) at ../libinterp/parse-tree/pt-stmt.h:201
#78 octave::tree_evaluator::eval(std::__1::shared_ptr<octave::tree_statement_list>&, bool) (this=0x0, this@entry=0x7fffb8056520, stmt_list=..., interactive=true) at ../libinterp/parse-tree/pt-eval.cc:985
#79 0x00007ffff73f5454 in octave::tree_evaluator::repl() (this=0x7fffb8056520) at ../libinterp/parse-tree/pt-eval.cc:804
#80 0x00007ffff7649877 in octave::interpreter::main_loop() (this=0x7fffb8056380) at ../libinterp/corefcn/interpreter.cc:1350
#81 octave::interpreter::execute() (this=0x7fffb8056380) at ../libinterp/corefcn/interpreter.cc:916
#82 0x00007ffff7d0baec in octave::interpreter_qobject::execute() (this=0x57c530) at ../libgui/src/interpreter-qobject.cc:89
#83 0x00007ffff50d1a99 in QObject::event(QEvent*) (this=0x57c530, e=0x7b0c40) at kernel/qobject.cpp:1314
#84 0x00007ffff5dae423 in QApplicationPrivate::notify_helper(QObject*, QEvent*) (this=<optimized out>, receiver=0x57c530, e=0x7b0c40) at kernel/qapplication.cpp:3637
#85 0x00007ffff7d6f204 in octave::octave_qapplication::notify(QObject*, QEvent*) (this=0x0, receiver=0x7000000101, ev=0x7f8fb884198f) at ../libgui/src/octave-qobject.cc:147
#86 0x00007ffff50a7358 in QCoreApplication::notifyInternal2(QObject*, QEvent*) (receiver=0x57c530, event=0x7b0c40) at kernel/qcoreapplication.cpp:1064
#87 0x00007ffff50aa8b6 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) (receiver=0x0, event_type=0, data=0x636400) at kernel/qcoreapplication.cpp:1821
#88 0x00007ffff50f8fa7 in postEventSourceDispatch(GSource*, GSourceFunc, gpointer) (s=0x7fffb80121f0) at kernel/qeventdispatcher_glib.cpp:277
#89 0x00007ffff051ae2f in g_main_context_dispatch () at /usr/lib64/libglib-2.0.so.0
#90 0x00007ffff056f508 in g_main_context_iterate.constprop () at /usr/lib64/libglib-2.0.so.0
#91 0x00007ffff05185f3 in g_main_context_iteration () at /usr/lib64/libglib-2.0.so.0
#92 0x00007ffff50f8a48 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) (this=0x7fffb806cd30, flags=...) at kernel/qeventdispatcher_glib.cpp:423
#93 0x00007ffff50a5d62 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) (this=this@entry=0x7fffcb7fd210, flags=..., flags@entry=...) at ../../include/QtCore/../../src/corelib/global/qflags.h:69
#94 0x00007ffff4ee89ca in QThread::exec() (this=<optimized out>) at ../../include/QtCore/../../src/corelib/global/qflags.h:121
#95 0x00007ffff4ee9bf9 in QThreadPrivate::start(void*) (arg=0x5137c0) at thread/qthread_unix.cpp:331
#96 0x00007ffff0a9f832 in start_thread () at /usr/lib64/libc.so.6
#97 0x00007ffff0a3f450 in clone3 () at /usr/lib64/libc.so.6

Dmitri.
--

If I compile w/ "-ggdb3 -O0" it does not crash.

If file is open with non-utf encoding, fprintf does not write anything into it:

octave:48> fid = fopen('05.txt', 'w', 'n', 'Windows-1252')
fid = 22
octave:49> fprintf(fid, 'ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû')
octave:50> f_error = fclose(fid)
f_error = 0

and file 05.txt. (I tried few other encodings with the same result). If I omit encoding, it would write utf-8 string:

octave:54> fid = fopen('06.txt', 'w', 'n')
fid = 22
octave:55> fprintf(fid, 'ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû')
octave:56> f_error = fclose(fid)
f_error = 0

$ file 06.txt
06.txt: UTF-8 Unicode text, with no line terminators
$ cat 06.txt
ÀÂÈÊÌàäéèêìîöòùûÀÂÈÊÌàäéèêìîöòùû

This is with AMD clang/libc

$ clang -v
AMD clang version 14.0.6 (CLANG: AOCC_4.0.0-Build#434 2022_10_28) (based on LLVM Mirror.Version.14.0.6)

Dmitri.
--

The previous comment was from the logs of a macOS 11 runner.
It looks like that test crashed on a macOS 12 runner:

fatal: caught signal Segmentation fault: 11 -- stopping myself...
/bin/sh: line 1: 81520 Segmentation fault: 11  ( /bin/sh ../run-octave --no-init-file --silent --no-history -p /Users/runner/work/octave/octave/.build/test/mex /Users/runner/work/octave/octave/.build/../test/fntests.m /Users/runner/work/octave/octave/.build/../test )
make[4]: *** [check-local] Error 139
  io.tst .........................................................
make[3]: *** [check-am] Error 2
make[2]: *** [check] Error 2
make[1]: *** [check-recursive] Error 1
make: *** [check] Error 2

Ooof.

Is there something related that is specifically different on macOS compared to other platforms?

The octave.space builder is back online and is currently part way through a build of revision 63038dcbd648.

The change that defaults to UTF-8 encoded output files (Matlab compatible) already landed on the stable branch (see comment #40).

If you are on Windows, you could wait for a nightly build from after 31 Mar 2023:
https://buildbot.octave.space/#/download

Unfortunately, the worker that usually builds those nightlies is currently down due to a power outage with no ETA when it will come back online:
https://octave.discourse.group/t/buildbot-tempo-question/4176/9

Alternatively, you could use the MSYS2 build artifact from the GitHub-CI like described here:
https://wiki.octave.org/Continuous_Build#GitHub_build_artifacts


For the patch from comment #46, you'd need to build Octave yourself.

The attached patch handles invalid UTF-8 better than the previous one and also fixes a couple of thinkos.

Trying to get this correct for all cases isn't trivial. I removed one FIXME and added five new ones. I guess that is another proof for the 80-20 rule: It takes about 20% of the time to get something 80% right. But 80% of the time to get the last 20%.

Anyway, I still think this patch is an improvement upon what we currently have because the remaining issues are in more niche cases compared to the more general issue before.

Comments and testing is still appreciated.


(file #54550)

There are still issues with the latest patch if the string to be written contains invalid UTF-8 at the end of the 32 byte buffer chunks.
I'm looking into it currently.

This solves the reported problem and is probably good enough for 8.x release. But the whole encoding issue remains buggy
(e.g. saving and loading a char array that contains non-utf-8
chars results in conversion of this array to utf-8 and thus increasing it size). So we should advise people os trying to stick to UTF-8 encoding everywhere.

Dmitri.
--

I pushed a change to the stable branch that defaults the encoding to UTF-8 when opening streams with `fopen`:
https://hg.savannah.gnu.org/hgweb/octave/rev/0ed6d1f01625

That fixes the issue with the scripts from comment #0 for me. The files are successfully written (in UTF-8).
That is compatible to newer versions of Matlab which also write in UTF-8 if no differing encoding is specified when opening the stream with `fopen`.

That will mitigate this regression for users that don't specify an encoding when opening a stream with `fopen`. I'd guess that is the large majority of cases.

With that in place, we could probably revert https://hg.savannah.gnu.org/hgweb/octave/rev/fcd97a68e5f7 to restore Matlab compatibility without re-introducing the regression for the large majority of cases.

Re-titling the report to reflect the remaining issue.

As I mentioned in comment #28 https://savannah.gnu.org/bugs/?63930#comment28 all the "t" mode does is changes (or not) line endings \n <--> \r\n.
From: https://www.mathworks.com/help/matlab/ref/fopen.html

<<<
On Windows® systems, in text mode:

Read operations that encounter a carriage return followed by a newline character ('\r\n') remove the carriage return from the input.

Write operations insert a carriage return before any newline character in the output.

Open or create a new file in text mode if you want to write to it in MATLAB and then open it in Microsoft® Notepad, or any text editor that does not recognize '\n' as a newline sequence. When writing to the file, end each line with '\r\n'. For an example, see fprintf. Otherwise, open files in binary mode for better performance.

>>>

The fact that "t" changed encodings in octave is manifestation of other bugs.

Dmitri.
--

I'm assuming that you mean "char arrays" when you write "strings".

That'd be a major change. We can't do that to fix the issue in a dot-release. But maybe you could bring that up in the next online meeting?

As a "low-hanging" short-term solution, defaulting to no conversion again (i.e., UTF-8 output independent of the value set for `__mfile_encoding__`) might help to reduce the number of users that are affected.

I might be missing the point. But why are you expecting less issues when converting a UTF-8 stream "on-the-fly" to UTF-16 compared to any other target encoding? The "end of buffer" issue would be exactly the same afaict.
Or are you voting for changing the internal representation of char arrays in Octave to UTF-16?

Sorry for the long text. But this is kind of complicated and we should try to give it some structure if we like to fix this.

Re-encoding the .m files doesn't necessarily mean that the output encoding changes. We should really separate this into "independent" steps in our minds:

1. The parser reads the code from the file.
In this step, the input encoding matters. That is what `__mfile_encoding__` was originally meant to affect.
In older versions of Matlab and Octave this was always the encoding matching the system locale. In newer versions of Matlab, it seems to use heuristics to parse files either in the encoding matching the system locale or UTF-8. In "new-ish" versions of Octave, the encoding for the parser can be set using the `__mfile_encoding__` function.

2. Content is written to an output stream with the `f*` family of functions.
In this step, the output encoding matters.
In Matlab, this always happened (and still does) using the encoding matching the system locale afaict. In older versions of Octave, `char` arrays were actually treated as byte arrays. So, they were just written as is. But that caused issues when we actually needed to know which string a given byte array represented. So, since some time `char` arrays are UTF-8 encoded strings in Octave. Since before Octave 8 (or 7?), the `f*` family of functions didn't convert anything. So, they either wrote whatever bytes the `char` array contained (in old-Octave) - or UTF-8 (in newer Octave).
However, the latter caused incompatibilities with Matlab (which writes in the system locale by default). (Or at least, it used to do that at some point.)
To be closer to Matlab's behavior, Octave was changed to use the encoding selected by `__mfile_encoding__` (i.e., the input encoding used by the parser) also for writing to files by default.
Since Octave now also implements the option to specify the encoding when opening a stream, it is also possible to specify an encoding differing from the encoding used by the parser.

There is also a third case where encoding matters: the encoding the built-in editor uses when saving files. Theoretically, that could be yet another differing encoding. But to avoid surprises, we decided to keep the encoding that the parser uses and the encoding the editor uses in sync.

Additionally, there is a difference between using the GUI and the CLI: The interpreter itself doesn't have options that persist between sessions. The GUI does. That means that if you change the encoding used by the built-in editor (and hence the parser), that setting will persist when you re-start Octave. In contrast, if you change the encoding used by the parser with `__mfile_encoding__` in the CLI, that setting won't persist (unless you do that in one of the startup files).

As a side note: The reason why this seems to be less of an issue for Matlab might be that they are using UTF-16 encoded strings for the internal representation of their char arrays. Most strings consist only (or mainly) of characters from within the BMP. So, they are represented by a single code unit in UTF-16 (compared to surrogate pairs for characters outside the BMP).
Octave on the other hand chose to use UTF-8 for which only ASCII characters can be represented with a single code unit (a byte). All other characters need 2 to 4 bytes to be represented. So, for Octave it is "much more likely" that a buffer might "happen to end" in the middle of a multi-byte sequence.


Skipping the encoding for byte streams is probably a step in the correct direction. But it might still make sense to check what Matlab is doing. E.g., does writing "ä" (or any other multi-byte character) to a byte stream with encoding set to "UTF-8" differ from a file that is written as a text stream with the same encoding?

For the remaining issue of how to correctly convert the encoding on-the-fly when writing to a stream, the following "solutions" come to mind:
1. Change the internal representation of character arrays in Octave to UTF-16 (from currently UTF-8). That would make it less likely for the internal buffer to end "in the middle of a code point". And it would also make Octave more similar to Matlab in that respect.
A couple of years back, we decided against that because we were trying to avoid character conversions on too many interfaces. But now, the conversion facilities in Octave are ready (and we are converting on many interfaces anyway). That won't be something we can even consider to do for a minor release.

2. Try to improve our `codecvt_u8` that manages the conversion from or to UTF-8. I haven't looked into that (and probably won't be able to do so in the nearest future).
But the changes would probably need to happen close to this FIXME note:
https://hg.savannah.gnu.org/hgweb/octave/file/fcd97a68e5f7/liboctave/util/oct-string.cc#l635

3. As a short term "solution", we could change the default for output streams back to UTF-8. If users need to write in a different encoding, using `fflush` on the stream often might be a (unreliable) work-around.

I rolled all the different fprintf tests in this thread into one file called fprintf_test.m. Initially that file was ISO-8859 encoded.

$ octave --eval "__mfile_encoding__ ('latin1'), __mfile_encoding__, fprintf_test"
ans = latin1

$ file *
01_D.txt:       ISO-8859 text
02_D.txt:       ISO-8859 text
04.txt:         ISO-8859 text, with very long lines (384), with no line terminators
05.txt:         ISO-8859 text, with very long lines (384), with no line terminators
06.txt:         ISO-8859 text, with very long lines (384), with no line terminators
07.txt:         ISO-8859 text, with very long lines (384), with no line terminators
fprintf_test.m: ISO-8859 text, with CRLF line terminators

Then I opened fprintf_test.m in a text editor and saved it as UTF-8 with Unix line endings.

$ gedit fprintf_test.m   # saved as UTF-8 with Unix line endings

$ file *
01_D.txt:       ISO-8859 text
02_D.txt:       ISO-8859 text
04.txt:         ISO-8859 text, with very long lines (384), with no line terminators
05.txt:         ISO-8859 text, with very long lines (384), with no line terminators
06.txt:         ISO-8859 text, with very long lines (384), with no line terminators
07.txt:         ISO-8859 text, with very long lines (384), with no line terminators
fprintf_test.m: Unicode text, UTF-8 text

Now running the same command again:

$ octave --eval "__mfile_encoding__ ('latin1'), __mfile_encoding__, fprintf_test"
ans = latin1

$ file *txt
01_D.txt: ISO-8859 text
02_D.txt: Unicode text, UTF-8 text
04.txt:   Unicode text, UTF-8 text, with very long lines (384), with no line terminators
05.txt:   Non-ISO extended-ASCII text, with very long lines (767), with no line terminators
06.txt:   Non-ISO extended-ASCII text, with very long lines (766), with no line terminators
07.txt:   Non-ISO extended-ASCII text, with very long lines (765), with no line terminators

Evidently some functions tacitly pick up on the m-file's encoding and some don't, some give UTF-8, and some give "extended ASCII" that's not ISO or Unicode.

Are we supposed to carry over the m-file's encoding for file operations?

Could someone test on Windows what happens to the text files when the source file changes encoding?

It passes for me too. I can also see that with "t" the file in
latin1 encoding (1-byte per char) and w/o it is in utf-8
(2-bytes per char).

$ od -c /tmp/oct-p1si8a
0000000   a   b   c   , 344 366 374  \n
0000010
$ od -c /tmp/oct-egUaDK
0000000   a   b   c   , 303 244 303 266 303 274  \n
0000013

May be it is good enough for now.

Dmitri.
--

https://www.mathworks.com/help/matlab/ref/fwrite.html

<<<
The MATLAB®char type is not a fixed size, and the number of bytes depends on the encoding scheme associated with the file. Set encoding with fopen.

>>>

Dmitri.
--

That is the only one.
As far as I know "t" mode means converting \n to \r\n . 

As always it is up to matlab compatibility.

If it were up to me it would apply only to char/string
formats in fscanf/fprintf.

It is a mess.

Dmitri.
--

It broke some BISTs in
file-io.cc-tst

(see buildbots e.g.)

Dmitri.
--

Thank you for working on this problem.

Just recall that I have additionally some random behaviour (Window 10).

In file #54500, only one special character is wrong (on last line). All others are ok.

Comment #8 describes some systematical behaviour for which each 16th character is wrong.

Right... I see the same:

$ file *txt
01_D_patched.txt:   Unicode text, UTF-8 text
01_D_unpatched.txt: ISO-8859 text
02_D_patched.txt:   Unicode text, UTF-8 text
02_D_unpatched.txt: ISO-8859 text

I tried tweaking the patch like this:

diff -r c735cce18c7a libinterp/corefcn/oct-stream.h
--- a/libinterp/corefcn/oct-stream.h    Thu Mar 23 16:22:59 2023 -0400
+++ b/libinterp/corefcn/oct-stream.h    Fri Mar 24 11:57:24 2023 -0400
@@ -127,7 +127,7 @@ public:

     // wrap the output stream with encoding conversion facet
     std::ostream *os = output_stream ();
-    if (os && *os)
+    if (os && *os && ! (m_mode & std::ios::binary))
       {
         convfacet_u8 *facet = new convfacet_u8 (m_encoding);
         std::wbuffer_convert<convfacet_u8, char> *converter

But it gave the same results as the previous patch. It fixes fwrite but fprintf needs some exception?

Please test the trial patch at https://savannah.gnu.org/bugs/index.php?63415#comment35

Yes, this is a serious bug and switching to utf-8 is a temp workaround.

Dmitri.
--

Change your editor encodings to "UTF-8"

Dmitri.
--

Bugs https://savannah.gnu.org/bugs/?63930 and https://savannah.gnu.org/bugs/?63931 are probably related.

When writing bytes with large value to a file (either with fprintf for text files or with fwrite for binary files) the desired value is sometimes replaced by 0x3F.

Thus, this has probably nothing to do with code pages and encoding...

I found some systematical behaviour of the problem. This is described below:

Test cases and comparisons are appended (fprintf_bug.tgz). I also join pictures of the
comparisons because I'm not sure that you see the same thing than me.

1)
If the m-file only writes special characters to the file, the proble does not appear.

Test case : fprintf_bug4.m => 04.txt

2)
Once a standard character is outputted (incl \n), each 16th character is wrong (replaced by char 3F).

Test case : fprintf_bug5.m   =>  05.txt


3)
If a second standard character is outputted, the problem disappear for the rest of the file.

Test case : fprintf_bug6.m   =>  06.txt

4)
if a third standard character is outputted, the problem appears again for the rest of the file.

Test case : fprintf_bug7.m   =>  07.txt


(file #54508)

Thanks.

The encoding of the strings is actually converted twice.
1. When parsing the .m file, the entire file is converted from codepage 1252 to UTF-8. (That is done so that Octave can potentially feed those strings to other libraries, like for regex matching or printing text in figures.)
2. When writing to the file, each character array is converted from UTF-8 to the target encoding. That is either the one specified when opening the file (with `fopen`), or, if none is specified, the encoding that is used for parsing .m files (the encoding returned by `__mfile_encoding__`). That was done to be compatible with Matlab.

I'm guessing that something might be going wrong in the second conversion if the output is buffered, and the buffer is flushed when the end of it happens to be "between" bytes of a multi-byte UTF-8 character...

Do I understand correctly that the two text files were created on two different PCs with different locale?

Instead of images of text, could you please attach the actual generated text files?

It seems that there might be an issue with the encoding of the text file being written by the fprintf function in Octave 8.1.0. The character on line 29 that is being printed incorrectly might be due to a mismatch in the encoding being used by the function and the encoding of the file.

One possible solution to this issue is to explicitly specify the encoding being used by the fprintf function. For example, if the text file is in UTF-8 encoding, the fprintf function can be called with the '-encoding' option set to 'UTF-8'

fprintf_bug1.m illustrates a strange behavior of the fprintf function which appeared with the new version of Octave (8.1.0).

The script writes 2 text files with the same content, except on line 5, but the difference is small.
One of the two files is written correctly, while the other one contains an error on another line (29) containing a special character.

The comparison between the 2 text file is plotted in the appended file File_comparison_1.png.

The character which is printed where a ? in the comparison is 0x3F instead of 0xFB.

Any small change in the m-file can modify the result. Therefore additional test files with the corresponding comparison.

I tried this on 2 different PCs with different codepages:
PS C:\> chcp
Active code page: 850

PS C:\> chcp
Active code page: 437