bugGNU Octave - Bugs: bug #66448, File encoding conversion errors on...

 
 

bug #66448: File encoding conversion errors on macOS 14

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Fri 15 Nov 2024 05:52:28 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Release: 
Operating System:  * Mac OS Fixed Release:  10.1.0
Planned Release:  10.1.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 18 Nov 2024 05:14:31 PM UTC, comment #11: 

Like already mentioned, the GitHub CI succeeded with that change.
Unfortunately, that means that Octave has a new dependency on macOS.
When it comes to potentially graft the gnulib update to the stable branch: I'm not sure if we'd like to change dependency requirements for minor releases. (It's not a "full-blown" API or ABI change though afaict.)
We could probably ask for the opinion of others during next online developer meeting.

Anyway, closing this report as fixed (in the CI rules).


Markus Mützel <mmuetzel>
Group administrator
Sun 17 Nov 2024 06:23:09 PM UTC, comment #10: 

Thanks for tracking this down.

I pushed the following change to the CI rules on macOS:
https://hg.savannah.gnu.org/hgweb/octave/rev/1de4eedfe2fe

Homebrew prints the following when installing the libiconv package:

==> libiconv
libiconv is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have libiconv first in your PATH, run:
  echo 'export PATH="/usr/local/opt/libiconv/bin:$PATH"' >> /Users/runner/.bash_profile

For compilers to find libiconv you may need to set:
  export LDFLAGS="-L/usr/local/opt/libiconv/lib"
  export CPPFLAGS="-I/usr/local/opt/libiconv/include"


I don't know what the "all kinds of trouble" is hinting at. But at least in preliminary tests, it avoids the failing tests in the CI.

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Sun 17 Nov 2024 05:25:17 PM UTC, comment #9: 

I added to configure
"--with-libiconv-prefix=/opt/homebrew/Cellar/libiconv/1.17"
and now it works again.

I can send you config.* files with and without, if that can be helpful.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 17 Nov 2024 05:14:11 PM UTC, comment #8: 

I have noticed that my hg top was not up to date. (I was using hg.octave.org). I recloned from savannah and now I see the same problems:


file-encoding/file-encoding.tst ................................ pass    4/11
                                                                   FAIL    3
                                                              REGRESSION   4


and



>>>>> processing /Users/dmitri/src/octave/test/file-encoding/file-encoding.tst
***** test
 help_str = get_help_text ("test_CP1252");
 ## The help text contains the string "ÄÖÜäöü ŠŽšž".  Compare to the
 ## corresponding UTF-8 byte sequence to make sure this test is independent
 ## of how this .tst file is interpreted.
 ref_str = char ([195 132 195 150 195 156 195 164 195 182 195 188 32 ...
                  197 160 197 189 197 161 197 190]);
 assert (strfind (help_str, ref_str));
!!!!! test failed
file_reader::get_input: converting from codepage 'windows-1252' to UTF-8: Function not implemented
***** test
 path_orig = path ();
 unwind_protect
   addpath ("CP1251");
   help_str = get_help_text ("test_CP1251");
   ## The help text contains the string "ДЦЬдць ЉЋљћ".  Compare to the
   ## corresponding UTF-8 byte sequence to make sure this test is independent
   ## of how this .tst of how this .tst file is interpreted.
   ref_str = char ([208 148 208 166 208 172 208 180 209 134 209 140 32 ...
                    208 137 208 139 209 153 209 155]);
   assert (strfind (help_str, ref_str));
 unwind_protect_cleanup
   path (path_orig);
   dir_encoding (canonicalize_file_name ("CP1251"), "delete");
 end_unwind_protect
!!!!! test failed
file_reader::get_input: converting from codepage 'windows-1251' to UTF-8: Function not implemented
***** test
 clear all;  # make sure files are re-parsed
 path_orig = path ();
 unwind_protect
   addpath (canonicalize_file_name ("CP1251"));
   help_str = get_help_text ("test_CP1251");
   ## The help text contains the string "ДЦЬдць ЉЋљћ".  Compare to the UTF-8
   ## byte sequence to make sure this test is independent of how this .tst
   ## file is interpreted.
   ref_str = char ([208 148 208 166 208 172 208 180 209 134 209 140 32 ...
                    208 137 208 139 209 153 209 155]);
   assert (strfind (help_str, ref_str));
 unwind_protect_cleanup
   path (path_orig);
   dir_encoding (canonicalize_file_name ("CP1251"), "delete");
 end_unwind_protect
!!!!! test failed
file_reader::get_input: converting from codepage 'windows-1251' to UTF-8: Function not implemented
***** test <*62780>
 ## wrap in "evalc" to suppress output to the log
 evalc ('oruntests (".");');
!!!!! regression: https://octave.org/testfailure/?62780
native2unicode: iconv() is not supported.  Installing GNU libiconv and then re-compiling Octave could fix this.
***** test <*62780>
 unwind_protect
   dir_encoding (canonicalize_file_name ("CP1251"), "windows-1251");
   ## use "evalc" to suppress output to the log
   evalc ('oruntests ("CP1251");');
 unwind_protect_cleanup
   dir_encoding (canonicalize_file_name ("CP1251"), "delete");
 end_unwind_protect
!!!!! regression: https://octave.org/testfailure/?62780
native2unicode: iconv() is not supported.  Installing GNU libiconv and then re-compiling Octave could fix this.
***** test <*62780>
 old_mfile_encoding = mfile_encoding ("windows-1251");
 unwind_protect
   ## use "evalc" to suppress output to the log
   evalc ('oruntests ("CP1251");');
 unwind_protect_cleanup
   mfile_encoding (old_mfile_encoding);
 end_unwind_protect
!!!!! regression: https://octave.org/testfailure/?62780
native2unicode: iconv() is not supported.  Installing GNU libiconv and then re-compiling Octave could fix this.
***** test <*62780>
 path_orig = path ();
 unwind_protect
   addpath (canonicalize_file_name ("CP1251"));
   ## use "evalc" to suppress output to the log
   evalc ('oruntests ("CP1251");');
 unwind_protect_cleanup
   path (path_orig);
   dir_encoding (canonicalize_file_name ("CP1251"), "delete");
 end_unwind_protect
!!!!! regression: https://octave.org/testfailure/?62780
native2unicode: iconv() is not supported.  Installing GNU libiconv and then re-compiling Octave could fix this.
>>>>> processing /Users/dmitri/src/octave/test/help/help.tst
>>>>> processing /Users/dmitri/src/octave/test/json/jsondecode.tst
>>>>> processing /Users/dmitri/src/octave/test/json/jsonencode.tst
***** testif HAVE_RAPIDJSON_PRETTYWRITER
 data = {'Hello'; 'World!'};
 exp  = do_string_escapes ([ '[\n', ...
                             '  "Hello",\n', ...
                             '  "World!"\n', ...
                             ']' ]);
 obs  = jsonencode (data, 'PrettyPrint', true);
 assert (isequal (obs, exp));

 exp  = '["Hello","World!"]';
 obs  = jsonencode (data, 'PrettyPrint', false);
 assert (isequal (obs, exp));
----- skipped test (missing feature)

***** testif HAVE_RAPIDJSON_PRETTYWRITER
 data = [1, 2; 3, 4];
 exp  = do_string_escapes ([ ...
 '[\n'                       ...
 '  [\n'                   ...
 '    1,\n'              ...
 '    2\n'               ...
 '  ],\n'                  ...
 '  [\n'                   ...
 '    3,\n'              ...
 '    4\n'               ...
 '  ]\n'                   ...
 ']' ]);
 obs  = jsonencode (data, 'PrettyPrint', true);
 assert (isequal (obs, exp));

 exp  = '[[1,2],[3,4]]';
 obs  = jsonencode (data, 'PrettyPrint', false);
 assert (isequal (obs, exp));
----- skipped test (missing feature)

>>>>> processing /Users/dmitri/src/octave/test/jupyter-notebook/jupyter-notebook.tst


% hg id
226a392f0818 @

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 17 Nov 2024 02:55:56 PM UTC, comment #7: 

I have Apple's cpu (M4). Do you have the same CommandLineTools
on all the systems? I use libiconv-1.17 from Homebrew.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 17 Nov 2024 02:06:54 PM UTC, comment #6: 

In case that should matter: The GitHub-hosted macos-13 runners are on Intel Silicon, the macos-14 runners are on Apple Silicon.

Maybe, the issue is specific to the architecture?

Markus Mützel <mmuetzel>
Group administrator
Sun 17 Nov 2024 12:52:55 AM UTC, comment #5: 

OK, the final build with all the usual config options enabled.

  while.tst ...................................................... pass    6/6    [ 0.003s /  0.003s]
                                                        total time (CPU / CLOCK)  [  91.8s /   93.7s]
Failure Summary:

  libinterp/corefcn/conv2.cc-tst ................................. pass   48/51
                                                    (reported bug) XFAIL   3
  libinterp/corefcn/data.cc-tst .................................. pass 1260/1266
                                                    (reported bug) XFAIL   6
  libinterp/corefcn/dlmread.cc-tst ............................... pass    7/9
                                                    (reported bug) XFAIL   2
  libinterp/corefcn/file-io.cc-tst ............................... pass  100/101
                                                    (reported bug) XFAIL   1
  libinterp/corefcn/gsvd.cc-tst .................................. pass   44/46
                                                    (reported bug) XFAIL   2
  libinterp/corefcn/inv.cc-tst ................................... pass   60/62
                                                    (reported bug) XFAIL   2
  libinterp/corefcn/mappers.cc-tst ............................... pass  414/420
                                                    (reported bug) XFAIL   6
  libinterp/corefcn/max.cc-tst ................................... pass  111/114
                                                    (reported bug) XFAIL   3
  libinterp/corefcn/pr-output.cc-tst ............................. pass   12/13
                                                    (reported bug) XFAIL   1
  libinterp/corefcn/regexp.cc-tst ................................ pass  121/122
                                                    (reported bug) XFAIL   1
  libinterp/corefcn/sparse-xpow.cc-tst ........................... pass    3/5
                                                    (reported bug) XFAIL   2
  libinterp/corefcn/strfns.cc-tst ................................ pass  126/129
                                                    (reported bug) XFAIL   3
  libinterp/octave-value/ov-java.cc-tst .......................... pass   13/15
                                                    (reported bug) XFAIL   2
  libinterp/parse-tree/pt-tm-const.cc-tst ........................ pass  202/203
                                                    (reported bug) XFAIL   1
  general/interp2.m .............................................. pass   60/62
                                                    (reported bug) XFAIL   2
  general/logspace.m ............................................. pass   10/12
                                                    (reported bug) XFAIL   2
  io/importdata.m ................................................ pass   25/26
                                                    (reported bug) XFAIL   1
  plot/appearance/camlookat.m .................................... pass    9/10
                                                    (reported bug) XFAIL   1
  plot/draw/isosurface.m ......................................... pass   29/30
                                                    (reported bug) XFAIL   1
  specfun/gammainc.m ............................................. pass   87/88
                                                    (reported bug) XFAIL   1
  specfun/realpow.m .............................................. pass    6/7
                                                    (reported bug) XFAIL   1
  statistics/mean.m .............................................. pass   79/80
                                                    (reported bug) XFAIL   1
  statistics/normalize.m ......................................... pass   99/101
                                                    (reported bug) XFAIL   2
  testfun/test.m ................................................. pass   56/57
                                                    (reported bug) XFAIL   1
  bug-44643/bug-44643.tst ........................................ pass    0/1
                                                    (reported bug) XFAIL   1
  bug-49379/bug-49379.tst ........................................ pass    0/1
                                                    (reported bug) XFAIL   1
  classdef/classdef.tst .......................................... pass   36/38
                                                    (reported bug) XFAIL   2
  classdef-debug/test-classdef-breakpoints.tst ................... pass    5/6
                                                    (reported bug) XFAIL   1
  fcn-handle/bug-51567.tst ....................................... pass    0/2
                                                    (reported bug) XFAIL   2
  eval-command.tst ............................................... pass   13/16
                                                    (reported bug) XFAIL   3
  io.tst ......................................................... pass  168/173
                                                    (reported bug) XFAIL   5

Summary:

  PASS                            19765
  FAIL                                0
  XFAIL (reported bug)               63
  SKIP (missing feature)              3
  SKIP (run-time condition)          62


WRT iconv:

config.log:ac_cv_header_iconv_h=yes
config.log:am_cv_func_iconv=yes
config.log:am_cv_func_iconv_summary='yes, in libiconv'
config.log:am_cv_func_iconv_works=yes
config.log:am_cv_lib_iconv=yes
config.log:gl_cv_iconv_nonconst=yes
config.log:gl_cv_next_iconv_h='<iconv.h>'
config.log:octave_cv_iconv_canonicalize=yes
config.log:octave_cv_iconvlist=yes
config.log:GNULIB_LINK_DEPS='         -liconv   -Wl,-framework -Wl,CoreFoundation -Wl,-framework -Wl,CoreServices'
config.log:ICONV_H='iconv.h'
config.log:LIBICONV='-liconv'



Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 16 Nov 2024 08:48:07 PM UTC, comment #4: 

Same on 10.0.0 Also, running "make check":

...
fcn-handle/object-method.tst ................................... pass    4/4    [ 0.007s /  0.007s]
  fcn-handle/package-function.tst ................................ pass    4/4    [ 0.002s /  0.002s]
  fcn-handle/shared-ctx.tst ...................................... pass    1/1    [ 0.006s /  0.006s]
  fcn-handle/static-method.tst ................................... pass    4/4    [ 0.002s /  0.002s]
  file-encoding/file-encoding.tst ................................ pass   11/11   [ 0.663s /  0.664s]
  help/help.tst .................................................. pass   21/21   [ 0.147s /  0.217s]
...


% hg id
40c84a2c3aa3 @

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 16 Nov 2024 07:42:06 PM UTC, comment #3: 

This is on MacOS 15.1 if that matters.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 16 Nov 2024 07:41:13 PM UTC, comment #2: 

On 9.2.1 (On MacOS) I see:

>>>>> /Users/dmitri/src/octave/test/file-encoding/file-encoding.tst
***** assert (dir_encoding ("."), "windows-1252")
!!!!! test failed
ASSERT errors for:  assert (dir_encoding ("."),"windows-1252")

  Location  |  Observed  |  Expected  |  Reason
     []        'utf-8'    'windows-1252'   Strings don't match
octave:8> ver
----------------------------------------------------------------------
GNU Octave Version: 9.2.1 (hg id: 563f1654764a)
GNU Octave License: GNU General Public License
Operating System: Darwin 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:06:57 PDT 2024; root:xnu-11215.41.3~3/RELEASE_ARM64_T6041 arm64
----------------------------------------------------------------------
no packages installed.


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 16 Nov 2024 11:32:08 AM UTC, comment #1: 

Might be related to iconv.

Before the gnulib update:

checking for iconv... yes
checking for working iconv... yes
checking how to link with libiconv... -liconv
checking whether iconv is compatible with its POSIX signature... yes


After the gnulib update:

checking for iconv... yes
checking for working iconv... no
checking whether iconv is compatible with its POSIX signature... yes


Markus Mützel <mmuetzel>
Group administrator
Fri 15 Nov 2024 05:52:28 PM UTC, original submission:  

Some tests have started to fail on macOS 14 after updating gnulib:
https://github.com/gnu-octave/octave/actions/runs/11843167971/job/33003488322#step:13:67271


>>>>> processing /Users/runner/work/octave/octave/test/file-encoding/file-encoding.tst
***** test
 help_str = get_help_text ("test_CP1252");
 ## The help text contains the string "ÄÖÜäöü ŠŽšž".  Compare to the
 ## corresponding UTF-8 byte sequence to make sure this test is independent
 ## of how this .tst file is interpreted.
 ref_str = char ([195 132 195 150 195 156 195 164 195 182 195 188 32 ...
                  197 160 197 189 197 161 197 190]);
 assert (strfind (help_str, ref_str));
!!!!! test failed
file_reader::get_input: converting from codepage 'windows-1252' to UTF-8: Function not implemented
***** test
 path_orig = path ();
 unwind_protect
   addpath ("CP1251");
   help_str = get_help_text ("test_CP1251");
   ## The help text contains the string "ДЦЬдць ЉЋљћ".  Compare to the
   ## corresponding UTF-8 byte sequence to make sure this test is independent
   ## of how this .tst of how this .tst file is interpreted.
   ref_str = char ([208 148 208 166 208 172 208 180 209 134 209 140 32 ...
                    208 137 208 139 209 153 209 155]);
   assert (strfind (help_str, ref_str));
 unwind_protect_cleanup
   path (path_orig);
   dir_encoding (canonicalize_file_name ("CP1251"), "delete");
 end_unwind_protect
!!!!! test failed
file_reader::get_input: converting from codepage 'windows-1251' to UTF-8: Function not implemented
***** test
 clear all;  # make sure files are re-parsed
 path_orig = path ();
 unwind_protect
   addpath (canonicalize_file_name ("CP1251"));
   help_str = get_help_text ("test_CP1251");
   ## The help text contains the string "ДЦЬдць ЉЋљћ".  Compare to the UTF-8
   ## byte sequence to make sure this test is independent of how this .tst
   ## file is interpreted.
   ref_str = char ([208 148 208 166 208 172 208 180 209 134 209 140 32 ...
                    208 137 208 139 209 153 209 155]);
   assert (strfind (help_str, ref_str));
 unwind_protect_cleanup
   path (path_orig);
   dir_encoding (canonicalize_file_name ("CP1251"), "delete");
 end_unwind_protect
!!!!! test failed
file_reader::get_input: converting from codepage 'windows-1251' to UTF-8: Function not implemented
***** test <*62780>
 ## wrap in "evalc" to suppress output to the log
 evalc ('oruntests (".");');
!!!!! regression: https://octave.org/testfailure/?62780
native2unicode: iconv() is not supported.  Installing GNU libiconv and then re-compiling Octave could fix this.
***** test <*62780>
 unwind_protect
   dir_encoding (canonicalize_file_name ("CP1251"), "windows-1251");
   ## use "evalc" to suppress output to the log
   evalc ('oruntests ("CP1251");');
 unwind_protect_cleanup
   dir_encoding (canonicalize_file_name ("CP1251"), "delete");
 end_unwind_protect
!!!!! regression: https://octave.org/testfailure/?62780
native2unicode: iconv() is not supported.  Installing GNU libiconv and then re-compiling Octave could fix this.
***** test <*62780>
 old_mfile_encoding = mfile_encoding ("windows-1251");
 unwind_protect
   ## use "evalc" to suppress output to the log
   evalc ('oruntests ("CP1251");');
 unwind_protect_cleanup
   mfile_encoding (old_mfile_encoding);
 end_unwind_protect
!!!!! regression: https://octave.org/testfailure/?62780
native2unicode: iconv() is not supported.  Installing GNU libiconv and then re-compiling Octave could fix this.
***** test <*62780>
 path_orig = path ();
 unwind_protect
   addpath (canonicalize_file_name ("CP1251"));
   ## use "evalc" to suppress output to the log
   evalc ('oruntests ("CP1251");');
 unwind_protect_cleanup
   path (path_orig);
   dir_encoding (canonicalize_file_name ("CP1251"), "delete");
 end_unwind_protect
!!!!! regression: https://octave.org/testfailure/?62780
native2unicode: iconv() is not supported.  Installing GNU libiconv and then re-compiling Octave could fix this.


They are still passing on macOS 13.

Might or might not be related to updating gnulib. The last CI run before the gnulib update still passed the tests.

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-11-18 mmuetzel Fixed ReleaseNone 10.1.0
    2024-11-18 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2024-11-17 mmuetzel StatusNone Ready For Test
        Planned ReleaseNone 10.1.0
    2024-11-16 mmuetzel Carbon-Copy- Added dasergatskov

    Back to the top

    Powered by Savane 3.15-4cd8.
    Corresponding source code