bugGNU Octave - Bugs: bug #65963, pkg install "fopen: encoding...

 
 

bug #65963: pkg install "fopen: encoding must be 'UTF-8'' warnings on macOS 14 AS

Submitter:  Andrew Janke <apjanke>
Submitted:  Sun 07 Jul 2024 02:52:27 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Ready For Test Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * 9.1.0 Operating System:  * Mac OS
Fixed Release:  None Planned Release:  9.3.0 (current stable)
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 08 Jul 2024 01:22:58 PM UTC, comment #4: 

I pushed a patch to the stable branch that should avoid the warning on installations that use libc++ if `fopen` is called with `system` as the value for the encoding identifier and if the locale charset is UTF-8:
https://hg.savannah.gnu.org/hgweb/octave/rev/a26c012c0860

It also uses a more verbose warning message as suggested on discourse.

It should be part of the next minor release of Octave 9.

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Mon 08 Jul 2024 07:59:48 AM UTC, comment #3: 

Responding to just a couple parts of this...

> Did you run your tests with the GUI or CLI?


That was all in the GUI. I ran some CLI tests just now. None of them have the warnings unless I set mfile_encoding('system') inside that session or one of the startup files.

> The default mfile_encoding should be "utf-8" on all platforms.


I think what happened is a couple years back, I called the old `__mfile_encoding__` on that one computer, using an older version of Octave, to get it set to 'system'. And that didn't have any impact until this newer 9.x version of Octave. I now think the macOS 12 vs. 14 and Intel vs. Apple Silicon stuff is a red herring. I can reproduce the same behavior on all of them if I call `mfile_encoding` on them.

> it would be best to avoid double-posting the same issue in multiple places


I'll stick to one or the other from now on.

Andrew Janke <apjanke>
Mon 08 Jul 2024 07:17:34 AM UTC, comment #2: 

This might be a couple of different bugs or misunderstandings.

  • When starting the Octave CLI, the GUI settings are not read. So, any changes for the encoding or language that were done in the GUI settings don't apply. If you'd like to change the default .m file encoding for the CLI, consider setting it in one of the startup files.


  • The value of `OCTAVE_HAVE_STRICT_ENCODING_FACET` is set in `oct-conf-post-private.h` with an explanatory comment:


#if defined (HAVE_LLVM_LIBCXX)
/* The stream encoding facet from libc++ is stricter than libstdc++ when
   it comes to reverting the stream.  Disable encoding conversion for file
   streams with libc++.
   FIXME: Maybe use a more specific test.  */
#  define OCTAVE_HAVE_STRICT_ENCODING_FACET 1
#endif


Patches for implementing an encoding conversion for file streams with libc++ are welcome. (I probably won't work on that because the only platform that I know that uses libc++ is macOS. And I don't have hardware to test anything with that OS. I'd be happy to review proposals though.)

  • Immediately before the code snippet you quoted, there is the following:


  // Valid names for encodings consist of ASCII characters only.
  std::transform (encoding.begin (), encoding.end (), encoding.begin (),
                  ::tolower);

  if (encoding == "system")
    encoding = octave_locale_charset_wrapper ();

#if defined (OCTAVE_HAVE_STRICT_ENCODING_FACET)
  if (encoding != "utf-8")
    {
      warning_with_id ("Octave:fopen:encoding-unsupported",
                       "fopen: encoding must be 'UTF-8' for this version");
      encoding = "utf-8";
    }
#endif


The original string `encoding` is transformed to lower-case. We probably need to keep that there to be able to reliably recognize "system" (case-insensitive). But it might well be that the locale_charset returns a value that is not all lower-case.
We'll probably need to convert that value to lower-case again for that warning to disappear for you.

  • The warning is probably correct in a sense that "on-the-fly" encoding conversions don't work on that platform (libc++, see above). You can still convert between encodings on "static" strings (i.e., `char` vectors in Octave) with `unicode2native` or `native2unicode`. I.e., open the file (e.g., in byte mode), read it's content as `int8`, then convert to a UTF-8 encoded `char` vector with `native2unicode`.


  • The default mfile_encoding should be "utf-8" on all platforms. See, e.g., the following for the default value of the GUI settings:

https://hg.savannah.gnu.org/hgweb/octave/file/d539aff8b327/libgui/src/gui-preferences-ed.cc#l234

gui_pref
ed_default_enc ("editor/default_encoding", QVariant ("UTF-8"));


Did you run your tests with the GUI or CLI?

PS: Imho, it would be best to avoid double-posting the same issue in multiple places (for whatever reason). Even if you cross-link to the other place (that's better than nothing), the discussion will inevitably be split and trying to follow it in the future might become more difficult than necessary...

Markus Mützel <mmuetzel>
Group administrator
Sun 07 Jul 2024 03:07:31 PM UTC, comment #1: 

I posted about this on octave.discourse.group, too, because I'm interested in getting some broader discussion about the general state of text encoding on recent Octave versions. It links to this ticket.

https://octave.discourse.group/t/m-file-encodings-utf-8-on-macos-and-fopen/5733

Andrew Janke <apjanke>
Sun 07 Jul 2024 02:52:27 PM UTC, original submission:  

Hi, folks.

Starting with Octave 9.1 and affecting 9.2, when I do any `pkg install -forge` installation, I'm getting a bunch of "warning: fopen: encoding must be 'UTF-8' for this version" warnings.

Only on macOS 14 on Apple Silicon; macOS 14 on Intel, and macOS 12 on either architecture, do not produce this warning. Don't know if that's a difference in the OS & arch, or just a difference in how these particular machines of mine are configured.

The warning looks like:


warning: fopen: encoding must be 'UTF-8' for this version
warning: called from
    install>extract_pkg at line 332 column 7
    install>create_pkgadddel at line 761 column 7
    install at line 239 column 7
    pkg at line 619 column 9


I get dozens or hundreds of them on each pkg install operation.

Diagnosis and guesses


The code around the fopen() at the bottom of that stack trace is, in 9.2.0:


function pkg = extract_pkg (nm, pat)

  fid = fopen (nm, "rt", "n", mfile_encoding ());
  pkg = "";
  if (fid >= 0)


On my macOS 14 AS machine (eilonwy, bare metal), `mfile_encoding()` returns 'system'. On my other machines (macOS 14 Intel, and macOS 12 either arch), mfile_encoding() returns 'utf-8'.

I'm guessing that fopen() doesn't like 'system' for its encoding argument? The helptext for fopen() says it supports 'native' as an argument there, but not 'system'. Looks like it has about the same meaning as mfile_encoding's 'system' value.

Discussion


Is 'system' supposed to be supported here? Should pkg's install() be doing some translation?

What's the default for mfile_encoding supposed to be (on macOS)? Is that set in a user configuration somewhere that persists between Octave sessions? Wondering if I have some setting on that one machine that's messing things up, either because I set it at some point, or it's left over from running an older version of Octave that set it somewhere persistent, or something like that.

I called `mfile_encoding('utf-8')` on eilonwy, the machine where it was returning 'system', and restarted Octave, and now it's returning 'utf-8'. And I see "default_encoding=UTF-8" in my ~/.config/octave/octave-gui.ini file. It switches to "default_encoding=SYSTEM (UTF-8)" if I run `mfile_encoding ('system')". After doing that, in CLI octave, mfile_encoding() returns 'utf-8', but in `octave --gui`, mfile_encoding() returns 'system'. That difference in behavior is a little surprising.

The warning says the encoding "must" be UTF-8. Is that true? Does fopen not support other encodings in this platform/situation?

I see this in libinterp/corefcn/file-io.cc in the Octave source. I don't know what this strict encoding facet stuff is. But it seems odd to limit it to UTF-8 in a general IO operation like fopen. Modern systems can still handle other encodings I think, and it's not uncommon to have or receive files that are in other encodings, like UTF-16 or non-Unicode encodings, even today in 2024.


#if defined (OCTAVE_HAVE_STRICT_ENCODING_FACET)
  if (encoding != "utf-8")
    {
      warning_with_id ("Octave:fopen:encoding-unsupported",
                       "fopen: encoding must be 'UTF-8' for this version");
      encoding = "utf-8";
    }
#endif


Also, it looks like that's a case-sensitive comparison, and the value must be lower-case "utf-8", but the error message is talking about upper case "UTF-8".

Test results


I tried this on various OS, arch, and Octave combinations over this weekend. Here's what I observed.

On AS (eilonwy):

  • In 9.2.0 brewed on macOS 14 AS: warnings
  • In Octave.app 9.2.0 beta1 on macOS 14 AS: same warnings
  • In Octave.app 9.1.0 beta4 on macOS 14 AS: same warnings
  • In Octave.app 8.4.0 on macOS 14 AS: no warning
  • In Octave.app 6.2.0 on macOS 14 AS: no warning
  • In Octave.app 4.4.1 on macOS 14 AS: `pkg install -forge control` fails at linker step; no warning up until then.


On AS (oabuild-as (macOS 12) VM on eilonwy):

  • In Octave.app 9.2.0 beta1 on macOS 12 AS: no warnings! huh.
  • In Octave.app 9.1.0 beta4 on macOS 12 AS: no warnings! huh.
  • In Octave.app 8.4.0 on macOS 12 AS: no warnings


On Intel (angharad):

  • Brewed octave 9.2.0 on macOS 14 Intel: no warnings
  • Octave.app 9.2.0 beta1 on macOS 14 Intel: no warnings
  • Octave.app 9.1.0 beta4 on macOS 14 Intel: no warnings
  • Octave.app 8.4.0 on macOS 14 Intel: no warnings
  • Octave.app 6.2.0 on macOS 14 Intel: no warnings



mfile_encoding() returns:

  • brewed Octave 9.2.0 on macOS 14 AS: 'system'
  • brewed Octave 9.2.0 on macOS 14 Intel: 'utf-8'
  • Octave.app 9.1.0 beta4 on macOS 12 AS (VM): 'utf-8'
  • Octave.app 9.1.0 beta4 on macOS 12 Intel (VM): 'utf-8'


Andrew Janke <apjanke>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by apjanke (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-07-08 mmuetzel StatusNone Ready For Test
        Planned ReleaseNone 9.3.0 (current stable)

    Back to the top

    Powered by Savane 3.13-b921.
    Corresponding source code