I pushed a patch to the stable branch that should avoid the warning on installations that use libc++ if `fopen` is called with `system` as the value for the encoding identifier and if the locale charset is UTF-8:
https://hg.savannah.gnu.org/hgweb/octave/rev/a26c012c0860

It also uses a more verbose warning message as suggested on discourse.

It should be part of the next minor release of Octave 9.

Marking as ready for test.

This might be a couple of different bugs or misunderstandings.

• When starting the Octave CLI, the GUI settings are not read. So, any changes for the encoding or language that were done in the GUI settings don't apply. If you'd like to change the default .m file encoding for the CLI, consider setting it in one of the startup files.

• The value of `OCTAVE_HAVE_STRICT_ENCODING_FACET` is set in `oct-conf-post-private.h` with an explanatory comment:

#if defined (HAVE_LLVM_LIBCXX)
/* The stream encoding facet from libc++ is stricter than libstdc++ when
   it comes to reverting the stream.  Disable encoding conversion for file
   streams with libc++.
   FIXME: Maybe use a more specific test.  */
#  define OCTAVE_HAVE_STRICT_ENCODING_FACET 1
#endif

Patches for implementing an encoding conversion for file streams with libc++ are welcome. (I probably won't work on that because the only platform that I know that uses libc++ is macOS. And I don't have hardware to test anything with that OS. I'd be happy to review proposals though.)

• Immediately before the code snippet you quoted, there is the following:

  // Valid names for encodings consist of ASCII characters only.
  std::transform (encoding.begin (), encoding.end (), encoding.begin (),
                  ::tolower);

  if (encoding == "system")
    encoding = octave_locale_charset_wrapper ();

#if defined (OCTAVE_HAVE_STRICT_ENCODING_FACET)
  if (encoding != "utf-8")
    {
      warning_with_id ("Octave:fopen:encoding-unsupported",
                       "fopen: encoding must be 'UTF-8' for this version");
      encoding = "utf-8";
    }
#endif

The original string `encoding` is transformed to lower-case. We probably need to keep that there to be able to reliably recognize "system" (case-insensitive). But it might well be that the locale_charset returns a value that is not all lower-case.
We'll probably need to convert that value to lower-case again for that warning to disappear for you.

• The warning is probably correct in a sense that "on-the-fly" encoding conversions don't work on that platform (libc++, see above). You can still convert between encodings on "static" strings (i.e., `char` vectors in Octave) with `unicode2native` or `native2unicode`. I.e., open the file (e.g., in byte mode), read it's content as `int8`, then convert to a UTF-8 encoded `char` vector with `native2unicode`.

• The default mfile_encoding should be "utf-8" on all platforms. See, e.g., the following for the default value of the GUI settings:

https://hg.savannah.gnu.org/hgweb/octave/file/d539aff8b327/libgui/src/gui-preferences-ed.cc#l234

gui_pref
ed_default_enc ("editor/default_encoding", QVariant ("UTF-8"));

Did you run your tests with the GUI or CLI?

PS: Imho, it would be best to avoid double-posting the same issue in multiple places (for whatever reason). Even if you cross-link to the other place (that's better than nothing), the discussion will inevitably be split and trying to follow it in the future might become more difficult than necessary...

Hi, folks.

Starting with Octave 9.1 and affecting 9.2, when I do any `pkg install -forge` installation, I'm getting a bunch of "warning: fopen: encoding must be 'UTF-8' for this version" warnings.

Only on macOS 14 on Apple Silicon; macOS 14 on Intel, and macOS 12 on either architecture, do not produce this warning. Don't know if that's a difference in the OS & arch, or just a difference in how these particular machines of mine are configured.

The warning looks like:

warning: fopen: encoding must be 'UTF-8' for this version
warning: called from
    install>extract_pkg at line 332 column 7
    install>create_pkgadddel at line 761 column 7
    install at line 239 column 7
    pkg at line 619 column 9

I get dozens or hundreds of them on each pkg install operation.

Diagnosis and guesses

The code around the fopen() at the bottom of that stack trace is, in 9.2.0:

function pkg = extract_pkg (nm, pat)

  fid = fopen (nm, "rt", "n", mfile_encoding ());
  pkg = "";
  if (fid >= 0)

On my macOS 14 AS machine (eilonwy, bare metal), `mfile_encoding()` returns 'system'. On my other machines (macOS 14 Intel, and macOS 12 either arch), mfile_encoding() returns 'utf-8'.

I'm guessing that fopen() doesn't like 'system' for its encoding argument? The helptext for fopen() says it supports 'native' as an argument there, but not 'system'. Looks like it has about the same meaning as mfile_encoding's 'system' value.

Discussion

Is 'system' supposed to be supported here? Should pkg's install() be doing some translation?

What's the default for mfile_encoding supposed to be (on macOS)? Is that set in a user configuration somewhere that persists between Octave sessions? Wondering if I have some setting on that one machine that's messing things up, either because I set it at some point, or it's left over from running an older version of Octave that set it somewhere persistent, or something like that.

I called `mfile_encoding('utf-8')` on eilonwy, the machine where it was returning 'system', and restarted Octave, and now it's returning 'utf-8'. And I see "default_encoding=UTF-8" in my ~/.config/octave/octave-gui.ini file. It switches to "default_encoding=SYSTEM (UTF-8)" if I run `mfile_encoding ('system')". After doing that, in CLI octave, mfile_encoding() returns 'utf-8', but in `octave --gui`, mfile_encoding() returns 'system'. That difference in behavior is a little surprising.

The warning says the encoding "must" be UTF-8. Is that true? Does fopen not support other encodings in this platform/situation?

I see this in libinterp/corefcn/file-io.cc in the Octave source. I don't know what this strict encoding facet stuff is. But it seems odd to limit it to UTF-8 in a general IO operation like fopen. Modern systems can still handle other encodings I think, and it's not uncommon to have or receive files that are in other encodings, like UTF-16 or non-Unicode encodings, even today in 2024.

#if defined (OCTAVE_HAVE_STRICT_ENCODING_FACET)
  if (encoding != "utf-8")
    {
      warning_with_id ("Octave:fopen:encoding-unsupported",
                       "fopen: encoding must be 'UTF-8' for this version");
      encoding = "utf-8";
    }
#endif

Also, it looks like that's a case-sensitive comparison, and the value must be lower-case "utf-8", but the error message is talking about upper case "UTF-8".

Test results

I tried this on various OS, arch, and Octave combinations over this weekend. Here's what I observed.

On AS (eilonwy):

• In 9.2.0 brewed on macOS 14 AS: warnings
• In Octave.app 9.2.0 beta1 on macOS 14 AS: same warnings
• In Octave.app 9.1.0 beta4 on macOS 14 AS: same warnings
• In Octave.app 8.4.0 on macOS 14 AS: no warning
• In Octave.app 6.2.0 on macOS 14 AS: no warning
• In Octave.app 4.4.1 on macOS 14 AS: `pkg install -forge control` fails at linker step; no warning up until then.

On AS (oabuild-as (macOS 12) VM on eilonwy):

• In Octave.app 9.2.0 beta1 on macOS 12 AS: no warnings! huh.
• In Octave.app 9.1.0 beta4 on macOS 12 AS: no warnings! huh.
• In Octave.app 8.4.0 on macOS 12 AS: no warnings

On Intel (angharad):

• Brewed octave 9.2.0 on macOS 14 Intel: no warnings
• Octave.app 9.2.0 beta1 on macOS 14 Intel: no warnings
• Octave.app 9.1.0 beta4 on macOS 14 Intel: no warnings
• Octave.app 8.4.0 on macOS 14 Intel: no warnings
• Octave.app 6.2.0 on macOS 14 Intel: no warnings

mfile_encoding() returns:

• brewed Octave 9.2.0 on macOS 14 AS: 'system'
• brewed Octave 9.2.0 on macOS 14 Intel: 'utf-8'
• Octave.app 9.1.0 beta4 on macOS 12 AS (VM): 'utf-8'
• Octave.app 9.1.0 beta4 on macOS 12 Intel (VM): 'utf-8'