Re-titling to reflect that we still can't use encodings which don't consist of a sequence of chars.

>  UTF-8 is an encoding that covers all Unicode characters. What do you mean by changing the default encoding from "utf-8" to "unicode"?

I mean, make the default external file encoding not strictly UTF-8, but a "meta-encoding" which means "Unicode, but choose/detect the actual UTF-whatever encoding on a per-file basis". So that you might have a project that has some .m files in UTF-8, and some files in UTF-16, and these encodings are autodetected when the individual files are read, based on their contents. And for output, previously-existing files are saved in their original encodings (to avoid introducing spurious changes into source control or whatever), and newly-created files are saved in a default Unicode encoding, probably UTF-8.

> Wrt your comment #10: Unfortunately(?) the C++ standard doesn't define any character encodings. To make matters worse wchar_t and wstring have different sizes on different platforms...

That's my understanding as well. And, yeah, "Unfortunately(?)" is right. It's a bummer that there's not standard support in base C/C++, but this Unicode stuff is so complex that whatever made it out of a C++ standards committee would probably be woefully outdated by the time it hit the streets.

What I really mean is:

• Define an EncodedFileReader class that:
  • detects file encodings using some heuristics, or takes an encoding explicitly supplied by the caller, and then
  • does line-oriented encoding-aware input reading (based on the initial encoding detection/specification) and encoding conversion, returning the results as UTF-8 strings inside `std::string` (or whatever Octave is using for its internal Unicode string representation). So the external encoding could be UTF-16, UTF-8, UTF-32, or whatever, as long as the Unicode library supports translation between that and the native Octave representation, and line feeds could be safely detected
    • (which maybe means slurping the entire input file in to memory, transcoding the entire thing at once, and then detecting line feeds. But that's probably not an issue because source code files are relatively small.)

My understanding is that Octave is currently using gnulib for Unicode support, and that gnulib provides encoding translations, but not encoding/character-set detection. So we'd have to roll our own encoding detection, or take a dependency on a library that supports it.

> While C++ 11 added u16string and u32string, I don't know of any iostreams that support these types out of the box.

I don't know of any standard C/C++ stuff that supports this either. AFAIK, base C/C++ Unicode support is not great.

So that means rolling our own encoding-detection and Unicode-aware iostream support (which might not be so bad if all you care about is UTF-8 + UTF-16 and BMP-only support), or taking a dependency on ICU4C or similar, or deferring this to later.

I'm wondering if this feature may actually be detrimental to the Octave user community, because it would allow (and maybe encourage) users to create M-code that is incompatible with M-code in other encodings.

Let's say I'm collaborating on a project with someone in Latvia and someone in China, and each of them have their systems set up to use their local non-Unicode encodings. I don't think there's a way for me to configure my Octave system in a way that the source code from both of them could be used in a single Octave session, if the source file encoding is a global Octave setting.

Perhaps it would be better to require that all Octave M-code source files be in Unicode encodings, or at least make that the default on all operating systems. That way all Octave .m code bases would be interoperable with all other Octave .m code bases, at least by default.

Using the system default code page would require users to be aware of the interoperability issue, and take explicit action to create their source code in Unicode format. That seems like an advanced operation that normal Octave users shouldn't have to be aware of.

Along the same lines, perhaps this:

> static std::string Vmfile_encoding = "utf-8";

should be a more generic "unicode" encoding, like this:

> static std::string Vmfile_encoding = "unicode";

to allow for easier integration of future support for autodetected UTF-16 and UTF-32 files?

I pushed the change here:
http://hg.savannah.gnu.org/hgweb/octave/rev/9cc1ca6538e3

Mike: You are right. UTF-16 files aren't read correctly.
The problem seems to be with octave_fgets which reads the file as an array of bytes (using char) and uses strlen to determine the string length. That means for most files using UTF-16 it reads only the first (half) character.

Even if we solved the strlen issue, it doesn't seem to be easy to read UTF-16 or UTF-32 files with std::fgets (e.g. stops reading at single byte \n which could be part of a valid 2-byte or 4-byte character).
Does anyone have an idea what we could do? Besides maybe document that we are not able to read .m files with these encodings and emit an error when trying to set them as the mfile_encoding.

Once we should be able to read the file, the conversion to UTF-8 should be working out-of-the-box.

The attached set fixes the second issue you mentioned.


(file #44137)

I pushed the changeset here:
http://hg.savannah.gnu.org/hgweb/octave/rev/82445187633e

Marking as ready for test.

The previous patches set the wrong default on Windows platforms. It should be the system codepage on Windows and utf-8 otherwise.
I'll update the patch as soon as I have something ready.

The attached file adds support for converting .m files with any character encoding to UTF-8 before passing its content to the lexer. The assumed .m file encoding can be changes with _mfile_encoding_.

I'll also attach "test_iso_8859_15.m" which is encoded with ISO-8859-15 (Western). With it I see the following results in an Octave session starting with the default .m file encoding "utf-8":

>> test_iso_8859_15
ans = W�hrend Adam lacht, jagen zw�lf Boxk�mpfer Eva quer �ber den gro�en Sylter Deich - f�r satte 12.345.667,89 �uro
>> __mfile_encoding__
ans = utf-8
>> __mfile_encoding__ ("iso-8859-15")
ans = iso-8859-15
>> clear test_iso_8859_15
>> test_iso_8859_15
ans = Während Adam lacht, jagen zwölf Boxkämpfer Eva quer über den großen Sylter Deich - für satte 12.345.667,89 €uro

While all non-ASCII characters are initially displayed with the replacement character, they are correctly read and displayed after the character encoding was set accordingly.