I set the mfile encoding to "system" (which is "CP1252" on my Windows locale) and ran the following commands (from patch #8696):

setenv('GNUTERM', 'qt')
graphics_toolkit gnuplot
h = text (0.5,0.6, 'H \0 \Gamma \aleph I')
set(h,'fontweight', 'bold')

The UTF-8 encoded letters showed correctly besides the current global mfile encoding being different.
So this change seems to work as expected.

Closing as fixed.

Looks I forgot to attach the patch.

Here it is.

(file #49932)

If the encoding that is used to read some .m file functions is changed, I see no way around re-parsing them (which is essentially the same as `clear THAT_FUNCTION`). That entails resetting persistent variables

After the first attempts on this, I'm starting to have doubts if using the `PKG_ADD` script is the best way to go ahead anyway. I like the approach of getting the file encoding when a directory is added to the path. That way the dir_encoding can be read and stored only ones instead of trying to get that information for each file that is parsed.
But when this information is provided with `PKG_ADD`, what seems to be happening is:
1. Parse all files in the new directory (including PKG_ADD).
2. Execute the PKG_ADD script (which might call `dir_encoding`).
3. Parse all files in the new directory again if the encoding has changed. (This has to be done manually with the current patch.)

Parsing the functions twice is not very efficient. And parsing the files with the wrong encoding in step 1 might lead to unexpected side effects (parse errors). Having functions that `mlock` themselves might also be problematic with that approach.

It would probably be better imho if the following happened when a new directory is added to the load path:
1. Look for a special file (that doesn't need to be parsed by Octave) in the new directory. Read and set the `dir_encoding` from information gathered from that file.
2. Parse all files with the correct encoding.
3. Potentially call the `PKG_ADD` script (or do whatever we decide elsewhere).

That's close to what Andrew proposed in comment #14.

Maybe that file could be named `.oct_settings`. That way we could add other directory specific settings in the future. (No idea what that could be. But who knows...)
The file itself could contain strings (one setting per line) that are easy to parse.
E.g.:

encoding=UTF-8

Does that approach look reasonable?

I don't think there is currently a way to force all functions defined in a particular directory to be reloaded.

Also, remember that when functions are reloaded, persistent variables will be re-initialized.  Is that OK in this instance?

The attached patch applies the directory specific m-file encoding to all files that would be added by an addpath command (including the special subfolders).
It also adds some documentation to the manual.

It should be mostly complete now.

I've only tested on Ubuntu. But I hope it equally works on other platforms.

If this were to be added to Octave, it would probably make sense to add PKG_ADD scripts to all folders in the repository that contain m-files. The directory specific m-file encoding should be set to UTF-8 in those files.

(file #49815)

Here's an idea: instead of doing a "recursive, overrideable" thing that would require traversing the directory tree, you could require that an `.encoding` file must be present at the exact directory level that's added as a `PATH` entry (or you do one source_encoding() function call per PATH entry), and it applies to the entire directory subtree below that path entry; no overrides allowed. You want to combine source trees with different encodings, you need to add them to the Octave `PATH` as separate entries. That way it costs you just one file access per directory tree, no matter  how deep the +package or `private/` hierarchy goes, and there's still a way to have sets of code with different encodings.

(Maybe this is something that was already discussed.)

Thanks for your feedback. Much appreciated.

Having the encoding apply to private, @class or (nested) +namespace folders that are made "available" by the addpath command is probably what a user would expect. I agree.

But we already agreed on the mailing list that we don't want to traverse the file tree to find some magic files in a "random" parent folder.

I'm unsure if the dir_encoding should also apply to any private, @class or (nested) +namespace folders in that directory.

One the one hand, it might make it easier for the user to write the necessary PKG_ADD file if those were included automatically. Something like this would suffice:

dir_encoding (fileparts (mfilename ("fullpath")), "iso8859-1");

On the other hand, it's probably also conceivable that a user would like to drop in a +namespace folder from another project that uses a different encoding. For this to work, it would be best if the PKG_ADD file contained a line for each (automatically loaded) subfolder.

Ah, gotcha. Okay, that makes sense.

> But I agree that the new goal of adding a function that can set the .m file encoding on a per-directory basis is better.

IMHO a function and PKG_ADD hooks for Octave Forge packages isn't the best approach: encoding is a property of a file or sets of files on disk, and it'd be better to express that declaratively with sidecar files that live next to them in the same file hierarchy, or some internal encoding indicator that's embedded in the files, instead of requiring procedural manipulation, especially if that procedural manipulation is tied into `pkg` packaging insteapd of being a property of arbitrary Octave source trees.

Here's one more edge case: Octave's BISTs can be defined in C++ *.cc source files in special %! lines inside C++ comments. How should they be interpreted?

E.g. prospective patch http://hg.savannah.gnu.org/hgweb/octave/rev/ccea3574f36b contains a test with "abc,äöü" strings.

See also bug #45597. I find it hard to decide what is the more important usecase for Windows Octave:

• exchange m-files with Windows Matlab (That's why the default encoding in Windows is currently "system". That's "my" usecase.)
• exchange m-files with Linux Octave, or with Windows Octave of other locales (proposal in this bug report, could be best covered with "utf8" encoding).

Octave has a hard life with respect to international (non-ASCII) characters on Windows and there are a lot of (partly overlapping) open bug reports covering many different issues.
It would make things easier if we used only one encoding consistently all over Octave.

At the moment, the standard encoding for the text editor is set to "System" on Windows. That points to a lot of different codepages dependent on the locale of Windows (and its standard codepage). UTF-8 is the default codepage on non-Windows systems. This makes it difficult to exchange Octave files containing non-ASCII characters between different platform (or different locales of Windows).

UTF-8 can be used to encode any glyph that is available in Unicode.
Can we also make it the default on Windows?