bugGNU Octave - Bugs: bug #55195, "help" for core...

 
 

bug #55195: "help" for core functions contains odd symbols for non-ASCII characters

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Mon 10 Dec 2018 05:48:53 PM UTC
   
 
Category:  Interpreter Severity:  4 - Important
Priority:  5 - Normal Item Group:  Regression
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 20 Oct 2020 12:39:56 PM UTC, comment #6: 

This is solved by changeset dff830c84726 for bug #49685:
https://hg.savannah.gnu.org/hgweb/octave/rev/dff830c84726

For the originally motivating example (`help sym`), the output is correct even if the mfile encoding is set to "system" (the default; "CP1252" on my Windows machine) when I add a file ".oct_config" with the following content to the package directory:

encoding=utf-8


Adding those files to the packages should be tracked in separate bug reports.

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Mar 2019 05:58:37 PM UTC, comment #5: 

BISTs are gathered from .cc sources during build. These files aren't affected by the mfile_encoding.
Also BISTs in .m files aren't read by the parser.
So this bug is exclusively about doc strings.
Please, see bug #53875 for the BISTs. We might need to re-open that bug if we are going to change the default how fopen reads text.

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Mar 2019 04:53:19 PM UTC, comment #4: 

Might not be able to postpone this too long: work on bug #55452 has checked in non-ASCII characters in the BIST code (e.g. http://hg.savannah.gnu.org/hgweb/octave/rev/ccea3574f36b) and that's probably subject to the same encoding considerations as help strings.

There may also be an issue with assuming that doc strings are encoded in UTF-8 while the file itself is in a different encoding: text editors don't handle files with mixed encodings. So you wouldn't be able to properly edit these when running in a non-UTF-8 locale if the source files are in system-default encoding.

Andrew Janke <apjanke>
Mon 07 Jan 2019 11:32:33 PM UTC, comment #3: 

Agree about postponing, and I think you should start a discussion on the maintainers mailing list about assumptions of default file encodings and the load path and so on. At least when I first saw this bug report I wasn't exactly clear on what the problem even is, since I'm GNU/Linux-centric and I just assume every file is UTF-8 encoded, no problem. So I think to a lot of developers this is very much a foreign concept and needs some general understanding about the implications.

Mike Miller <mtmiller>
Group Member
Sun 30 Dec 2018 12:05:13 PM UTC, comment #2: 

I had a look at the code for the lexer/parser again and  couldn't find anything that would indicate whether a file is from core Octave, an (OF) package or from a directory added by the user.

Maybe it would be handy if we would keep track of this. E.g., we cannot warn at the moment if a user (unintentionally) overwrites a core function if it is an .m file. Maybe "load_path" should distinguish if a file is "internal" or "external". And that information could be fed down to the parser (via "fcn_file_in_path"?).

Since this would be a bigger change and non-ASCII characters never really worked correctly on previous version of Octave anyway (on non-UTF-8 systems), this change should probably be postponed until after the Octave 5 release.

Markus Mützel <mmuetzel>
Group administrator
Fri 28 Dec 2018 11:48:04 PM UTC, comment #1: 

No answers, just a couple observations after looking at this for a minute.

Octave assumes that all texinfo text that goes into the user manual (including all core function doc strings) is UTF-8 (see the @documentencoding declaration in doc/interpreter/octave.texi).

Only one core function doc string contains its own @documentencoding declaration, the 'slash' function. The content of the 'slash' function doc string is entirely ASCII encoded. The rest of the core functions are implicitly UTF-8 because the complete user manual is declared to be UTF-8.

Only two core function doc strings use any non-ASCII characters in their raw markup. The 'strftime' function uses a single '±' character, and the 'unicode_idx' function (new in Octave 5) uses a 'ä' character. These are both built-in functions, so I'm not sure whether mfile_encoding has anything to do with their doc strings. Does it?

I think that we should be able to assume that all core doc strings are UTF-8, whether or not they contain a @documentencoding declaration.

Mike Miller <mtmiller>
Group Member
Mon 10 Dec 2018 05:48:53 PM UTC, original submission:  

TL;DR:
Character encoding strikes again. Does the lexer keep track of whether .m files are from core?



When Octave is configured to use an mfile_encoding other than UTF-8, help text of function files that are encoded in UTF-8 is displayed with odd characters. On Windows, this happens with Octave's default settings. Other systems aren't affected by default (but only if the user configures to use a different encoding).

E.g.: "help sym" displays a lot of scrambled characters. That is because that file is encoded in UTF-8 but we assume it to be encoded in the configured mfile_encoding. Converting it from SYSTEM (CP1252 in my case) to UTF-8 creates these odd characters.

This is a regression. (Before, we didn't worry about encoding but had problems handling string vectors from user functions or interacting with the file system.)

That conversion is done in input.cc in function "file_reader::get_input".
Can we differentiate between .m files from the core or packages (which probably always are UTF-8) on the one hand and user created .m files (which could have any encoding) on the other hand at that point? Does the lexer keep track of this?

What about texinfo settings such as "@documentencoding UTF-8"? Should we parse for them and do the conversion only conditionally?
Should we skip the help text in the conversion completely? In that case, we might have to move the conversion elsewhere (to the lexer?).

Alternatively, we could revert the conversion in "help.m" (only if we discover an "@documentencoding" command?) for functions from the core or from packages.

But text in strings in functions from core Octave or from packages are probably encoded in UTF-8 as well (independent from the current mfile_encoding). So we shouldn't convert functions from core or packages at all and only do the codepage conversion on user functions.

This might also affect how we should open function files from core Octave (or from packages) in the embedded editor.

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-10-20 mmuetzel StatusPostponed Fixed
        Open/ClosedOpen Closed
        Dependencies- Depends on bugs #49685
    2019-01-07 mtmiller StatusNone Postponed

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code