bugGNU Octave - Bugs: bug #53842, Handle m-files with arbitrary...


bug #53842: Handle m-files with arbitrary character encoding

Submitted by:  Markus Mützel <mmuetzel>
Submitted on:  Sat 05 May 2018 06:12:02 PM UTC  
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Ready For Test Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Any

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Sat 12 May 2018 09:07:20 PM UTC, comment #9:

I pushed the change here:

Markus Mützel <mmuetzel>
Project Member
Fri 11 May 2018 08:04:11 PM UTC, comment #8:

Change looks good, I think you can push that small fix.

I think it's fine to defer handling of UTF-16 and UTF-32 encoded files until the future, and either document or not the lack of support for now.

See also bug #53786.

Mike Miller <mtmiller>
Project Administrator
Thu 10 May 2018 07:08:59 PM UTC, comment #7:

Mike: You are right. UTF-16 files aren't read correctly.
The problem seems to be with octave_fgets which reads the file as an array of bytes (using char) and uses strlen to determine the string length. That means for most files using UTF-16 it reads only the first (half) character.

Even if we solved the strlen issue, it doesn't seem to be easy to read UTF-16 or UTF-32 files with std::fgets (e.g. stops reading at single byte \n which could be part of a valid 2-byte or 4-byte character).
Does anyone have an idea what we could do? Besides maybe document that we are not able to read .m files with these encodings and emit an error when trying to set them as the mfile_encoding.

Once we should be able to read the file, the conversion to UTF-8 should be working out-of-the-box.

The attached set fixes the second issue you mentioned.

(file #44137)

Markus Mützel <mmuetzel>
Project Member
Thu 10 May 2018 01:27:19 AM UTC, comment #6:

This works for me on Debian with an m-file encoded in iso8859-1 or latin1.

This does not work for me on Debian with an m-file encoded in utf-16. Should that be possible with this change so far?

Also, I think functions that take a string argument setting usually return the previous value, not the new value. So for example

both of these return values should be 'utf-8'.

Mike Miller <mtmiller>
Project Administrator
Wed 09 May 2018 07:27:46 PM UTC, comment #5:

I pushed the changeset here:

Marking as ready for test.

Markus Mützel <mmuetzel>
Project Member
Wed 09 May 2018 06:12:20 PM UTC, comment #4:

With this patch, the default m-file encoding is "system" on Windows platforms. That is the same as the default for the file editor.

(file #44132)

Markus Mützel <mmuetzel>
Project Member
Tue 08 May 2018 12:25:53 PM UTC, comment #3:

The previous patches set the wrong default on Windows platforms. It should be the system codepage on Windows and utf-8 otherwise.
I'll update the patch as soon as I have something ready.

Markus Mützel <mmuetzel>
Project Member
Sun 06 May 2018 08:15:21 PM UTC, comment #2:

The attached modified patch avoids including the gnulib header in the wrapper.h file. This is done by using "void *" instead of "iconv_t" as proposed by jwe on IRC.

(file #44106)

Markus Mützel <mmuetzel>
Project Member
Sat 05 May 2018 06:25:54 PM UTC, comment #1:

The attached file adds support for converting .m files with any character encoding to UTF-8 before passing its content to the lexer. The assumed .m file encoding can be changes with _mfile_encoding_.

I'll also attach "test_iso_8859_15.m" which is encoded with ISO-8859-15 (Western). With it I see the following results in an Octave session starting with the default .m file encoding "utf-8":

While all non-ASCII characters are initially displayed with the replacement character, they are correctly read and displayed after the character encoding was set accordingly.

Markus Mützel <mmuetzel>
Project Member
Sat 05 May 2018 06:12:02 PM UTC, original submission:

As discussed on the mailing list [1], Octave isn't aware of character encoding yet.
This bug will track the part of reading m-files with arbitrary encoding.
I will post a patch as soon as I have a bug number.

[1] http://octave.1599824.n4.nabble.com/Handle-encoding-of-Octave-strings-td4687603.html

Markus Mützel <mmuetzel>
Project Member


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #44137:  bug53842_previous_setting.patch added by mmuetzel (939B - application/octet-stream)
file #44132:  bug53842_mfile_encoding_v3.patch added by mmuetzel (7KiB - application/octet-stream)
file #44106:  bug53842_mfile_encoding_v2.patch added by mmuetzel (6KiB - application/octet-stream)


Depends on the following items: None found

   bug dependencies.


Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-05-10 mmuetzel Attached File- => Added bug53842_previous_setting.patch, #44137
    2018-05-09 mmuetzel StatusPatch Submitted => Ready For Test
    2018-05-09 mmuetzel Attached File- => Added bug53842_mfile_encoding_v3.patch, #44132
        StatusIn Progress => Patch Submitted
    2018-05-08 mmuetzel StatusPatch Submitted => In Progress
    2018-05-06 mmuetzel Attached File- => Added bug53842_mfile_encoding_v2.patch, #44106
        StatusNone => Patch Submitted
    2018-05-06 mmuetzel Dependencies- => bugs #53646 is dependent

    Back to the top

    Powered by Savane 3.3