bugGNU Octave - Bugs: bug #55452, fopen() does not support encoding...

 
 

bug #55452: fopen() does not support encoding argument

Submitter:  Andrew Janke <apjanke>
Submitted:  Sat 12 Jan 2019 02:29:42 AM UTC
   
 
Category:  Octave Function Severity:  1 - Wish
Priority:  3 - Low Item Group:  Feature Request
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 14 Jan 2020 08:33:55 AM UTC, comment #34: 

I opened a new bug #57596 for the open question about the "len" argument of "fgetl" and "fgets".

Markus Mützel <mmuetzel>
Group administrator
Sat 04 Jan 2020 04:45:50 AM UTC, comment #33: 

File a new bug about the "len" argument if it is necessary.

This bug is resolved.  Marking as Fixed and closing report.

Rik <rik5>
Group administrator
Fri 25 Oct 2019 03:56:53 PM UTC, comment #32: 

I pushed the changes in file #46481 here:
http://hg.savannah.gnu.org/hgweb/octave/rev/8edca821f170

We could still leave this bug open to decide if the "len" argument for "fgetl" and "fgets" should refer to input bytes (as it is now) or to "character entities".
But maybe we should open a new bug for that remaining point.

Marking this bug as ready for test because the original issue (encoding argument for fopen) is implemented.

Markus Mützel <mmuetzel>
Group administrator
Sun 21 Jul 2019 06:11:00 PM UTC, comment #31: 

I rebased file #46480 to the current default branch and pushed it here:
http://hg.savannah.gnu.org/hgweb/octave/rev/1c8b20731af4

"fgetl" and "fgets" with a "len" argument still need more thinking. Reading the correct number of bytes from a source isn't trivial for variable-byte encodings. Maybe file #46481 could be applied as an intermediate step to support "fgetl" end "fgets" with one argument (and for encodings that use one byte per character).

Interestingly, I have to specify the byte order for the UTF-16 examples in Andrew's test suite or the tests fail (i.e. I have to rename those files to include UTF-16BE).
I recently re-installed Ubuntu 19.04. Maybe that or some other changes on my system might have caused this issue.

Markus Mützel <mmuetzel>
Group administrator
Wed 13 Mar 2019 07:40:29 AM UTC, comment #30: 

Hard to say without testing. I don't see anything in the docs specifying either of those exactly.

Andrew Janke <apjanke>
Wed 13 Mar 2019 07:31:55 AM UTC, comment #29: 

I never found out if Matlab really uses UTF-16 or just UCS-2. What does Matlab do when reading (or writing?) a character outside the BMP like e.g. the violin key 𝄞 (U+1D11E)?
When reading UTF-8 encoded multi-byte characters from a file with fgets does it advance 2 or more bytes if necessary?

Markus Mützel <mmuetzel>
Group administrator
Wed 13 Mar 2019 07:06:17 AM UTC, comment #28: 

Here's what I'm concerned about: if "fgetl" and "fgets" do encoded input, but their "len" arguments are expressed in bytes instead of characters:

Is "len" the number of input bytes (from the file) or output bytes (in the returned string)?

If "len" is the number of input bytes, and you specify a length that leaves off reading in the middle of a multi-byte sequence, what happens? If you just leave it as is, the file pointer is no longer a correct position for starting decoding again. And the bytes that were read are not a complete valid string in their input encoding.

Consider the example of a file containing "Hello!\nGoodbye!\n" encoded in UTF-16 little-endian. ("samples/hello - UTF-16 LE.txt" in my test repo.) What should happen if you open it and do "str1 = fgetl(fid, 3); str2 = fgetl(fid, 3);"?

Andrew Janke <apjanke>
Wed 13 Mar 2019 06:46:20 AM UTC, comment #27: 


> Would it be possible to add test cases for writing to files with different encodings to your test suite? Maybe tests with "fprintf" (should be working mostly) and "fputs" (probably fails?).


Sure. I should have some time for this tomorrow night.

> Do you know of encodings that encode numerals like "3" with different bytes than 7bit ASCII?


Well, UTF-16 does. But it's just putting the same int value into a wider 16-bit space, like the 7-bit ASCII value plus a null padding byte. That's all I can think of.

Andrew Janke <apjanke>
Wed 13 Mar 2019 06:39:15 AM UTC, comment #26: 

Hmm. If we're going to change "fgetl", it seems like "fgets" should change too. The only difference between "fgetl" and "fgets" is whether they keep the newline character.

It might make more sense to track Matlab's "fgets" behavior instead of POSIX C's "fgets". And according to the Matlab doco, it reads characters in an encoding-aware manner. https://www.mathworks.com/help/matlab/ref/fgets.html (See "Tips" down at the bottom.) Interpreting the Matlab doco in this light is a little hard, because Matlab uses 2-byte UTF-16 chars, so a Matlab char is typically a single Unicode character, and vice versa.

I think you're right about POSIX C, though: when they say "character", I think they usually mean C char.

Andrew Janke <apjanke>
Sun 10 Mar 2019 03:24:27 PM UTC, comment #25: 

Wrt "fgets": I am still not certain we should change its behaviour.
At the moment it is (more or less) POSIX conform in behaviour. [1]
It might be unexpected if it returned an "unpredictable" number of bytes. Even more so if it also did some kind of automatic conversion.
The documentation isn't very precise in what is meant with "character". Sometimes that can refer to a C-char (i.e. a byte or one element of a character array) and sometimes to an "abstract character" (approx. in its Unicode definition). The former meaning is probably implied in the majority of cases.

[1]: http://pubs.opengroup.org/onlinepubs/009695399/functions/fgets.html

Markus Mützel <mmuetzel>
Group administrator
Sun 10 Mar 2019 03:06:25 PM UTC, comment #24: 

Would it be possible to add test cases for writing to files with different encodings to your test suite? Maybe tests with "fprintf" (should be working mostly) and "fputs" (probably fails?).

Do you know of encodings that encode numerals like "3" with different bytes than 7bit ASCII?
It would be interesting to add test cases for these as well. Something like fprintf (fid, "%.3f", pi);

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Mar 2019 10:16:09 PM UTC, comment #23: 

Builds now with those patches. I've updated the octave-default-with-55452 formula.

Same failures – just textscan on UTF-16 with a BOM.

I'm guessing the BOM is causing it to erroneously detect a string/delimiter boundary at the very beginning of the string.


>> [fid,msg] = fopen('encoded-files/ex-001/txt-UTF-16.txt', 'r', 'native', 'UTF-16')
fid =  19
msg =
>> str = textscan(fid, '%s', 'Delimiter','')
str =
{
  [1,1] =
  {
    [1,1] =
    [2,1] = Hello, world!
  }
}

>>


Andrew Janke <apjanke>
Sat 09 Mar 2019 09:42:07 PM UTC, comment #22: 

The attached patch "bug55452_fopen_convert_stream.patch" moves the encoding conversion to after the formatting is done.

The second patch "bug55452_fgetl_v2.diff" is the remainder of the previous patch from comment #14 that adds basic support for fgetl (and fgets).

(file #46480, file #46481)

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Mar 2019 08:34:17 PM UTC, comment #21: 

Patch file #46477 no longer applies to the default branch. Is it still needed?

> Octave's internal string encoding is UTF-8.


If this is the case, then I don't think you need to worry about printf format strings; the C printf() family is UTF-8 compatible. Just the final transcode step to the external encoding is needed.

Andrew Janke <apjanke>
Sat 09 Mar 2019 08:27:49 PM UTC, comment #20: 

I pushed the part of the patch that fixes the validation of the name for the encoding here:
http://hg.savannah.gnu.org/hgweb/octave/rev/7506884fc158


Octave's internal string encoding is UTF-8.

You are right: We probably need to do the conversion after we formatted the string and before actually passing it on to the file stream.
The fprintf logic is not generic enough for all cases.
I am working on a patch for this right now.

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Mar 2019 07:31:18 PM UTC, comment #19: 

I've added a special `octave-default-with-55452` formula to make it easier to install this Octave test variant on macOS.


brew tap octave-app/octave-app
brew install octave-default-with-55452
/usr/local/opt/octave-default-with-55452/bin/octave



Andrew Janke <apjanke>
Sat 09 Mar 2019 07:15:21 PM UTC, comment #18: 


> Could you please try with the attached patch. It should fix the weird error message and add preliminary support for "fgetl".


I expanded the test to use both fgetl() and textscan(), and ran it against Octave default built with your patch. That cleared up most of the failures. Only failures now are textscan() coming back with an empty string for the UTF-16 with byte order mark cases.


    FAIL: ex-001 UTF-16  - textscan
    FAIL: ex-003 UTF-16  - textscan


Attaching file with full results.

(file #46478)

Andrew Janke <apjanke>
Sat 09 Mar 2019 06:40:56 PM UTC, comment #17: 


> I was thinking that Shift-JIS was a multi-byte encoding (like UTF-16 for the BMP). But after a little bit of reading, it looks like it is a variable byte encoding like UTF-8 and encodes most of ASCII with the very same byte. So it might actually work without too much of a hassle.


Almost. Shift-JIS encodes 7-bit ASCII characters as the same single bytes. But you also have to worry about the other direction: individual bytes of 2-byte Shift-JIS byte sequences aliasing to ASCII characters. And there it could be a problem: bytes 0x40–0x7E may appear as the second byte of a 2-byte Shift-JIS byte sequence. And that range may include characters that appear in `printf()` formatting sequences, including all the alphabetical characters.

It's a little hard for me to puzzle out whether this means a valid Shift-JIS-encoded printf format string could be misinterpreted, because no Shift-JIS multibyte sequences contain bytes that alias to '%', and valid printf control strings do not allow non-ASCII characters between the '%' and its following specifier character.

Regardless, I don't think we should be focusing on particular encoding cases like this in finding a code approach: we should look for a general solution that handles any encoding without special-case logic.

(And when I say “ASCII” here I mean strict 7-bit ASCII.)

Andrew Janke <apjanke>
Sat 09 Mar 2019 06:29:18 PM UTC, comment #16: 


> I was wondering about format strings like "äöü %s %.3f", i.e. format strings that don't consist exclusively of format specifiers. We probably would have to somehow detect which parts of the format string are specifiers and convert only the remaining parts before we pass the string to the C functions.


I think this depends on what encoding Octave is storing its strings in internally. If it's UTF-8, you're in the clear, because the multi-byte sequences never use bytes that are valid ASCII characters.

If Octave is internally using the system default encoding, I think you technically have to transcode it to UTF-8 or whatever the C/C++ string processing functions are using for the placeholder-substitution logic.

Then in either case you need to transcode it again to your final output encoding.

BUT - every single-byte encoding I'm aware of is ASCII-compatible, and since Octave's internal strings are built on chars and are not currently doing transcoding, I think you're stuck with "single-byte-compatible" encodings, and that concern doesn't apply.

And at the C level, `sprintf()` and friends are not encoding-aware, so I think you're stuck using an ASCII-compatible encoding with them anyway.

Andrew Janke <apjanke>
Sat 09 Mar 2019 05:50:55 PM UTC, comment #15: 

I was thinking that Shift-JIS was a multi-byte encoding (like UTF-16 for the BMP). But after a little bit of reading, it looks like it is a variable byte encoding like UTF-8 and encodes most of ASCII with the very same byte. So it might actually work without too much of a hassle.

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Mar 2019 05:37:15 PM UTC, comment #14: 

I forgot that "textscan" has been worked on already (see comment #5).

Could you please try with the attached patch. It should fix the weird error message and add preliminary support for "fgetl".

I was wondering about format strings like "äöü %s %.3f", i.e. format strings that don't consist exclusively of format specifiers. We probably would have to somehow detect which parts of the format string are specifiers and convert only the remaining parts before we pass the string to the C functions.
Unless, "%s" (or any other format specifier) encodes to the same byte sequence in these encodings as it would in ASCII (like it does for UTF-8).
Supporting UTF-16 or UTF-32 probably just gets weird. Maybe we should defer those to later.

(file #46477)

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Mar 2019 04:44:09 PM UTC, comment #13: 


> Is that related to the "@" in the file name?


Could be. That's where it occurs. But I checked it in the debugger and the encoding part before the "@" is being correctly stripped out, and looks like an ordinary `"UTF-16"` to me.

Andrew Janke <apjanke>
Sat 09 Mar 2019 04:43:11 PM UTC, comment #12: 


> I only checked with fprintf (fid, "%s", string), and fscanf (fid, "%s") before. I didn't have a look at "fgetl" yet.


Looks like we'll need a "fgetl()", "fgets()", and/or "textscan()" before this test can really run. "fscanf()" elides whitespace in its input when doing `fscanf (fid, '%s')`.

"fgetl()" would be more convenient, because making sure the example files don't have newlines at the ends of the files is a bit of a bother.

> There doesn't seem to be a convenient function to get the number of characters in a string straight away (or I forgot about it). "numel" returns the number of bytes in the char array.


I'm really just looking for the number of bytes here, for quick identity-checking where null bytes are not apparent in the output. I'm just calling it "chars" because it's counting Octave chars. Which are really just bytes at this point.

Your fputs()/fprintf()/fgetl()/fscanf() ideas make sense.

> I am not sure how to handle "fgets": Should we just read one byte and return that? Or should we make sure that we read one character (whatever the number of bytes necessary)?


I would say, since fgets() is line-oriented, and that seems to be the family of functions we should make encoding-aware, that it should return characters: `fgets (fid, len)` should read len characters from the input, however many bytes that is in the input encoding, and return len characters, however many bytes that is in Octave's internal/native encoding. That's consistent with a naive reading of the documentation for fgets(). But it will probably cause a compatibility break with current uses of fgets(), because currently I suspect Octave conflates bytes and characters here, so the len input is currently interpreted as a number of bytes (Octave chars), not characters.

For that matter, maybe this family of functions should be renamed "text-oriented", since it includes fgets(), which is not line-aware.

> Should we just read one byte and return that?


I don't think that's possible in an encoding-aware world, because a given character may be represented by different bytes, and even different numbers of bytes, in the input and Octave internal encodings.

> I never worked with multi-byte encodings like SHIFT-JIS. How do they encode ASCII characters?


Different encodings do it differently. Some encodings are fixed-width (like Big5 or UTF-16 (basically) or UTF-32) and just use the same number of bytes for every character, so ASCII characters get encoded as multiple bytes with 0-padding. Some encodings (like Shift-JIS or UTF-8) use different numbers of bytes for different character ranges. In these, you check the value of the first byte of a byte sequence to determine how many bytes there will be in it; its range encodes/implies the length of the byte sequence for that character.

UTF-16 has the additional complication that it's actually a variable-width encoding: each UTF-16 "code unit" is 2 bytes, and most characters are encoded as a single code unit, but exotic characters outside the "Basic Multilingual Plane" are encoded as "surrogate pairs" of two code units. Many UTF-16 implementations (including Matlab and old versions of Java) just ignore this complexity and pretend that UTF-16 is fixed-width.

Andrew Janke <apjanke>
Sat 09 Mar 2019 04:13:21 PM UTC, comment #11: 

I crossed with you comment #9: Is that related to the "@" in the file name?

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Mar 2019 04:10:59 PM UTC, comment #10: 

Thanks for your testing.
I only checked with fprintf (fid, "%s", string), and fscanf (fid, "%s") before. I didn't have a look at "fgetl" yet. It looks like these functions take different code paths.

If I replace the function "slurp_file_one_line" in your test suite with the following, the results look a little bit better:

function out = slurp_file_one_line (file, encoding)
  try
    [fh, msg] = fopen (file, "r", "native", encoding);
    if fh < 0
      error ("Failed opening file for reading: %s: %s", msg, file);
    endif
    # out = fgetl (fh);
    out = fscanf (fh, "%s");
    fclose (fh);
    out = out(:)';
  catch err
    err
    out = "";
  end_try_catch

endfunction



On Windows, there seem to be at least two more different bugs:

>> run_bug_55452_tests
Running fixed-text encoded file test ex-001:
Reference text: Hello,world! (12 chars)
running: ex-001 ISO-8859-1
  decoded: Hello,world! (12 chars)
  ok: ex-001 ISO-8859-1
running: ex-001 ISO-8859-15
  decoded: Hello,world! (12 chars)
  ok: ex-001 ISO-8859-15
running: ex-001 KOI8-R
  decoded: Hello,world! (12 chars)
  ok: ex-001 KOI8-R
running: ex-001 SHIFT_JIS
  decoded: Hello,world! (12 chars)
  ok: ex-001 SHIFT_JIS
running: ex-001 UTF-16
  decoded: Hello,world! (12 chars)
  ok: ex-001 UTF-16
running: ex-001 UTF-16 no-bom
err =

  scalar structure containing the fields:

    message = fopen: conversion from codepage 'utf-16' not supported
    identifier =
    stack =

      3x1 struct array containing the fields:

        file
        name
        line
        column
        scope


  decoded:  (0 chars)
  FAIL: ex-001 UTF-16 no-bom
Running fixed-text encoded file test ex-002:
Reference text: あありりががととうう丸丸 (18 chars)
running: ex-002 SHIFT_JIS
  decoded: あありりががととうう丸丸 (18 chars)
  ok: ex-002 SHIFT_JIS
running: ex-002 UTF-16
  decoded: あありりががととうう丸丸 (18 chars)
  ok: ex-002 UTF-16
Running fixed-text encoded file test ex-003:
Reference text: KaßnerÖkonomSchöpsÜbermutMüller (36 chars)
running: ex-003 ISO-8859-1
  decoded: KaßnerÖkonomSchöpsÜbermutMüller (36 chars)
  ok: ex-003 ISO-8859-1
running: ex-003 UTF-16
  decoded: KaßnerÖkonomSchöpsÜbermutMüller (36 chars)
  ok: ex-003 UTF-16


There doesn't seem to be a convenient function to get the number of characters in a string straight away (or I forgot about it). "numel" returns the number of bytes in the char array. Maybe "max (unicode_idx (str))" would be more correct.

Back on topic:
In the f* family of functions, I think that "fwrite" and "fread" should ignore the encoding and just handle "pure bytes".
"fputs" and "fprintf" (%s format arguments and the format string itself) should probably convert to the specified encoding.
"fgetl" and "fscanf" (%s format arguments) should be converted from the specified encoding.
I am not sure how to handle "fgets": Should we just read one byte and return that? Or should we make sure that we read one character (whatever the number of bytes necessary)?

Please let me know if I'm missing something.

I never worked with multi-byte encodings like SHIFT-JIS. How do they encode ASCII characters? I am wondering if fprintf correctly treads the format string on current default.

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Mar 2019 03:30:59 PM UTC, comment #9: 

Oddly, if I change the `fgetl ()` in my test to `textscan ()`, then it errors out with `error: fopen: conversion from codepage 'utf-16' not supported` on one of the passes, and the previous pass decodes a 0-long string. That's strange, because I wouldn't expect using textscan() to affect the success of subsequent fopen() calls.

Changed test code:


function out = slurp_file_one_line (file, encoding)
  [fh, msg] = fopen (file, "r", "native", encoding);
  if fh < 0
    error ("Failed opening file for reading: %s: %s", msg, file);
  endif
  %out = fgetl (fh);
  lines = textscan (fh, "%s", "Delimiter","");
  out = lines{1}{1};
  fclose (fh);
  out = out(:)';
endfunction


Result:


>> run_bug_55452_tests
Running fixed-text encoded file test ex-001:
Reference text: Hello, world! (13 chars)
running: ex-001 ISO-8859-1
  decoded: Hello, world! (13 chars)
  ok: ex-001 ISO-8859-1
running: ex-001 ISO-8859-15
  decoded: Hello, world! (13 chars)
  ok: ex-001 ISO-8859-15
running: ex-001 KOI8-R
  decoded: Hello, world! (13 chars)
  ok: ex-001 KOI8-R
running: ex-001 SHIFT_JIS
  decoded: Hello, world! (13 chars)
  ok: ex-001 SHIFT_JIS
running: ex-001 UTF-16
  decoded:  (0 chars)
  FAIL: ex-001 UTF-16
running: ex-001 UTF-16 no-bom
error: fopen: conversion from codepage 'utf-16' not supported
error: called from
    run_bug_55452_tests>slurp_file_one_line at line 51 column 13
    run_bug_55452_tests>run_fixed_text_encoded_file_test at line 31 column 18
    run_bug_55452_tests at line 11 column 5


Andrew Janke <apjanke>
Sat 09 Mar 2019 02:54:10 PM UTC, comment #8: 

Got a build of the current default and ran my test. A couple failures:


>> run_bug_55452_tests
Running fixed-text encoded file test ex-001:
Reference text: Hello, world! (13 chars)
running: ex-001 ISO-8859-1
  decoded: Hello, world! (13 chars)
  ok: ex-001 ISO-8859-1
running: ex-001 ISO-8859-15
  decoded: Hello, world! (13 chars)
  ok: ex-001 ISO-8859-15
running: ex-001 KOI8-R
  decoded: Hello, world! (13 chars)
  ok: ex-001 KOI8-R
running: ex-001 SHIFT_JIS
  decoded: Hello, world! (13 chars)
  ok: ex-001 SHIFT_JIS
running: ex-001 UTF-16
  decoded: ��Hello, world! (28 chars)
  FAIL: ex-001 UTF-16
running: ex-001 UTF-16 no-bom
  decoded: Hello, world! (26 chars)
  FAIL: ex-001 UTF-16 no-bom
Running fixed-text encoded file test ex-002:
Reference text: ありがとう丸 (18 chars)
running: ex-002 SHIFT_JIS
  decoded: ���肪�Ƃ��� (12 chars)
  FAIL: ex-002 SHIFT_JIS
running: ex-002 UTF-16
  decoded: 0B0�0L0h0FN8 (13 chars)
  FAIL: ex-002 UTF-16
Running fixed-text encoded file test ex-003:
Reference text: Kaßner Ökonom Schöps Übermut Müller (40 chars)
running: ex-003 ISO-8859-1
  decoded: Ka�ner �konom Sch�ps �bermut M�ller (35 chars)
  FAIL: ex-003 ISO-8859-1
running: ex-003 UTF-16
  decoded: ��Ka�ner �konom Sch�ps �bermut M�ller (73 chars)
  FAIL: ex-003 UTF-16


Looks like a couple things going on here:

- The BOM in UTF-16 files looks like it's being propagated to the decoded string. That probably shouldn't happen.

- UTF-16 encoded text is being turned in to too many chars. Looks like a \0 char is getting inserted between each ASCII-like char.


>> fh = fopen('encoded-files/ex-001/txt-UTF-16.txt'); line = fgetl (fh); fclose (fh);
>> line
line = ��Hello, world!
>> line == 0
ans =
  0  0  1  0  1  0  1  0  1  0  1  0  1  0  1  0  1  0  1  0  1  0  1  0  1  0  1  0

>>


- ISO-8859-1 doesn't seem to get converted to UTF-8.

This brings up another question: How can I read an entire text file in, without having to iterate over doing a fgetl() on each line? `fscanf (fid, "%s")`? Would `fread (fid, '*char')` be expected to work? (What are the semantics for reading chars with fread() on a stream with a non-native encoding?)

Andrew Janke <apjanke>
Sat 09 Mar 2019 01:26:22 PM UTC, comment #7: 

One minor doco issue: I think we should refer to these as "encodings" instead of "code pages". Code pages are vendor-specific means for identifying encodings; we're using the more-or-less-universal names for them here. And in modern practice, where I've seen people use "code page", they usually mean "Windows code page".

Andrew Janke <apjanke>
Sat 09 Mar 2019 01:08:45 PM UTC, comment #6: 

I created a little test for this: https://github.com/apjanke/octave-test-55452. Feel free to add more example encoded text files.

I'll set up a build with these patches to run overnight and see how the test works out tomorrow.

Andrew Janke <apjanke>
Sat 09 Feb 2019 07:15:05 PM UTC, comment #5: 

Pushed the part that actually adds the encoding to the "fopen" function here:
http://hg.savannah.gnu.org/hgweb/octave/rev/c13143821eef

And the support for encodings of streams in textscan here:
http://hg.savannah.gnu.org/hgweb/octave/rev/ccea3574f36b

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Feb 2019 04:29:38 PM UTC, comment #4: 

Second part with the back-end for basic encoding conversions in file streams is pushed here:
http://hg.savannah.gnu.org/hgweb/octave/rev/29ea2369971d

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Feb 2019 03:02:34 PM UTC, comment #3: 

I'll be breaking down this changeset into smaller, better digestible chunks.
First part pushed here:
http://hg.savannah.gnu.org/hgweb/octave/rev/2dc190401eeb

Markus Mützel <mmuetzel>
Group administrator
Sun 03 Feb 2019 03:02:16 PM UTC, comment #2: 

The attached patch reduces some duplicate code wrt the previous one by adding new functions "octave::string::u8_from_encoding" and "octave::string::u8_to_encoding".
It also adds support for codepages to "textscan".

(file #46170)

Markus Mützel <mmuetzel>
Group administrator
Sat 02 Feb 2019 07:21:07 PM UTC, comment #1: 

The attached patch adds support for opening files with specified codepage.
I only did some basic testing using codepage ISO8859-1 with fprintf and fscanf. Further tests are necessary. At least, "make check" still passes.

(file #46168)

Markus Mützel <mmuetzel>
Group administrator
Sat 12 Jan 2019 02:29:42 AM UTC, original submission:  

Matlab's fopen() supports a fourth input and output argument, encoding, that specifies the encoding the file is in.

https://www.mathworks.com/help/matlab/ref/fopen.html


fileID = fopen(filename,permission,machinefmt,encodingIn)
...
[filename,permission,machinefmt,encodingOut] = fopen(fileID)


It would be nice if Octave supported this. That would allow you to read in or write out files that are not in your locale's native encoding. That would be useful, e.g., when doing international data analysis projects, or if you have data files in mixed encodings for some other reason.

Andrew Janke <apjanke>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #46480:  bug55452_fopen_convert_stream.patch added by mmuetzel (7KiB - application/octet-stream)
file #46481:  bug55452_fgetl_v2.diff added by mmuetzel (556B - application/octet-stream)
file #46477:  bug55452_fgetl.diff added by mmuetzel (1KiB - application/octet-stream)
file #46170:  bug55452_fopen_codepage_v2.patch added by mmuetzel (31KiB - application/octet-stream)
file #46168:  bug55452_fopen_codepage.patch added by mmuetzel (28KiB - application/octet-stream)

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by jwe (Updated the item)
  • -email is unavailable- added by apjanke (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 17 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-01-04 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2019-10-25 mmuetzel StatusIn Progress Ready For Test
    2019-10-23 mmuetzel Dependencies- bugs #57107 is dependent
    2019-03-09 mmuetzel Attached File- Added bug55452_fopen_convert_stream.patch, #46480
        Attached File- Added bug55452_fgetl_v2.diff, #46481
    2019-03-09 apjanke Attached File- Added run_bug_55452_tests-results-with-patch-46477.txt, #46478
    2019-03-09 mmuetzel Attached File- Added bug55452_fgetl.diff, #46477
        StatusReady For Test In Progress
    2019-03-05 mmuetzel Dependencies- bugs #55826 is dependent
    2019-02-09 mmuetzel StatusPatch Submitted Ready For Test
    2019-02-03 mmuetzel Attached File- Added bug55452_fopen_codepage_v2.patch, #46170
    2019-02-02 mmuetzel Attached File- Added bug55452_fopen_codepage.patch, #46168
        StatusNone Patch Submitted
    2019-01-12 jwe Severity3 - Normal 1 - Wish
        Priority5 - Normal 3 - Low
    2019-01-12 jwe Item GroupMatlab Compatibility Feature Request

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code