bugGNU Octave - Bugs: bug #49222, octave-io 2.4.3: xls2oct with...

 
 

bug #49222: octave-io 2.4.3: xls2oct with "OCT" interface lost the ability to read german umlauts or °

Submitter:  Andreas Weber <andy1978>
Submitted:  Thu 29 Sep 2016 02:29:49 PM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  3 - Low Item Group:  Regression
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * other Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 23 Oct 2016 07:05:09 PM UTC, comment #23: 

I've just uploaded io-2.4.4. to the package release tracker with the fixes I outlined in comment #21.
That means that in this new release the default operation as regards encoding and XML processing is back to what it was in io-2.4.0.  But for Windoze 7 and below (and maybe Win8 & Win8.1, I don't know) a way out is available to deal with UTF-8 encodings.

Closing this bug report as fixed.

(I'll admit that this all took a bit more work than I expected back in comment #1 :-)
But I'm glad with the results.)
Markus, thanks again for your contributions !

Philip Nienhuis <philipnienhuis>
Group Member
Fri 21 Oct 2016 08:55:22 PM UTC, comment #22: 

I pushed the fixes I outline in comment #21. To that end the OPTIONS field "retain_nonascii" has been renamed into "convert_utf".

Unless a better idea for fixing this on Windows comes up very soon, I'll do some more testing and make an io-2.4.4 release this weekend.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 21 Oct 2016 08:14:57 PM UTC, comment #21: 

Taking into account bug #43099, I think I'll adapt the fixes I pushed so that the utf82native conversions and vice versa need to be requested explicitly by the user with the options struct in oct2xls / oct2ods / xls2oct / ods2oct functions.
That leaves users on older Windows versions at least an option to display umlauts etc. in the terminal.

On Windows 10 it all seems to function properly with qt5 & the patches in bug #43099 (see there).

Philip Nienhuis <philipnienhuis>
Group Member
Sun 16 Oct 2016 09:25:20 PM UTC, comment #20: 

Markus,
I'm not so sure the patches do the wrong thing.

With what I know now, after some experimenting, is that what is explained in the wiki is not so much that Octave uses UTF-8 encoding, but is merely "transparent" - it just conveys string contents to lower level functions and often that works out well. I get the impression all of that is more accidental.

As stated in bug #46855, writing w/o converting non-alphanumeric chars to file gives problems when opening the file in LO or Excel. With your patches, that issue is solved.

On Windows everything works fine now.
On Linux it is different. If I read Andreas' test file I get  before your patch the proper characters in the terminal (Konsole), but in the Variable Editor (patch #8656) I see garbage. With your patch I get garbage in the terminal but proper text in the Variable Editor. So also on Linux there's a problem, or at least something going on, with encoding and/or codepages in the GUI terminal, I guess.
So it could well be that Andreas was just lucky that things worked well for him with io-2.4.0

I just pushed a next step, as also the Java based interfaces couldn't properly display Andreas' test .xlsx file contents in the Octave terminal on Windows. So I expanded the call to tidyxml to all non-COM interfaces.

I think the various Java spreadsheet I/O interfaces all have methods to assess the file encoding. I do not know about COM/ActiveX.
.xlsx, .gnumeric and .ods indicate the encoding in the top line of each individual XML file, as stated in comment #1.

Maybe we should transfer the discussion to the maintainers ML?

BTW I uploaded an io-2.4.4a dev snapshot to my dropbox. If you want you can have a try with it:
https://www.dropbox.com/s/0vchnxamjavkfwu/io-2.4.4a.tar.gz?dl=0
Note that in that dev snapshot there's a new option field in oct2xls/xls2oct/oct2ods/ods2oct "retain_nonascii" to bypass the encoding steps.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 16 Oct 2016 07:53:11 PM UTC, comment #19: 

Philip,

I am no longer sure whether this was the right thing to do.
Judging from the link [1] you provided in bug #49348, I guess that UTF-8 is Octave's "default encoding" (if it can be considered to have a default encoding).
That would mean we shouldn't convert to "Unicode". In fact, I guess the current approach only worked because QT seems to interpret invalid UTF-8 as ISO-8859-1 (latin-1). At least this is what I guess after reading [2]. This also explains why the command window would display fine after switching to codepage 1252 (which basically is the same as ISO-8859-1 for its printable characters).

This seems to leave us with 3 possibilities for reading strings from the file:
1- Read the byte stream from the xls and treat it directly as UTF-8.
2- Read the byte stream from the xls, check whether it is valid UTF-8 and strip invalid sequences.
3- Read the encoding and byte stream from the xls, convert it from that encoding to UTF-8.

After that, it might be possible to convert the resulting strings to whatever codepage the user has set.

I would like to vote for option 1, until a user stumbles over a file that isn't encoded in UTF-8 and complains. I would also prefer to always use UTF-8 in Octave and consider it a bug if that does not display right. It looks like QT5 does a much better job of supporting that encoding even in Windows.

The same holds for writing to files: Octave strings should always be encoded in UTF-8. Thus, it should be possible to directly write them to the file. But it might be reasonable to check for invalid sequences to avoid problems like bug #46855.

Sorry for providing you with wrong patches.

What do you think what should best be done?

[1] http://wiki.octave.org/International_Characters_Support#The_state_of_Octave
[2] https://en.wikipedia.org/wiki/UTF-8#Invalid_byte_sequences

Markus Mützel <mmuetzel>
Group administrator
Sat 15 Oct 2016 10:13:40 AM UTC, comment #18: 

In this message
https://lists.gnu.org/archive/html/help-octave/2010-04/msg00093.html
a proposed solution to properly display umlauts etc in the GUI terminal on Windows is to emit the command:

[Status, Text] = dos ("chcp 1252")


This solution works here on Win7, Octave 4.3.0+ on US-English locale.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 14 Oct 2016 08:42:04 PM UTC, comment #17: 

markus

I pushed your patch here:
http://hg.code.sf.net/p/octave/io/rev/9ebebc6503c0
with the following edits (most of it style nitpicking, apologies):

  1. Mützel => Muetzel as my mingw Mercurial doesn't want to accept umlauts


  1. Indentation of function body contents
  2. single => double quotes
  3. fix texinfo typo (@var[...} => @var{...}) in tidyxml.m
  4. first help sentence should just be one sentence explaining what the function does. I/O arguments explanations come after an empty line


Thanks very much for your contribution!

I'll leave this bug report open for a while; other interfaces may also suffer from conversion problems.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 14 Oct 2016 04:43:04 PM UTC, comment #16: 

I agree the use of general conversion functions would be a very good solution.
Before I push I'd like to test it a bit better myself.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 13 Oct 2016 05:17:42 PM UTC, comment #15: 

The current approach is quite flexible and can easily be extended to other encodings if this should be necessary: Just pass a function handle for that encoding to tidyxml and it should work.
Attached is yet another patch that adds the encoding and decoding of UTF-8 to ods2oct and oct2ods as well.
I also added a fixme that shows how this could work if native2unicode and unicode2native was available in Octave.

I would guess the patch should now be ready to be pushed.

(file #38720)

Markus Mützel <mmuetzel>
Group administrator
Thu 13 Oct 2016 01:08:58 PM UTC, comment #14: 

@Markus,
I have no idea when JWE thinks it's time to release 4.2. But maybe we shouldn't bother and just do our thing in our own time frame.

tidyxml.m (I couldn't think up a better name at the time :-) ) could have an option to distinguish de- and re-coding, yes.
But I have no objections at all to rename it into utf82unicode and unicode2utf8, in fact that might be a superior solution.

AFAICS all XML files in .xlsx (and .ods) have a first XML definitions line mentioning the encoding, usually that is UTF8. A check + conditional call for that is easy in principle but requires a bit of digging around in the general OCT interface code to find out where to put it. I'd rather do that on a subfile-by-subfile basis (subfiles = files in .xlsx archive) as long as I'm not certain that all files in a .xlsx archive have the same encoding. The ECMA docs are huge and I found that MS-Excel doesn't always conform to those.
Figuring out will probably take quite a bit of time and won't be ready in a jiffy.

The .ods counterparts are easily patched (I think they already call tidyxml).
I have plans (and actually almost all code ready) to fold oct2xls/oct2ods into one piece, same for xls2oct and ods2oct (and odsread/xlsread xlswrite/odswrite), to simplify things for when they (ever) go to core Octave.
ods2oct / oct2ods would be wrappers to their xls siblings then, similar for the other pairs.

BTW: I found that also some of the Java-based interfaces portray the same issues (POI / .xlsx and UNO). For now I'm only concerned about the OCT interface as that deals with XML only. Once it works there I'll look at fixing the other interfaces.

NB: I (and I suppose Andreas as well) really appreciate your quick fixes, thanks very much!

Philip Nienhuis <philipnienhuis>
Group Member
Thu 13 Oct 2016 07:26:10 AM UTC, comment #13: 

Found two instances where I used "end" insetead of "endif". Attached is the updates patch.

(file #38717)

Markus Mützel <mmuetzel>
Group administrator
Wed 12 Oct 2016 10:35:58 PM UTC, comment #12: 

Philip,

attached is a patch which adds the conversion in both directions.
I only added the conversion for xls2oct and oct2xls because I am not sure which file formats would actually need it. But it should be pretty straight forward to add it for *.ods as well.

I also don't know whether it is a good idea to unconditionally decode from UTF-8. Is it possible that the source file uses a different codepage?

(file #38716)

Markus Mützel <mmuetzel>
Group administrator
Wed 12 Oct 2016 09:00:35 PM UTC, comment #11: 

Philip,

thank you for reviewing and accepting so quickly.

Would it be OK to pass an additional argument to tidyxml to indicate in which direction the conversion should be done (UTF-8 to Unicode or vice versa)?
Matlab has the functions native2unicode and unicode2native which can be used to do these conversions. Maybe we could add functions for utf82unicode and unicode2utf8 until the former are available in core Octave.
If you can still wait a few days with the release, I will have a look whether I can come up with something.

Markus Mützel <mmuetzel>
Group administrator
Wed 12 Oct 2016 07:28:10 PM UTC, comment #10: 

@Markus,

Thank you very much, that cset helps out a lot - I can confirm what you write. I pushed it with some minor edits here:
http://hg.code.sf.net/p/octave/io/rev/e670260a712b

Now, writing to file still goes wrong. I suppose the conversion should go the other way round then. Note that tidyxml is also called in oct2xls.m.
If I comment out the call to tidyxml in oct2xls.m, write the contents of Andreas' file to disk and open it in Excel, I get complaints about illegal XML characters.
IIRC one of the reasons I made tidyxml.m is that Excel (or LibreOffice, I don't remember) didn't accept characters outside the range [32..127] in XML text contents. Although I could find that nowhere in the ECMA specs.

For now I think the situation is acceptable.
Unless a solution is soon found for writing/converting unicode back to acceptable XML text I think I'll make a new io release Very Soon (hopefully in time for the final Octave-4.2 mxe binaries to be made by JWE)
I think a note in oct2xls.m's texinfo header that XML chars outside 32..128] will be stripped when writing to file might be sufficient may be useful.

What do you think?

Philip Nienhuis <philipnienhuis>
Group Member
Wed 12 Oct 2016 03:34:47 PM UTC, comment #9: 

The problem here is that tidyxml does not convert UTF-8 to Unicode.

Additionally, Octave does not support Unicode. The highest char that does not fail is char(255). Nonetheless, these characters seem to map correctly on my system (Windows 10, German locale).

Attached is a patch for tidyxml that converts double byte UTF-8 characters to Unicode (and drops multi-byte chars with more than two bytes because Octave doesn't support Unicode anyway). With this approach at least the first two Unicode blocks (basic latin and latin-1 supplement) can be read correctly.

The characters do not display correctly in the "Command Window" (probably a codepage problem) but display fine in the "Workspace" for me.

(file #38713)

Markus Mützel <mmuetzel>
Group administrator
Wed 12 Oct 2016 10:13:24 AM UTC, comment #8: 

IIRC there are several bug reports about unicode in Octave on Windows, but I'd have to search first. I hope to find some time for that this week.

At first it was a little surprise to me to see Octave properly displaying unicode, even if in a bash-based terminal. But of course it is Octave's string handling functions that are unicoode-unaware, the rest should be OK. Except for mxe, that is. But Octave-on-Windows invokes an underlying cmd.exe that (AFAIK) is unicode-aware as well. Go figure.

Maybe we'd better move this discussion to the maintainers ML, that has a wider audience so we may have a better chance for a usable answer there.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 12 Oct 2016 09:49:00 AM UTC, comment #7: 

Philip, thank you for sharing your investigation.

Using "ispc" in tidyxml would break comaptibility of scripts between Windows and GNU/Linux. So although this would be an easy fix for my problem I wouldn't do this.

Using and option for xls2oct (and the others which use tidyxml) like "remove_non_ASCII" (because that is what it does, it removes non 7-bit ASCII chars) seems to be the cleanest approach, default to "=0, No".

Of course the best fix would be to fix reading unicode in the MXE build. Do you have an idea where they get lost?. Does the other interfaces (POI, UNO) work with äöüß°?. I don't have a windoze OS at home so I can first try this next week on work.

Andreas Weber <andy1978>
Group Member
Tue 11 Oct 2016 07:32:40 PM UTC, comment #6: 

Andreas,

It turns out that is the OS that makes the difference.

On Linux, after commenting out the call to tidyxml.m at L.198 in xls2oct (in io-2.4.3), I get the results you're after. Locale settings etc make little difference.

On Windows I get the results I showed in comment #2

I'm at loss at what to do now.
Octave formally doesn't support double-byte characters; yet the GUI seems to have no problems there, nor does konsole (I'm running Mageia-5 with KDE).
There are a few ways to proceed:

1. Enclose the calls to tidyxml in a "if (ispc)" block

2. Make calling tidyxml configurable using the "options" arguments in xls2oct & oct2xls (& ods2oct & oct2ods) with a suitable default (which one?)

3. Same as 2, but with different defaults on Linux (don't call tidyxml) and Windows (do call tidyxml).  Which IMO gets very messy.

Suggestions?

Philip Nienhuis <philipnienhuis>
Group Member
Tue 11 Oct 2016 06:55:15 AM UTC, comment #5: 

Philip,
with cset "release 2.4.0 921:19abe4f284ff" (I had to fix octave_config_info.USE_64_BIT_IDX_T -> _have_feature_ for Octave 4.1.0+) it works with the CLI and the GUI.

If I use Octave 4.2.0-rc2 and install the newest io 2.4.3 and replace the .m files for the OCT interface in inst it also works.

LANG=de_DE.utf8
Debian GNU/Linux 8.5 (jessie)

I also tried it with "LANG=nl_NL.utf8" (which I guess is your default locale) and it works

Andreas Weber <andy1978>
Group Member
Mon 10 Oct 2016 07:11:33 PM UTC, comment #4: 

ping Andreas

I'm still interested to have a fix for this, if at all possible.

In the last weeks I have tried a few things, a.o., recreate the circumstances where you could get (w. io-2.4.0)

et =
{
  [1,1] = 90°
  [2,1] = äöü
  [3,1] =  1234
}


but I could only get this:

{
  [1,1] = 90┬░
  [2,1] = ├ñ├╢├╝
  [3,1] =  1234
}



I suspect it must be s/th with locale but I have no further clues.
Did you run the CLI or the GUI, and what are your system settings as regards locale? (and what system do you have anyway)

Philip Nienhuis <philipnienhuis>
Group Member
Sun 02 Oct 2016 11:59:16 AM UTC, comment #3: 

BTW,
the output in comment #2 is exactly the same I get with io-2.4.0.

Philip Nienhuis <philipnienhuis>
Group Member
Sat 01 Oct 2016 11:52:36 AM UTC, comment #2: 

Andreas,

Changing L.42 of tidyxml.m (in io package) from

    ostr = char (istr(find (istr > 31 & istr < 127)));

into
+vrbatim+
    ostr = char (istr(find (istr > 31 & istr < 255)));
-verbatim-
I get with your spreadsheet:


>> [~, ~, rw] = xlsread ('bug_49222_AW_foobar2.xlsx', 1, '', 'oct')
Checking requested interface(s):
OCT*; (* = default interface)
rw =
{
  [1,1] = 90┬░
  [2,1] = ├ñ├╢├╝
  [3,1] =  1234
}
>> uint8 (rw {2, 1})
ans =

  195  164  195  182  195  188


which clearly shows the double-byte issue.

I played around on my box with various language settings for Windows and Octave but I couldn't get the proper output.



Philip Nienhuis <philipnienhuis>
Group Member
Thu 29 Sep 2016 06:25:37 PM UTC, comment #1: 

Yes the change was done deliberately, indeed already for io-2.4.1 (see the NEWS).

For the OCT interface spreadsheet I/O scripts there is no way to distinguish double-byte chars from single-byte chars. I found that the OCT interface returned gibberish for several spreadsheets and produced illegible stuff when reading or (re)writing .xlsx spreadsheets produced or processed by LibreOffice or Excel. Those programs often changed the encoding to unicode.

So I guess in a way you were lucky that you didn't hit problems. But I did.

Maybe there is a trick using the encoding pointer in the first line of the various XML files in the .xlsx archives. All I ever saw there was "UTF-8".
I'd be happy to include a fix but it must be proven thoroughly that it works reliably in all cases.

The fact that OCT reads xml files using regexp may be interfering here. regexp is much faster and has less memory overhead than a "real" XML parser but of course every advantage comes at a price.
There is an xmlread in the io package but it's Java-based and has Java overhead. If Java is considered a good option after all then I'd suggest to turn to Apache POI anyway as the POI interface works fine and reliably, has much less maintenance overhead, and Apache POI itself is actively maintained.

As there are workarounds (Apache POI, UNO) I'll lower priority.
I'll leave this report open but if no fix comes up in -say- 2 months I think I'll close it with "won't fix".

BTW I see that you (Andreas) submitted the report but your name isn't listed in "Originator name" ?

Philip Nienhuis <philipnienhuis>
Group Member
Thu 29 Sep 2016 02:29:49 PM UTC, original submission:  

With version 2.4.0 of io it was possible to read äöüß° from xlsx using OCT interface:


pkg load io;
xls = xlsopen ("foobar2.xlsx", 0, "OCT");
ret = xls2oct (xls, "Tabelle1", "A1:A3")
xlsclose (xls);

Checking requested interface(s):
OCT*; (* = default interface)
ret =
{
  [1,1] = 90°
  [2,1] = äöü
  [3,1] =  1234
}


I made a bisect, the changeset which broke this is:

Änderung:        929:6d8a647d1b8c
Nutzer:          Philip Nienhuis <prnienhuis@users@sf.net>
Datum:           Fri Jan 15 22:42:31 2016 +0100
Zusammenfassung: OCT interface: conversions for character I/O from/to XML-based file formats


Andreas Weber <andy1978>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #38720:  bug49222_utf8_v2b.patch added by mmuetzel (11KiB - text/x-patch)
file #38717:  bug49222_utf8_v2a.patch added by mmuetzel (8KiB - text/x-patch)
file #38716:  bug49222_utf8_v2.patch added by mmuetzel (8KiB - text/x-diff)
file #38713:  bug49222_utf8.patch added by mmuetzel (2KiB - text/x-patch)
file #38622:  foobar2.xlsx added by andy1978 (4KiB - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by andy1978 (adding package maintainer)
  • -email is unavailable- added by andy1978 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 16 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-10-23 philipnienhuis StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2016-10-14 philipnienhuis StatusIn Progress Ready For Test
    2016-10-13 mmuetzel Attached File- Added bug49222_utf8_v2b.patch, #38720
    2016-10-13 mmuetzel Attached File- Added bug49222_utf8_v2a.patch, #38717
    2016-10-12 mmuetzel Attached File- Added bug49222_utf8_v2.patch, #38716
    2016-10-12 philipnienhuis StatusNone In Progress
    2016-10-12 mmuetzel Attached File- Added bug49222_utf8.patch, #38713
    2016-09-29 philipnienhuis Priority5 - Normal 3 - Low
        Release4.0.3 other
        Operating SystemGNU/Linux Any
    2016-09-29 andy1978 Summaryoctave-io xls2oct with &quot;OCT&quot; interface lost the ability to read german umlauts or ° octave-io 2.4.3: xls2oct with "OCT" interface lost the ability to read german umlauts or °
    2016-09-29 andy1978 Carbon-Copy- Added philipnienhuis
    2016-09-29 andy1978 Carbon-CopyRemoved philip -
    2016-09-29 andy1978 Carbon-Copy- Added philip
    2016-09-29 andy1978 Attached File- Added foobar2.xlsx, #38622

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code