bugGNU Octave - Bugs: bug #41326, textread gives unnecessary warning...

 
 

bug #41326: textread gives unnecessary warning on umlauts

Submitter:  Hartmut <hardy>
Submitted:  Thu 23 Jan 2014 09:13:08 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Inaccurate Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 20 Dec 2018 06:46:07 PM UTC, comment #10: 

I think this issue can be CLOSED now (as discussed in comment #9).

Hartmut <hardy>
Sun 29 Apr 2018 05:47:57 PM UTC, comment #9: 

I have re-tested with with Octave 4.3.92 (Ubuntu linux) and 4.3.91 (Win 7):

  • I have never ever seen this warning message again. So I think the original idea of this bug report is obsolete and the issue should be CLOSED.
  • There is still no character encoding that leads to a working text import (with an umlaut) under linux and windows simultanously. This is a pity, but probably due to the fact that both operating systems have different default text encodings, and Octave does not have any  internationalzation in this respect (except maybe by accident, see comment #1). But this is probably been taken care of in many other bug reports that are still open today.
  • I have observed a reproducible crash of current Octave when importing texts with a certain text encoding (under both operating systems)! This is new, Octave 4.2.1 did not do this yet. I will open a separate bug report for this.
Hartmut <hardy>
Fri 27 Apr 2018 06:53:19 PM UTC, comment #8: 

Can you summarize the status of this issue for 4.4 or 5?

The summary says that a warning is printed, but I do not get a warning with Octave 4.2 or 4.4 on Debian GNU/Linux.

Is this a Windows-only character encoding issue now? Is this a duplicate now of the several other open bug reports about Windows and character encodings?

Mike Miller <mtmiller>
Group Member
Sat 19 Nov 2016 11:01:31 AM UTC, comment #7: 

This issue is still present in Octave 4.2.0 (linux and Windows).

Hartmut <hardy>
Fri 31 Jan 2014 09:28:01 AM UTC, comment #6: 

I've now also tested on Windows. Octave behaves a bit less tolerant there:

  • official Windows built 3.6.2: a couple of character encodings work nicely to import my 'ü'. They are for example Windows-1252 and ISO-8859-1. With other character encodings (like UTF-8) I only get funny symbols. (Tested on a german Win7 machine.)


  • The not so nice behavior in 3.6.2 is, that I found NO character encoding at all that did import the 'ü' properly in Linux AND in Windows at the same time. That makes life harder to use Octave independet of operating systems (like I do for instructional courses.)


  • inofficial Windows built 3.8.0 (mxe-octave), in terminal mode as well as in GUI mode: Here it is easy. Octave 3.8 NEVER imported my 'ü' properly, in any previously tested character encoding. That was disappointing :(
Hartmut <hardy>
Mon 27 Jan 2014 08:11:17 AM UTC, comment #5: 

For the record, after testing I have come to the following conclusion (at least for my linux system): there indeed is no encoding handling in octave, but interaction through the terminal (don't know about the GUI) -- supposedly via external libraries or terminal emulators -- assumes (and works correct with) UTF-8. So you can say a='ü' and it will display correctly, whos will show you that it occupies two bytes, and double(a) gives you the UTF-8 representation of this code point (and this also holds for unicode code points that do not coincide with ISO 8859-1, as is here the case). So when saving your text files in UTF-8, this interfaces nicely with fread/fwrite, strread...
This is a very desirable behaviour, whatever matlab may do, and there is definitely no need for any magic involving encoding guessing (such as text editors do) on the octave side.

Anonymous
Sat 25 Jan 2014 10:53:12 AM UTC, comment #4: 

Sorry for the confusion. I re-tested thorougly, and there seems to be NO difference between textscan and textread regarding the import of umlauts.

I've tested the following text encodings (all give the same warning message):

  • no encoding: textscan and textread work  OK  with an ü (This might be due to my system wide setting of the encoding on my linux machine.)
  • UTF-8: textscan and textread work   OK   with an ü
  • Windows 1252: textscan and textread work   NOT OK   with an ü
  • ISO 8859-1: textscan and textread work  NOT OK  with an ü


I can accept the initial comment of Jordi that there is no handling of character encodings implemented in Octave at all. If this is accepted to be this way, then just close this bug please.

Hartmut <hardy>
Thu 23 Jan 2014 10:52:43 PM UTC, comment #3: 

So it worked with textscan.

That's intriguing, as both textread and textscan invoke the very same backend for the actual text processing, i.e., strread.m

Furthermore, as to file encoding etc. I can't find any real difference between textscan.m (which is handed a file id by the user) and textread.m (which itself creates the file id). In the end both read the same text string and hand it to strread.m

Philip Nienhuis <philipnienhuis>
Group Member
Thu 23 Jan 2014 10:01:03 PM UTC, comment #2: 

You are right: My original csv file was coded in UTF-8. Then it worked fine as described.

Once I save the umlaut in a file with "Windows-1252/WinLatin" for example, I only get a funny symbol instead of my umlaut.

Note: When I switch to textscan command instead, like aa = textscan('test.csv','%s') , then I do get a nice umlaut even with the WinLatin encoding.

Is this a bug? I mean: Can Matlab read umlauts properly with textread / textscan? There seem to be Matlab commands like detect_encoding and unicode2native in Matlab. This might suggest that textread/textscan in Matlab isn't any better either.

Hartmut <hardy>
Thu 23 Jan 2014 09:20:48 PM UTC, comment #1: 


> So I assume that the given warning is unnecesssary. Octave does import umlauts properly.


Octave has no internationalisation support except by accident. The warning is not unnecessary, you just got lucky in this case. Whatever encoding your umlaut was in was also the representation that Octave chose to use for it when displaying it.

Try a different encoding, both multibyte and single byte, and see if you notice a difference with this warning.

Jordi Gutiérrez Hermoso <jordigh>
Group Member
Thu 23 Jan 2014 09:13:08 PM UTC, original submission:  

This is how to produce the behavior. It happend on the current release 3.8.0. of octave:

  • create a textfile "test.csv" that simply has one umlaut character in it. For example ü
  • use textread to import this into an octave variable: a = textread('test.csv','%s')
  • You will get a warning message "warning: range error for conversion to character value"
  • Also the text will properly be imported. The variable a does contain the umlaut afterwards.


So I assume that the given warning is unnecesssary. Octave does import umlauts properly.

Additionally the similar command strread will not give this warning on the same job. b = strread('ü','%s') just executes without any warning, and the umlaut is also properly read.

Hartmut <hardy>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by saturn
  • -email is unavailable- added by jordigh (Posted a comment)
  • -email is unavailable- added by hardy (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-12-20 mtmiller Carbon-CopyRemoved 80942 -
    2018-12-20 mtmiller StatusNeed Info Fixed
        Open/ClosedOpen Closed
    2018-08-23 saturn Carbon-Copy- Added saturn
    2018-04-27 mtmiller Release3.8.0 dev
    2014-01-23 jordigh StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code