bugGNU Octave - Bugs: bug #41326, textread gives unnecessary warning...

 
 

bug #41326: textread gives unnecessary warning on umlauts

Submitted by:  Hartmut <hardy>
Submitted on:  Thu 23 Jan 2014 09:13:08 PM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Inaccurate Result
Status: Need InfoAssigned to: None
Originator Name: Open/Closed: Open
Release: 3.8.0Operating System: Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 19 Nov 2016 11:01:31 AM UTC, comment #7:

This issue is still present in Octave 4.2.0 (linux and Windows).

Hartmut <hardy>
Fri 31 Jan 2014 09:28:01 AM UTC, comment #6:

I've now also tested on Windows. Octave behaves a bit less tolerant there:

  • official Windows built 3.6.2: a couple of character encodings work nicely to import my 'ü'. They are for example Windows-1252 and ISO-8859-1. With other character encodings (like UTF-8) I only get funny symbols. (Tested on a german Win7 machine.)
  • The not so nice behavior in 3.6.2 is, that I found NO character encoding at all that did import the 'ü' properly in Linux AND in Windows at the same time. That makes life harder to use Octave independet of operating systems (like I do for instructional courses.)
  • inofficial Windows built 3.8.0 (mxe-octave), in terminal mode as well as in GUI mode: Here it is easy. Octave 3.8 NEVER imported my 'ü' properly, in any previously tested character encoding. That was disappointing :(
Hartmut <hardy>
Mon 27 Jan 2014 08:11:17 AM UTC, comment #5:

For the record, after testing I have come to the following conclusion (at least for my linux system): there indeed is no encoding handling in octave, but interaction through the terminal (don't know about the GUI) -- supposedly via external libraries or terminal emulators -- assumes (and works correct with) UTF-8. So you can say a='ü' and it will display correctly, whos will show you that it occupies two bytes, and double(a) gives you the UTF-8 representation of this code point (and this also holds for unicode code points that do not coincide with ISO 8859-1, as is here the case). So when saving your text files in UTF-8, this interfaces nicely with fread/fwrite, strread...
This is a very desirable behaviour, whatever matlab may do, and there is definitely no need for any magic involving encoding guessing (such as text editors do) on the octave side.

Anonymous
Sat 25 Jan 2014 10:53:12 AM UTC, comment #4:

Sorry for the confusion. I re-tested thorougly, and there seems to be NO difference between textscan and textread regarding the import of umlauts.

I've tested the following text encodings (all give the same warning message):

  • no encoding: textscan and textread work OK with an ü (This might be due to my system wide setting of the encoding on my linux machine.)
  • UTF-8: textscan and textread work OK with an ü
  • Windows 1252: textscan and textread work NOT OK with an ü
  • ISO 8859-1: textscan and textread work NOT OK with an ü

I can accept the initial comment of Jordi that there is no handling of character encodings implemented in Octave at all. If this is accepted to be this way, then just close this bug please.

Hartmut <hardy>
Thu 23 Jan 2014 10:52:43 PM UTC, comment #3:

So it worked with textscan.

That's intriguing, as both textread and textscan invoke the very same backend for the actual text processing, i.e., strread.m

Furthermore, as to file encoding etc. I can't find any real difference between textscan.m (which is handed a file id by the user) and textread.m (which itself creates the file id). In the end both read the same text string and hand it to strread.m

Philip Nienhuis <philipnienhuis>
Project Member
Thu 23 Jan 2014 10:01:03 PM UTC, comment #2:

You are right: My original csv file was coded in UTF-8. Then it worked fine as described.

Once I save the umlaut in a file with "Windows-1252/WinLatin" for example, I only get a funny symbol instead of my umlaut.

Note: When I switch to textscan command instead, like aa = textscan('test.csv','%s') , then I do get a nice umlaut even with the WinLatin encoding.

Is this a bug? I mean: Can Matlab read umlauts properly with textread / textscan? There seem to be Matlab commands like detect_encoding and unicode2native in Matlab. This might suggest that textread/textscan in Matlab isn't any better either.

Hartmut <hardy>
Thu 23 Jan 2014 09:20:48 PM UTC, comment #1:

> So I assume that the given warning is unnecesssary. Octave does import umlauts properly.


Octave has no internationalisation support except by accident. The warning is not unnecessary, you just got lucky in this case. Whatever encoding your umlaut was in was also the representation that Octave chose to use for it when displaying it.

Try a different encoding, both multibyte and single byte, and see if you notice a difference with this warning.

Jordi Gutiérrez Hermoso <jordigh>
Project Administrator
Thu 23 Jan 2014 09:13:08 PM UTC, original submission:

This is how to produce the behavior. It happend on the current release 3.8.0. of octave:

  • create a textfile "test.csv" that simply has one umlaut character in it. For example ü
  • use textread to import this into an octave variable: a = textread('test.csv','%s')
  • You will get a warning message "warning: range error for conversion to character value"
  • Also the text will properly be imported. The variable a does contain the umlaut afterwards.

So I assume that the given warning is unnecesssary. Octave does import umlauts properly.

Additionally the similar command strread will not give this warning on the same job. b = strread('ü','%s') just executes without any warning, and the umlaut is also properly read.

Hartmut <hardy>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jordigh (Posted a comment)
  • -unavailable- added by hardy (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 23 Jan 2014 09:20:48 PM UTCjordighStatusNone=>Need Info

    Back to the top


    Powered by Savane 3.1-cleanup1