bugGNU Octave - Bugs: bug #58813, Document that csvread, dlmread...

 
 

bug #58813: Document that csvread, dlmread expect regular ASCII files rather than UTF-8.

Submitted by:  None
Submitted on:  Thu 23 Jul 2020 09:47:08 PM UTC  
 
Category:  Documentation Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Documentation
Status:  Confirmed Assigned to:  None
Originator Name:  Adrian Barnes Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  5.2.0
Operating System:  Any

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 24 Jul 2020 06:17:25 AM UTC, comment #2: 

I didn't check in detail yet. But I believe that these functions currently expect files to be encoded in UTF-8 (but without BOM). ASCII is a sub-set of UTF-8. So ASCII files are also handled correctly.
It would probably be not too hard to skip the BOM.

Markus Mützel <mmuetzel>
Project Member
Fri 24 Jul 2020 03:15:17 AM UTC, comment #1: 

I tested with Matlab and it has the same issue: it fails to read the UTF-8 CSV file.  I think we should resolve this by documenting that csvread and dlmread expect regular ASCII files as input.

Rik <rik5>
Project Administrator
Thu 23 Jul 2020 09:47:08 PM UTC, original submission:  

Greetings, I have noticed that csvread incorrectly returns the first cell as ‘0’, when the csv file was saved from MS Excel as ‘UTF-8 csv’, but works okay when the export type is ‘CSV’. So I suspect something is going wrong with how csvread works with the ‘byte order mark’ that MS Excel is now using for this export type. While I am not implying csvread contains a bug (it is probably meant to work only with ascii csv), it may be appropriate to add a caveat to the documentation, i.e. csvread’s function reference.

To replicate using GNU Octave 5.2 for linux or for windows (I have tried both), and Excel 365, create a csv file csvu.csv such as [1 2 3; 4 5 6] and save as utf8 csv, and the same numbers saved as plain (ascii) csva.csv. In Octave, use csvread on both files and observe the first value is 0 from csvu.csv and 1 from csva.csv.

I originally raised this on the GNU/Octave discourse page:
https://octave.discourse.group/t/csvread-1st-cell-incorrect-with-utf-8-csv-files-from-ms-excel/99/2

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49537:  csvu.csv added by None (24B - text/csv)
file #49538:  csva.csv added by None (21B - text/csv)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-07-24 rik5 CategoryOctave Function => Documentation
        Item GroupNone => Documentation
        StatusNone => Confirmed
        Summarycsvread fails to handle 1st cell of CSV as exported by MS Excel as 'utf-8 CSV' due to byte order mark => Document that csvread, dlmread expect regular ASCII files rather than UTF-8.
    2020-07-23 None Attached File- => Added csvu.csv, #49537
        Attached File- => Added csva.csv, #49538

    Back to the top


    Powered by Savane 3.5