bugGNU Octave - Bugs: bug #58813, "csvread" and...

 
 

bug #58813: "csvread" and "dlmread" should strip BOM from UTF-8 encoded files

Submitter:  None
Submitted:  Thu 23 Jul 2020 09:47:08 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Fixed Assigned to:  None
Originator Name:  Adrian Barnes Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 04 Sep 2020 01:22:15 AM UTC, comment #4: 

I reviewed the patch and made a few small additions (a BIST test for this bug report, a "const" qualifier on the BOM variable).  See http://hg.savannah.gnu.org/hgweb/octave/rev/65fde9fe3b60.

Marking as fixed and closing report.

Rik <rik5>
Group administrator
Fri 28 Aug 2020 02:12:31 PM UTC, comment #3: 

The attached patch strips the BOM if needed.

(file #49719)

Markus Mützel <mmuetzel>
Group administrator
Fri 24 Jul 2020 06:17:25 AM UTC, comment #2: 

I didn't check in detail yet. But I believe that these functions currently expect files to be encoded in UTF-8 (but without BOM). ASCII is a sub-set of UTF-8. So ASCII files are also handled correctly.
It would probably be not too hard to skip the BOM.

Markus Mützel <mmuetzel>
Group administrator
Fri 24 Jul 2020 03:15:17 AM UTC, comment #1: 

I tested with Matlab and it has the same issue: it fails to read the UTF-8 CSV file.  I think we should resolve this by documenting that csvread and dlmread expect regular ASCII files as input.

Rik <rik5>
Group administrator
Thu 23 Jul 2020 09:47:08 PM UTC, original submission:  

Greetings, I have noticed that csvread incorrectly returns the first cell as ‘0’, when the csv file was saved from MS Excel as ‘UTF-8 csv’, but works okay when the export type is ‘CSV’. So I suspect something is going wrong with how csvread works with the ‘byte order mark’ that MS Excel is now using for this export type. While I am not implying csvread contains a bug (it is probably meant to work only with ascii csv), it may be appropriate to add a caveat to the documentation, i.e. csvread’s function reference.

To replicate using GNU Octave 5.2 for linux or for windows (I have tried both), and Excel 365, create a csv file csvu.csv such as [1 2 3; 4 5 6] and save as utf8 csv, and the same numbers saved as plain (ascii) csva.csv. In Octave, use csvread on both files and observe the first value is 0 from csvu.csv and 1 from csva.csv.

I originally raised this on the GNU/Octave discourse page:
https://octave.discourse.group/t/csvread-1st-cell-incorrect-with-utf-8-csv-files-from-ms-excel/99/2

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49719:  bug58813_dlmread_strip_BOM.patch added by mmuetzel (1KiB - application/octet-stream)
file #49537:  csvu.csv added by None (24B - text/csv)
file #49538:  csva.csv added by None (21B - text/csv)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 14 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-09-04 rik5 StatusPatch Submitted Fixed
        Open/ClosedOpen Closed
    2020-08-28 mmuetzel Release5.2.0 dev
    2020-08-28 mmuetzel Attached File- Added bug58813_dlmread_strip_BOM.patch, #49719
        CategoryDocumentation Octave Function
        Item GroupDocumentation Unexpected Error or Warning
        StatusConfirmed Patch Submitted
        SummaryDocument that csvread, dlmread expect regular ASCII files rather than UTF-8. "csvread" and "dlmread" should strip BOM from UTF-8 encoded files
    2020-07-24 rik5 CategoryOctave Function Documentation
        Item GroupNone Documentation
        StatusNone Confirmed
        Summarycsvread fails to handle 1st cell of CSV as exported by MS Excel as 'utf-8 CSV' due to byte order mark Document that csvread, dlmread expect regular ASCII files rather than UTF-8.
    2020-07-23 None Attached File- Added csvu.csv, #49537
        Attached File- Added csva.csv, #49538

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code