bugGNU Octave - Bugs: bug #58813, "csvread" and...


bug #58813: "csvread" and "dlmread" should strip BOM from UTF-8 encoded files

Submitted by:  None
Submitted on:  Thu 23 Jul 2020 09:47:08 PM UTC  
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Fixed Assigned to:  None
Originator Name:  Adrian Barnes Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  dev
Operating System:  Any

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Fri 04 Sep 2020 01:22:15 AM UTC, comment #4: 

I reviewed the patch and made a few small additions (a BIST test for this bug report, a "const" qualifier on the BOM variable).  See http://hg.savannah.gnu.org/hgweb/octave/rev/65fde9fe3b60.

Marking as fixed and closing report.

Rik <rik5>
Project Administrator
Fri 28 Aug 2020 02:12:31 PM UTC, comment #3: 

The attached patch strips the BOM if needed.

(file #49719)

Markus Mützel <mmuetzel>
Project Member
Fri 24 Jul 2020 06:17:25 AM UTC, comment #2: 

I didn't check in detail yet. But I believe that these functions currently expect files to be encoded in UTF-8 (but without BOM). ASCII is a sub-set of UTF-8. So ASCII files are also handled correctly.
It would probably be not too hard to skip the BOM.

Markus Mützel <mmuetzel>
Project Member
Fri 24 Jul 2020 03:15:17 AM UTC, comment #1: 

I tested with Matlab and it has the same issue: it fails to read the UTF-8 CSV file.  I think we should resolve this by documenting that csvread and dlmread expect regular ASCII files as input.

Rik <rik5>
Project Administrator
Thu 23 Jul 2020 09:47:08 PM UTC, original submission:  

Greetings, I have noticed that csvread incorrectly returns the first cell as ‘0’, when the csv file was saved from MS Excel as ‘UTF-8 csv’, but works okay when the export type is ‘CSV’. So I suspect something is going wrong with how csvread works with the ‘byte order mark’ that MS Excel is now using for this export type. While I am not implying csvread contains a bug (it is probably meant to work only with ascii csv), it may be appropriate to add a caveat to the documentation, i.e. csvread’s function reference.

To replicate using GNU Octave 5.2 for linux or for windows (I have tried both), and Excel 365, create a csv file csvu.csv such as [1 2 3; 4 5 6] and save as utf8 csv, and the same numbers saved as plain (ascii) csva.csv. In Octave, use csvread on both files and observe the first value is 0 from csvu.csv and 1 from csva.csv.

I originally raised this on the GNU/Octave discourse page:



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #49719:  bug58813_dlmread_strip_BOM.patch added by mmuetzel (1KiB - application/octet-stream)
file #49537:  csvu.csv added by None (24B - text/csv)
file #49538:  csva.csv added by None (21B - text/csv)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 14 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-09-04 rik5 StatusPatch Submitted => Fixed
        Open/ClosedOpen => Closed
    2020-08-28 mmuetzel Release5.2.0 => dev
    2020-08-28 mmuetzel Attached File- => Added bug58813_dlmread_strip_BOM.patch, #49719
        CategoryDocumentation => Octave Function
        Item GroupDocumentation => Unexpected Error or Warning
        StatusConfirmed => Patch Submitted
        SummaryDocument that csvread, dlmread expect regular ASCII files rather than UTF-8. => "csvread" and "dlmread" should strip BOM from UTF-8 encoded files
    2020-07-24 rik5 CategoryOctave Function => Documentation
        Item GroupNone => Documentation
        StatusNone => Confirmed
        Summarycsvread fails to handle 1st cell of CSV as exported by MS Excel as 'utf-8 CSV' due to byte order mark => Document that csvread, dlmread expect regular ASCII files rather than UTF-8.
    2020-07-23 None Attached File- => Added csvu.csv, #49537
        Attached File- => Added csva.csv, #49538

    Back to the top

    Powered by Savane 3.5