bugGNU Octave - Bugs: bug #59848, dlmread does not support CR line...

 
 

bug #59848: dlmread does not support CR line endings (Classic Mac)

Submitter:  Hartmut <hardy>
Submitted:  Sat 09 Jan 2021 02:54:51 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  2 Item Group:  Inaccurate Result
Status:  Wont Fix Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 6.1.0 Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 13 Jan 2021 09:54:57 PM UTC, comment #10: 

Thinking twice about the origin of this csv-file with CR line terminators: I now think that my student probably downloaded this csv file from a official website that provides historical weather data (with the first column being the date information), the actual data was from the year 2020. And I personally can well imagine that those places still generate csv files with quite ancient PC infrastructure.

Nevertheless, I agree that this is a "very in the corner" case, since current OS do not generate this kind of file any more. You could leave the bug report open for future reference, or maybe even just update the bug report title to something more understandable and then close the report as "wont't fix".

Thank you for your help. And also sorry for the "noise", I was assuming a more general problem in our dlmread function.

Hartmut <hardy>
Wed 13 Jan 2021 09:16:38 PM UTC, comment #9: 

I don't want to be pessimistic, but I doubt it is worth fixing this when there are easy workarounds.  See this Wikipedia article on line terminators (https://en.wikipedia.org/wiki/Newline).  The last major operating system to use CR as the line terminator was Mac OS-9 and that was deprecated in 2001.  It doesn't feel necessary to me to support 20-year old systems.

A one-line conversion, which makes a backup of the original file, is


perl -i.bak -pe 's/\r/\n/g' FILENAME


What system was your student using that put CR as the line terminator?  As I noted, not modern operating system is using this convention so it is a wonder that they were able to do this.

More likely is that they were generating the CSV file manually and they directly put in carriage returns with printf ("\r") as the line endings instead of using the neutral newline character printf ("\n") which is translated by the OS to the correct character.  If true, I would suggest that this is something the programmer got wrong rather than Octave.

We can leave this bug report open in case someone does want to try fixing it.

Rik <rik5>
Group administrator
Wed 13 Jan 2021 07:42:10 PM UTC, comment #8: 

Yes, that's the key point. Thanks for noting it!

When I change the line endings in my csv files (from currently CR) to CRLF, then everything works fine. I then get the expected (and Matlab compatible) results with my demo code in comment #0.

The history of those csv files is as follows: They were sent to me by email. It might well be that these file were originally created on a Mac OS (but I cannot be 100% sure, because it was not me creating them). I then massivly edited the content of those csv files in order to prepare something small and compact for this bug report. But I assume that this file editing (under Windows as well as under Linux) did not change the existing line endings (i.e. leaving it CR).

MY CONCLUSION: As a result, I would now reformulate the essence of this bug report: "The dlmread function does not work properly with data files that have CR line endings." This is a comptibility issue, because Matlab treats those csv files with CR just fine. And this is also a usability issue, because I understand Octave as a cross platform tool, that should be able to accept files (at least data files like csv) also from other OS origin.

(My use case is the following: I demand the generation of data files and corresponding m-files as a kind of homework. I then let the resulting code and their corresponding data files run on my PC. And I would like Octave to work properly, regardless of the OS that the person preparing the files uses. This mostly works fine, in 99% of the cases. But in this corner case it seems to fail.)

Can we fix this?

Hartmut <hardy>
Tue 12 Jan 2021 11:21:03 PM UTC, comment #7: 

I can reproduce the problem with the data files uploaded by Hartmut.

If I execute 'file dataA.csv' I get


dataA.csv: UTF-8 Unicode (with BOM) text, with CR line terminators


I think this is the source of the problem.  The line terminators are neither UNIX (LF) nor Windows (CR LF), but old-style Macintosh line terminators (CR).

I used a text editor and search and replace function to substitute UNIX LF for the CR characters and then dlmread works correctly.

What program generated this file and can you have it create more appropriate line terminators?

Rik <rik5>
Group administrator
Tue 12 Jan 2021 08:29:54 PM UTC, comment #6: 

I have tested Rik's suggestens from comment #5 on my linux system with Octave 6.1 release:

1) Starting the Octave GUI with the "-f" flag did NOT change my results.

2) I am using a GERMAN locale on both the linux as well as the windows system. Can you give me a hint on how to (temporarily) change this to English, in order to test if this improves my results?

3) I will attach both csv FILES and a short m-file (...doublecheck.m) that reproduces this behavior for me. It would be nice to get some feedback if anyone else can reproduce this with those files. (The file endings in the csv files should be "CR" only, according to the Geany text editor.)

(file #50710, file #50711, file #50712)

Hartmut <hardy>
Tue 12 Jan 2021 05:09:12 PM UTC, comment #5: 

As with several others, I get the correct result with Octave version 6.1.  Marking status as "Works For Me".

@Hartmut: Here are three possible suggestions to try.

1) Start Octave with '-f' flag so that no user initialization files are read.  Perhaps you have something in a .octaverc file that is changing the behavior of Octave.

2) What locale are you using?  It is possible that there might be an interaction with the text-reading routines in dlmread.

3) Can you upload the exact file dataA.csv that you are using?  Perhaps there are hidden characters, or the file has CRLF line endings even on Linux, or something else.

Rik <rik5>
Group administrator
Sat 09 Jan 2021 07:20:24 PM UTC, comment #4: 

I tried it here on ubuntu and octave 7.0.0
It worked perfectly, no unusual results.

Doug Stewart <dastew>
Sat 09 Jan 2021 07:05:27 PM UTC, comment #3: 

My observations in comment #0 were done with Octave 6.1 under Ubuntu Linux (official tar-ball, self compiled) and also under Windows 10 (exe installer). I could even reproduce this behavior now with Octave 5.2 under Linux. Can anyone else reproduce this?

Hartmut <hardy>
Sat 09 Jan 2021 06:22:20 PM UTC, comment #2: 

Same with current stable (hg id 59696b24ea53)



octave:1> myfile = "dataA.csv";
octave:2> data1 = dlmread(myfile,',',1,2)
data1 =

    8.5000    5.4000   81.0000
    6.4000    4.9000   90.0000

octave:3> data2 = dlmread(myfile,',',0,2)
data2 =

    9.4000    5.8000   78.0000
    8.5000    5.4000   81.0000
    6.4000    4.9000   90.0000

octave:4> myfile = "dataB.csv";
octave:5> data1 = dlmread(myfile,',',1,2)
data1 =

    8.5000    5.4000   81.0000
    6.4000    4.9000   90.0000

octave:6> data2 = dlmread(myfile,',',0,2)
data2 =

    9.4000    5.8000   78.0000
    8.5000    5.4000   81.0000
    6.4000    4.9000   90.0000



Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 09 Jan 2021 06:10:16 PM UTC, comment #1: 

FWIW, in Octave-7.0.0 I get:

>> dlmread('csvA.csv',',',1,2)
ans =
    8.5000    5.4000   81.0000
    6.4000    4.9000   90.0000

>> dlmread('csvB.csv',',',0,2)
ans =
    9.4000    5.8000   78.0000
    8.5000    5.4000   81.0000
    6.4000    4.9000   90.0000


which AFAICS looks good.

Philip Nienhuis <philipnienhuis>
Group Member
Sat 09 Jan 2021 02:54:51 PM UTC, original submission:  

Here is how to reproduce this behavior of dlmread. I used those two demo data files:

dataA.csv:

2020-12-21 00:00:00,2020-12-21 01:00:00,9.4,5.8,78
2020-12-21 01:00:00,2020-12-21 02:00:00,8.5,5.4,81
2020-12-21 02:00:00,2020-12-21 03:00:00,6.4,4.9,90


dataB.csv:

a,b,9.4,5.8,78
c,d,8.5,5.4,81
e,f,6.4,4.9,90


and the following small m-script:

clear
myfile = "dataA.csv";
%myfile = "dataB.csv";

data1 = dlmread(myfile,',',1,2) % skip first row and two first columns
data2 = dlmread(myfile,',',0,2) % only skip to first columns


With dataA.csv I get
as result in data1

  • in Matlab: the expected numerical result (2x3 matrix)
  • in Octave: an empty matrix.

and as result in data2

  • in Matlab: the expected numerical result (3x3 matrix)
  • in Octave: an 1x11 complex array of nonsense numbers


With dataB.csv I get
as result in data1

  • in Octave: an empty matrix

as result in data2

  • in Octave: a 1x11 array of the correct numbers plus some additional zeros

(I have not testes the Matlab result here)

I know that dlmread can only return fully numeric results. But in this case the expected result is fully numeric, because just the two first columns, that dlmread should skip, contain non-numeric values. That's why I think dlmread should be able to properly function in this case (and in Matlab this does work fine).

I also know that I could work around this with textread or other data import functions. But it feels wrong that in this quite simple case (just numeric numbers to read) our dlmread function (and therefore probably also csvread) fails. (And the "next generation" data import function matrixread is not yet implemented in Octave.)

Hartmut <hardy>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50710:  dataA.csv added by hardy (156B - text/csv)
file #50711:  dataB.csv added by hardy (45B - text/csv)
file #50712:  dlmread_bug_doublecheck.m added by hardy (232B - text/x-objcsrc)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by dastew (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by hardy (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-01-14 rik5 StatusConfirmed Wont Fix
        Open/ClosedOpen Closed
        Summarydlmread returns wrong values when skipping non-numerical columns dlmread does not support CR line endings (Classic Mac)
    2021-01-13 rik5 Priority5 - Normal 2
    2021-01-12 rik5 StatusWorks For Me Confirmed
    2021-01-12 hardy Attached File- Added dataA.csv, #50710
        Attached File- Added dataB.csv, #50711
        Attached File- Added dlmread_bug_doublecheck.m, #50712
    2021-01-12 rik5 StatusNone Works For Me

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code