bugGNU Octave - Bugs: bug #33580, dlmread does not interpret CR as a...


bug #33580: dlmread does not interpret CR as a newline

Submitted by:  None
Submitted on:  Thu 16 Jun 2011 05:07:19 PM UTC  
Category:  Octave Function Severity:  2 - Minor
Priority:  3 - Low Item Group:  Incorrect Result
Status:  Confirmed Assigned to:  None
Originator Name:  Dan Whitney Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  dev
Operating System:  Mac OS

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Tue 25 Oct 2016 05:46:21 PM UTC, comment #6: 

textread.m was mentioned in comment #4.  textread.m should correctly asses the EOL character(s) by itself if it isn't explicitly specified in the input args. If this doesn't work a specific bug report is welcome (please include example file(s) and show us the call to textread) and we'll have a look.

Philip Nienhuis <philipnienhuis>
Project Member
Sun 23 Oct 2016 05:37:05 AM UTC, comment #5: 

Octave internally uses C++ getline() function to get a line of text.  This reads characters up to a newline and then discards the newline.  Because no one sat down to agree on this, a newline is CR on MAC, LF on UNIX, and CRLF on Windows.  The C++ standard library on the platform where Octave is executing determines which character(s) are interpreted as newline.  It is the user's responsibility, and always has been, to change text files to the correct encoding when moving between operating systems.  This is not Octave specific in any way.

There are utilities for this like dos2unix and unix2dos.  Use these, or a simple script like the Perl one posted earlier, when transferring files between operating systems. 

Rik <rik5>
Project Administrator
Thu 01 Mar 2012 09:47:33 PM UTC, comment #4: 

I would like to chime in. While we are (not) waiting for a fix, can we at least get a small note in help on csvread, dlmread, textread describing this peculiarity? I have spend about an hour trying to find why I'm getting unexpected results and stumbled on this bug report only by an accident - a simple note would save a lot of time!

Maksym <petrenkomaxim>
Fri 17 Jun 2011 07:04:35 AM UTC, comment #3: 

> The most direct fix that works is to open the file in Excel 2011 and
> save as Windows formatted text.

Simple byte substitution seems more direct to me. You're using a Unix
system, and one of the guiding principles of Unix philosphy is that
text is the default format because it's the format that all Unix tools
can understand. It is not necessary to open Excel to manipulate a text
CSV file or to use Word to needlessly give formatting to a plaintext
Octave session as RTF. Perl as above is one simple example of how to
do this without all that Microsoft faff.

> The suggested fix perl -pi -e 's,\r,\n,gs' shanghai_node_list2.csv did
> not work.

I didn't think you would type the perl command in Octave, because I
had forgotten that Octave has a perl command. I meant in bash. Double
up the backslashes if you type it from Octave:

     perl -pi -e 's,\\r,\\n,gs' shanghai_node_list2.csv

Regardless, while I think it's still a valid point about making
dlmread and other Octave commands recognise Mac OS 9 line endings, I'm
marking this bug low priority for now. I hope you agree. If you don't,
I'll revert its severity.

Jordi GutiƩrrez Hermoso <jordigh>
Project Administrator
Fri 17 Jun 2011 06:20:20 AM UTC, comment #2: 

The suggested fix perl -pi -e 's,r,n,gs' shanghai_node_list2.csv did not work. See attached record of octave interactions.

The most direct fix that works is to open the file in Excel 2011 and save as Windows formatted text.

I am using Mac OS 10.6.7 but I am dependent on Office for creating saved files. Whatever it uses is what I get. Word, but not Excel, permits me to choose the EOL character when saving txt files and no Office app lets me pick the EOL character for csv files.

(file #23531)

Daniel Whitney <danwhitney>
Thu 16 Jun 2011 06:38:21 PM UTC, comment #1: 

The advertising (I imagine you mean the docstring) doesn't specify anything about CR being a newline. The last commonly-used operating system that used CR as a newline was Mac OS 9. Are you really using this OS?

If you're using a newer POSIX-compatible OS (e.g. Mac OS X), the following workaround may simplify your life a little to convert CR to LF:

     perl -pi -e 's,\r,\n,gs' shanghai_node_list2.csv

It may be desirable to intrpret CR as a newline as well. The current implementation relies on the C++'s standard library default interpretation of a newline, i.e. looks for the \n character. Patching dlmread to interpret CR as a newline, therefore, isn't entirely a triviality. The patch would optimally figure out the line ending of the file and use that as a delimiter for the C++ stdlib getline() function.

Jordi GutiƩrrez Hermoso <jordigh>
Project Administrator
Thu 16 Jun 2011 05:07:19 PM UTC, original submission:  

The documentation simply says to use dlmread as though there was nothing to it but to do it. But in fact dlmread (and csvread) wants LF only as the end of line marker and will not recoginze CR or (I think) CRLF.  A simple excel csv file ends its lines in CR and dlmread reads it in as one long string instead of a set of lines.  Matlab reads such files effortlessly.

It seems to be a shame that such a simple command does not work.  I have lots of network models saved as excel csv files and I find I have to convert each one to a text file using Word and save them as LF only text files.  This does not ruin my life but it is a needless pain.

I recorded this but as v 3.2.3 but v 3.4.0 does the same thing.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #23531:  CR fix failed.rtf added by danwhitney (5KiB - text/rtf - file contains failed and successful trials at Octave command line)
file #23529:  shanghai_node_list2.csv added by None (661B - text/csv - The csv file is read incorrectly. Its size is 1 x 198. The file shanghai_mode_list5 is read correctly. Its size is 50 x 8.)
file #23530:  shanghai_node_list5.txt added by None (778B - text/plain - The csv file is read incorrectly. Its size is 1 x 198. The file shanghai_mode_list5 is read correctly. Its size is 50 x 8.)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mtmiller (Updated the item)
  • -email is unavailable- added by petrenkomaxim (Posted a comment)
  • -email is unavailable- added by danwhitney (Updated the item)
  • -email is unavailable- added by jordigh (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-10-23 rik5 Open/ClosedOpen => Closed
    2014-01-19 mtmiller CategoryNone => Octave Function
        Release3.2.3 => dev
    2011-06-17 jordigh Severity3 - Normal => 2 - Minor
        Priority5 - Normal => 3 - Low
    2011-06-17 danwhitney Attached File- => Added CR fix failed.rtf, #23531
    2011-06-16 jordigh StatusNone => Confirmed
        Summarydlmread does not function as advertised => dlmread does not interpret CR as a newline
    2011-06-16 None Attached File- => Added shanghai_node_list2.csv, #23529
        Attached File- => Added shanghai_node_list5.txt, #23530

    Back to the top

    Powered by Savane 3.5