bugGNU Octave - Bugs: bug #42025, dlmread unable to recognize...

 
 

bug #42025: dlmread unable to recognize emptyvalues at start of line with whitespace separator

Submitter:  Philip Nienhuis <philipnienhuis>
Submitted:  Tue 01 Apr 2014 09:08:27 PM UTC
   
 
Category:  Libraries Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  None
Originator Name:  Philip Nienhuis Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 25 Oct 2016 05:28:50 PM UTC, comment #19: 

Wow, this has turned out to be a very long and difficult problem.

I checked in a final cset (http://hg.savannah.gnu.org/hgweb/octave/rev/68d5c4759783) that does not create an extra column when the input line ends with a separator.  I also made some changes that will improve the efficiency of dlmread because I avoid constantly re-sizing the output data matrix for the first 32 rows.

A final question, what happens if you try and read imp6.txt and specify a delimiter that is incorrect such as a comma?  In Octave, I get a complex result which seems strange.


dlmread ('imp6.txt', ',')
ans =

    1 +  2i
   11 + 22i


I will open up a new bug report for that issue if Matlab does something different.

Closing this report.

Rik <rik5>
Group administrator
Tue 25 Oct 2016 07:49:45 AM UTC, comment #18: 

Matlab r2016b, r2014a:

>> dlmread ('imp6.txt', ' ')
ans =
     0     0     0     0
     0     1     2     0
    11    22     0     0
     0     0     0     0
>> dlmread ('imp3.txt', ' ')
ans =
     1    11     0
     0    22     0
     0     3    33
     0     0     0


A trailing space on last line or not makes no difference as to nr. of columns, not even presence of the last line itself (i.e., L.3 ending with EOF)

Philip Nienhuis <philipnienhuis>
Group Member
Mon 24 Oct 2016 08:11:46 PM UTC, comment #17: 

sorry no time today at work. Next chance tomorrow.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 23 Oct 2016 09:35:46 PM UTC, comment #16: 

In the mean time, I've checked in a bunch of changes that improve dlmread here (http://hg.savannah.gnu.org/hgweb/octave/rev/471cc4268677).

Rik <rik5>
Group administrator
Sun 23 Oct 2016 04:30:12 PM UTC, comment #15: 

yep I'll try to test at work tomorrow.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 23 Oct 2016 02:47:50 PM UTC, comment #14: 

When you re-test imp3.txt, could you also try the attached imp6.txt?

Under Octave, I now get this.  I don't know if the final colmun is Matlab-compatible.  I've made the encoding for imp6.txt CRLF since I assume you are doing testing on Windows.  If not, use dos2unix on imp6.txt before testing.


octave:2> dlmread ('imp6.txt', ' ')
ans =

    0    0    0    0    0
    0    1    2    0    0
   11   22    0    0    0
    0    0    0    0    0




(file #38797)

Rik <rik5>
Group administrator
Sun 23 Oct 2016 04:35:20 AM UTC, comment #13: 

I've got something that works for everything except this set of tests


>> dlmread ('imp3.txt', ' ')
ans =
     1    11     0
     0    22     0
     0     3    33
>> dlmread ('imp4.txt', ' ')
ans =
     0     0    22     0
     1    11     0     0
     0     0     3    33
     0     0     0     0  % apparently one space on L.4


Both imp3.txt and imp4.txt end with a line that contains a single space.  So, why isn't the result for imp3.txt this


     1    11     0
     0    22     0
     0     3    33
     0     0     0


Could you re-test imp3.txt, after making sure it is the one attached to this bug report?

Rik <rik5>
Group administrator
Fri 21 Oct 2016 07:54:02 PM UTC, comment #12: 

I cannot find any imp3.txt on my boxes other than the one I recently uploaded. This bug report is 2.5 years old, in the mean time my PC's have been upgraded several times. I think I've entered it from my box at work, and especially that one is free game (beyond my own control) :-(

Yet I'm surprised I get different results than in April 2014; I can't explain, sorry.

Anyway I'd suggest to just proceed, Rik, with the changes you had in mind. If they also solve Carnë's problem I think this bug can be considered fixed.
After all, the Matlab results from comment #10 are very consistent.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 21 Oct 2016 04:27:46 PM UTC, comment #11: 

I'll wait to make any changes until I get confirmation that imp3.txt and imp4.txt are the correct examples to use.  I have a pretty good idea how to replicate the outputs from Philip in comment #10 if that is indeed the correct path.

Rik <rik5>
Group administrator
Thu 20 Oct 2016 10:07:27 AM UTC, comment #10: 

(just a quick try)
With the imp3.txt and imp4.txt in this bug report, I get with
2012a, 204a and 2016b:

>> format compact
>> dlmread ('imp3.txt', ' ')
ans =
     1    11     0
     0    22     0
     0     3    33
>> dlmread ('imp4.txt', ' ')
ans =
     0     0    22     0
     1    11     0     0
     0     0     3    33
     0     0     0     0  % apparently one space on L.4
>> dlmread ('imp3.txt')
ans =
     1    11
    22     0
     3    33
>> dlmread ('imp4.txt')
ans =
    22     0
     1    11
     3    33


Replacing CRLF with just LF makes no difference.
It could be that the imp3.txt I recently sent was different than the original one that motivated this bug report. I'll look at that later.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 19 Oct 2016 07:09:56 PM UTC, comment #9: 

@Philip: Try to read the original file (imp3.txt) with the latest version 2016b.  I don't think there have been changes so you should get the same answer of


1 11

0 22

3 33


Also, try downloading imp4.txt from this bug report and executing


dlmread ('imp4.txt', ' ')




(file #38768)

Rik <rik5>
Group administrator
Tue 18 Oct 2016 08:21:18 PM UTC, comment #8: 

I can try in 2012a, 2014a and 2016b, but maybe only Thursday at the earliest.
Exactly what would you like me to check?

AFAICS Matlab didn't change behavior in those versions.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 17 Oct 2016 10:24:05 PM UTC, comment #7: 

We could get a check.  Philip apparently has access to 2014a.

The problem is that whitespace is supposed to be ignored at the start of lines.  But what if the delimiter is whitespace itself?  There is some code to partially work around this by setting an internal variable sepflag, but it is all rather complicated.

In the imp3.txt file, one of the lines is


" 3 33"


When read with


dlmread ('imp3.txt', ' ')


I would think that the first space is treated as a delimiter which means the output should be


0  3  33


Instead, Matlab eats the whitespace at the beginning of the line and returns


3  33


But the line above this in imp3.txt is


" 22"


In this case, Matlab doesn't eat the whitespace and actually returns


0  22


It is almost like you would need to parse any line beginning with a whitespace delimiter twice in order to determine which way gave you the "best" outcome.

Rik <rik5>
Group administrator
Mon 17 Oct 2016 10:11:22 PM UTC, comment #6: 

Actually, my last post shows that it works fine when the separator is not whitespace. The cause seems to be this change http://hg.savannah.gnu.org/hgweb/octave/rev/99b00a9a5147 which fixed bug #32260 with the exact opposite behaviour (ignore trailing whitespace).

That bug had used Matlab R2009 so maybe Matlab has changed behaviour since? Or maybe the issue is not on the trailing whitespace but on automatically guessing the delimiter (so it only ignores trailing whitespace that is not used as delimiter)?

Carnë Draug <carandraug>
Group Member
Mon 17 Oct 2016 10:02:32 PM UTC, comment #5: 

I just bumped into this bug now too. An easier example that does not use whitespace as separate for sake of copy/paste:


$ cat foo
1a2a3
5aa7
a8a9
$ octave
octave:1> dlmread ("foo", "a")
ans =

   1   2   3
   5   0   7
   0   8   9


Carnë Draug <carandraug>
Group Member
Sat 08 Oct 2016 09:03:10 AM UTC, comment #4: 

I had forgotten about this report, a bit dumb of me not to upload imp3.txt at the time.
Anyway here it is.

To be sure I just checked, Matlab (2014a) still behaves the same.

(file #38684)

Philip Nienhuis <philipnienhuis>
Group Member
Fri 07 Oct 2016 10:50:34 PM UTC, comment #3: 

This bug report is quite old, but still might be fixed.  Can you upload the file imp3.txt so that it can be tested?  There seems to have been some issues with the verbatim tag in savannah which make me unsure of what the input file is exactly.

Rik <rik5>
Group administrator
Sat 05 Apr 2014 07:55:15 PM UTC, comment #2: 

(Hmmm apparently savannah doesn't like me.....)

Another try:

ML r2012a gives for first call (correctly, I think):


1 11

0 22

3 33


Philip Nienhuis <philipnienhuis>
Group Member
Sat 05 Apr 2014 07:53:20 PM UTC, comment #1: 

Re-reading, I spotted a copy/paste error, sorry for that:

Corrected:

ML r2012a gives for first call (correctly, I think):

1 11

  1. 22


3 33

Philip Nienhuis <philipnienhuis>
Group Member
Tue 01 Apr 2014 09:08:27 PM UTC, original submission:  

File imp3.txt:

1 11
 22
3 33

(no trailing spaces after 2nd column)

Octave:

>> dlmread ('imp3.txt', ' ')
ans =

    1   11
   22    0
    3   33

>> dlmread ('imp3.txt')
ans =

    1   11
   22    0
    3   33


ML r2012a gives for first call (correctly, I think):

1   11

  1.   22

3   33

and for the second (w/o specified delimiter)

 1   11
22    0
 3   33

so Octave is ML-compatible in the second case.

(FYI, I've seen bug #32260. That is about dlm behavior for leading whitespace when no delimiter is specified)

Philip Nienhuis <philipnienhuis>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #38797:  imp6.txt added by rik5 (20B - text/plain)
file #38768:  imp4.txt added by rik5 (23B - text/plain)
file #38684:  imp3.txt added by philipnienhuis (19B - text/plain)

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by philipnienhuis (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-10-25 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2016-10-23 rik5 StatusNeed Info Ready For Test
    2016-10-23 rik5 Dependencies- bugs #42023 is dependent
    2016-10-23 rik5 Attached File- Added imp6.txt, #38797
    2016-10-19 rik5 Attached File- Added imp4.txt, #38768
    2016-10-17 carandraug Summarydlmread unable to recognize emptyvalues at start of line dlmread unable to recognize emptyvalues at start of line with whitespace separator
    2016-10-17 carandraug Summarydlmread doesn't properly treat leading space as empty values for space delimiter dlmread unable to recognize emptyvalues at start of line
    2016-10-08 philipnienhuis Attached File- Added imp3.txt, #38684
    2016-10-07 rik5 StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code