bugGNU Octave - Bugs: bug #38317, textread & textscan: different...

 
 

bug #38317: textread & textscan: different results for empy format strings than MATLAB

Submitter:  None
Submitted:  Tue 12 Feb 2013 05:03:17 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  philipnienhuis
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 30 Mar 2013 07:31:42 PM UTC, comment #19: 

Thanks for the investigation.

As far as I'm concerned this particular issue won't be fixed anymore in the current strread/textread/textscan versions.

This behavior is similar to bug #38587 and is related to the way text files are parsed. Nothing to be done about it, except a binary (x)textscan.

Philip Nienhuis <philipnienhuis>
Group Member
Sat 30 Mar 2013 03:06:18 PM UTC, comment #18: 

“test_textread.txt” in the previous post should be replaced with “empty_spots.txt”.

Stig <default>
Sat 30 Mar 2013 03:02:38 PM UTC, comment #17: 

I have yet find the Matlab behaviour we are looking at here documented anywhere, so I am just comparing the results from Octave to the corresponding results from Matlab.

(The files empty_spots.txt, empty_lines.txt have been attached.)

Matlab 7.14.0.739 (R2012a)

>> X = textread('empty_spots.txt', '', 'headerlines', 6)

X =

         0    5.6900    4.3700    0.5400
    1.0000    0.5800    0.5900    0.0100
    2.0000    0.4300    0.4200         0
    3.0000    0.9300    0.5400    0.3000
    4.0000    0.1200    0.9900    0.7000


Octave

octave:1> X = textread('test_textread.txt', '', 'headerlines', 6)
X =

   0.00000   5.69000   4.37000   0.54000
   1.00000   0.58000   0.59000   0.01000
   2.00000   0.43000   0.42000   3.00000
   0.93000   0.54000   0.30000   4.00000
   0.12000   0.99000   0.70000   0.00000


Obviously they don't behave in the same way.

If we take a look at the the file empty_spots.txt, it will be natural (in my opinion) to think of the the numbers as constituting a 5x4 matrix with one element missing (at 3,4). This view seems to be in accordance with Matlab's behaviour, which just fills in a 0 at that position.

Over to empty_lines.txt:

Matlab 7.14.0.739 (R2012a)

>> X = textread('empty_lines.txt', '', 'headerlines', 6)

X =

         0    5.6900    4.3700    0.5400
    1.0000    0.5800    0.5900    0.0100
    2.0000    0.4300    0.4200    0.3000
    3.0000    0.9300    0.5400    0.3000
    4.0000    0.1200    0.9900    0.7000


Octave

octave:1> X = textread('empty_lines.txt', '', 'headerlines', 6)
X =

   0.00000   5.69000   4.37000   0.54000
   0.00000   0.00000   1.00000   0.58000
   0.59000   0.01000   0.00000   2.00000
   0.43000   0.42000   0.30000   3.00000
   0.93000   0.54000   0.30000   0.00000
   4.00000   0.12000   0.99000   0.70000
   0.00000   0.00000   0.00000   0.00000
   0.00000   0.00000   0.00000   0.00000


Matlab, as we can see if we compare the output to the content in the input file, ignores all empty lines, whereas Octave for every empty line adds a 0 to the resulting matrix. If necessary it (Octave) also fills in more zeros to complete the last row in the matrix. (If one of the five empty lines at the end of empty_lines.txt is removed, Octave outputs a 7x4 matrix instead of an 8x4 matrix).

Stig <default>
Fri 29 Mar 2013 07:44:46 PM UTC, comment #16: 

Hopefully that major event was a positive one.

Anyway, does textread.m work correctly for you now? (You can get it directly from the repo, see
http://hg.savannah.gnu.org/hgweb/octave/file/a53f56e16627/scripts/io )

Philip Nienhuis <philipnienhuis>
Group Member
Fri 29 Mar 2013 02:16:33 PM UTC, comment #15: 

Hello. This is the OP. I just want to apologize for not replying in the thread. Some major events happened in my life, which took this off my mind for a while. Sorry about that.

Great effort from the dev, nevertheless.

Stig <default>
Fri 22 Mar 2013 04:50:47 PM UTC, comment #14: 

Fixed in this changeset:

http://hg.savannah.gnu.org/hgweb/octave/rev/0cbe330f39a2

This will be part of the next stable version of Octave.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 20 Mar 2013 07:47:50 PM UTC, comment #13: 

Ha Rik, no I haven't pushed yet because:

1. I haven't got feedback from the OP, nor from anyone else, about proper functioning of the modified functions

2. I have some more patches to current textread/textscan lying around (mostly tests). First those, than these patches

3. There's some (minor) interference from patch #7969

(4. And of course, lack of time. Busy week)

I plan to finish this report Friday.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 20 Mar 2013 05:50:15 PM UTC, comment #12: 

Philip, have you pushed your patches?  Can this report be closed now?

Rik <rik5>
Group administrator
Fri 15 Mar 2013 04:04:10 PM UTC, comment #11: 

Here are provisional patches for textread.m and textscan.m, providing the behavior of ML as far as I could deduce it from 2013a.

textscan.m behaves a bit differently from textread.m in that it doesn't return a rectangular array but just the right number of columns (cell arrays) in returned output cell array, and it returns NaN in empty fields.
The "EmptyValue" and "CollectOutput" parameters can be used to adapt this.

I'll wait a few days for your comments until I push these changes.


(file #27608, file #27609)

Philip Nienhuis <philipnienhuis>
Group Member
Sun 24 Feb 2013 10:42:27 AM UTC, comment #10: 

(Edited title to better reflect exact issue;
Changed Item group to Matlab compatibility)

I don't understand exactly what you mean, sorry.
Do you have some example Matlab commands + small data file outlining the problem? I have ML at hand so I can try both Octave and ML.

As to data files containing lines with varying number of data fields, please note that those cannot be read by Octave's current textread.m and textscan.m
A binary (compiled) textscan() might do that but it is still in the making (even primordial yet).

BTW, can you point us to Matlab documentation outlining this behavior of textread (and textscan)?
AFAICS we're still discussing undocumented (yet IMO useful) ML features.

FYI, last week I have patched textscan.m as well (it behaves a bit differently than textread.m with empty format strings) and fixed textread.m's tests (I've mixed up a few things in the version you tried). I've also patched another (yet unreported) bug in textread.m.

I'm awaiting your examples before I proceed any further.

Thanks

Philip Nienhuis <philipnienhuis>
Group Member
Sat 16 Feb 2013 08:41:29 PM UTC, comment #9: 

Two things I noticed:

MATLAB puts a 0 in "empty spots", but your solution tries to fit the elements into an array with the number of columns as the number of entries on the first line. (In MATLAB the number of columns corresponds to the number of entries on the line with most entries, it seems.)

Specifying 0 to 'headerlines' gives a warning: "warning: strread: unknown property 'headerlines'".

Stig <default>
Sat 16 Feb 2013 07:47:38 PM UTC, comment #8: 

Seems to be working. Great.

Stig <default>
Thu 14 Feb 2013 10:42:10 PM UTC, comment #7: 

Some observations: AFAICS in ML this only works if:
- the format string is empty
 AND
- you only specify one or no output argument (nargout <= 1)
 AND
- you only have numerical output (after headerline processing).

Attached is a patched textread.m that mimics ML's (AFAICS undocumented) behavior.
Tests added and the bug I mentioned in comment #1 fixed (this bug fix is already pushed, also for textscan.m).

Please try this textread.m (put it somewhere in your Octave search path, e.g., in your current working directory) and please report back if it works for you.

I'm also busy amending textscan.m similarly but there I've got to investigate possible interference with CollectOutput param.

Unfortunately I ran out of time to even prepare a changeset - hopefully next week I can do that & finish textscan.m

(file #27466)

Philip Nienhuis <philipnienhuis>
Group Member
Wed 13 Feb 2013 07:41:37 PM UTC, comment #6: 

Yeah you're right, Savannah (and Nabble) remove multiple whitespace, sorry for overlooking that.

Anyway I've already fixed a somewhat related bug in 3.6.4-rc2 (I'll push it later). Hopefully next week I'll fix the original issue.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 13 Feb 2013 06:33:40 PM UTC, comment #5: 

I see. The columns were aligned when I copied the output from the program. I thought the whitespace was removed by the software on this site because I didn't use the verbatim markup (which I should have done).

This is what it really looks like:


>> textread('deaths.txt', '', 'headerlines', 3)

ans =

         0    5.6900    4.3700
    1.0000    0.5800    0.5900
    2.0000    0.4300    0.4200
    3.0000    0.3100    0.2900
    4.0000    0.2300    0.2100
    5.0000    0.1800    0.1500
    6.0000    0.1500    0.1200
    7.0000    0.1500    0.1000
    8.0000    0.1500    0.1000
    9.0000    0.1700    0.0600
   10.0000    0.2000    0.0600
   11.0000    0.1900    0.0800
   12.0000    0.2000    0.0900
.
.
.


Just doing a test down here. If the columns below are aligned, I must have messed up somewhere else.

>> textread('deaths.txt', '', 'headerlines', 3)


ans =

         0    5.6900    4.3700
    1.0000    0.5800    0.5900
    2.0000    0.4300    0.4200
    3.0000    0.3100    0.2900
    4.0000    0.2300    0.2100
    5.0000    0.1800    0.1500
    6.0000    0.1500    0.1200
    7.0000    0.1500    0.1000
    8.0000    0.1500    0.1000
    9.0000    0.1700    0.0600
   10.0000    0.2000    0.0600
   11.0000    0.1900    0.0800
   12.0000    0.2000    0.0900

Stig <default>
Wed 13 Feb 2013 05:04:12 PM UTC, comment #4: 

That doesn't match the output you gave in your original post.
An echo of ML's output would have shown aligned columns then, not something like:

ans =

  1. 5.6900 4.3700

1.0000 0.5800 0.5900
:

(in ML r2013a I get neatly aligned data columns, see comment #2)

Or did you simply copy the file contents into your bug report?

Philip Nienhuis <philipnienhuis>
Group Member
Wed 13 Feb 2013 11:38:06 AM UTC, comment #3: 

MATLAB version is 7.14.0.739 (R2012a).

Excuse me if I misunderstand (I am new to MATLAB and Octave), but to me it seems that I get numerical output in both MATLAB and Octave.

From MATLAB:


>> X = textread('deaths.txt', '', 'headerlines', 3);
>> class(X)

ans =

double

>> size(X)

ans =

   100     3



Thanks for the effort.

Stig <default>
Tue 12 Feb 2013 11:20:42 PM UTC, comment #2: 

Upon closer inspection in your report I now see that you actually got text output rather than numerical output. Which IMO would be a ML bug as ML's default format is %f (numeric) rather than %s (string).

However, with ML r2013a prerelease I get:

>>  textread('deaths.txt', '', 'headerlines', 3)

ans =

         0    5.6900    4.3700
    1.0000    0.5800    0.5900
    2.0000    0.4300    0.4200
    3.0000    0.3100    0.2900
:

(i.e., a 100x3 double array).

What ML version do you have?

Anyway AFAICS this isn't documented ML behavior. Yet I think it would be easy (and IMO useful) to patch strread.m to get similar behavior.

FWIW, textscan.m behaves differently than Octave as well. I'll look into that too:

>> fid = fopen ('deaths.txt');
>> textscan (fid, '', 'headerlines', 3)
ans =
    [100x1 double]    [100x1 double]    [100x1 double]

while Octave does:

octave:2> fid = fopen ('deaths.txt');
octave:3> C = textscan (fid, '', 'headerlines', 3)
C =
{
  [1,1] =

       0.00000
       5.69000
       4.37000
       1.00000
       0.58000
       0.59000
       2.00000
       0.43000
       0.42000
       3.00000
       0.31000
       0.29000
:
<snip>
octave:4> size (C{1})
ans =
   300     1


Title adapted to include textscan.

Please have some patience.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 12 Feb 2013 10:44:53 PM UTC, comment #1: 

I'll have a look.
May take some time as I'm a bit busy now and away next week.

In 3.6.4-rc2 I get a crash. Hmmmm...

Thanks for reporting.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 12 Feb 2013 05:03:17 PM UTC, original submission:  

Hello.

(The file deaths.txt is attached.)

If i issue the following command in MATLAB, I get this result:

>> textread('deaths.txt', '', 'headerlines', 3)


ans =

         0    5.6900    4.3700
    1.0000    0.5800    0.5900
    2.0000    0.4300    0.4200
    3.0000    0.3100    0.2900
    4.0000    0.2300    0.2100
    5.0000    0.1800    0.1500
    6.0000    0.1500    0.1200
    7.0000    0.1500    0.1000
    8.0000    0.1500    0.1000
    9.0000    0.1700    0.0600
   10.0000    0.2000    0.0600
   11.0000    0.1900    0.0800
   12.0000    0.2000    0.0900
   13.0000    0.2600    0.1400
   14.0000    0.3000    0.1800
   15.0000    0.4400    0.2500
   16.0000    0.5900    0.2700
   17.0000    0.7500    0.2900
   18.0000    0.8600    0.3400
   19.0000    0.8600    0.3500
   20.0000    0.8300    0.3500
   21.0000    0.8200    0.3100
   22.0000    0.8100    0.2700
   23.0000    0.8300    0.2700
   24.0000    0.8600    0.2700
...

In Octave however, I get this result:

octave:1> textread('deaths.txt', '', 'headerlines', 3)
ans =

     0.00000
     5.69000
     4.37000
     1.00000
     0.58000
     0.59000
     2.00000
     0.43000
     0.42000
     3.00000
     0.31000
     0.29000
     4.00000
     0.23000
     0.21000
     5.00000
     0.18000
     0.15000
     6.00000
     0.15000
     0.12000
     7.00000
     0.15000
     0.10000
     8.00000
     0.15000
     0.10000
     9.00000
     0.17000
     0.06000
    10.00000
     0.20000
     0.06000
    11.00000
     0.19000
...

Is this a bug or a feature? Should they behave the same?

Thank you.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #27724:  empty_lines.txt added by default (323B - text/plain)
file #27725:  empty_spots.txt added by default (303B - text/plain)
file #27466:  textread.m added by philipnienhuis (9KiB - text/x-objcsrc)
file #27451:  deaths.txt added by None (4KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by default (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 16 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-03-30 default Attached File- Added empty_lines.txt, #27724
        Attached File- Added empty_spots.txt, #27725
    2013-03-30 rik5 Carbon-CopyRemoved 72865 -
    2013-03-22 philipnienhuis StatusPatch Submitted Fixed
        Open/ClosedOpen Closed
    2013-03-16 philipnienhuis StatusNeed Info Patch Submitted
    2013-03-15 philipnienhuis Attached File- Added textread_#38317.patch, #27608
        Attached File- Added textscan_#38317.patch, #27609
    2013-02-24 philipnienhuis Item GroupNone Matlab Compatibility
        Summarytextread &amp; textscan give different results in MATLAB and Octave textread & textscan: different results for empy format strings than MATLAB
    2013-02-14 philipnienhuis Attached File- Added textread.m, #27466
        StatusConfirmed Need Info
    2013-02-12 philipnienhuis SummaryTextread gives different results in MATLAB and Octave textread & textscan give different results in MATLAB and Octave
    2013-02-12 philipnienhuis StatusNone Confirmed
        Assigned toNone philipnienhuis
    2013-02-12 None Attached File- Added deaths.txt, #27451

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code