bugGNU Octave - Bugs: bug #38317, textread & textscan: different...

 
 

bug #38317: textread & textscan: different results for empy format strings than MATLAB

Submitted by:  None
Submitted on:  Tue 12 Feb 2013 05:03:17 PM UTC  
 
Category: InterpreterSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Matlab Compatibility
Status: FixedAssigned to: Philip Nienhuis <philipnienhuis>
Originator Name: Originator Email: -unavailable-
Open/Closed: ClosedRelease: dev
Operating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 30 Mar 2013 07:31:42 PM UTC, comment #19:

Thanks for the investigation.

As far as I'm concerned this particular issue won't be fixed anymore in the current strread/textread/textscan versions.

This behavior is similar to bug #38587 and is related to the way text files are parsed. Nothing to be done about it, except a binary (x)textscan.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sat 30 Mar 2013 03:06:18 PM UTC, comment #18:

“test_textread.txt” in the previous post should be replaced with “empty_spots.txt”.

Stig <default>
Sat 30 Mar 2013 03:02:38 PM UTC, comment #17:

I have yet find the Matlab behaviour we are looking at here documented anywhere, so I am just comparing the results from Octave to the corresponding results from Matlab.

(The files empty_spots.txt, empty_lines.txt have been attached.)

Matlab 7.14.0.739 (R2012a)

Octave

Obviously they don't behave in the same way.

If we take a look at the the file empty_spots.txt, it will be natural (in my opinion) to think of the the numbers as constituting a 5x4 matrix with one element missing (at 3,4). This view seems to be in accordance with Matlab's behaviour, which just fills in a 0 at that position.

Over to empty_lines.txt:

Matlab 7.14.0.739 (R2012a)

Octave

Matlab, as we can see if we compare the output to the content in the input file, ignores all empty lines, whereas Octave for every empty line adds a 0 to the resulting matrix. If necessary it (Octave) also fills in more zeros to complete the last row in the matrix. (If one of the five empty lines at the end of empty_lines.txt is removed, Octave outputs a 7x4 matrix instead of an 8x4 matrix).

Stig <default>
Fri 29 Mar 2013 07:44:46 PM UTC, comment #16:

Hopefully that major event was a positive one.

Anyway, does textread.m work correctly for you now? (You can get it directly from the repo, see
http://hg.savannah.gnu.org/hgweb/octave/file/a53f56e16627/scripts/io )

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Fri 29 Mar 2013 02:16:33 PM UTC, comment #15:

Hello. This is the OP. I just want to apologize for not replying in the thread. Some major events happened in my life, which took this off my mind for a while. Sorry about that.

Great effort from the dev, nevertheless.

Stig <default>
Fri 22 Mar 2013 04:50:47 PM UTC, comment #14:

Fixed in this changeset:

http://hg.savannah.gnu.org/hgweb/octave/rev/0cbe330f39a2

This will be part of the next stable version of Octave.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Wed 20 Mar 2013 07:47:50 PM UTC, comment #13:

Ha Rik, no I haven't pushed yet because:

1. I haven't got feedback from the OP, nor from anyone else, about proper functioning of the modified functions

2. I have some more patches to current textread/textscan lying around (mostly tests). First those, than these patches

3. There's some (minor) interference from patch #7969

(4. And of course, lack of time. Busy week)

I plan to finish this report Friday.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Wed 20 Mar 2013 05:50:15 PM UTC, comment #12:

Philip, have you pushed your patches? Can this report be closed now?

Rik <rik5>
Project Administrator
Fri 15 Mar 2013 04:04:10 PM UTC, comment #11:

Here are provisional patches for textread.m and textscan.m, providing the behavior of ML as far as I could deduce it from 2013a.

textscan.m behaves a bit differently from textread.m in that it doesn't return a rectangular array but just the right number of columns (cell arrays) in returned output cell array, and it returns NaN in empty fields.
The "EmptyValue" and "CollectOutput" parameters can be used to adapt this.

I'll wait a few days for your comments until I push these changes.

(file #27608, file #27609)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sun 24 Feb 2013 10:42:27 AM UTC, comment #10:

(Edited title to better reflect exact issue;
Changed Item group to Matlab compatibility)

I don't understand exactly what you mean, sorry.
Do you have some example Matlab commands + small data file outlining the problem? I have ML at hand so I can try both Octave and ML.

As to data files containing lines with varying number of data fields, please note that those cannot be read by Octave's current textread.m and textscan.m
A binary (compiled) textscan() might do that but it is still in the making (even primordial yet).

BTW, can you point us to Matlab documentation outlining this behavior of textread (and textscan)?
AFAICS we're still discussing undocumented (yet IMO useful) ML features.

FYI, last week I have patched textscan.m as well (it behaves a bit differently than textread.m with empty format strings) and fixed textread.m's tests (I've mixed up a few things in the version you tried). I've also patched another (yet unreported) bug in textread.m.

I'm awaiting your examples before I proceed any further.

Thanks

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sat 16 Feb 2013 08:41:29 PM UTC, comment #9:

Two things I noticed:

MATLAB puts a 0 in "empty spots", but your solution tries to fit the elements into an array with the number of columns as the number of entries on the first line. (In MATLAB the number of columns corresponds to the number of entries on the line with most entries, it seems.)

Specifying 0 to 'headerlines' gives a warning: "warning: strread: unknown property 'headerlines'".

Stig <default>
Sat 16 Feb 2013 07:47:38 PM UTC, comment #8:

Seems to be working. Great.

Stig <default>
Thu 14 Feb 2013 10:42:10 PM UTC, comment #7:

Some observations: AFAICS in ML this only works if:
- the format string is empty
AND
- you only specify one or no output argument (nargout <= 1)
AND
- you only have numerical output (after headerline processing).

Attached is a patched textread.m that mimics ML's (AFAICS undocumented) behavior.
Tests added and the bug I mentioned in comment #1 fixed (this bug fix is already pushed, also for textscan.m).

Please try this textread.m (put it somewhere in your Octave search path, e.g., in your current working directory) and please report back if it works for you.

I'm also busy amending textscan.m similarly but there I've got to investigate possible interference with CollectOutput param.

Unfortunately I ran out of time to even prepare a changeset - hopefully next week I can do that & finish textscan.m

(file #27466)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Wed 13 Feb 2013 07:41:37 PM UTC, comment #6:

Yeah you're right, Savannah (and Nabble) remove multiple whitespace, sorry for overlooking that.

Anyway I've already fixed a somewhat related bug in 3.6.4-rc2 (I'll push it later). Hopefully next week I'll fix the original issue.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Wed 13 Feb 2013 06:33:40 PM UTC, comment #5:

I see. The columns were aligned when I copied the output from the program. I thought the whitespace was removed by the software on this site because I didn't use the verbatim markup (which I should have done).

This is what it really looks like:

Just doing a test down here. If the columns below are aligned, I must have messed up somewhere else.

>> textread('deaths.txt', '', 'headerlines', 3)


ans =

0 5.6900 4.3700
1.0000 0.5800 0.5900
2.0000 0.4300 0.4200
3.0000 0.3100 0.2900
4.0000 0.2300 0.2100
5.0000 0.1800 0.1500
6.0000 0.1500 0.1200
7.0000 0.1500 0.1000
8.0000 0.1500 0.1000
9.0000 0.1700 0.0600
10.0000 0.2000 0.0600
11.0000 0.1900 0.0800
12.0000 0.2000 0.0900

Stig <default>
Wed 13 Feb 2013 05:04:12 PM UTC, comment #4:

That doesn't match the output you gave in your original post.
An echo of ML's output would have shown aligned columns then, not something like:

ans =

  1. 5.6900 4.3700

1.0000 0.5800 0.5900
:

(in ML r2013a I get neatly aligned data columns, see comment #2)

Or did you simply copy the file contents into your bug report?

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Wed 13 Feb 2013 11:38:06 AM UTC, comment #3:

MATLAB version is 7.14.0.739 (R2012a).

Excuse me if I misunderstand (I am new to MATLAB and Octave), but to me it seems that I get numerical output in both MATLAB and Octave.

From MATLAB:

Thanks for the effort.

Stig <default>
Tue 12 Feb 2013 11:20:42 PM UTC, comment #2:

Upon closer inspection in your report I now see that you actually got text output rather than numerical output. Which IMO would be a ML bug as ML's default format is %f (numeric) rather than %s (string).

However, with ML r2013a prerelease I get:
(i.e., a 100x3 double array).

What ML version do you have?

Anyway AFAICS this isn't documented ML behavior. Yet I think it would be easy (and IMO useful) to patch strread.m to get similar behavior.

FWIW, textscan.m behaves differently than Octave as well. I'll look into that too:
while Octave does:

Title adapted to include textscan.

Please have some patience.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Tue 12 Feb 2013 10:44:53 PM UTC, comment #1:

I'll have a look.
May take some time as I'm a bit busy now and away next week.

In 3.6.4-rc2 I get a crash. Hmmmm...

Thanks for reporting.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Tue 12 Feb 2013 05:03:17 PM UTC, original submission:

Hello.

(The file deaths.txt is attached.)

If i issue the following command in MATLAB, I get this result:

>> textread('deaths.txt', '', 'headerlines', 3)


ans =

0 5.6900 4.3700
1.0000 0.5800 0.5900
2.0000 0.4300 0.4200
3.0000 0.3100 0.2900
4.0000 0.2300 0.2100
5.0000 0.1800 0.1500
6.0000 0.1500 0.1200
7.0000 0.1500 0.1000
8.0000 0.1500 0.1000
9.0000 0.1700 0.0600
10.0000 0.2000 0.0600
11.0000 0.1900 0.0800
12.0000 0.2000 0.0900
13.0000 0.2600 0.1400
14.0000 0.3000 0.1800
15.0000 0.4400 0.2500
16.0000 0.5900 0.2700
17.0000 0.7500 0.2900
18.0000 0.8600 0.3400
19.0000 0.8600 0.3500
20.0000 0.8300 0.3500
21.0000 0.8200 0.3100
22.0000 0.8100 0.2700
23.0000 0.8300 0.2700
24.0000 0.8600 0.2700
...

In Octave however, I get this result:

octave:1> textread('deaths.txt', '', 'headerlines', 3)
ans =

0.00000
5.69000
4.37000
1.00000
0.58000
0.59000
2.00000
0.43000
0.42000
3.00000
0.31000
0.29000
4.00000
0.23000
0.21000
5.00000
0.18000
0.15000
6.00000
0.15000
0.12000
7.00000
0.15000
0.10000
8.00000
0.15000
0.10000
9.00000
0.17000
0.06000
10.00000
0.20000
0.06000
11.00000
0.19000
...

Is this a bug or a feature? Should they behave the same?

Thank you.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #27724:  empty_lines.txt added by default (323B - text/plain)
file #27725:  empty_spots.txt added by default (303B - text/plain)
file #27466:  textread.m added by philipnienhuis (9kB - text/x-objcsrc)
file #27451:  deaths.txt added by None (4kB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by default (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 16 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 30 Mar 2013 03:03:31 PM UTCdefaultAttached File-=>Added empty_lines.txt, #27724
      Attached File-=>Added empty_spots.txt, #27725
    Sat 30 Mar 2013 02:54:43 PM UTCrik5Carbon-CopyRemoved 72865=>-
    Fri 22 Mar 2013 04:50:47 PM UTCphilipnienhuisStatusPatch Submitted=>Fixed
      Open/ClosedOpen=>Closed
    Sat 16 Mar 2013 11:54:54 AM UTCphilipnienhuisStatusNeed Info=>Patch Submitted
    Fri 15 Mar 2013 04:04:10 PM UTCphilipnienhuisAttached File-=>Added textread_#38317.patch, #27608
      Attached File-=>Added textscan_#38317.patch, #27609
    Sun 24 Feb 2013 10:42:27 AM UTCphilipnienhuisItem GroupNone=>Matlab Compatibility
      Summarytextread & textscan give different results in MATLAB and Octave=>textread & textscan: different results for empy format strings than MATLAB
    Thu 14 Feb 2013 10:42:10 PM UTCphilipnienhuisAttached File-=>Added textread.m, #27466
      StatusConfirmed=>Need Info
    Tue 12 Feb 2013 11:20:42 PM UTCphilipnienhuisSummaryTextread gives different results in MATLAB and Octave=>textread & textscan give different results in MATLAB and Octave
    Tue 12 Feb 2013 10:44:53 PM UTCphilipnienhuisStatusNone=>Confirmed
      Assigned toNone=>philipnienhuis
    Tue 12 Feb 2013 05:03:17 PM UTCNoneAttached File-=>Added deaths.txt, #27451

    Back to the top


    Powered by Savane 3.1-cleanup