bugGNU Octave - Bugs: bug #41630, Importdata function fails with ...

 
 

bug #41630: Importdata function fails with Unable to determine delimiter error

Submitted by:  None
Submitted on:  Tue 18 Feb 2014 07:22:12 AM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Matlab Compatibility
Status: FixedAssigned to: Philip Nienhuis <philipnienhuis>
Originator Name: Ravikiran ChiruOriginator Email: -unavailable-
Open/Closed: ClosedRelease: 3.8.0
Operating System: Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 26 Apr 2014 03:26:10 PM UTC, comment #8:

I applied your patch here (http://hg.savannah.gnu.org/hgweb/octave/rev/99d1ef340de4).

The stable branch is supposed to be just that, quite stable. The three big reasons to put a patch on stable are:

1) Wildly incorrect result rendering a function unusable, for example 2*2 = 5.

2) Correct input generates a segfault, that should never happen.

3) Regression, code that used to work no longer does.

Matlab compatibility isn't a criteria we consider. Generally, Matlab is a moving target so we do our best with each release and then put changes on the development branch. I know that jwe has been trying to get Octave on a more frequent release schedule; Not as bad as Firefox or Chrome which are every 6 weeks, but more frequent. The introduction of the GUI is throwing things off though. I think we should push to have the 4.2 release no more than 6 months after the GUI release so that the significant number of changes that are accumulating don't wait for a full year to year and a half.

Rik <rik5>
Project Administrator
Sat 26 Apr 2014 01:52:35 PM UTC, comment #7:

OK here's a patch that limits the search for EOL to the first 4096 bytes of the file.

On another note, my previous patch was pushed on default, which may take a looong time to get released.
As this bug was a ML incompatibility, could there be a motive for pushing it also to stable or gui-release?

I noted that the current patched importdata version (in default) doesn't work on octave < 4.1.0+ as numfields (called in the very last stanza of importdata.m's main func) isn't implemented there.

(file #31256)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sun 20 Apr 2014 08:56:36 PM UTC, comment #6:

The benchmarking I did showed that find was 10X faster than index. The function index is an m-file and if you look at the implementation it uses strfind; In either case, find or index, the entire buffer is being searched. I think it would be a good idea to check just the first 4K for an EOL delimiter.

Rik <rik5>
Project Administrator
Sun 20 Apr 2014 08:38:12 PM UTC, comment #5:

Thanks for reviewing & pushing.

Is find() really faster, also on big files? I can imagine that index() is ploughing through the entire file if no matching EOL is found.

Originally I had index() search only in the first
min (filelength, 4096)
bytes to limit excessive search time; I dropped it but I cannot recall anymore why :-( (hmmm aging memory I suppose)
If you want I can make a cs to reintroduce this.

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Sat 19 Apr 2014 04:28:46 AM UTC, comment #4:

I modified the patch to use find() rather than index() as it was faster. I checked it in here (http://hg.savannah.gnu.org/hgweb/octave/rev/271873b2f44f). Closing report.

Rik <rik5>
Project Administrator
Wed 02 Apr 2014 06:56:55 PM UTC, comment #3:

The attached changeset fixes this for me. I get the same results as with ML r2012a.

Note that there are a few other ML incompatibilities I'm working on, but these depend on a number of dlmread issues (see bugs #42022, #42023 and #42025).

I've added a few tests as well.

(file #31107)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Wed 26 Mar 2014 01:40:40 PM UTC, comment #2:

Confirmed, I hit a similar issue at work today.

BTW, the contents of the text file you attached differ from your bug report. But I don't think that matters.

Trying your example in Matlab neatly gives the output I suppose you expected (note I copied the file contents from your report, not your file):

Right, looks good. Now trying with two columns of text and a delimiter:

File contents:
Landmark-1 HHH
Landmark-2 HHH
Landmark-3 HHH
Landmark-4 HHH
Landmark-5 HHH

and moreover with a file containing:

1223_889&
123745==2515@3

ML gives:

...so I conclude that if Matlab's importdata() can't distill any numeric data from the file, it simply assumes it consists of all text and it'll return the lines it originally interpreted to be headerlines as file contents.
Little wonder as I think importdata.m is primarily meant to return numeric data, rather than text strings. You'd need to carefully read between the lines in the ML docs, but it's there.

You can see that if ML doesn't detect any numeric data, it simply ignores the explicitly specified delimiters in the file and returns a 1-column cellstr array.

I'll prepare a changeset later on (hopefully this week).

Another changeset is required for newly supported spreadsheet formats (a.o., gnumeric), some of which are now supported w/o external dependencies.

In the ML docs there's mention of another importdata output format, having headers above the data columns. I don't need it now, but if it's not too involved I'll have a go at that one as well.

--------------
OS set to "any" (I hit this on Windows);

Item Group => ML compatibility (as I think Octave isn't in error per se; it's just IMO inconsistent ML behavior we'd need to mimic)

Philip Nienhuis <philipnienhuis>
Project MemberIn charge of this item.
Tue 18 Feb 2014 07:49:30 AM UTC, comment #1:

One more detail:

When I tried the older version of importdata.m taken from http://sourceforge.net/p/octave/feature-requests/41/

I used tab "\t" as the delimiter.

Ravikiran Chiru <mailravi3390>
Tue 18 Feb 2014 07:22:12 AM UTC, original submission:

I am using octave 3.8.0-6 OS X mavericks dmg downloaded from

http://sourceforge.net/projects/octave/files/Octave%20MacOSX%20Binary/2013-12-30%20binary%20installer%20of%20Octave%203.8.0%20for%20OSX%2010.9.1%20%28beta%29/

I tried to import a plain text file (attached landmarks_list.txt) which looks like:

Landmark-1
Landmark-2
Landmark-3
Landmark-4
Landmark-5

i.e it has no headers and a single column.

Loading this file using any of the following

A=importdata("landmarks_list.txt")
A=importdata("landmarks_list.txt","t")
A=importdata("landmarks_list.txt","t",0)
A=importdata("landmarks_list.txt"," ")
A=importdata("landmarks_list.txt"," ",0)

all fail with:

error: importdata: Unable to determine delimiter
error: called from:
error: /usr/local/octave/3.8.0/share/octave/3.8.0/m/io/importdata.m at line 214, column 5
error: /usr/local/octave/3.8.0/share/octave/3.8.0/m/io/importdata.m at line 132, column 38

I then tried an older version of importdata.m taken from http://sourceforge.net/p/octave/feature-requests/41/
and it worked like a charm.
There seems to be some issue with the delimiter detection.

NOTE: There is not path/file not found issue

NOTE: I have tried this with multicolumn files with a valid "t" delimiter and this fails too with the latest importdata and it passes with the older version.

G'day
Chiru

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #31107:  importdata_41630.cs added by philipnienhuis (2KiB - text/x-csharp)
file #30593:  landmark_list.txt added by None (78B - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by mailravi3390 (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 10 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 26 Apr 2014 01:52:35 PM UTCphilipnienhuisAttached File-=>Added importdata_41630_#2.cs, #31256
    Sat 19 Apr 2014 04:28:46 AM UTCrik5StatusPatch Submitted=>Fixed
      Open/ClosedOpen=>Closed
    Wed 02 Apr 2014 06:56:55 PM UTCphilipnienhuisAttached File-=>Added importdata_41630.cs, #31107
      StatusIn Progress=>Patch Submitted
    Wed 26 Mar 2014 01:40:40 PM UTCphilipnienhuisItem GroupIncorrect Result=>Matlab Compatibility
      StatusNone=>In Progress
      Assigned toNone=>philipnienhuis
      Operating SystemMac OS=>Any
    Tue 18 Feb 2014 07:22:12 AM UTCNoneAttached File-=>Added landmark_list.txt, #30593

    Back to the top


    Powered by Savane 3.1-cleanup1