bugGNU Octave - Bugs: bug #49630, importdata very slow when read...

 
 

bug #49630: importdata very slow when read large files of text containing data

Submitted by:  None
Submitted on:  Thu 17 Nov 2016 08:50:40 AM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Performance
Status: FixedAssigned to: None
Originator Name: Dinesh RajOriginator Email: -unavailable-
Open/Closed: ClosedRelease: dev
Operating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 19 Nov 2016 05:25:08 AM UTC, comment #2:

I made a change here (http://hg.savannah.gnu.org/hgweb/octave/rev/1c73344fd348) so that only the first 25 lines are read. If we haven't found any numeric data by that time then it is probably a text file and should just be imported as one.

Marking as fixed and closing report.

Rik <rik5>
Project Administrator
Thu 17 Nov 2016 12:25:26 PM UTC, comment #1:

The command works fine in the development version. But I agree that importdata.m should work a bit more efficient.

A little debug session shows that in L. 168 and up it doesn't read just "a first few rows" but rather reads all rows. I waited 5 minutes (no response, Octave busy reading) and then had a lunch break. After that Octave had finished reading 217952 labels.
Then, in L.208 & up it again reads & parses the entire file.
I haven't tried beyond L.235 and up, as I noted that the input file is to be read another time (w. dlmread) & parsed again.

FYI, it is much more efficient to read these files using textscan

(Title, Item group & release adapted)

Philip Nienhuis <philipnienhuis>
Project Member
Thu 17 Nov 2016 08:50:40 AM UTC, original submission:

octave's importdata function could not read a large file containing labels for pixels of an image. The text file contains
2,17,952 labels. Matlab can read the file in a few seconds and variable containing the data appears in the workspace.

Octave kept me waiting for half an hour and never read the file.
I'm using octave cli (4.0.0) on windows 7. I have observed the same behaviour in ubuntu 16.04.

I have attached the file I was trying to read.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #39000:  imageLabels.txt added by None (2MiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 19 Nov 2016 05:25:08 AM UTCrik5StatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
    Thu 17 Nov 2016 12:25:26 PM UTCphilipnienhuisItem GroupMatlab Compatibility=>Performance
      StatusNone=>Confirmed
      Release4.0.0=>dev
      Summaryimportdata cannot read large files of text containing data, whereas matlab does=>importdata very slow when read large files of text containing data
    Thu 17 Nov 2016 08:50:40 AM UTCNoneAttached File-=>Added imageLabels.txt, #39000

    Back to the top


    Powered by Savane 3.1-cleanup1