bugGNU Octave - Bugs: bug #30468, dlmread - performance problems

 
 

bug #30468: dlmread - performance problems

Submitter:  Alexander Renz <veloci>
Submitted:  Sat 17 Jul 2010 09:53:49 AM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Other
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 3.2.4 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 15 Feb 2012 11:06:41 PM UTC, comment #4: 

The code improvements have already been incorporated into Octave.  To see the increased performance just use any recent Octave version >= 3.4.0.

Rik <rik5>
Group administrator
Wed 15 Feb 2012 10:36:47 PM UTC, comment #3: 

That sounds awesome, thank you for the improvement !
But how can I use your file ? How can I use it in octave ?

Would you by chance know a page that would explain that ?
I'm a new octave user and I barely know how to use what's already there and not more....


thanks

Tripmania <tripmania>
Tue 20 Jul 2010 11:11:18 AM UTC, comment #2: 

I have pushed a couple of improvements to dlmread that should make it usable for large data sizes:
http://hg.savannah.gnu.org/hgweb/octave/rev/3d5c6b84ddaf

Using Rik's script for input, I now see the following timings:

octave:1> tic; m = load ('matrix.csv'); toc
Elapsed time is 0.567143 seconds.
octave:2> tic; m = dlmread ('matrix.csv'); toc
Elapsed time is 0.29008 seconds.

thanks for reporting.

Jaroslav Hajek <highegg>
Mon 19 Jul 2010 06:24:14 AM UTC, comment #1: 

Yes, dlmread is absolutely atrocious.  The file is written in C++, so one would expect it to be of relatively high performance, but it isn't.

I'm attaching a small Perl script which makes a test file which is a matrix of 100,000 rows X 5 columns.  This was useful as input for testing.

Incidentally, Octave's normal 'load' command can handle tab-separated data.  Until dlmread is fixed, you will have better luck with that.  It won't handle header rows, however, so you would need to delete those.

For example, I benchmarked Octave's 'load' command on the 100K test matrix.

tic; x = load("matrix.csv"); bm = toc
bm =  0.74381

Less than 1 second to load the data.  After that you could simply throw out the column you don't need.

x(:,1) = [];


(file #21005)

Rik <rik5>
Group administrator
Sat 17 Jul 2010 09:53:49 AM UTC, original submission:  

The dlmread function seems to have a performace lack, if used for analysing large data files (number of rows > 100000). The processing time increases exponetially, compared to the time needed for lesser data sets.
Comparison:
In my case i've used dlmread for data files with a fixed number of header rows followed by >300000 rows/ 5 columns separated by tabs. From these i've read only columns 2-5 starting after the header rows and continuing to the end of the file. For that i calculate a range vector in advance and allocate memory in a zero matrix (data = zeros(datarange)). The process is started with

data = dlmread(filename,'\t',r);

I have tried this approach in Octave and Matlab. In Matlab i get results after a couple of moments for the whole data set. In Octave the process is started and doesn't seem to end.
I suspected having a problem with dlmread, so i tried to change the parameters- separator, range vector. The result was, that dlmread of Octave works fast for smaller data sets.
Now i'm using a crutch for being able to read the big data sets: I process the data in smaller parts and combine it piece by piece in the preallocated data matrix.

(reference: http://octave.1599824.n4.nabble.com/reading-data-from-ascii-files-td2291167.html#a2291167)

Alexander Renz <veloci>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #21005:  mk_file.pl added by rik5 (160B - text/x-perl)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tripmania (Posted a comment)
  • -email is unavailable- added by highegg (Posted a comment)
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by veloci (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-07-20 highegg Open/ClosedOpen Closed
    2010-07-19 rik5 Attached File- Added mk_file.pl, #21005
        CategoryNone Interpreter
        Item GroupNone Other
        StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code