bugGNU Octave - Bugs: bug #30468, dlmread - performance problems

 
 

bug #30468: dlmread - performance problems

Submitted by:  Alexander Renz <veloci>
Submitted on:  Sat 17 Jul 2010 09:53:49 AM UTC  
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Other
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  3.2.4 Operating System:  Microsoft Windows

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Wed 15 Feb 2012 11:06:41 PM UTC, comment #4:

The code improvements have already been incorporated into Octave.  To see the increased performance just use any recent Octave version >= 3.4.0.

Rik <rik5>
Project Administrator
Wed 15 Feb 2012 10:36:47 PM UTC, comment #3:

That sounds awesome, thank you for the improvement !
But how can I use your file ? How can I use it in octave ?

Would you by chance know a page that would explain that ?
I'm a new octave user and I barely know how to use what's already there and not more....

thanks

Tripmania <tripmania>
Tue 20 Jul 2010 11:11:18 AM UTC, comment #2:

I have pushed a couple of improvements to dlmread that should make it usable for large data sizes:
http://hg.savannah.gnu.org/hgweb/octave/rev/3d5c6b84ddaf

Using Rik's script for input, I now see the following timings:

octave:1> tic; m = load ('matrix.csv'); toc
Elapsed time is 0.567143 seconds.
octave:2> tic; m = dlmread ('matrix.csv'); toc
Elapsed time is 0.29008 seconds.

thanks for reporting.

Jaroslav Hajek <highegg>
Mon 19 Jul 2010 06:24:14 AM UTC, comment #1:

Yes, dlmread is absolutely atrocious.  The file is written in C++, so one would expect it to be of relatively high performance, but it isn't.

I'm attaching a small Perl script which makes a test file which is a matrix of 100,000 rows X 5 columns.  This was useful as input for testing.

Incidentally, Octave's normal 'load' command can handle tab-separated data.  Until dlmread is fixed, you will have better luck with that.  It won't handle header rows, however, so you would need to delete those.

For example, I benchmarked Octave's 'load' command on the 100K test matrix.

tic; x = load("matrix.csv"); bm = toc
bm =  0.74381

Less than 1 second to load the data.  After that you could simply throw out the column you don't need.

x(:,1) = [];

(file #21005)

Rik <rik5>
Project Administrator
Sat 17 Jul 2010 09:53:49 AM UTC, original submission:

The dlmread function seems to have a performace lack, if used for analysing large data files (number of rows > 100000). The processing time increases exponetially, compared to the time needed for lesser data sets.
Comparison:
In my case i've used dlmread for data files with a fixed number of header rows followed by >300000 rows/ 5 columns separated by tabs. From these i've read only columns 2-5 starting after the header rows and continuing to the end of the file. For that i calculate a range vector in advance and allocate memory in a zero matrix (data = zeros(datarange)). The process is started with

data = dlmread(filename,'\t',r);

I have tried this approach in Octave and Matlab. In Matlab i get results after a couple of moments for the whole data set. In Octave the process is started and doesn't seem to end.
I suspected having a problem with dlmread, so i tried to change the parameters- separator, range vector. The result was, that dlmread of Octave works fast for smaller data sets.
Now i'm using a crutch for being able to read the big data sets: I process the data in smaller parts and combine it piece by piece in the preallocated data matrix.

(reference: http://octave.1599824.n4.nabble.com/reading-data-from-ascii-files-td2291167.html#a2291167)

Alexander Renz <veloci>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #21005:  mk_file.pl added by rik5 (160B - text/x-perl)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tripmania (Posted a comment)
  • -email is unavailable- added by highegg (Posted a comment)
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by veloci (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-07-20 highegg Open/ClosedOpen => Closed
    2010-07-19 rik5 Attached File- => Added mk_file.pl, #21005
        CategoryNone => Interpreter
        Item GroupNone => Other
        StatusNone => Confirmed

    Back to the top


    Powered by Savane 3.4