patchGNU Octave - Patches: patch #8593, Faster version of dlmread(),...

 
 

patch #8593: Faster version of dlmread(), supporting escaped delimiters

Submitter:  None
Submitted:  Mon 12 Jan 2015 11:19:56 PM UTC
   
 
Category:  None Priority:  5 - Normal
Status:  Wont Do Privacy:  Public
Assigned to:  None Originator Email:  -email is unavailable-
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 14 Jan 2015 09:04:11 PM UTC, comment #5: 

Yes, I've had several reports that csv2cell (BTW: not my code) is blazingly fast.

The fact that read_dlm_esc() can handle variable-length lines (in terms of fields) is a good point in favor. csv2cell blindly assumes equal line length, so doesn't need to check, that may explain some speed advantage.

I think that as soon as .csv files get too complex, it is better to invoke spreadsheet programs (by functions from the io package) as those are built to handle "soup". Just IMO.

Good luck!

Philip Nienhuis <philipnienhuis>
Group Member
Tue 13 Jan 2015 10:46:44 PM UTC, comment #4: 

Thanks for the heads up Philip.

I had to spend some time getting octave running again after a new xilinx install hosed my octave install (fragile library dependancies), so I could run a quick comparison test.

So just FYI, my code runs significantly slower than csv2cell(). On a test CSV file I found that csv2cell() parsed in 0.016 seconds and read_dlm_esc() parsed in 0.292 seconds. However, read_dlm_read() will handle lines with a variable number of fields, which I required for the project I was working on. So, the good news is that it wasn't completely wasted effort.

I'll keep my github repo active, just incase anyone finds it useful.

Also a heads up to anyone that uses the read_dlm_esc() code. I took a quick look at it again, and the starting column has not been implemented yet. I have a place holder parameter, but no functionality for it.

As for maybe adding to csv2cell(), I'll have to see how my schedule works out in the next year or so.

Thanks,
Russ

Russell Brinkmann <rbb>
Tue 13 Jan 2015 10:24:39 PM UTC, comment #3: 

I was afraid of that :-(

But I really do appreciate your hard work. Your code didn't look bad at all (but I'm no c++ coder).

The 4092 row size limit has been discussed in the help ML a while back; for me it's just a parameter, could be 32 Kb as well, it was just there when csv2cell/cell2csv were moved from the miscellaneous package to io some years back.


If you still have appetite to help develop Octave, there are some close alternatives you might be interested in:

- As regards the column selector (and optionally row selector), that would be a great addition; could you implement that for csv2cell?

- On the maintainers ML at irregular times the discussion about a binary textscan() pops up (it's now an m-file and the work horse strread,m became so complex that it is almost unmaintanable - I know because I rewrote large parts of it).
I figured csv2cell could be a start for a binary textscan. 2 or 3 years ago the lead dev supplied a _xtextscan_.cc skeleton that could already do some input parsing, the actual read code was the big missing piece.


Once again, thanks for your efforts.
Closing report, but you can still post here if you like.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 13 Jan 2015 10:09:43 PM UTC, comment #2: 

Well, crap. Looks like I duplicated a bunch of effort.

I haven't benchmarked my code vs csv2cell(), but I doubt it will be a significant speed improvement. I haven't thoroughly tested it yet, but my code does have a parameter for which column to start reading, and I think I support more than 4096 rows.

However, these small (and relatively un-tested) additions are probably not worth maintaining a separate code base.

Thanks for pointing out the duplication.

Please disregard the patch submission.

Thanks,
Russ

Russell Brinkmann <rbb>
Tue 13 Jan 2015 07:22:10 PM UTC, comment #1: 

(io pkg maintainer here)  Thanks for the contribution, Russ.

Just a perhaps silly question:
What is the functional difference with csv2cell (also in the io package)?
AFAICS csv2cell can do the very same, i.e., handle various delimiters (specified on the command line), process input fields between double quotes (or other text protection characters as specified on the command line), read mixed csv files (numeric data and text), and skip headerlines.
Just type "help csv2cell"

Philip Nienhuis <philipnienhuis>
Group Member
Mon 12 Jan 2015 11:19:56 PM UTC, original submission:  

Hi,

I've created read_dlm_esc.cc which will compile into a .oct file, and read character delimited files. I've created a github repository for it:

https://github.com/rbb/read_dlm_esc

I've also attached the .cc file.

Hopefully this will make a useful addition to the io package:

http://sourceforge.net/p/octave/io/ci/default/tree/

Thanks,
Russ

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #32813:  read_dlm_esc.cc added by None (4KiB - text/x-c++src)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rbb (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-01-13 philipnienhuis StatusNone Wont Do
        Open/ClosedOpen Closed
    2015-01-12 None Attached File- Added read_dlm_esc.cc, #32813

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code