bugGNU Octave - Bugs: bug #58618, [octave forge] (io) csv2cell does...

 
 

bug #58618: [octave forge] (io) csv2cell does not read all rows

Submitter:  Andreas Bertsatos <pr0m1th3as>
Submitted:  Fri 19 Jun 2020 04:12:17 AM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * other Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 27 Jul 2020 03:34:42 PM UTC, comment #5: 

Max line length set to 32768 and fix pushed here:
http://hg.code.sf.net/p/octave/io/rev/3648f48d23db

Closing report.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 19 Jun 2020 07:32:30 PM UTC, comment #4: 

updated the source file with

#define MAXSTRINGLENGTH 16384

and it works like a charm, thanks. I am currently working with HOG descriptors, which tend to result to rather large vectors. Although, my case file was just little bit above the previous limit, I consider a 4-fold increase to be the minimum by today's standards. I run a simple tic toc test by repeatedly (100 times) reading a reasonably sized csv (file size = 337.1KB; cell size = 151x290) that works well under both versions of csv2cell and the performance difference was negligible

   2.74406 seconds @MSL 16384 vs 2.61971 sec @MSL 4096

Additional testing with MAXSTRINGLENGTH 32768 returned 2.97757 seconds.
So, either 16384 or 32768 will do I think, although the latter seems like an overkill to me. I hope somebody else, who regularly uses large csv files, might share an opinion on this matter.

Recompiling csv2cell.cc as Philip suggested, didn't work because it produced fatal error due to missing "config.h" dependency. So the workaround was to make the change inside the package source file and re-install it locally.

Many thanks :)

Andreas Bertsatos <pr0m1th3as>
Fri 19 Jun 2020 09:10:55 AM UTC, comment #3: 

That should read:

"It's easy to recompile csv2cell yourself after adapting this value; just a:

mkoctfile csv2cell.cc

will do."

(lousy foldable keyboard here, but my family hates the sound of typing on normal keyboards so I have to make do with it :-) )

Philip Nienhuis <philipnienhuis>
Group Member
Fri 19 Jun 2020 09:08:19 AM UTC, comment #2: 

csv2cell's read buffer indeed is set to 4096 chars, see L. 193:

#define MAXSTRINGLENGTH 4096

It's easy to recompile csv2cell yourself after adapting this value; just a:

mkoctfile csv2cll.cc will do.

Beware that you first have to unload the io package and do a "clear -f" to unload the resident csv2cell incarnation.

Noting that this value was set there years ago when processing .csv data files with extremely long lines wasn't done very often, is there any reasonable value you'd like to suggest for this value now?

Philip Nienhuis <philipnienhuis>
Group Member
Fri 19 Jun 2020 07:59:06 AM UTC, comment #1: 

Line 56 is the first line with more than 4096 characters (it has 4149), line 88 is the second with 4229, apart from them all are below 4096. Presumably this is a problem that some read buffer is allocated only to 4096=2^12 bytes.

Michael Leitner <mleitner>
Fri 19 Jun 2020 04:12:17 AM UTC, original submission:  

Hi,

unexpectedly, I realized that csv2cell partially imports this particular csv file (attached), which was created by cell2csv function in the first place. It only reads up to row 56 for no apparent reason. If I delete any row after row 56, the result is the same. If I delete row 56, then it imports up to row 88.
Deleting any random column does not affect this behavior.

Apparently, csvread function reads the entire file (150x642).
The csv appears intact in libreoffice calc and in text editor.

I ve been using csv2cell/cell2csv for a long time and handled more than 10k similar files with no issue at all. Unfortunately, I can't trace the reason for this unpredictable behaviour.
Presumably, there is a bug in csv2cell which is triggered by this particular file.

Andreas Bertsatos <pr0m1th3as>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49309:  test.csv added by pr0m1th3as (501KiB - text/csv)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mleitner (Posted a comment)
  • -email is unavailable- added by pr0m1th3as (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-07-27 philipnienhuis StatusNeed Info Fixed
        Open/ClosedOpen Closed
    2020-06-19 philipnienhuis Release5.1.0 other
    2020-06-19 philipnienhuis StatusNone Need Info
        Operating SystemGNU/Linux Any
        Summarycsv2cell does not read all rows [octave forge] (io) csv2cell does not read all rows
    2020-06-19 pr0m1th3as Attached File- Added test.csv, #49309

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code