bugGNU Octave - Bugs: bug #63655, [octave forge] (io) csv2cell line...

 
 

bug #63655: [octave forge] (io) csv2cell line length limit

Submitter:  Duane Kaufman <duanekaufman>
Submitted:  Fri 13 Jan 2023 03:23:46 PM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Wont Fix Assigned to:  None
Originator Name:  Duane Kaufman Open/Closed:  * Closed
Release:  * other Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 13 Jan 2023 10:05:03 PM UTC, comment #3: 

It's not exactly a bug, no. Anyway I won't change csv2cell's max line length.

csv2cell's max line length is hard-coded to 32768 characters, so that is already halfway your 72 K line length.
The reason for (this admittedly arbitrary) 32 K is to avoid excessive RAM usage during processing of otherwise simple .csv files with just a few data per line.
It's easy to change that limit and recompile csv2cell.cc using mkoctfile (don't forget to first do a "clear -f" or "clear csv2cell" to wipe csv2cell from Octave's function table in memory).
You can easily extract csv2cell.cc from a downloaded io package.

Reading your issue I wonder if it isn't better to simply use textscan or s/th along the lines of:


fid = fopen (<filename.cc>, "r");
fgetline (fid); # Get rid of / read beyond first line.
                # Maybe repeat to skip other header lines
data = fscanf (fid, <parameters>);
:
fclose (fid);


(assuming fscanf can skip commas or other delimiters, I never tried)

Adapting title
Updating fields (forge package, not core Octave)
Closing report

Philip Nienhuis <philipnienhuis>
Group Member
Fri 13 Jan 2023 04:41:02 PM UTC, comment #2: 

Sorry for the noise. I will post in the Octave Discourse Group.

Duane Kaufman <duanekaufman>
Fri 13 Jan 2023 03:33:00 PM UTC, comment #1: 

This is not a bug. Please use https://octave.discourse.group/
for support questions.

I think the limit comes from (old?) Excel specs.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 13 Jan 2023 03:23:46 PM UTC, original submission:  

Hello,
I'm currently trying to use Octave to analyze data taken with a photoacoustic research device.
The device writes out data into a csv file, which can be read by Excel, or LibreOffice. The waveform data however is placed in a single row, and can at times consist of over 10k ADC readings, which leads to a line length at times of over 72k characters.
The native csv2cell packaged in the io package cannot handle this.
When I modify a local version to have a MAXSTRINGLENGTH of 131072, I can read the files I need to.
Since this value is only for a single character buffer allocation, why can't this be set much larger by default?

Duane Kaufman <duanekaufman>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by duanekaufman (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-01-13 philipnienhuis StatusNone Wont Fix
        Open/ClosedOpen Closed
        Release7.3.0 other
        Summaryio -- csv2cell line length limit [octave forge] (io) csv2cell line length limit

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code