bugGNU Octave - Bugs: bug #42579, textscan error out of memory or...

 
 

bug #42579: textscan error out of memory or dimension too large

Submitter:  None
Submitted:  Wed 18 Jun 2014 04:16:17 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  3 - Low Item Group:  Performance
Status:  Fixed Assigned to:  None
Originator Name:  Chris Mumby Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * 3.8.1
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 19 Mar 2016 10:33:38 PM UTC, comment #6: 

See
https://savannah.gnu.org/patch/index.php?8783
for a binary textscan that is now in core.

Closing report

Philip Nienhuis <philipnienhuis>
Group Member
Wed 25 Jun 2014 06:58:33 PM UTC, comment #5: 

FYI, some comments from the one who rewrote and extended textscan.m & friends to what it is now.

textscan has been implemented as an m-file function. Its work is actually done behind the scenes by strread.m, another m-file function.

The way the whole setup worked and still works now isn't very efficient; lots of intermediate results on cellstr arrays with many memory copies. Yet I'm surprised that a 37 MB file expands into 1.5-2 GB in RAM.
All this inefficiency exists because of the many, many, many options offered by textscan and strread. As far as I'm concerned, for many complicated and big files one had better resort to Perl or so.

This textscan/strread situation will only get better when a binary (=compiled) textscan has been implemented. ATM no one has the time and priority for it.
2 or 3 years ago JWE has issued a first patch that could recognize input parameters; the rest still has to be written. Having experience with the m-file versions, I suppose it's a lot of work that may easily take a year or more to mature and reach/surpass current ML compatibility.

Why isn't csv2cell usable for you? I've worked on that (I'm also the maintainer of the io package where csv2cell and cell2csv live) and as I have plans to improve those two functions I could use some input.

Anyway I'll mark this bug as postponed, lower priority, and clean up several other statuses (like: regression -> performance - it was never better than it is now; OS -> Any - same issues on Linux; Interpreter -> Octave function)

Philip Nienhuis <philipnienhuis>
Group Member
Mon 23 Jun 2014 12:13:36 PM UTC, comment #4: 

Thanks for the reply.

I confirm that if I reduce the amount of lines that it starts to work, I scaled the 37mb file back to 30mb and it worked. However, you are right, the memory usage in Windows for octave-cli-3.8.1.exe jumps to about 1.3GB and then goes down, and then starts heading towards 1GB; therefore, the textscan function is not memory efficient (does seem strange given that you tell textscan the format). I suspect that if there was a 64bit version for windows the 37mb might work as there would be more than 2GB of process memory available to Octave.

It's not a bug per se, but 36mb ascii file is nothing serious, Matlab has not problem, and in other languages it isn't a problem (37mb is tiny), so I do see it as a problem. If textscan can't be fixed in its current form, and a C++ rewrite isn't an option, this textscan limitation should be documented, or the recommendation not to use textscan. I am trying to have .m files that can be used with Matlab (what we use now) and Octave, while I have had some success, in general, it is proving difficult and it is a barrier (so I can't use csv2cell). For large datasets I do use something more efficient, like binary file formats, but 37mb in text does not need to do this conversion.

I do hope textscan can be fixed. I have programmed quite a bit on Windows, but not on Linux, (and I can't find instructions on how to compile on Windows) so if there is more information needed I might be able to help.


Chris Mumby <chrismumby>
Fri 20 Jun 2014 06:07:03 PM UTC, comment #3: 

dlmread is a c++ written function, textscan is a .m written function. It could be possible that you'll hit the out of memory border with textscan much earlier than with dlmread.

First, you should try to split the file in a smaller one


split -l 1000 file.name


This should split the file in sevaral 1000 lines long file. Try to read one of those with your methode.
When it works, then it is imho not a bug in textscan.

If you can read all small files into you workspace in sequence, than maybe just a textscan rewrite in C++ can be fix that.

Second, you could try csv2cell from io package too. It is C++ written OCT function, but has some limitations too. Or maybe use fread and regexp...or any other function which is not a script function.

Third, for example, I can't load a  1471219 row x 23 columns csv file into memory (64bit Linux, 8GB ram) too (OOM). Than you have to load those big data into a database and just access the data you really need for the moment.

Markus Bergholz <markuman>
Thu 19 Jun 2014 08:38:24 PM UTC, comment #2: 

When attaching a file (a 7z of the data file to keep it below the 16mb limit), it uploads but doesn't appear.

Chris Mumby <chrismumby>
Thu 19 Jun 2014 11:01:45 AM UTC, comment #1: 

Sorry, I must off got the Markup wrong, the code was:

fid=fopen('data.dat','r')
data=textscan(fid, '%f %f %f %f')
fclose(fid)

Chris Mumby <chrismumby>
Wed 18 Jun 2014 04:16:17 PM UTC, original submission:  

Loading a 37mb file using textscan In Octave 3.6.4 and Octave 3.8.1 I get the following error: "error: out of memory or dimension too large for Octave's index type".

If I replace textscan with dlmread it works fine. Using the command line debugger (only way I could track down exactly where the error was being generated), I had tracked down the line it throws the error, it is in strread.m line 404 which reads 'rxp_wsp = sprintf ("[%s]+", white_spaces);'

Use the following code with the example file in the attached 7z file.

fid = fopen(isofilename);
data=textscan(fid,  '%f %f %f %f %f %f %f %f');
fclose(fid);
+verbatim+

I have only used Octave on Windows and I have tried converting the line endings to Windows one but that doesn't help. I have Matlab at work and this is a barrier to using Octave to run the script without changing all the scripts to dlmread.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by markuman (Posted a comment)
  • -email is unavailable- added by chrismumby (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-03-19 philipnienhuis StatusPostponed Fixed
        Open/ClosedOpen Closed
    2014-06-25 philipnienhuis CategoryInterpreter Octave Function
        Priority5 - Normal 3 - Low
        Item GroupRegression Performance
        StatusNone Postponed
        Operating SystemMicrosoft Windows Any

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code