bugGNU Octave - Bugs: bug #42579, textscan error out of memory or...

 
 

bug #42579: textscan error out of memory or dimension too large

Submitted by:  None
Submitted on:  Wed 18 Jun 2014 04:16:17 PM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 3 - LowItem Group: Performance
Status: PostponedAssigned to: None
Originator Name: Chris MumbyOriginator Email: -unavailable-
Open/Closed: OpenRelease: 3.8.1
Operating System: Any

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Wed 25 Jun 2014 06:58:33 PM UTC, comment #5:

FYI, some comments from the one who rewrote and extended textscan.m & friends to what it is now.

textscan has been implemented as an m-file function. Its work is actually done behind the scenes by strread.m, another m-file function.

The way the whole setup worked and still works now isn't very efficient; lots of intermediate results on cellstr arrays with many memory copies. Yet I'm surprised that a 37 MB file expands into 1.5-2 GB in RAM.
All this inefficiency exists because of the many, many, many options offered by textscan and strread. As far as I'm concerned, for many complicated and big files one had better resort to Perl or so.

This textscan/strread situation will only get better when a binary (=compiled) textscan has been implemented. ATM no one has the time and priority for it.
2 or 3 years ago JWE has issued a first patch that could recognize input parameters; the rest still has to be written. Having experience with the m-file versions, I suppose it's a lot of work that may easily take a year or more to mature and reach/surpass current ML compatibility.

Why isn't csv2cell usable for you? I've worked on that (I'm also the maintainer of the io package where csv2cell and cell2csv live) and as I have plans to improve those two functions I could use some input.

Anyway I'll mark this bug as postponed, lower priority, and clean up several other statuses (like: regression -> performance - it was never better than it is now; OS -> Any - same issues on Linux; Interpreter -> Octave function)

Philip Nienhuis <philipnienhuis>
Project Member
Mon 23 Jun 2014 12:13:36 PM UTC, comment #4:

Thanks for the reply.

I confirm that if I reduce the amount of lines that it starts to work, I scaled the 37mb file back to 30mb and it worked. However, you are right, the memory usage in Windows for octave-cli-3.8.1.exe jumps to about 1.3GB and then goes down, and then starts heading towards 1GB; therefore, the textscan function is not memory efficient (does seem strange given that you tell textscan the format). I suspect that if there was a 64bit version for windows the 37mb might work as there would be more than 2GB of process memory available to Octave.

It's not a bug per se, but 36mb ascii file is nothing serious, Matlab has not problem, and in other languages it isn't a problem (37mb is tiny), so I do see it as a problem. If textscan can't be fixed in its current form, and a C++ rewrite isn't an option, this textscan limitation should be documented, or the recommendation not to use textscan. I am trying to have .m files that can be used with Matlab (what we use now) and Octave, while I have had some success, in general, it is proving difficult and it is a barrier (so I can't use csv2cell). For large datasets I do use something more efficient, like binary file formats, but 37mb in text does not need to do this conversion.

I do hope textscan can be fixed. I have programmed quite a bit on Windows, but not on Linux, (and I can't find instructions on how to compile on Windows) so if there is more information needed I might be able to help.

Chris Mumby <chrismumby>
Fri 20 Jun 2014 06:07:03 PM UTC, comment #3:

dlmread is a c++ written function, textscan is a .m written function. It could be possible that you'll hit the out of memory border with textscan much earlier than with dlmread.

First, you should try to split the file in a smaller one

This should split the file in sevaral 1000 lines long file. Try to read one of those with your methode.
When it works, then it is imho not a bug in textscan.

If you can read all small files into you workspace in sequence, than maybe just a textscan rewrite in C++ can be fix that.

Second, you could try csv2cell from io package too. It is C++ written OCT function, but has some limitations too. Or maybe use fread and regexp...or any other function which is not a script function.

Third, for example, I can't load a 1471219 row x 23 columns csv file into memory (64bit Linux, 8GB ram) too (OOM). Than you have to load those big data into a database and just access the data you really need for the moment.

Markus Bergholz <markuman>
Thu 19 Jun 2014 08:38:24 PM UTC, comment #2:

When attaching a file (a 7z of the data file to keep it below the 16mb limit), it uploads but doesn't appear.

Chris Mumby <chrismumby>
Thu 19 Jun 2014 11:01:45 AM UTC, comment #1:

Sorry, I must off got the Markup wrong, the code was:

fid=fopen('data.dat','r')
data=textscan(fid, '%f %f %f %f')
fclose(fid)

Chris Mumby <chrismumby>
Wed 18 Jun 2014 04:16:17 PM UTC, original submission:

Loading a 37mb file using textscan In Octave 3.6.4 and Octave 3.8.1 I get the following error: "error: out of memory or dimension too large for Octave's index type".

If I replace textscan with dlmread it works fine. Using the command line debugger (only way I could track down exactly where the error was being generated), I had tracked down the line it throws the error, it is in strread.m line 404 which reads 'rxp_wsp = sprintf ("[%s]+", white_spaces);'

Use the following code with the example file in the attached 7z file.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by markuman (Posted a comment)
  • -unavailable- added by chrismumby (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 25 Jun 2014 06:58:33 PM UTCphilipnienhuisCategoryInterpreter=>Octave Function
      Priority5 - Normal=>3 - Low
      Item GroupRegression=>Performance
      StatusNone=>Postponed
      Operating SystemMicrosoft Windows=>Any

    Back to the top


    Powered by Savane 3.1-cleanup