Wed 25 Jun 2014 06:58:33 PM UTC, comment #5:
FYI, some comments from the one who rewrote and extended textscan.m & friends to what it is now.
textscan has been implemented as an m-file function. Its work is actually done behind the scenes by strread.m, another m-file function.
The way the whole setup worked and still works now isn't very efficient; lots of intermediate results on cellstr arrays with many memory copies. Yet I'm surprised that a 37 MB file expands into 1.5-2 GB in RAM.
All this inefficiency exists because of the many, many, many options offered by textscan and strread. As far as I'm concerned, for many complicated and big files one had better resort to Perl or so.
This textscan/strread situation will only get better when a binary (=compiled) textscan has been implemented. ATM no one has the time and priority for it.
2 or 3 years ago JWE has issued a first patch that could recognize input parameters; the rest still has to be written. Having experience with the m-file versions, I suppose it's a lot of work that may easily take a year or more to mature and reach/surpass current ML compatibility.
Why isn't csv2cell usable for you? I've worked on that (I'm also the maintainer of the io package where csv2cell and cell2csv live) and as I have plans to improve those two functions I could use some input.
Anyway I'll mark this bug as postponed, lower priority, and clean up several other statuses (like: regression -> performance - it was never better than it is now; OS -> Any - same issues on Linux; Interpreter -> Octave function)
|
Mon 23 Jun 2014 12:13:36 PM UTC, comment #4:
Thanks for the reply.
I confirm that if I reduce the amount of lines that it starts to work, I scaled the 37mb file back to 30mb and it worked. However, you are right, the memory usage in Windows for octave-cli-3.8.1.exe jumps to about 1.3GB and then goes down, and then starts heading towards 1GB; therefore, the textscan function is not memory efficient (does seem strange given that you tell textscan the format). I suspect that if there was a 64bit version for windows the 37mb might work as there would be more than 2GB of process memory available to Octave.
It's not a bug per se, but 36mb ascii file is nothing serious, Matlab has not problem, and in other languages it isn't a problem (37mb is tiny), so I do see it as a problem. If textscan can't be fixed in its current form, and a C++ rewrite isn't an option, this textscan limitation should be documented, or the recommendation not to use textscan. I am trying to have .m files that can be used with Matlab (what we use now) and Octave, while I have had some success, in general, it is proving difficult and it is a barrier (so I can't use csv2cell). For large datasets I do use something more efficient, like binary file formats, but 37mb in text does not need to do this conversion.
I do hope textscan can be fixed. I have programmed quite a bit on Windows, but not on Linux, (and I can't find instructions on how to compile on Windows) so if there is more information needed I might be able to help.
|
Fri 20 Jun 2014 06:07:03 PM UTC, comment #3:
dlmread is a c++ written function, textscan is a .m written function. It could be possible that you'll hit the out of memory border with textscan much earlier than with dlmread.
First, you should try to split the file in a smaller one
This should split the file in sevaral 1000 lines long file. Try to read one of those with your methode.
When it works, then it is imho not a bug in textscan.
If you can read all small files into you workspace in sequence, than maybe just a textscan rewrite in C++ can be fix that.
Second, you could try csv2cell from io package too. It is C++ written OCT function, but has some limitations too. Or maybe use fread and regexp...or any other function which is not a script function.
Third, for example, I can't load a 1471219 row x 23 columns csv file into memory (64bit Linux, 8GB ram) too (OOM). Than you have to load those big data into a database and just access the data you really need for the moment.
|
Wed 18 Jun 2014 04:16:17 PM UTC, original submission:
Loading a 37mb file using textscan In Octave 3.6.4 and Octave 3.8.1 I get the following error: "error: out of memory or dimension too large for Octave's index type".
If I replace textscan with dlmread it works fine. Using the command line debugger (only way I could track down exactly where the error was being generated), I had tracked down the line it throws the error, it is in strread.m line 404 which reads 'rxp_wsp = sprintf ("[%s]+", white_spaces);'
Use the following code with the example file in the attached 7z file.
|