bugGNU Octave - Bugs: bug #58345, Emit error and don't parse scripts...


bug #58345: Emit error and don't parse scripts > 1GB in size

Submitted by:  None
Submitted on:  Sun 10 May 2020 12:20:21 PM UTC  
Category:  Interpreter Severity:  3 - Normal
Priority:  2 Item Group:  Feature Request
Status:  Fixed Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  dev
Operating System:  GNU/Linux

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Tue 26 May 2020 02:41:02 PM UTC, comment #7: 

Final item for this bug report.  I checked in a change so that the parser looks at the file size before parsing and emits an error if it is greater than 512MB.  This should be enough to prevent random segfaults in the interpreter.  See https://hg.savannah.gnu.org/hgweb/octave/rev/6870068fb34e.

Marking as fixed and closing report.

Rik <rik5>
Project Administrator
Wed 13 May 2020 03:17:46 PM UTC, comment #6: 

Rik, that was very comprehensive! Thank you! I had been using the C++ versions of textscan and fgetl before but I had not tried dlmread and that would be the best way to eliminate my bottleneck in reading data from file.

Unfortunately in my case, the data coming in is from an external C++ application, and editing its source to write Octave-readable binary formats is impractical at present (my Octave code is only one downstream client for its output data, among many). But dlmread and dlmwrite are very promising. Thank you again.

Wed 13 May 2020 03:11:35 AM UTC, comment #5: 

In previous versions of Octave textscan was an m-file, and hence slow.  In newer versions textscan is written in C++ and might be fast enough for your needs.  Similarly, fgetl is written in C++, but generally it is wrapped in a for loop in an m-file and loops in interpreted languages are slow (unless there is a JIT compiler).

I would look at storing the data in a uniform format so that you could take advantage of some of the routines written in C++.

As an example, for 2-D matrices.  If you just write a text file that contains numbers separated by spaces you can use 'load' to have the interpreter read in all the values.

For example, I made this file and called it A.var (attached to the bug report as well).

1 3.1415 2
-1e2 10 2.718

I can then do

octave:1> load A.var
octave:2> A
A =

     1.0000     3.1415     2.0000
  -100.0000    10.0000     2.7180

However, it turns out that even load() is ~3X slower than dlmread().

To test, I first created an array of 1GB in size

octave:13> sz = ceil (sqrt (1e9/8));
octave:14> x = rand (sz, sz);
octave:15> whos x
Variables visible from the current scope:

variables in scope: top scope

   Attr Name        Size                     Bytes  Class
   ==== ====        ====                     =====  =====
        x       11181x11181             1000118088  double

Then I wrote it out to a space-separated file.

octave:18> dlmwrite ('x.var', x, ' ');

The resulting file is 2.3GB in size.  Next, I tried reading it back in with dlmread

octave:23> tic; x = dlmread ('x.var', ' '); toc
Elapsed time is 107.864 seconds.

That's okay, I guess.  The result was worse for a straight load.

octave:26> tic; load ('x.var'); toc
Elapsed time is 342.51 seconds.

Even with textscan now written in C++, it is far too slow.

octave:41> tic; c = textscan (fid, fmt); toc
Elapsed time is 1911.16 seconds.

Clearly, however, the winner is to use a binary format rather than a text one.

octave:29> tic; save -binary x.bin x; toc
Elapsed time is 0.811615 seconds.
octave:30> clear x
octave:32> tic; load x.bin; toc
Elapsed time is 1.38077 seconds.

For this purpose fwrite()/fread() are just about as fast as well.

Rik <rik5>
Project Administrator
Tue 12 May 2020 11:36:57 PM UTC, comment #4: 

OP here. I do not have access to Matlab, but the solution you've outlined is very reasonable.

I was writing them as scripts in the first place because I found that textscan or fgetl or techniques were the bottleneck in my application, and formatting it as a script to have the interpreter directly inhale the data as a named variable was faster, but it of course was not meant to do that for large data.

Tue 12 May 2020 09:36:52 PM UTC, comment #3: 

This seems reasonbale.  The Mathworks, in their answer, recommend the same strategy of separating the variables (data) from the processing code.

I suppose one thing we could do is check the file size before attempting to parse the m-file and just issue an error if it is too large.  This would be nicer than an abrupt crash.

Re-titling to reflect that.

Rik <rik5>
Project Administrator
Tue 12 May 2020 09:08:09 PM UTC, comment #2: 

It seems that the limit in Matlab is 128MB:

Guillaume <gyom>
Tue 12 May 2020 08:57:43 PM UTC, comment #1: 

Scripts are designed to manipulate data, rather than be the data.  I guess I'm not surprised that you found some internal limit somewhere in the interpreter.

For reference, does Matlab process your script?  I bet they also have a hard time.

I would probably re-architect to have your script file load the necessary data as the first step, and then process it.  It's two files, rather than one, but still not that inconvenient.

Rik <rik5>
Project Administrator
Sun 10 May 2020 12:20:21 PM UTC, original submission:  

I was programmatically creating several script files that had the output of an unrelated program  and Octave code to further manipulate them, essentially being self-contained scripts with both code and data, the data coming from the upstream program. One of those script files was over 1GB because it had that much data. Upon invoking that script, Octave crashed with no warning. I was monitoring memory usage through top, and it never exceeded 9%, but it crashed none the same and sent me back to the bash prompt.

I can work around this, but I wanted to report this data point about the limit of what the interpreter can handle. The file which crashed had one matrix of size 19100000 x 12, each element being a positive integer from 1 to 3320. The only code was "arr = [" at the start and a closing "];" at the end.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #49062:  A.var added by rik5 (25B - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by gyom (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-05-26 rik5 StatusConfirmed => Fixed
        Open/ClosedOpen => Closed
    2020-05-13 rik5 Attached File- => Added A.var, #49062
    2020-05-12 rik5 Priority1 - Later => 2
        Item GroupSegfault, Bus Error, etc. => Feature Request
        StatusNone => Confirmed
        SummaryCrash on executing a 1GB script file => Emit error and don't parse scripts > 1GB in size
    2020-05-12 rik5 CategoryNone => Interpreter
        Priority5 - Normal => 1 - Later

    Back to the top

    Powered by Savane 3.5