bugGNU Octave - Bugs: bug #33680, System stalled by using 'strread'

 
 

bug #33680: System stalled by using 'strread'

Submitter:  None
Submitted:  Thu 30 Jun 2011 03:21:31 PM UTC
   
 
Category:  Libraries Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Fixed Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * 3.4.2
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 22 Jul 2011 08:18:31 PM UTC, comment #4: 

This has been addressed in two changesets.  The problem was not regexp but strtrim.  Changing the internals of that function has led to a 15X speedup.  This change was propagated to the 3.4.2 branch in this changeset (http://hg.savannah.gnu.org/hgweb/octave/rev/a319f6835e9b).  You will need to build from source in order to see the change.

Alternatively, the change above and the additional work that Philip did is now on the development branch of Mercurial.  You will need to build from source to see the change again.

Rik <rik5>
Group administrator
Fri 01 Jul 2011 06:25:22 PM UTC, comment #3: 

Thank you for quickly reporting back.

regexp is a beast, yes, but I think it was invoked there for a reason (namely, to be able to split also on %*4s or %*6f -like format specifiers).
With my strsplit() trick, future proper implementation of those "skip forward" format specifiers becomes more awkward.
But if Octave already chokes on 14 MB files, what will happen if I feed it our 1.2 GB text files?
For now, with strsplit() instead of regexp(), no existing functionality is lost and textread/textscan/strread work faster while needing less resources. Should be good enough.


BTW: The problem with newline-as-delimiter is this:

Matlab docs say:
default whitespace for textscan.m: "\b\t" (no space!)
    "       "          textread.m: " \b\t" (note space)
    "       "          strread.m:  "\b\r\n\t" (no space!)
Then, ML docs also say:
default delimiter for textscan.m: whitespace
    "       "         textread.m: none
    "       "         strread.m:  one or more whitespace chars
Combining this, for default delimiters we have (still according to the ML docs)
textscan.m: "\b\t" (note: no space! and no newline)
textread.m: no default value (i.e., no newline or space)
strread.m:  "\b\r\n\t" (no space! but incuding a newline)

For (text) files one can expect records to be separated by newlines, so having a newline as implied default delimiter is somewhat logical. Indeed, AFAICS textscan.m and textread.m do seem to always use them as delimiter (Matlab does this too).
Test case 1 in textscan.m shows that it work this way (and in Matlab this test case gives the same results).
So, as newline always seems to be implied delimiter when reading from file, why aren't they listed as default delimiter?

For strings this is less obvious (I can imagine one wants to read substrings containing newlines, each substring delimited by tabs, from some long text string).

<UPDATE>
In the mean time, I got an answer:
It turns out that end-of-line (EOL) (which is system-dependent), is treated as a separate delimiter (indeed!), to be controlled by the "endofline" parameter.
By coincidence I had already implemented parts of this parameter's handling, but for the wrong reason. Now that I know what it's purpose really is, this last "missing feature" can finally be fixed as well.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 01 Jul 2011 11:04:53 AM UTC, comment #2: 

Thank you philipnienhuis. Now the textread works fine!
The problem seem to be solved.

I think the regexp was the problem. 'regexp' leads to a memory overrun. I don't know why.
I did not know the 'strsplit' command. The memory load is much lesser than with 'regexp'. 

At the moment I have not problems with some newline delimiters. I have add some newlines in my input files. This input change don't change the result of 'textread'. Sorry, but I don't see a problem with newlines in the input files. What do you mean?

christian <sterngucker>
Fri 01 Jul 2011 08:50:43 AM UTC, comment #1: 

I have been working on strread.m, textread.m and textscan.m for some time to make them more ML-compatible, (properly) implement some ML features, and fix a few bugs, see #31380, #32720 and #33536. (At a low pace as I'm occupied with many things at the moment)

The offending statement you mentioned can more easily be replaced by a simple strplit(), provided that before invoking it some cleanup is performed as regards whitespace etc.
But that may take loads of memory, too.

I've attached my current versions of strread.m and textread.m  (textscan.m is still in the making), I'd like to know if these work better for you.
So please report back!

Please note that there is some confusion (at my side) on how to treat newlines. In strread.m, a newline was always added to the delimiters but I doubt whether this is correct.
The ML docs are a bit unclear if not contradictory about whether newlines are always treated as delimiters, or not. I've asked on some Matlab forums but got no response yet.
If you need newline delimiters, with these test versions you might need to supply a
 ..., "delimiter", "<any delimiters you need>\n"
parameter/value pair on the call to strread / textread / textscan


(file #23604, file #23605)

Philip Nienhuis <philipnienhuis>
Group Member
Thu 30 Jun 2011 03:21:31 PM UTC, original submission:  

The routine 'strread' the function in line 244

[~, ~, ~, out] = regexp (text, pat);

is extrem memory consuming. The memory will reservated until octave is closed. Big files over 14MB will lead to
a running out of memory failure of a system with 1GB RAM.

The History:

I use

+varbatim+
textread[Z(:,1),Z(:,2),Z(:,4),Z(:,3)]=textread('File.txt','%f %f %f %f','headerlines',11)
-verbatim-

The File.txt is 14 MB large. I found that regexp is the origin of the problem. After 'regexp' started the system stalled.

The command "whos" in octave shows a normal memory usage of the big input files, but while the 'strread' routine is running the machine memory filled up until the harddrive SWAP starts.The complete system hang up and over an hour later octave ends with a memory failure message.

I have rewrite the 'strread' to avoid 'regexp', but this routine functioning only for my files.
An example of my file you are attached on this posting.
In order to run this file, you need the command:
+varbatim+
[Ct(:,2),Ct(:,1), tmp]=textread_cm('110527kali_t.txt','%f %f %f','headerlines',10);
-verbatim-
and the two rewritten octave files.

Thanks for help!




Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #23604:  textread.m added by philipnienhuis (4KiB - text/x-objective-c)
file #23605:  strread.m added by philipnienhuis (14KiB - text/x-objective-c)
file #23593:  110527kali_t.txt added by None (1KiB - text/plain)
file #23594:  strread_cm.m added by None (9KiB - text/x-matlab)
file #23595:  textread_cm.m added by None (3KiB - text/x-matlab)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by sterngucker (Posted a comment)
  • -email is unavailable- added by philipnienhuis (Updated the item)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-07-22 rik5 StatusNone Fixed
        Open/ClosedOpen Closed
    2011-07-01 philipnienhuis Attached File- Added textread.m, #23604
        Attached File- Added strread.m, #23605
    2011-06-30 None Attached File- Added 110527kali_t.txt, #23593
        Attached File- Added strread_cm.m, #23594
        Attached File- Added textread_cm.m, #23595

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code