bugGNU Octave - Bugs: bug #38616, memory leak in regexprep

 
 

bug #38616: memory leak in regexprep

Submitter:  None
Submitted:  Thu 28 Mar 2013 08:17:36 AM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Need Info Assigned to:  None
Originator Name:  ce Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 13 Oct 2013 04:38:19 AM UTC, comment #10: 

We are getting ready to do the 3.8 release.  Can you verify whether this bug is still a problem for you with the development code from Mercurial?  My guess is that this is no longer a problem.

Rik <rik5>
Group administrator
Thu 11 Apr 2013 09:15:00 PM UTC, comment #9: 

Correction: results from last comment was a build of the latest stable -> 3.6.4 .
Building latest tip now.

Christoph Ellenberger <ellchr00>
Thu 11 Apr 2013 08:54:29 PM UTC, comment #8: 

Ok I managed to build the latest tip on my box (Gentoo Core Duo E6750) at home.

I found that the following worked as well without a glitch:

s="0123456789";
for i=1:21, s=cstrcat(s,s);,endfor

s = regexprep(s,'0','2');
s = regexprep(s,'1','2');
s = regexprep(s,'2','3');


but with your second suggestion I get the out of memory (As I don't have much experience with gdb...did I do it right?):


octave:1> s = repmat ('2', 1, 2^23);
octave:2> s = regexprep (s, '2', '3');
error: memory exhausted or requested size too large for range of Octave's index type -- trying to return to prompt
octave:2>
Program received signal SIGINT, Interrupt.
0xb7fe0424 in __kernel_vsyscall ()
(gdb) bt
#0  0xb7fe0424 in __kernel_vsyscall ()
#1  0xb642fd23 in __read_nocancel () from /lib/libpthread.so.0
#2  0xb606574b in rl_getc () from /lib/libreadline.so.6
#3  0xb6075f70 in rl_explicit_arg () from /lib/libreadline.so.6


free shows the following before I start:

            total       used       free     shared    buffers     cached
Mem:       3373344     543364    2829980          0      15568     150248
-/+ buffers/cache:     377548    2995796
Swap:       987992     393768     594224


Additionally I tested the inital problem. This gives no Error here. This seems to be strictly Windows related and wil need the help of someone able to build under Windows.


octave:1> s="0123456789";
octave:2> for i=1:21, s=cstrcat(s,s);,endfor
octave:3>
octave:3> s = regexprep(s,'0','2');
octave:4> s = regexprep(s,'1','2');
octave:5> clear all
octave:6> s="0123456789";
octave:7> for i=1:21, s=cstrcat(s,s);,endfor
octave:8> save "TestData.dat" s
octave:9> fid = fopen("TestData.dat","r");
octave:10> s = fscanf(fid,'%c');
octave:11> fclose(fid);
octave:12> s = regexprep(s,'0','2');
octave:13> clear all
octave:14> fid = fopen("TestData.dat","r");
octave:15> s = fscanf(fid,'%c');
octave:16> s = regexprep(s,'0','2');
octave:17> clear all
octave:18> fid = fopen("TestData.dat","r");
octave:19> s = fscanf(fid,'%c');
octave:20> fid = fopen("TestData.dat","r");
octave:21> s = fscanf(fid,'%c');


Christoph Ellenberger <ellchr00>
Thu 11 Apr 2013 03:01:13 PM UTC, comment #7: 

At least on Linux you can use the 'free' command at the command line to see how much memory you have.  This will give you both actual memory and swap memory.  If 2^23 is bigger than actual + swap then that would be the obvious problem, independent of Octave.

Rik <rik5>
Group administrator
Thu 11 Apr 2013 01:00:25 PM UTC, comment #6: 

Actually the 3.7.2+ I am running on a i5 750 (64 bit) [windows xp] but it is compiled for i686-pc-mingw32 from Michael Goffioul (VS2010) and here unfortunately I do not have a build environment. But at home under Linux I can do the tests (given some time).

And already 2^22 seems to be too big, as
s = repmat ('2', 1, 2^22);
s = regexprep (s, '2', '3');
gives me the "out of memory..." error.

There might be probably 2 things, as the indexing problem in my oppinion wouldn't explain for the initial error, that after the clear all, the second read-in with fscanf would fail.

Christoph Ellenberger <ellchr00>
Wed 10 Apr 2013 08:12:08 PM UTC, comment #5: 

I started by attempting to reproduce the segfault you describe with this code.


s="0123456789";
for i=1:21, s=cstrcat(s,s);,endfor

s = regexprep(s,'0','2');
s = regexprep(s,'1','2');
s = regexprep(s,'2','3');


Unfortunately this works fine for me with the system I have available for testing (3.6.4 on x86_64 Linux).

Are you running on a 32-bit or 64-bit architecture?  This would affect the size of pointers which could be a problem.

Second, the final segfault is replacing 3/10 of the entries in s with a new value.  3/10 of the size of s is 6291456 or approximately 2^22.6.  So in order to untangle things, does the following standalone code also cause a segfault?


s = repmat ('2', 1, 2^23);
s = regexprep (s, '2', '3');


Finally, would it be possible for you to compile with debugging and run under gdb to produce a backtrace?

Something like the following should do it assuming you are using a Bourne sh.


CFLAGS="-g -O0"
CXXFLAGS="-g -O0"
# That is the letter 'O' followed by the numerical digit zero.
export CFLAGS
export CXXFLAGS
./configure --your_options_here
make
# The following runs the newly built octave under gdb
run-octave -g
s="0123456789";
for i=1:21, s=cstrcat(s,s);,endfor

s = regexprep(s,'0','2');
s = regexprep(s,'1','2');
s = regexprep(s,'2','3');
# When this produces a segmentation fault, get a backtrace with 'bt'
bt
# And attach the results to this bug report.


Rik <rik5>
Group administrator
Thu 28 Mar 2013 04:53:02 PM UTC, comment #4: 

I also attach TestData3.dat after:

fid = fopen("TestData.dat","r");
s = fscanf(fid,'%c');
fclose(fid);
s = regexprep(s,'0','2');
save "TestData3.dat" s


(file #27703)

Christoph Ellenberger <ellchr00>
Thu 28 Mar 2013 04:37:59 PM UTC, comment #3: 

Sure. As all files are not really small (20 MB) I zipped them. Surprisingly for me is that the binary version of a string is not really binary... All are the same also size(s) is always giving the same answer.

(file #27702)

Christoph Ellenberger <ellchr00>
Thu 28 Mar 2013 02:53:25 PM UTC, comment #2: 

Could you upload some of the intermediate files to see if there is a problem with them?

I would like to see TestData.dat, TestData1.dat, and TestData2.dat after executing the following:


s="0123456789";
for i=1:21, s=cstrcat(s,s);,endfor
save "TestData.dat" s
save -binary "TestData1.dat" s
clear all

fid = fopen("TestData.dat","r");
s = fscanf(fid,'%c');
save -binary "TestData2.dat" s


Rik <rik5>
Group administrator
Thu 28 Mar 2013 09:22:08 AM UTC, comment #1: 

As a reference, I show the original discussion on octave-help list:

http://octave.1599824.n4.nabble.com/memory-leak-in-regexprep-windows-3-7-2-VS-td4651218.html

Tatsuro MATSUOKA <tmacchant>
Thu 28 Mar 2013 08:17:36 AM UTC, original submission:  

The following code produces an out of memory / memory exhausted error on octave 3.7.2+(VS), 3.6.2(VS, MinGW), 3.4.3 (MinGW) and 3.2.4 (MinGW) as well as on octave-3.6.4 under Linux:

#**************************************
s="0123456789";
for i=1:21, s=cstrcat(s,s);,endfor
 
s = regexprep(s,'0','2');
s = regexprep(s,'1','2');
s = regexprep(s,'2','3');
#**************************************

Also the following (it seems to be something persistent):
********
s="0123456789";
for i=1:21, s=cstrcat(s,s);,endfor
save "TestData.dat" s

***************************
fid = fopen("TestData.dat","r");
s = fscanf(fid,'%c');
warning: range error for conversion to character value
fclose(fid);
s = regexprep(s,'0','2');
clear all
fid = fopen("TestData.dat","r");
s = fscanf(fid,'%c');
****************************

results in (octave-3.6.2 VS and 3.6.2 MinGW [win 7 64 bit Home premium, 4GB memory])
error: memory exhausted or requested size too large for range of Octave's index type -- trying to return to prompt
and (octave 3.7.2+ VS) [win 7 64 bit Home premium, 4GB memory] and [Win XP, 4GB]
error: out of memory or dimension too large for Octave's
index type
but succeds in Octave-3.6.2 Cygwin, -3.2.4 MinGW and -3.4.3 MinGW



Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #27703:  TestData3.zip added by ellchr00 (40KiB - application/x-zip-compressed)
file #27702:  TestData.zip added by ellchr00 (120KiB - application/x-zip-compressed)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ellchr00 (Updated the item)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by tmacchant (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-10-17 rik5 Open/ClosedOpen Closed
    2013-03-28 ellchr00 Attached File- Added TestData3.zip, #27703
    2013-03-28 ellchr00 Attached File- Added TestData.zip, #27702
    2013-03-28 rik5 StatusNone Need Info
    2013-03-28 tmacchant Carbon-CopyRemoved -email is unavailable- -
    2013-03-28 tmacchant Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code