bugGNU Octave - Bugs: bug #37509, `load -z` is very slow

 
 

bug #37509: `load -z` is very slow

Submitter:  Elias Pipping <pipping>
Submitted:  Fri 05 Oct 2012 04:28:49 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Regression
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 3.6.3 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 12 Oct 2012 03:52:18 PM UTC, comment #11: 

Okay, turns out my patch wasn't exactly the same, and I was invalidating buffers. I have pushed Mike's correct patch:

    http://hg.savannah.gnu.org/hgweb/octave/graph/3ae8c1ee7365

Closing this report. I think it's time to release 3.6.4 now?

Jordi Gutiérrez Hermoso <jordigh>
Group Member
Fri 12 Oct 2012 03:13:25 AM UTC, comment #10: 

I extracted the changeset and applied it directly to the development branch and it works for me.

Rik <rik5>
Group administrator
Sun 07 Oct 2012 09:13:34 PM UTC, comment #9: 

That is confusing. I just did a clean build of stable with this cset applied to double-check and it works for me:


$ ./run-octave -qf --path $HOME --eval "testcase(10000)"
Direct: 1.254416e-02
Indirect: 2.540994e-02


Mike Miller <mtmiller>
Group Member
Sun 07 Oct 2012 06:37:18 PM UTC, comment #8: 

Mike, I am confused, because although I came upon your exact cset independently and also thought it was the right way, it is erroring out for me!


octave:2> load random.dat.gz
error: load: failed to load matrix constant
error: load: trouble reading ascii file 'random.dat.gz'
error: load: reading file random.dat.gz


I'm at a loss. You say it worked for you on both machines you were testing previously? Perhaps there's an unrelated bug here.

Jordi Gutiérrez Hermoso <jordigh>
Group Member
Sat 06 Oct 2012 03:25:43 PM UTC, comment #7: 

Attached change fixes both problems for me. Anyone have any other test cases to run against this?

(file #26712)

Mike Miller <mtmiller>
Group Member
Sat 06 Oct 2012 12:32:23 AM UTC, comment #6: 

More interesting, only RHEL5 32-bit:


octave:1> version
ans = 3.6.3
octave:2> octave_config_info ("canonical_host_type")
ans = i686-pc-linux-gnu
octave:3> load random.dat.gz
error: load: failed to load matrix constant
error: load: trouble reading ascii file `random.dat.gz'
error: load: reading file random.dat.gz


meanwhile...


octave:1> version
ans = 3.6.3
octave:2> octave_config_info ("canonical_host_type")
ans = x86_64-unknown-linux-gnu
octave:3> load random.dat.gz


produces no error.

Mike Miller <mtmiller>
Group Member
Fri 05 Oct 2012 09:10:39 PM UTC, comment #5: 

Interesting, I see the same thing on my Debian development setup, but on RHEL5 I see a completely different error when it tries to load a gz-compressed ASCII file as in your testcase


octave-3.6.3:1> load random.dat.gz
error: load: failed to load matrix constant
error: load: trouble reading ascii file `random.dat.gz'
error: load: reading file random.dat.gz


This error does not occur in 3.6.1 but does occur in 3.6.3.

If I change your script to save with -ascii or -binary (keeping -z), this error goes away, and the slowdown also goes away.

Mike Miller <mtmiller>
Group Member
Fri 05 Oct 2012 07:23:28 PM UTC, comment #4: 

I suspect this cset is to blame:

http://hg.savannah.gnu.org/hgweb/octave/rev/980e2d5c83f7

Indeed, on 3.6.2 we do not see this problem. The issue seems to be that that istream::tellg call ends up in gzfilebuf::seekoff:

http://hg.savannah.gnu.org/hgweb/octave/file/d174210ce1ec/src/zfstream.cc#l462

which is calling gzseek thousands of times, which is acknowledged to be extremely slow:

http://library.developer.nokia.com/topic/S60_3rd_Edition_Cpp_Developers_Library/GUID-2AD4ABC3-4061-4698-AF45-1DEC83466E64/html/zlib-gzseek-1.html

http://www.gzip.org/zlib/manual.html#gzseek

since this is a confirmed regression, we should fix this before the 3.6.4 release.

Jordi Gutiérrez Hermoso <jordigh>
Group Member
Fri 05 Oct 2012 04:49:01 PM UTC, comment #3: 

When I say `load -z`, I really mean `load` called on a gzip-compressed file. Sorry for the confusion.

Elias Pipping <pipping>
Fri 05 Oct 2012 04:43:18 PM UTC, comment #2: 

And another try :)


function testcase(n)
system("rm -f random.dat random.dat.gz");
A=rand(1,n);
save -z random.dat.gz A;
# Use -z to uncompress
tic;
load random.dat.gz;
fprintf(stderr, "Direct: %en", toc());
# Uncompress manually
tic;
system("gunzip random.dat.gz");
load random.dat;
fprintf(stderr, "Indirect: %en", toc());


Elias Pipping <pipping>
Fri 05 Oct 2012 04:42:23 PM UTC, comment #1: 

Here's the sample code again with proper markup (hopefully, since there's no preview).

+verbatim
function testcase(n)
system("rm -f random.dat random.dat.gz");
A=rand(1,n);
save -z random.dat.gz A;
# Use -z to uncompress
tic;
load random.dat.gz;
fprintf(stderr, "Direct: %en", toc());
# Uncompress manually
tic;
system("gunzip random.dat.gz");
load random.dat;
fprintf(stderr, "Indirect: %en", toc());
-verbatim

Elias Pipping <pipping>
Fri 05 Oct 2012 04:28:49 PM UTC, original submission:  

The following script

<<<<SNIP
function testcase(n)

system("rm -f random.dat random.dat.gz");

A=rand(1,n);
save -z random.dat.gz A;

# Use -z to uncompress
tic;
load random.dat.gz;
fprintf(stderr, "Direct: %en", toc());

# Uncompress manually
tic;
system("gunzip random.dat.gz");
load random.dat;
fprintf(stderr, "Indirect: %en", toc());
<<<<SNAP

creates a random 1xn matrix, and writes it to a gzip-compressed file. Then it compares how long it takes to

  (1) read it using `load -z` and to
  (2) uncompress it through a call to gunzip and then `load` it.

While the overhead for the call to gzip dominates at first, with n=10000, `load -z` is three orders of magnitudes slower than the indirect approach for me:

% octave --eval 'testcase(1000)' >/dev/null
Direct: 2.528620e-01
Indirect: 1.927996e-02

% octave --eval 'testcase(10000)' >/dev/null
Direct: 2.424446e+01
Indirect: 3.546500e-02

I can reproduce this with octave 3.6.3 on two machines.

Octave 3.2.4, which is installed on one of those machines as well, does not show this behaviour:

% /usr/bin/octave --eval 'testcase(10000)' >/dev/null
Direct: 1.430511e-02
Indirect: 1.741195e-02

Elias Pipping <pipping>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #26712:  oct-fix-load-z.diff added by mtmiller (975B - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by jordigh (Posted a comment)
  • -email is unavailable- added by pipping (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-10-12 jordigh StatusPatch Submitted Fixed
        Open/ClosedOpen Closed
    2012-10-06 mtmiller Attached File- Added oct-fix-load-z.diff, #26712
        StatusConfirmed Patch Submitted
    2012-10-05 jordigh StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code