bugGNU Octave - Bugs: bug #37509, `load -z` is very slow

 
 

bug #37509: `load -z` is very slow

Submitted by:  Elias Pipping <pipping>
Submitted on:  Fri 05 Oct 2012 04:28:49 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Regression
Status: FixedAssigned to: None
Originator Name: Open/Closed: Closed
Release: 3.6.3Operating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Fri 12 Oct 2012 03:52:18 PM UTC, comment #11:

Okay, turns out my patch wasn't exactly the same, and I was invalidating buffers. I have pushed Mike's correct patch:

http://hg.savannah.gnu.org/hgweb/octave/graph/3ae8c1ee7365

Closing this report. I think it's time to release 3.6.4 now?

Jordi Gutiérrez Hermoso <jordigh>
Project Administrator
Fri 12 Oct 2012 03:13:25 AM UTC, comment #10:

I extracted the changeset and applied it directly to the development branch and it works for me.

Rik <rik5>
Project Administrator
Sun 07 Oct 2012 09:13:34 PM UTC, comment #9:

That is confusing. I just did a clean build of stable with this cset applied to double-check and it works for me:

Mike Miller <mtmiller>
Project Administrator
Sun 07 Oct 2012 06:37:18 PM UTC, comment #8:

Mike, I am confused, because although I came upon your exact cset independently and also thought it was the right way, it is erroring out for me!

I'm at a loss. You say it worked for you on both machines you were testing previously? Perhaps there's an unrelated bug here.

Jordi Gutiérrez Hermoso <jordigh>
Project Administrator
Sat 06 Oct 2012 03:25:43 PM UTC, comment #7:

Attached change fixes both problems for me. Anyone have any other test cases to run against this?

(file #26712)

Mike Miller <mtmiller>
Project Administrator
Sat 06 Oct 2012 12:32:23 AM UTC, comment #6:

More interesting, only RHEL5 32-bit:

meanwhile...

produces no error.

Mike Miller <mtmiller>
Project Administrator
Fri 05 Oct 2012 09:10:39 PM UTC, comment #5:

Interesting, I see the same thing on my Debian development setup, but on RHEL5 I see a completely different error when it tries to load a gz-compressed ASCII file as in your testcase

This error does not occur in 3.6.1 but does occur in 3.6.3.

If I change your script to save with -ascii or -binary (keeping -z), this error goes away, and the slowdown also goes away.

Mike Miller <mtmiller>
Project Administrator
Fri 05 Oct 2012 07:23:28 PM UTC, comment #4:

I suspect this cset is to blame:

http://hg.savannah.gnu.org/hgweb/octave/rev/980e2d5c83f7

Indeed, on 3.6.2 we do not see this problem. The issue seems to be that that istream::tellg call ends up in gzfilebuf::seekoff:

http://hg.savannah.gnu.org/hgweb/octave/file/d174210ce1ec/src/zfstream.cc#l462

which is calling gzseek thousands of times, which is acknowledged to be extremely slow:

http://library.developer.nokia.com/topic/S60_3rd_Edition_Cpp_Developers_Library/GUID-2AD4ABC3-4061-4698-AF45-1DEC83466E64/html/zlib-gzseek-1.html

http://www.gzip.org/zlib/manual.html#gzseek

since this is a confirmed regression, we should fix this before the 3.6.4 release.

Jordi Gutiérrez Hermoso <jordigh>
Project Administrator
Fri 05 Oct 2012 04:49:01 PM UTC, comment #3:

When I say `load -z`, I really mean `load` called on a gzip-compressed file. Sorry for the confusion.

Elias Pipping <pipping>
Fri 05 Oct 2012 04:43:18 PM UTC, comment #2:

And another try :)

Elias Pipping <pipping>
Fri 05 Oct 2012 04:42:23 PM UTC, comment #1:

Here's the sample code again with proper markup (hopefully, since there's no preview).

+verbatim
function testcase(n)
system("rm -f random.dat random.dat.gz");
A=rand(1,n);
save -z random.dat.gz A;
# Use -z to uncompress
tic;
load random.dat.gz;
fprintf(stderr, "Direct: %en", toc());
# Uncompress manually
tic;
system("gunzip random.dat.gz");
load random.dat;
fprintf(stderr, "Indirect: %en", toc());
-verbatim

Elias Pipping <pipping>
Fri 05 Oct 2012 04:28:49 PM UTC, original submission:

The following script

<<<<SNIP
function testcase(n)

system("rm -f random.dat random.dat.gz");

A=rand(1,n);
save -z random.dat.gz A;

# Use -z to uncompress
tic;
load random.dat.gz;
fprintf(stderr, "Direct: %en", toc());

# Uncompress manually
tic;
system("gunzip random.dat.gz");
load random.dat;
fprintf(stderr, "Indirect: %en", toc());
<<<<SNAP

creates a random 1xn matrix, and writes it to a gzip-compressed file. Then it compares how long it takes to

(1) read it using `load -z` and to
(2) uncompress it through a call to gunzip and then `load` it.

While the overhead for the call to gzip dominates at first, with n=10000, `load -z` is three orders of magnitudes slower than the indirect approach for me:

% octave --eval 'testcase(1000)' >/dev/null
Direct: 2.528620e-01
Indirect: 1.927996e-02

% octave --eval 'testcase(10000)' >/dev/null
Direct: 2.424446e+01
Indirect: 3.546500e-02

I can reproduce this with octave 3.6.3 on two machines.

Octave 3.2.4, which is installed on one of those machines as well, does not show this behaviour:

% /usr/bin/octave --eval 'testcase(10000)' >/dev/null
Direct: 1.430511e-02
Indirect: 1.741195e-02

Elias Pipping <pipping>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #26712:  oct-fix-load-z.diff added by mtmiller (975B - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by jordigh (Posted a comment)
  • -unavailable- added by pipping (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 12 Oct 2012 03:52:18 PM UTCjordighStatusPatch Submitted=>Fixed
      Open/ClosedOpen=>Closed
    Sat 06 Oct 2012 03:25:43 PM UTCmtmillerAttached File-=>Added oct-fix-load-z.diff, #26712
      StatusConfirmed=>Patch Submitted
    Fri 05 Oct 2012 07:23:28 PM UTCjordighStatusNone=>Confirmed

    Back to the top


    Powered by Savane 3.1-cleanup1