bugGNU Octave - Bugs: bug #61092, Tangible performance degradation...

 
 

bug #61092: Tangible performance degradation on dev between 6th July and 27th of August

Submitter:  Hendrik K <koerhen>
Submitted:  Tue 31 Aug 2021 06:25:31 AM UTC
   
 
Category:  Performance Severity:  2 - Minor
Priority:  3 - Low Item Group:  Regression
Status:  Works For Me Assigned to:  None
Originator Name:  koerhen Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 04 Sep 2021 11:35:35 AM UTC, comment #5: 

Additional investigation & checks I did:

I wanted to eliminate build machine issues, so I took a new clean Ubuntu installation (using a portable USB SSD ubuntu install on my PC) and compiled octave on this fresh ubuntu image with current dev and July changeset 29857 and run the script bm_61092.m below on the new fresh Ubuntu SSD install with three octave versions:

a) Current dev
===========
0.1435
1.2833e-03

b) Fresh octave image from June changeset 29857:
===========
0.1407
1.0131e-03
0.1435/ 0.1407 ~2%
Some small scale degradation but this may be within margin.

c) Existing image I built in July (same changeset 29857) and which still runs nicely on my machine
=====================
0.1346
1.1270e-03
0.1355/0.1263 ~ 7.2%.

I now think that the additional difference of 5% from (b) to (c) could have been caused by my build environment at that time which may have been temporarily and unknowingly to me (by other activities) tuned (CFLAGS?!,static libraries?, others?) to AMD Ryzen compilation.

At least this is a reasonable explanation for me why the same source code produces different performance results on the same HW. 

So I guess that the bug report can be closed.

Hendrik K <koerhen>
Thu 02 Sep 2021 09:55:23 PM UTC, comment #4: 

I can't reproduce this.  I used the attached script bm_61092.m shown below.


N = 100;  # number of trials (> 50 for Central Limit Theorem to apply)

bm = zeros (N, 1);   # pre-declare benchmark data array

## Run one inner loop to engage CPU frequency scaling
for i = 1:1e5
  x = i * 2;
  x = x .^ 2;
  x = x - 3 * x;
endfor

## Run benchmark
for j = 1:N
  tic ();
  for i = 1:1e5
    x = i * 2;
    x = x .^ 2;
    x = x - 3 * x;
  endfor
  bm(j) = toc ();
endfor

disp (mean (bm));
disp (std (bm));


Results are


DEV : 9/2/21
mean = .177
std  = .009
-----------------------------------------------------------------
STABLE : 6.3.0
mean = .19
std  = .01
-----------------------------------------------------------------
DEV : 7/6/21
mean = .18
std  = .01


For 7/6/21 the actual changeset ID was 29857:0b01806bb663.

(file #51864)

Rik <rik5>
Group administrator
Thu 02 Sep 2021 04:21:16 AM UTC, comment #3: 

Thanks for those further experiments 👍

Maybe someone comes up with an idea what could be the root of this issue.

The best approach to isolate the "guilty" commit(s) was a proper bisection search/compile/test.  This unfortunately takes some time.

Kai Torben Ohlhus <siko1056>
Group Member
Thu 02 Sep 2021 04:07:16 AM UTC, comment #2: 

You are welcome and good question:

Two tests:
a) Using the underlying math libs : no performance change
(not surprisingly though...).
tic;
mat = (1:10000) .* ones(10000);
for k=1:10
ts(k) = toc();
mat = sqrt(mat);
ts(k) = toc() - ts(k);
endfor
disp (mean(ts));


Image 6th of July: 0.3886s
Image 27th of August: 0.3883s

b) Using vectorized code: Tangible (but less than loop) degradation.
tic;
mat = (1:10000) .* ones(10000);
for k=1:10
ts(k) = toc();
mat(1:end-1,:) = mat(1:end-1,:) - mat(2:end,:);
ts(k) = toc() - ts(k);
endfor
disp (mean(ts));


Image 6th of July: 0.7135s
Image 27th of August: 0.7332s
0.7332/0.7135 = 1.0276

Hendrik K <koerhen>
Thu 02 Sep 2021 02:24:56 AM UTC, comment #1: 

Thanks for the observation 👍

However I am not sure how to deal with this observation.  As you said many commits happened in those almost two months.  Does this performance degradation also happen for vectorized code?  I mean for-loops with scalar computations have never been a "strength" of Octave 😓

Kai Torben Ohlhus <siko1056>
Group Member
Tue 31 Aug 2021 06:25:31 AM UTC, original submission:  

I regularly build (every 6-8 weeks) my own images using mercurial dev repository on my linux box (Ubuntu 20.04.3 LTS AMD Ryzen 9 9300X) and noticed a tangible (6%-9%) performance degradation between my octave image from the 7th of July and from the 27th of August this year for all octave programs.

Build is Plain vanilla following https://wiki.octave.org/Building

A quick check/test confirmed this:
tic;
for k=1:10
ts(k) = toc();
for i=1:1000000
  j = i * 2;
  j = j .^ 2;
  j = j - 3 * j;
endfor
ts(k) = toc() - ts(k);
endfor
disp (mean(ts));


Image 6th of July: 1.2132s
Image 27th of August: 1.3054s
1.3054/1.2132 = 107.6%.

Note: I had a quick look at the commits occurred between the two dates but could not identify anything pinpointing to the root of the degradation (I am not an Octave developer though)

Hendrik K <koerhen>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #51864:  bm_61092.m added by rik5 (424B - text/x-matlab)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by koerhen (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-09-05 rik5 Open/ClosedOpen Closed
    2021-09-02 rik5 Attached File- Added bm_61092.m, #51864
        StatusNeed Info Works For Me
    2021-09-02 siko1056 Severity3 - Normal 2 - Minor
        Priority5 - Normal 3 - Low
        StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code