bugGNU Octave - Bugs: bug #34234, two tests fail in speed.m (MinGW)

 
 

bug #34234: two tests fail in speed.m (MinGW)

Submitted by:  Tatsuro MATSUOKA <tmacchant>
Submitted on:  Wed 07 Sep 2011 07:48:00 AM UTC  
 
Category: NoneSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: FixedAssigned to: None
Originator Name: Open/Closed: Closed
Release: devOperating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 11 Oct 2011 02:27:29 AM UTC, comment #19:

On mingw

scripts/testfun\speed.m PASS 4/4

Tatsuro MATSUOKA <tmacchant>
Fri 07 Oct 2011 09:01:35 AM UTC, comment #18:

on cygwin

scripts/testfun/speed.m PASS 4/4

marco atzeri <matzeri>
Thu 06 Oct 2011 10:05:59 PM UTC, comment #17:

This function was in bad shape. I ended up changing lots of code in this changeset (http://hg.savannah.gnu.org/hgweb/octave/rev/5d3d802c772e). Tests should pass now, but I won't close this report until I get confirmation from Marco or Tatsuro.

Rik <rik5>
Project Administrator
Thu 06 Oct 2011 01:22:06 AM UTC, comment #16:

dbstop ("speed", 255);
octave:8> [order, n, T_f1, T_f2] = speed ("sum (x)", "", [100, 1000], "v = 0; for i =1:length (x), v += x(i); end")
stopped in /pub/hg/octave_build/../octave/scripts/testfun/speed.m at line 255
255: tailidx = ceil(length(__test_n)/2):length(__test_n);
debug> __test_n
__test_n = [](1x0)

-------------------------------------------------------
but much earlier

debug> dbnext
stopped in /pub/hg/octave_build/../octave/scripts/testfun/speed.m at line 191
191: assert (__test_n >= 1);
debug> __test_n
__test_n =

Columns 1 through 11:

100 118 139 164 193 228 268 316 373 439 518

Columns 12 through 15:

611 720 848 1000
------------------------------------------------------
stopped in /pub/hg/octave_build/../octave/scripts/testfun/speed.m at line 244
244: zidx = (__tnew < 100eps | __torig < 100eps);
debug> __tnew
__tnew =

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

debug> __torig
__torig =

Columns 1 through 6:

9.7704e-04 2.9299e-03 9.7609e-04 1.9529e-03 1.9541e-03 1.9529e-03

Columns 7 through 12:

1.9529e-03 4.8831e-03 6.8359e-03 7.8130e-03 8.7891e-03 1.0742e-02

Columns 13 through 15:

6.8362e-03 6.8359e-03 8.7891e-03

marco atzeri <matzeri>
Wed 05 Oct 2011 07:21:47 PM UTC, comment #15:

This is odd. I guess take it back one more step. I used the following code.

Rik <rik5>
Project Administrator
Wed 05 Oct 2011 06:03:23 PM UTC, comment #14:

octave:7> debug_on_error ()
ans = 1
octave:8> [order, n, T_f1, T_f2] = speed ("sum (x)", "", [100, 1000], "v = 0; for i =1:length (x), v += x(i); end")
error: speed: subscript indices must be either positive integers or logicals
error: called from
speed at line 256 column 5
stopped in /pub/hg/octave_build/../octave/scripts/testfun/speed.m at line 256
256: p = polyfit (log (__test_n(tailidx)), log (__tnew(tailidx)), 1);
debug> tailidx
tailidx = 0
debug> __tnew(tailidx)
error: speed: subscript indices must be either positive integers or logicals
debug> __tnew
__tnew = [](1x0)
debug> __test_n
__test_n = [](1x0)

marco atzeri <matzeri>
Wed 05 Oct 2011 04:06:01 PM UTC, comment #13:

I've re-opened this bug. The first error about the size of T_F2 and n_sz is annoying but I have fixed it in this changeset (http://hg.savannah.gnu.org/hgweb/octave/rev/d5c8c2fe2eed)

For the warning "subscript indices must be either positive integers or logicals" I will need either Tatsuro or Marco to debug that.

I would use the following code

and then just use "dbstep" until you come across the line which is generating the error message.

Rik <rik5>
Project Administrator
Wed 05 Oct 2011 09:22:33 AM UTC, comment #12:

On MinGW I have gotten the similar results as those in the Cygwin.

BTW, I will not able to commit until 10/11(JST) due to my job.

Tatsuro MATSUOKA <tmacchant>
Wed 05 Oct 2011 06:42:37 AM UTC, comment #11:

still a problem on the first test as T_f2 is not evaluated:

octave:17> test /pub/hg/octave/scripts/testfun/speed.m
***** test
[order, n, T_f1, T_f2] = speed ("airy (x)", "x = rand (n, 10)", [100, 1000]);
assert (isstruct (order));
assert (size (order), [1, 1]);
assert (fieldnames (order), {"p"; "a"});
assert (isnumeric (n));
assert (length (n) > 10);
n_sz = size (n);
assert (isnumeric (T_f1));
assert (size (T_f1), n_sz);
assert (isnumeric (T_f2));
assert (size (T_f2), n_sz);
!!!!! test failed
assert (size (T_f2),n_sz) expected
1 12
but got
1 15
values do not match

----------------------------------------------------
The second test is still failing, but it is not clear to me why:

octave:18> test /pub/hg/prova/fails/test_speed2.m
***** test
[order, n, T_f1, T_f2] = speed ("sum (x)", "", [100, 1000], "v = 0; for i = 1:length (x), v += x(i); end");
assert (isstruct (order));
assert (size (order), [1, 1]);
assert (fieldnames (order), {"p"; "a"});
assert (isnumeric (n));
assert (length (n) > 10);
n_sz = size (n);
assert (size (n), [1, 15]);
assert (isnumeric (T_f1));
assert (size (T_f1), n_sz);
assert (isnumeric (T_f2));
assert (size (T_f2), n_sz);
!!!!! test failed
subscript indices must be either positive integers or logicals

marco atzeri <matzeri>
Wed 05 Oct 2011 04:57:34 AM UTC, comment #10:

Fixed in this changeset (http://hg.savannah.gnu.org/hgweb/octave/rev/b48e11427ac0). I went with the approach of testing for a range of N (10-15) rather than an exact N=15.

Rik <rik5>
Project Administrator
Tue 04 Oct 2011 10:25:34 PM UTC, comment #9:

>Tatsuro, could you try the following code and see what it reports?


I have tried three times

Tatsuro MATSUOKA <tmacchant>
Tue 04 Oct 2011 04:18:41 PM UTC, comment #8:

I know what the problem is. The question now is what the best fix for the issue is.

According to the documentation for speed, the vector return value N is

The key is that the execution time needs to be greater than zero. On operating systems with low resolution timers the elapsed time reported will often be zero when it was really just too small to measure.

Tatsuro provided me the execution data from his test and the measured times were

On my Linux machine, the times were

As you can see, the MinGW timer is accurate to 1 millisecond. The Linux timer seems to have sub-millisecond accuracy (although it's still probably accurate to only 0.2 millisecond)

Another problem with the demo is that even on a machine with good resolution timers, if the machine is exceedingly quick then the elapsed time will be zero.

So, the first question is whether there is a better resolution timer available on MinGW and Cygwin and how we can access it. The current speed.m uses time(). We could try using tic() and toc() but they might still rely internally on the time() system call.

Tatsuro, could you try the following code and see what it reports?

The other possibility is to change the speed test functions to perform more work in each loop and guarantee that there will be a meaningful delay to measure. Another choice would be to validate the test less harshly. Instead of returning exactly 15 time points it should return some number of elapsed times in the range 5-15. This might be easier and more forgiving since Linux is reporting 15 values, MinGW is reporting 13 values, and Cygwin is reporting 11 values.

Rik <rik5>
Project Administrator
Tue 04 Oct 2011 03:54:42 PM UTC, comment #7:

similar fault on cygwin

!!!!! test failed
assert (size (n),[1, 15]) expected
1 15
but got
1 11
values do not match

the test can be modified from:

%! assert (isnumeric (n));
%! assert (size (n), [1, 15]);
%! assert (isnumeric (T_f1));
%! assert (size (T_f1), [1, 15]);
%! assert (isnumeric (T_f1));
%! assert (size (T_f2), [1, 15]);

to

%! assert (isnumeric (n));
%! assert (size (n,1), 1);
%! assert ((size (n,2)>10));
%! assert (isnumeric (T_f1));
%! assert (size (T_f1,1), 1);
%! assert ((size (T_f1,2)>10));
%! assert (isnumeric (T_f2));
%! assert (size (T_f2,1), 1);
%! assert ((size (T_f2,2)>10));

marco atzeri <matzeri>
Tue 04 Oct 2011 03:18:37 AM UTC, comment #6:

I am uploading "speed.ver" file.

(file #24094)

Tatsuro MATSUOKA <tmacchant>
Mon 03 Oct 2011 04:58:25 PM UTC, comment #5:

I think I know what is happening for one of the tests. Could you run the following code and upload the file it produces to this bug report?

Upload the "speed.var" file.

Rik <rik5>
Project Administrator
Mon 03 Oct 2011 09:12:00 AM UTC, comment #4:

Reference:
changeset 13263:a156263b5509
date Sun Oct 02 20:13:33 2011 +0200

Currently the FAILs are the same as the original submission.
scripts/testfun\speed.m PASS 2/4 FAIL 2

I do not figure out the reason why second test FAIL has been reversed.

Tatsuro MATSUOKA <tmacchant>
Mon 03 Oct 2011 04:16:11 AM UTC, comment #3:

What is the current result for

For the tip (13263:a156263b5509) I get

Rik <rik5>
Project Administrator
Tue 13 Sep 2011 12:04:18 AM UTC, comment #2:

With dev source (changeset 13131:9207cdd7ff40), the second error was disappeared

Tatsuro MATSUOKA <tmacchant>
Thu 08 Sep 2011 04:13:05 AM UTC, comment #1:

For the first test,

size of 'n' seems to be [1 15] ????

For the second test,

????

Tatsuro MATSUOKA <tmacchant>
Wed 07 Sep 2011 07:48:00 AM UTC, original submission:

A new test failure in line.m of the recent source,
(After changeset 13108 d44d7f3c7998)

I have not see the details yet but report the fail.

Tatsuro MATSUOKA <tmacchant>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #24094:  speed.var added by tmacchant (543B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by rik5
  • -unavailable- added by matzeri (Posted a comment)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by tmacchant (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 9 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 07 Oct 2011 03:21:31 PM UTCrik5StatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
    Wed 05 Oct 2011 04:06:01 PM UTCrik5StatusFixed=>Confirmed
      Open/ClosedClosed=>Open
    Wed 05 Oct 2011 04:57:34 AM UTCrik5StatusNeed Info=>Fixed
      Open/ClosedOpen=>Closed
    Tue 04 Oct 2011 04:18:41 PM UTCrik5Carbon-Copy-=>Added jwe
    Tue 04 Oct 2011 03:18:37 AM UTCtmacchantAttached File-=>Added speed.var, #24094
    Mon 03 Oct 2011 04:16:11 AM UTCrik5StatusNone=>Need Info

    Back to the top


    Powered by Savane 3.1-cleanup1