patchGNU Octave - Patches: patch #8119, Allow variable tolerance and...

 
 

patch #8119: Allow variable tolerance and improve error messages for assert.m script

Submitter:  Dan Sebald <sebald>
Submitted:  Wed 10 Jul 2013 09:02:05 AM UTC
   
 
Category:  None Priority:  5 - Normal
Status:  Done Privacy:  Public
Assigned to:  None Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 16 Aug 2013 03:41:41 AM UTC, comment #25: 

Yes, probably so.  I didn't realize there would be much change when starting though.  Nonetheless, it's nice to see it now near what the FIXME had in mind.

I thought about ruling out multiple errors for the same index, but felt it OK as is.  I don't mind it either way really.

That's right, there's another "error" at the top of the function.  "## Perhaps, say which elements failed?" -- yes it could probably be more explicit, but what to say?  Also, rather than decrement the depth level and error at the place of the conditional, another approach would be to just define the error message and let it fall through to the end at which point the error will occur because errmsg is not empty, i.e.,


 if (nargin == 1)
   ## Perhaps, say which elements failed?
   errmsg = sprintf ("assert %s failed", in);
 else
   errmsg = (varargin{:});
 end_if


Thanks for all the work on this.

Dan Sebald <sebald>
Thu 15 Aug 2013 07:29:24 PM UTC, comment #24: 

We should have started out by writing the tests and then verifying that we were actually getting the desired behavior.  I wrote a changeset which did that and fixed multiple bugs uncovered (http://hg.savannah.gnu.org/hgweb/octave/rev/a28c0d73e253).

Rik <rik5>
Group administrator
Wed 14 Aug 2013 10:05:29 PM UTC, comment #23: 

OK, thanks.

I double-checked some things and the call depth mod isn't quite right.  (The problem with "test assert" is that though it appropriately fails in the right location, it is hard to tell whether it is failing in the right way.)  The call_depth starts at zero and does an "if (call_depth > 0)" with the consequence that an error() always occurs.  Try:


x = {[3], [1,2,3]; 100+100*eps, "dog"};
y = x; y(1,1) = [2];  y(1,2) = [0, 2, 3]; y(2,1) = 101; y(2,2) = "cat";
assert (x, y)


With the attached change, call_depth is always incremented and always decremented at the end (otherwise subsequent calls to the function will fail).


(file #28844)

Dan Sebald <sebald>
Wed 14 Aug 2013 07:18:34 PM UTC, comment #22: 

Dan, this has moved far enough along that I have committed it to the repository(http://hg.savannah.gnu.org/hgweb/octave/rev/afd235a206a2).  I followed it up with a patch that extends the vector error checking to relative errors(http://hg.savannah.gnu.org/hgweb/octave/rev/684ccccbc15d).

Rik <rik5>
Group administrator
Mon 12 Aug 2013 08:46:26 PM UTC, comment #21: 

%!error assert ([2;2;3;3],[1;2;3;4]);

changed to

%!error assert ([2,2,3,3],[1,2,3,4]);


(file #28828)

Dan Sebald <sebald>
Mon 12 Aug 2013 08:41:04 PM UTC, comment #20: 

I was waiting on any response to a question on the discussion list about this behavior:


>> a = [11 22]
a =

   11   22

>> a ([1; 2])
ans =

   11   22


but haven't heard anything, so I've gone ahead and used (:) to columnize results where necessary.

I've had to add more tests.  Consider the following scenario:


assert ([(complex (NA, 1)) (complex (2, NA))], [(complex (NA, 2)) 2])


There is 1 disagreement of NA, and one disagreement between the imaginary parts.  However isfinite(complex(NA,1)) is zero, so the numerical test will not check whether complex (NA, 1) equals complex (NA, 2).  So, I think we need to break out the numerical tests into a real and imaginary test.  Here is the result:


>> assert ([(complex (NA, 1)) (complex (2, NA))], [(complex (NA, 2)) 2])
error: ASSERT errors for:  assert ([(complex (NA, 1)), (complex (2, NA))],[(complex (NA, 2)), 2])

  Location  |  Observed  |  Expected  |  Reason
   (1,2)         2NAi          2         'NA' mismatch
   (1,2)         2NAi          2         'NaN' mismatch
   (1,1)        NA+1i        NA+2i       Abs err NA exceeds tol 0


The reason text "Abs err NA exceeds tol 0" should probably read "Numbers are not equivalent", but I think we want to retain the extent of disagreement for the numerical case.

So, in the attached patch I'm taking a slightly different direction from previous versions.  For the numeric tests, I've first processed the matrices by converting any shared NA, NaN or +/-Inf to 0.  That way, the tests can continue with a similar behavior to before, but also go further by comparing any real or imaginary portion for which the other component is "!isfinite()".  (When discrepancies are printed out, the NA, NaN and +/-Inf are retained.)  Here are some samples (the 2NAi is a bug, reported here https://savannah.gnu.org/bugs/index.php?39773):


>> assert ([(complex (NA, 1)) (complex (2, NA))], [(complex (NA, 2)) 2])
error: ASSERT errors for:  assert ([(complex (NA, 1)), (complex (2, NA))],[(complex (NA, 2)), 2])

  Location  |  Observed  |  Expected  |  Reason
   (1,2)         2NAi          2         'NA' mismatch
   (1,2)         2NAi          2         'NaN' mismatch
   (1,1)        NA+1i        NA+2i       Abs err 1 exceeds tol 0

>> assert (complex (Inf, 0.2), complex (-Inf, 0.2 + 2*eps), eps)
error: ASSERT errors for:  assert (complex (Inf, 0.2),complex (-Inf, 0.2 + 2 * eps),eps)

  Location  |  Observed  |  Expected  |  Reason
   (1,1)       Inf+0.2i    -Inf+0.2000000000000005i    'Inf' mismatch
   (1,1)       Inf+0.2i    -Inf+0.2000000000000005i    Abs err 4.44089e-16 exceeds tol 2.22045e-16


I think I like the way that behaves because it catches everything for which there is disagreement, plus the reason description is more sensical.  What do you think.

I also added sub (:, matsize != 1) to shrink the indexing, i.e.,:


  [tmp{:}] = ind2sub (matsize, erridx (:));
  subs = [tmp{:}];
  if (numel (matsize) == 2)
    subs = subs (:, matsize != 1);
  endif


so the output looks like:


>> assert ([2;2;3;3],[1;2;3;4]);
error: ASSERT errors for:  assert ([2; 2; 3; 3],[1; 2; 3; 4])

  Location  |  Observed  |  Expected  |  Reason
    (1)           2            1         Abs err 1 exceeds tol 0
    (4)           3            4         Abs err 1 exceeds tol 0
>> assert ([6;6;7;7],[5;6;7;8])
error: ASSERT errors for:  assert ([6; 6; 7; 7],[5; 6; 7; 8])

  Location  |  Observed  |  Expected  |  Reason
    (1)           6            5         Abs err 1 exceeds tol 0
    (4)           7            8         Abs err 1 exceeds tol 0
>> assert (1, 2)
error: ASSERT errors for:  assert (1,2)

  Location  |  Observed  |  Expected  |  Reason
     ()           1            2         Abs err 1 exceeds tol 0



(file #28827)

Dan Sebald <sebald>
Mon 12 Aug 2013 07:02:02 AM UTC, comment #19: 

I'm aware of a half dozen bugs that need to be fixed.  Although several cases correctly fail, they don't fail in the correct way.  Easy fixes, but requires time.

Dan Sebald <sebald>
Sun 11 Aug 2013 10:03:41 PM UTC, comment #18: 

OK, this has become, "I can code that program in two loops".

I've incorporated your most recent index construction proposal along with some code changes to remove loops in the other err structure cells.  Perhaps it is too verbose this new way, but I'm fine with it.  I didn't really test performance, but I think it probably can't get too much better than this.  There is some duplication within the if-statements--maybe you can see a way to put that into a function format, but that is tricky because we're adding to err, not assigning err.

I've added some tests for multidimensional matrices, which really is not a separate condition just checking the general behavior of the index construction.

I've changed the other non-numerical tests to have the same indexing scheme as the numerical tests.  (Previous to this change it was the linear index that was reported...now it is the tuple index.)

I've changed the common prose to:

Abs err %g exceeds tol %g\n"
Max rel err %g exceeds tol %g"

I also noticed that the test associated with:

## Try to avoid problems comparing strange values like Inf+NaNi.

was incomplete in the sense that it really didn't test all the scenarios that could arise, so I made changes.  Rather than try to incorporate a "isnan" test in that stage, I thought that if the NA and NaN tests were thorough, then really there is no need to check/rule out any isnan within the Inf test.  By checking the real() and the imag() for all three of these exception values, everything is covered I think.  (The "Inf" test needs to check the sign of "Inf" as well.)

For the dimensions, I placed the (D1xD2x...) alongside the matrix symbol in the "Observed" and "Expected" columns, similar to what one sees with


>> zeros(3,0)
ans = [](3x0)


i.e.,


>> assert (zeros (2,0,2), zeros (2,0))
error: ASSERT errors for:  assert (zeros (2, 0, 2),zeros (2, 0))

  Location  |  Observed  |  Expected  |  Reason
     .         O(2x0x2)      E(2x0)      Dimensions don't match


How do you like that?

I think we are now getting to the point of tweaking things.  For example, I would sort of prefer the leftmost tuple value be monotonic as opposed to the rightmost tuple value:


>> assert (A.*(A!=2),A)
error: ASSERT errors for:  assert (A .* (A != 2),A)

  Location  |  Observed  |  Expected  |  Reason
   (2,1)          0            2         Abs err 2 exceeds tol 0
   (1,2)          0            2         Abs err 2 exceeds tol 0


but for now I'm fine with it.  And there is still the question of how to be more explicit about the recursive tests:


>> assert (x, y)
error: ASSERT errors for:  assert (cond {i},expected {i},tol)

  Location  |  Observed  |  Expected  |  Reason
     []          dog          cat        Strings don't match


other than "fine as is".

(file #28813)

Dan Sebald <sebald>
Thu 08 Aug 2013 07:28:00 PM UTC, comment #17: 

You can use the same trick that you used on your code, pulling the sub2ind call out of the loop, on my code as well.  The following runs in equivalent time without the eval call.


erridx = find (A != B);
err.reason = cell (numel (erridx), 1);
tmp = cell (1, ndims (A));
[tmp{:}] = ind2sub (size (A), erridx);
subs = [tmp{:}];

for i = 1:numel (erridx)
  loc = sprintf ("%d,", subs(i,:));
  err.reason{i} = ["(" loc(1:end-1) ")"];
endfor


It is faster for large erridx, but slower than the original code for small numbers of elements.  The crossover is ~18 elements.  At that point the total time is ~4 milliseconds.  So, probably it's not worth switching over given that most assert statements never report errors, and if they do it is only for a few elements.

If you really want to speed things up you can do a time/memory tradeoff and bring the sprintf out of the loop as well.  One call to sprintf to generate a whole list of strings and then index into the strings in the loop.  This produces a 4X savings on large matrices, but those aren't the norm.  On benchmarking, the crossover is still around 18 elements.

Rik <rik5>
Group administrator
Wed 07 Aug 2013 11:20:51 PM UTC, comment #16: 

I think I knew that already about deletion versus indexing, from decades ago perhaps.

Generally, eval() is slow (maybe it needs to be improved internally), but using it could be to advantage if it achieves something in a more efficient way than can be done otherwise.  I sought a way to keep the eval outside of a loop.  The amount of material inside of a loop is often the most critical thing when the loop has many iterations, and avoiding an interpreted loop by using a loop internal to the command is best.  Your proposal has ind2sub() and sprintf() inside the loop.

We can put the two evals into one eval:


oarg = sprintf ("i%d ", 1:ndims (A));
eval (cstrcat ("[", oarg, "] = ind2sub (size (A), find (A != B)); ", ...
               "idx = [", oarg, "];"));
for i = 1:length (i1)
  loc = sprintf("%d,", idx (i,:));
end


Here are some timing comparisons for when the loop is small:


>> A = ones(4);
>> B = zeros (size (A));
>>
>> err.reason = cell (sizeof (A), 1);
>> t_start = cputime ();
>>
>> oarg = sprintf ("i%d ", 1:ndims (A));
>> eval (cstrcat ("[", oarg, "] = ind2sub (size (A), find (A != B)); ", ...
               "idx = [", oarg, "];"));
>> for i = 1:length (i1)
  loc = sprintf("%d,", idx (i,:));
  err.reason {i} = ["(" loc(1:end-1) ")"];
endfor
>>
>> cputime () - t_start
ans =  0.049992
>>
>> err.reason = cell (sizeof (A), 1);
>> t_start = cputime ();
>>
>> erridx = find (A != B);
>> sz = size (A);
>> nd = ndims (A);
>> subs = cell (1, nd);
>> for idx = erridx(:).'
  [subs{:}] = ind2sub (sz, idx);
  loc = sprintf("%d,", subs{:});
  err.reason {i} = ["(" loc(1:end-1) ")"];
endfor
>>
>> cputime () - t_start
ans =  0.043994


and for when the loop is large:


>> A = ones(10,10,10,10,10);
>> B = zeros (size (A));
>>
>> err.reason = cell (sizeof (A), 1);
>> t_start = cputime ();
>>
>> oarg = sprintf ("i%d ", 1:ndims (A));
>> eval (cstrcat ("[", oarg, "] = ind2sub (size (A), find (A != B)); ", ...
               "idx = [", oarg, "];"));
>> for i = 1:length (i1)
  loc = sprintf("%d,", idx (i,:));
  err.reason {i} = ["(" loc(1:end-1) ")"];
endfor
>>
>> cputime () - t_start
ans =  7.2319
>>
>> err.reason = cell (sizeof (A), 1);
>> t_start = cputime ();
>>
>> erridx = find (A != B);
>> sz = size (A);
>> nd = ndims (A);
>> subs = cell (1, nd);
>> for idx = erridx(:).'
  [subs{:}] = ind2sub (sz, idx);
  loc = sprintf("%d,", subs{:});
  err.reason {i} = ["(" loc(1:end-1) ")"];
endfor
>>
>> cputime () - t_start
ans =  9.4526


Which is slightly faster isn't that critical really as this pertains to the case where a test fails, which is supposed to be rare.

I will make another pass at the changeset (may be this weekend).

Dan Sebald <sebald>
Wed 07 Aug 2013 02:57:01 PM UTC, comment #15: 

You've found out most of the answers already, but here goes.

The find() syntax is the way it is, and we can't change it without breaking Matlab compatibility.

I used the 'dimstr(end) = []' more as an example of pseudo-code that would work rather than finished code--I also prefer dimstr(1:end-1).  Incidentally, I've benchmarked indexing versus deletion and indexing is faster and thus preferred generally.  Another way to write this would be to treat the first index as special, rather than the last.  This also works:


sz = size (X);
dimstr = [sprintf("%d",sz(1)) sprintf("x%d",sz(2:end))];


For the indices, I would stay away from eval because it is quite slow in general.  The sample code I suggested uses sub2ind, but remains within the current interpreter.

Rik <rik5>
Group administrator
Wed 07 Aug 2013 08:07:47 AM UTC, comment #14: 

Apparently that isn't a bug.  The three output argument find is:

[i, j, v] = find (A)

and returns values in the third column.  I would think that the output arguments would be the various dimensions and if there were one greater output than dimensions that would be for the values.

No matter, just use ind2sub with an extra function call:


oarg = sprintf ("i%d ", 1:ndims (A));
eval (cstrcat ("[", oarg, "] = ind2sub (size (A), find (A != B));"));
eval (cstrcat ("idx = [", oarg, "];"));
for i = 1:length (i1)
  loc = sprintf("%d,", idx (i,:)) (1:end-1);
end


which works correctly:


>> oarg = sprintf ("i%d ", 1:ndims (A));
>> eval (cstrcat ("[", oarg, "] = ind2sub (size (A), find (A != B));"));
>> eval (cstrcat ("idx = [", oarg, "];"));
>> for i = 1:length (i1)
  loc = sprintf("%d,", idx (i,:)) (1:end-1)
end
loc = 1,1,1
loc = 2,2,2


Dan Sebald <sebald>
Wed 07 Aug 2013 06:21:32 AM UTC, comment #13: 

Let's try that again...

> Follow-up Comment #11, patch #8119 (project octave):


> 1) Instead of simply saying the dimensions don't match we could actually list
> them.  Sample format would be "1x3x5".  The following code will create this
> format.


That would be a good idea.


> dimstr = sprintf ("%dx", size (cond));
> dimstr(end) = []; # remove trailing 'x'


Good use of sprintf, but rather than removing the last character that way, just reference dimstr(1:end-1) at the location it is used.  This would work too:

dimstr = sprintf ("%dx", size (cond)) (1:end-1);

but somehow I prefer using the (1:end-1) construct at the point the array is used.


> 2) Absolute error uses the expression "Exceeds abs tol 0 by ..." while
> relative tolerance uses "Max rel err %g exceeds tol %g".  They should both use
> the same format whatever it is.


I saw that, but didn't feel like changing it until we thought things were heading in the right direction.  I'll have to look closely at what that last variation is doing.  Maybe I could express this more accurately and briefly.


> 3) Multi-dimensional arrays aren't handled properly.
>
>
> x = zeros (2,2,3);
> y = x;
> y(1,2,3) = 1;
> assert (x,y)
> error: ASSERT errors for:  assert (x,y)
>
>    Location  |  Observed  |  Expected  |  Reason
>     [1,6]          0            1         Exceeds abs tol 0 by 1


I didn't think to check three dimensions.


> It's kind of lame, but I think you can use sub2ind to do this.
>
>
> erridx = find (x != y);
> if (any (erridx))
>    sz = size (x);
>    nd = ndims (x);
>    subs = cell (1, nd);
>    for idx = erridx(:).'
>      [subs{:}] = ind2sub (sz, idx);
>      loc = ['(' sprintf("%d,", subs{:}) ];
>      loc(end) = ')';
>      ... print location and error ...
>    endfor
> endif


Let me think it over.  Maybe you noticed one small change I made between versions, which was to use the multiple output argument version of find().  I had computed the index similar to what you are suggesting (it's not difficult), but I found


          [I, J] = find (A != B & k);


much more convenient to work with.  And I notice now that this works:


>> A = ones(2,2,3);
>> B = A;
>> B(1,1,1) = 0;
>> B(2,2,2) = 0;
>> [i j k] = find (A != B)
i =

   1
   2

j =

   1
   4

k =

   1
   1


But doesn't that look like a bug?  No dimensions of the matrix in this example go beyond 3, so how could j be 4?  i j and k should be the same, [1 2]'.

Let's assume that is a bug and that we should be able to get the correct indeces that way.  So, maybe it would be more convenient to use something like:


oarg = sprintf ("i%d ", 1:ndims (A));
eval (cstrcat ("[", oarg, "] = find (A != B);"));
eval (cstrcat ("idx = [", oarg, "];"));
for i = 1:length (i1)
  loc = sprintf("%d,", idx (i,:)) (1:end-1);
end


How's that?  For this example, I'll print the results of some of the above commands and intermediate strings:


>> oarg = sprintf ("i%d ", 1:ndims (A))
oarg = i1 i2 i3
>> cstrcat ("[", oarg, "] = find (A != B);")
ans = [i1 i2 i3 ] = find (A != B);
>> eval (cstrcat ("[", oarg, "] = find (A != B);"));
>> cstrcat ("idx = [", oarg, "];")
ans = idx = [i1 i2 i3 ];
>> eval (cstrcat ("idx = [", oarg, "];"));
>> for i = 1:length (i1)
  loc = sprintf("%d,", idx (i,:)) (1:end-1)
end
loc = 1,1,1
loc = 2,4,1


Dan Sebald <sebald>
Wed 07 Aug 2013 06:17:28 AM UTC, comment #12: 


> Follow-up Comment #11, patch #8119 (project octave):


> 1) Instead of simply saying the dimensions don't match we could actually list
> them.  Sample format would be "1x3x5".  The following code will create this
> format.


That would be a good idea.


> dimstr = sprintf ("%dx", size (cond));
> dimstr(end) = []; # remove trailing 'x'


Good use of sprintf, but rather than removing the last character that way, just reference dimstr(1:end-1) at the location it is used.  This would work too:

dimstr = sprintf ("%dx", size (cond)) (1:end-1);

but somehow I prefer using the (1:end-1) construct at the point the array is used.


> 2) Absolute error uses the expression "Exceeds abs tol 0 by ..." while
> relative tolerance uses "Max rel err %g exceeds tol %g".  They should both use
> the same format whatever it is.


I saw that, but didn't feel like changing it until we thought things were heading in the right direction.  I'll have to look closely at what that last variation is doing.  Maybe I could express this more accurately and briefly.


> 3) Multi-dimensional arrays aren't handled properly.
>
>
> x = zeros (2,2,3);
> y = x;
> y(1,2,3) = 1;
> assert (x,y)
> error: ASSERT errors for:  assert (x,y)
>
>    Location  |  Observed  |  Expected  |  Reason
>     [1,6]          0            1         Exceeds abs tol 0 by 1


I didn't think to check three dimensions.


> It's kind of lame, but I think you can use sub2ind to do this.
>
>
> erridx = find (x != y);
> if (any (erridx))
>    sz = size (x);
>    nd = ndims (x);
>    subs = cell (1, nd);
>    for idx = erridx(:).'
>      [subs{:}] = ind2sub (sz, idx);
>      loc = ['(' sprintf("%d,", subs{:}) ];
>      loc(end) = ')';
>      ... print location and error ...
>    endfor
> endif


Let me think it over.  Maybe you noticed one small change I made between versions, which was to use the multiple output argument version of find().  I had computed the index similar to what you are suggesting (it's not difficult), but I found


          [I, J] = find (A != B & k);
-verbatim+

much more convenient to work with.  And I notice now that this works:

+verbatim+
>> A = ones(2,2,3);
>> B = A;
>> B(1,1,1) = 0;
>> B(2,2,2) = 0;
>> [i j k] = find (A != B)
i =

   1
   2

j =

   1
   4

k =

   1
   1


But doesn't that look like a bug?  No dimensions of the matrix in this example go beyond 3, so how could j be 4?  i j and k should be the same, [1 2]'.

Let's assume that is a bug and that we should be able to get the correct indeces that way.  So, maybe it would be more convenient to use something like:


oarg = sprintf ("i%d ", 1:ndims (A));
eval (cstrcat ("[", oarg, "] = find (A != B);"));
eval (cstrcat ("idx = [", oarg, "];"));
for i = 1:length (i1)
  loc = sprintf("%d,", idx (i,:)) (1:end-1);
end


How's that?  For this example, I'll print the results of some of the above commands and intermediate strings:


>> oarg = sprintf ("i%d ", 1:ndims (A))
oarg = i1 i2 i3
>> cstrcat ("[", oarg, "] = find (A != B);")
ans = [i1 i2 i3 ] = find (A != B);
>> eval (cstrcat ("[", oarg, "] = find (A != B);"));
>> cstrcat ("idx = [", oarg, "];")
ans = idx = [i1 i2 i3 ];
>> eval (cstrcat ("idx = [", oarg, "];"));
>> for i = 1:length (i1)
  loc = sprintf("%d,", idx (i,:)) (1:end-1)
end
loc = 1,1,1
loc = 2,4,1


Dan Sebald <sebald>
Wed 07 Aug 2013 12:31:37 AM UTC, comment #11: 

This is getting there.  I can see a few more improvements.

1) Instead of simply saying the dimensions don't match we could actually list them.  Sample format would be "1x3x5".  The following code will create this format.


dimstr = sprintf ("%dx", size (cond));
dimstr(end) = []; # remove trailing 'x'


2) Absolute error uses the expression "Exceeds abs tol 0 by ..." while relative tolerance uses "Max rel err %g exceeds tol %g".  They should both use the same format whatever it is.

3) Multi-dimensional arrays aren't handled properly.


x = zeros (2,2,3);
y = x;
y(1,2,3) = 1;
assert (x,y)
error: ASSERT errors for:  assert (x,y)

  Location  |  Observed  |  Expected  |  Reason
   [1,6]          0            1         Exceeds abs tol 0 by 1


It's kind of lame, but I think you can use sub2ind to do this.


erridx = find (x != y);
if (any (erridx))
  sz = size (x);
  nd = ndims (x);
  subs = cell (1, nd);
  for idx = erridx(:).'
    [subs{:}] = ind2sub (sz, idx);
    loc = ['(' sprintf("%d,", subs{:}) ];
    loc(end) = ')';
    ... print location and error ...
  endfor
endif



Rik <rik5>
Group administrator
Mon 05 Aug 2013 11:12:37 PM UTC, comment #10: 

OK, I get it now.  There was actually a bug or two still in the script.  Attached is a new version with all output packed into the error message so that test.m's try/catch can handle the display.


>> test clf
  ***** xtest
 hf = figure ("visible", "off");
 unwind_protect
   plot (1:10);
   set (hf, "papertype", "tabloid");
   clf (hf);
   assert (isempty (get (gcf, "children")));
   assert (get (hf, "papertype"), "tabloid");
   plot (1:10);
   clf (hf, "reset");
   kids = get (hf, "children");
   assert (isempty (get (gcf, "children")));
   assert (get (hf, "papertype"), "usletter");
 unwind_protect_cleanup
   close (hf);
 end_unwind_protect
!!!!! known failure
ASSERT errors for:  assert (get (hf, "papertype"),"usletter")

  Location  |  Observed  |  Expected  |  Reason
     []        tabloid      usletter     Strings don't match
PASSES 3 out of 3 tests (1 expected failures)


So that's an incremental step in the right direction.  I did find what seems like a bug in test.m.  I'll submit a new report for that.

The one thing I can think of to do would be something more descriptive for the recursive assert calls.  For example, here is the output of a recursive cell test:


>> assert (x, y)
error: ASSERT errors for:  assert (cond {i},expected {i},tol)

  Location  |  Observed  |  Expected  |  Reason
     1            3            2         Exceeds abs tol 0 by 1

ASSERT errors for:  assert (cond {i},expected {i},tol)

  Location  |  Observed  |  Expected  |  Reason
     1           100          101        Exceeds abs tol 0 by 1

ASSERT errors for:  assert (cond {i},expected {i},tol)

  Location  |  Observed  |  Expected  |  Reason
     1            1            0         Exceeds abs tol 0 by 1

ASSERT errors for:  assert (cond {i},expected {i},tol)

  Location  |  Observed  |  Expected  |  Reason
     []          dog          cat        Strings don't match


Notice how the arguments of the "assert" command are generic cell contents.  One could parse the original command for the variable names and either:

1) Create local variables with names similar to input... problematic if the variable matches something in the local space.

2) Temporarily redefine the "in" string... problematic if more than one level of recursion because original contents of "in" is lost.

We'll leave that for later.


(file #28770)

Dan Sebald <sebald>
Mon 05 Aug 2013 08:13:17 PM UTC, comment #9: 

The ones that look odd are %!xtests which are known to fail.


 test clf

ASSERT errors for:  assert (get (hf, "papertype"),"usletter")

  Location  |  Expected  |  Observed  |  Reason
     []        usletter     tabloid      Strings don't match

  ***** xtest
 hf = figure ("visible", "off");
 unwind_protect
   plot (1:10);
   set (hf, "papertype", "tabloid");
   clf (hf);
   assert (isempty (get (gcf, "children")));
   assert (get (hf, "papertype"), "tabloid");
   plot (1:10);
   clf (hf, "reset");
   kids = get (hf, "children");
   assert (isempty (get (gcf, "children")));
   assert (get (hf, "papertype"), "usletter");
 unwind_protect_cleanup
   close (hf);
 end_unwind_protect
!!!!! known failure
Strings don't match
PASSES 3 out of 3 tests (1 expected failure)


But if I run with the old assert


 test clf
  ***** xtest
 hf = figure ("visible", "off");
 unwind_protect
   plot (1:10);
   set (hf, "papertype", "tabloid");
   clf (hf);
   assert (isempty (get (gcf, "children")));
   assert (get (hf, "papertype"), "tabloid");
   plot (1:10);
   clf (hf, "reset");
   kids = get (hf, "children");
   assert (isempty (get (gcf, "children")));
   assert (get (hf, "papertype"), "usletter");
 unwind_protect_cleanup
   close (hf);
 end_unwind_protect
!!!!! known failure
assert (get (hf, "papertype"),"usletter") expected
usletter
but got
tabloid
PASSES 3 out of 3 tests (1 expected failure)


So I think we need to make sure that the ASSERT messages are printed after test.m has had a chance to print out the code block.

Rik <rik5>
Group administrator
Mon 05 Aug 2013 07:25:16 PM UTC, comment #8: 

I ran "make check" using the proposed assert.m and found the following:


  scripts/general/interpft.m .............................
ASSERT errors for:  assert (max (abs (imag (interpft ([1:8], 20)))),0,2 * eps)

  Location  |  Expected  |  Observed  |  Reason
     1        2.2542e-15       0         Exceeds abs tol 4.4409e-16 by 1.8101e-15


ASSERT errors for:  assert (max (abs (imag (interpft ([1:8], 21)))),0,2 * eps)

  Location  |  Expected  |  Observed  |  Reason
     1        3.0634e-15       0         Exceeds abs tol 4.4409e-16 by 2.6193e-15


ASSERT errors for:  assert (max (abs (imag (interpft ([1:9], 20)))),0,2 * eps)

  Location  |  Expected  |  Observed  |  Reason
     1        3.8512e-15       0         Exceeds abs tol 4.4409e-16 by 3.4071e-15


ASSERT errors for:  assert (max (abs (imag (interpft ([1:9], 21)))),0,2 * eps)

  Location  |  Expected  |  Observed  |  Reason
     1        4.6244e-15       0         Exceeds abs tol 4.4409e-16 by 4.1803e-15

 PASS   11/15   FAIL 4

...

  scripts/general/num2str.m ..............................
ASSERT errors for:  assert (num2str (1e23),"100000000000000000000000")

  Location  |  Expected  |  Observed  |  Reason
     []       100000000000000000000000   99999999999999991611392    Strings don't match

 PASS   23/23

(Not sure why this indicates all 23 pass if there was a failure.  Testing the "contrapositive" perhaps.)

...

  scripts/plot/clf.m .....................................
ASSERT errors for:  assert (get (hf, "papertype"),"usletter")

  Location  |  Expected  |  Observed  |  Reason
     []        usletter     tabloid      Strings don't match

 PASS    3/3

(Again, all 3 pass.)

...

  scripts/polynomial/residue.m ...........................
ASSERT errors for:  assert (br,b,1e-8)

  Location  |  Expected  |  Observed  |  Reason
     .            E            O         Dimensions don't match

 PASS    5/5

(All pass)

...

Summary:

  PASS     11010
  FAIL         7
  XFAIL        4
  SKIPPED      9


So, the first occurrence has 4 fails.  The other three occurrences indicate no fails, but the tally of summary statistics indicates 7 fials.  Seems odd.

Dan Sebald <sebald>
Mon 05 Aug 2013 02:36:11 AM UTC, comment #7: 

I revamped things a bit and created a table format output.  There is probably some tweaking to do, but it is close to what I think we are converging on.

With regards to what we discussed, this patch achieves:

1) Accumulating errors in one test depth and across multiple recursion depths.  Waiting until the very end to print out results and give just one single error message indicating either "general errors" or the error message if there was only a single error message.

2) Displaying only the associated values that are different and related info in what I think is a nicely formatted table.  If the compared items are an array, the index is one dimensional.  If the compared items are a matrix, the index is two dimensional.  If the compared items are cells, the index is '{}'.  If the compared items are a structure, the index is '.'.  Something like that.

3) Multiple tables for recursions.

If one does "test assert", all the results where there was a fail will be displayed.  Here are some that are more noteworthy:


octave:226> test assert

ASSERT errors for:  assert (zeros (3, 0),zeros (0, 2))

  Location  |  Expected  |  Observed  |  Reason
     .            E            O         Dimensions don't match


ASSERT errors for:  assert (3 + 2 * eps,3,eps)

  Location  |  Expected  |  Observed  |  Reason
     1            3            3         Exceeds abs tol 2.2204e-16 by 2.2204e-16


ASSERT errors for:  assert ([2; 2; 3],[1; 2; 3])

  Location  |  Expected  |  Observed  |  Reason
     1            2            1         Exceeds abs tol 0 by 1


ASSERT errors for:  assert (100 + 300 * eps,100,-2 * eps)

  Location  |  Expected  |  Observed  |  Reason
     ()           E            O         Max rel err 7.10543e-16 exceeds tol 4.44089e-16


ASSERT errors for:  assert (NaN,1)

  Location  |  Expected  |  Observed  |  Reason
     1           NaN           1         'NaN' mismatch


ASSERT errors for:  assert (NA,1)

  Location  |  Expected  |  Observed  |  Reason
     1            NA           1         'NA' mismatch
     1            NA           1         'NaN' mismatch


ASSERT errors for:  assert (-Inf,Inf)

  Location  |  Expected  |  Observed  |  Reason
     1           -Inf         Inf        'Inf' mismatch


ASSERT errors for:  assert ("dog","cat")

  Location  |  Expected  |  Observed  |  Reason
     []          cat          dog        Strings don't match


ASSERT errors for:  assert (3,"dog")

  Location  |  Expected  |  Observed  |  Reason
     []          dog           3         Expected string, but observed number


ASSERT errors for:  assert (cellstr ("dog"),"dog")

  Location  |  Expected  |  Observed  |  Reason
     []          dog           {}        Expected string, but observed cell


ASSERT errors for:  assert (cell2struct ({"dog"; 3}, {"pet", "age"}, 1),"dog")

  Location  |  Expected  |  Observed  |  Reason
     []          dog           []        Expected string, but observed struct


ASSERT errors for:  assert (x,3)

  Location  |  Expected  |  Observed  |  Reason
     ()           E            O         Class struct != double


ASSERT errors for:  assert (cond {i},expected {i},tol)

  Location  |  Expected  |  Observed  |  Reason
     []          cat          dog        Strings don't match


ASSERT errors for:  assert (cond {i},expected {i},tol)

  Location  |  Expected  |  Observed  |  Reason
     1            3            2         Exceeds abs tol 0 by 1


ASSERT errors for:  assert (cond {i},expected {i},tol)

  Location  |  Expected  |  Observed  |  Reason
     1           100          101        Exceeds abs tol 0 by 1


ASSERT errors for:  assert (cond {i},expected {i},tol)

  Location  |  Expected  |  Observed  |  Reason
     1            1            0         Exceeds abs tol 0 by 1


ASSERT errors for:  assert (cond {i},expected {i},tol)

  Location  |  Expected  |  Observed  |  Reason
     []          cat          dog        Strings don't match


ASSERT errors for:  assert (y1,y2 + eps * 1e-70,eps (y1))

  Location  |  Expected  |  Observed  |  Reason
     1          1e-80        1e-80       Exceeds abs tol 1.8727e-96 by 2.2204e-86
     2          1e-78        1e-78       Exceeds abs tol 1.1985e-94 by 2.2204e-86
     3          1e-76        1e-76       Exceeds abs tol 1.5341e-92 by 2.2204e-86
     4          1e-74        1e-74       Exceeds abs tol 1.9636e-90 by 2.2201e-86
     5          1e-72        1e-72       Exceeds abs tol 1.2567e-88 by 2.2244e-86

PASSES 60 out of 60 tests


Let me know what you think and we'll improve the formatting some.

Dan


(file #28761)

Dan Sebald <sebald>
Fri 02 Aug 2013 06:54:30 PM UTC, comment #6: 

I wasn't aware of the plus format, but I'm not sure it is that informative.  It's definitely condensed, but it is difficult to pick out exactly what entry it might be in a matrix.

I like your idea of a tabular form of display.  I've been trying to think of a way to reuse existing display features, but it is probably easier to just write a loop and create a table of this sort.  What I like about it is that there is only one position entry that tells right away what element is being compared, rather than have to look back and forth between two "cell display" lists to find the, say, [2,1] and [2,1].  The numbers that do not match are right next to one another.  Also, the last column gives insight into what might be wrong, especially if it were to include the degree of failure.  If I saw just a tiny discrepancy, I might guess it is a machine precision issue.

I think the column names are good ("observed" rather than "got", perhaps) and maybe just a little more condensed explanation.  I think there'd be sufficient space in 80 characters.  For example:


Location  |  Expected  |  Observed  |  Reason
[1,2]           2            2.0       Exceeds abs tol 0 by 2.174e-25


Dan Sebald <sebald>
Fri 02 Aug 2013 06:00:03 PM UTC, comment #5: 

What about using the 'plus' format for the differences matrix?


x = [1 2; 3 4]; y = [1 3; 2 4];
tol = 0.1;
result = x - y;
bad_idx = abs (result) > tol;
format plus "** ";
disp (bad_idx);
format short;


I like the idea of displaying the indices of the mismatches.  I think it might be even better if we moved from a vertical to a horizontal format.  Right now, it is


Expected:
...
...
...
Got
...
...
...
Tolerance exceeded by
...
...
...


But, I think this would be better.


Location  |  Expected      |     Got     |  Reason
[1,2]           2                 3         Exceeds absolute tolerance 0


We should use shorter names, but this is the idea.

Rik <rik5>
Group administrator
Fri 02 Aug 2013 04:10:37 PM UTC, comment #4: 

The color attribute would be a really nice feature, and her are some examples of controlling the color before using an ostream:

http://www.cplusplus.com/forum/beginner/5830/

but it just seems like a lot of work given the many Octave classes and associate print methods, at least more work than what the benefit would be.

Dan Sebald <sebald>
Fri 02 Aug 2013 08:13:07 AM UTC, comment #3: 

Yeah, I suppose you are right.  A vector containing only the different values with no position information is somewhat arcane.

Printing zeros for entries that match is easy enough, but it's a little more confusing than, say, "*".  I better way to show the elements that are different would be to display the contents similar to the current "display()", but make the font color for those elements that are different red.  Not sure that is possible right now.  A simple way to do this might be modifying "display()" to have optional input color matrices the same dimensions as the matrix, e.g.,:

display (X, R, G, B)

The color used for a particular element is gotten via the corresponding elements in R, G and B.

Another approach that might be more condensed would be to write the contents of the matrices that are different as text strings similar to what a cell looks like.  That way, people will be used to the format.  Your example would look like:


Expected:

exp =
{
  [2,1] =  2
  [1,2] =  3
}

but got:

obs =
{
  [2,1] =  3
  [1,2] =  2
}


As far as the stack trace depth, I typically only ever use the first level, i.e., the first trace statement following the error message.  It would nice, generally, to have a configuration option for the number of levels/calls to display in the trace.

Dan Sebald <sebald>
Fri 02 Aug 2013 12:35:48 AM UTC, comment #2: 

I took a look at this patch.  It seems generally like a good idea, although I see a need for some tweaking before applying.

I benchmarked it and it doesn't slow down the test suite, so performance is not a concern.

Two things I had a trouble with
1) With matrices, not vectors or scalars, I have a hard time seeing what is different
2) There are now two levels of stack trace reported.

For the first issue, try


obs = [1 3; 2 4];
exp = [1 2; 3 4];

assert (obs, exp)
error: assert (obs,exp) expected
   2
   3
but got
   3
   2
values do not match
error: called from:
error:   /home/rik/wip/Projects_Mine/octave-dev/scripts/testfun/assert.m at line 356, column 5
error:   /home/rik/wip/Projects_Mine/octave-dev/scripts/testfun/assert.m at line 202, column 9


With the old code this results in


error: assert (obs, exp) expected
   1   3
   2   4
but got
   1   2
   3   4
values do not match
error: called from:
error:   /home/rik/downloads/local/share/octave/3.7.5/m/testfun/assert.m at line 235, column 5


Maybe there is a way to pretty print the matrix with an '*' or some other character by the values which are mismatching.  Another possibility would be to print out a matrix of zeros, sized the same as obs and exp, with entries only in positions where a mismatch was found.  Thinking about it, this sounds easier.

The extra stack on the backtrace is only slightly annoying, but I'd still like to get rid of it.  Often tests are already buried pretty deeply in the call stack and it all gets reported when assert errors out.

Rik <rik5>
Group administrator
Thu 11 Jul 2013 01:05:58 AM UTC, comment #1: 

I missed a couple "iserror" occurrences.  New patch with minor tweaks.

(file #28536)

Dan Sebald <sebald>
Wed 10 Jul 2013 09:02:05 AM UTC, original submission:  

Only the comparisons which fall outside of tolerance are displayed.  The error messages are formatted better and make sense in terms of what is out of tolerance and by how much.

For that reason I've removed the FIXME about formatting at the top of the script file.  Let's tweak the output format as desired and be done with that.  I think it is pretty close as changed.

Instead of keeping track of error state via variable "iserror" all the way to the end of the routine, it uses a subfunction for printing the error.  In addition to a little more flexibility of output error message, this has the added advantage of displaying the line number about where the tests fails.  With the former approach, the line number was always somewhere at the end of the file.

The varying tolerance comes in handy for testing function results for which there is a wide numerical range--when a large tolerance value (say eps) doesn't really test most of the routine and a small tolerance value (say eps(min(x)) is so small that the vast majority of the other values will fail.  To illustrate, here is a test for the recent changes to binocdf.m:


>> p = 1-1e-3;
>> pval = binopdf ([0:50]', 50, p);
>> assert (cumsum (pval), binocdf ([0:50]', 50, p), 1000 * eps (pval))


Notice that the comparisons are within tolerance for all values, and I've had to relax the tolerance a bit because as we discovered with the discussions about binocdf there are different numerical techniques at the heart of the two routines binopdf and binocdf.

The following is what results on binocdf.m prior to the recent change:


>> assert (cumsum (pval), binocdf ([0:50]', 50, p), 1000 * eps (pval))
error: assert (cumsum (pval),binocdf ([0:50]', 50, p),1000 * eps (pval)) expected
   1.0000e-150
   4.9951e-146
   1.2226e-141
   1.9542e-137
   2.2940e-133
   2.1084e-129
   1.5798e-125
   9.9203e-122
   5.3270e-118
   2.4835e-114
   1.0172e-110
   3.6955e-107
   1.1999e-103
   3.5039e-100
    9.2515e-97
    2.2182e-93
    4.8477e-90
    9.6862e-87
    1.7741e-83
    2.9851e-80
    4.6225e-77
    6.5974e-74
    8.6884e-71
    1.0567e-67
    1.1877e-64
    1.2341e-61
    1.1855e-58
    1.0528e-55
    8.6403e-53
    6.5488e-50
    4.5801e-47
    2.9523e-44
    1.7514e-41
    9.5451e-39
    4.7686e-36
    2.1781e-33
    9.0685e-31
    3.4287e-28
    1.1721e-25
    3.6041e-23
    9.9053e-21
    2.4146e-18
    5.1720e-16
    9.6196e-14
    1.5303e-11
    2.0408e-09
    2.2198e-07
but got
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   0.0000e+00
   5.5511e-16
   9.6145e-14
   1.5303e-11
   2.0408e-09
   2.2198e-07
maximum absolute error 1e-150 exceeds tolerance 1.35666e-163
maximum absolute error 4.9951e-146 exceeds tolerance 8.89103e-159
maximum absolute error 1.2226e-141 exceeds tolerance 1.45671e-154
maximum absolute error 1.95425e-137 exceeds tolerance 2.38667e-150
maximum absolute error 2.294e-133 exceeds tolerance 3.91032e-146
maximum absolute error 2.10842e-129 exceeds tolerance 3.20333e-142
maximum absolute error 1.57977e-125 exceeds tolerance 2.62417e-138
maximum absolute error 9.92031e-122 exceeds tolerance 2.14972e-134
maximum absolute error 5.32698e-118 exceeds tolerance 8.80525e-131
maximum absolute error 2.48351e-114 exceeds tolerance 3.60663e-127
maximum absolute error 1.01725e-110 exceeds tolerance 1.47728e-123
maximum absolute error 3.69551e-107 exceeds tolerance 6.05092e-120
maximum absolute error 1.19988e-103 exceeds tolerance 2.47846e-116
maximum absolute error 3.50395e-100 exceeds tolerance 5.07588e-113
maximum absolute error 9.25151e-97 exceeds tolerance 1.03954e-109
maximum absolute error 2.21823e-93 exceeds tolerance 4.25796e-106
maximum absolute error 4.84772e-90 exceeds tolerance 8.7203e-103
maximum absolute error 9.68615e-87 exceeds tolerance 1.78592e-99
maximum absolute error 1.7741e-83 exceeds tolerance 3.65756e-96
maximum absolute error 2.98511e-80 exceeds tolerance 3.74534e-93
maximum absolute error 4.62254e-77 exceeds tolerance 7.67046e-90
maximum absolute error 6.59738e-74 exceeds tolerance 7.85455e-87
maximum absolute error 8.68837e-71 exceeds tolerance 1.60861e-83
maximum absolute error 1.05672e-67 exceeds tolerance 1.64722e-80
maximum absolute error 1.1877e-64 exceeds tolerance 1.68675e-77
maximum absolute error 1.23407e-61 exceeds tolerance 1.72723e-74
maximum absolute error 1.18551e-58 exceeds tolerance 1.76869e-71
maximum absolute error 1.05282e-55 exceeds tolerance 1.81114e-68
maximum absolute error 8.64034e-53 exceeds tolerance 1.8546e-65
maximum absolute error 6.54884e-50 exceeds tolerance 9.49557e-63
maximum absolute error 4.58011e-47 exceeds tolerance 9.72346e-60
maximum absolute error 2.95232e-44 exceeds tolerance 4.97841e-57
maximum absolute error 1.75142e-41 exceeds tolerance 2.54895e-54
maximum absolute error 9.54508e-39 exceeds tolerance 1.30506e-51
maximum absolute error 4.76856e-36 exceeds tolerance 6.68191e-49
maximum absolute error 2.17814e-33 exceeds tolerance 3.42114e-46
maximum absolute error 9.06846e-31 exceeds tolerance 1.75162e-43
maximum absolute error 3.42871e-28 exceeds tolerance 4.48416e-41
maximum absolute error 1.17214e-25 exceeds tolerance 2.29589e-38
maximum absolute error 3.60415e-23 exceeds tolerance 5.87747e-36
maximum absolute error 9.90534e-21 exceeds tolerance 1.50463e-33
maximum absolute error 2.41464e-18 exceeds tolerance 3.85186e-31
maximum absolute error 3.7911e-17 exceeds tolerance 9.86076e-29
maximum absolute error 5.02326e-17 exceeds tolerance 1.26218e-26
maximum absolute error 7.983e-18 exceeds tolerance 3.23117e-24
maximum absolute error 5.36716e-17 exceeds tolerance 4.1359e-22
maximum absolute error 1.89715e-17 exceeds tolerance 2.64698e-20


Notice that only the comparisons that failed are shown so that one doesn't have to weed through all the values.  Also notice that the comments about the error and tolerance now make sense, and furthermore the tolerance nicely scales relative to the magnitude of the expected item.  In this case, Rik knew about the string of zeros at the front.  However, I think this sort of setup would be a good test for the plethora of other routines using 1 - betainc().

Dan Sebald <sebald>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #28844:  octave-assert_call_depth-2013aug14.patch added by sebald (2KiB - application/octet-stream)
file #28827:  octave-assert_vector_tolerance-2013aug12.patch added by sebald (18KiB - application/octet-stream)
file #28813:  octave-assert_vector_tolerance-2013aug11.patch added by sebald (17KiB - application/octet-stream)
file #28770:  octave-assert_vector_tolerance-2013aug05.patch added by sebald (15KiB - application/octet-stream)
file #28761:  octave-assert_vector_tolerance-2013aug04.patch added by sebald (15KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by sebald (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-08-14 sebald Attached File- Added octave-assert_call_depth-2013aug14.patch, #28844
    2013-08-14 rik5 StatusNone Done
        Open/ClosedOpen Closed
    2013-08-12 sebald Attached File- Added octave-assert_vector_tolerance-2013aug12_2.patch, #28828
    2013-08-12 sebald Attached File- Added octave-assert_vector_tolerance-2013aug12.patch, #28827
    2013-08-11 sebald Attached File- Added octave-assert_vector_tolerance-2013aug11.patch, #28813
    2013-08-05 sebald Attached File- Added octave-assert_vector_tolerance-2013aug05.patch, #28770
    2013-08-05 sebald Attached File- Added octave-assert_vector_tolerance-2013aug04.patch, #28761
    2013-07-11 sebald Attached File- Added octave-assert_vector_tolerance-2013jul11.patch, #28536
    2013-07-10 sebald Attached File- Added octave-assert_vector_tolerance-2013jul10.patch, #28527

    Back to the top

    Powered by Savane 3.13-caa5.
    Corresponding source code