bugGNU Octave - Bugs: bug #49507, test: allow unit tests to signal...

 
 

bug #49507: test: allow unit tests to signal that a test is expected to fail based on arbitrary runtime check

Submitted by:  Mike Miller <mtmiller>
Submitted on:  Tue 01 Nov 2016 05:58:09 PM UTC  
 
Category: Octave FunctionSeverity: 1 - Wish
Priority: 5 - NormalItem Group: Feature Request
Status: FixedAssigned to: None
Originator Name: Open/Closed: Closed
Release: devOperating System: Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun 19 Mar 2017 04:08:58 PM UTC, comment #6:

That change looks like it does what I was proposing. I haven't tried it but it looks right to me.

Mike Miller <mtmiller>
Project Administrator
Sun 19 Mar 2017 06:27:03 AM UTC, comment #5:

Has this bug been fixed by jwe's changeset here:

For example, part of jwe's cset looks like this now:

Rik <rik5>
Project Administrator
Tue 06 Dec 2016 05:51:07 PM UTC, comment #4:

Something like the attached patch might do the job. It uses the error id proposed by Mike to make test skippable conditionally on runtime.
I also attached a small test file the uses that error id (unconditionally).

Still un-documented.

(file #39162, file #39163)

Markus Mützel <mmuetzel>
Project Member
Tue 06 Dec 2016 03:13:15 AM UTC, comment #3:

Oh yeah, there might be packages using the HAVE_XYZ syntax which would mean a headache to track down any change there.

Dan Sebald <sebald>
Tue 06 Dec 2016 03:05:35 AM UTC, comment #2:

I looked at test.m a bit. I understand the proposal, but I'm not sure how well that would work. I suppose it is doable. ID variable is not well documented in the error() function, so I'm not sure if that is supposed to be a printf ID (i.e., output) or an error type. So, would test.m be looking for a particular ID? Or would it have to scan the error message for a word such as "skip"?

Another thought is if the "testif" feature were slightly more robust, it might cover everything needed. For example, negation of test requirements and evaluation of strings might provide everything needed. For example, in a related gnuplot issue for publish.tst, a solution might be:

%!testif {"!HAVE_OSMESA", "any(strcmp(available_graphics_toolkits, 'gnuplot')"}

which would check for HAVE_OSMESA defined to 0 in a slight variation of the current test.m behavior, but also it would evaluate any string not of that form.

In some sense it would be simpler if instead of the slightly complex code for decomposing HAVE_OSMESA there were simply an Octave function called HAVE('OSMESA') that basically does the same thing. That would mean that there is then no special consideration for HAVE_XYZ, e.g.,

%!testif {"!HAVE('OSMESA')", "any(strcmp(available_graphics_toolkits, 'gnuplot')"}

I would think this is a very straightforward modification of test.m. Also there doesn't seem to be but a dozen script files using the HAVE_XYZ syntax in their tests? Those could be modified easily with opening the file in an editor and using FIND/REPLACE.

Dan Sebald <sebald>
Wed 02 Nov 2016 01:17:51 AM UTC, comment #1:

As a first stab at a proposal, I think it would be nice to use a particular error ID that is defined by Octave to be caught by the test function. A test that needs to depend on some feature at runtime could do something like

This would require the test.m function to look for that particular error ID (using lasterror, since we don't implement exception objects yet), and set appropriate local variables to indicate that this is a skipped test.

This error ID would be documented and advertised so that Forge packages and user code could also use this error ID to get the same results using test.

Other thoughts?

Mike Miller <mtmiller>
Project Administrator
Tue 01 Nov 2016 05:58:09 PM UTC, original submission:

This was discussed on the mailing list but not copied to a bug report.

It would be nice if unit test blocks could run arbitrary code and based on some runtime condition, raise a particular error or return a particular exit status that indicates to the test runner that the test is expected to fail / skip / do not report as a failure.

This came up in the context of Java tests. All tests are conditioned on HAVE_JAVA, which is fine as a compile-time condition, but also need to test whether the JRE can be loaded at runtime.

Similar tests could occur for existence of gnuplot, or existence of a DISPLAY / headless environment (Octave function have_window_system()), or for existence of optional functions, runtime libraries, etc.

Mike Miller <mtmiller>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #39162:  bug49507_skiptests.patch added by mmuetzel (1KiB - application/octet-stream)
file #39163:  test_skiptest.m added by mmuetzel (162B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by mmuetzel (Updated the item)
  • -unavailable- added by sebald (Posted a comment)
  • -unavailable- added by cbm
  • -unavailable- added by mtmiller (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 20 Mar 2017 04:58:41 PM UTCrik5StatusNeed Info=>Fixed
      Open/ClosedOpen=>Closed
    Sun 19 Mar 2017 06:27:03 AM UTCrik5StatusConfirmed=>Need Info
    Tue 06 Dec 2016 05:51:07 PM UTCmmuetzelAttached File-=>Added bug49507_skiptests.patch, #39162
      Attached File-=>Added test_skiptest.m, #39163
    Tue 01 Nov 2016 06:50:50 PM UTCcbmCarbon-Copy-=>Added cbm

    Back to the top


    Powered by Savane 3.1-cleanup1