bugGNU Octave - Bugs: bug #56325, Make runtests Matlab compatible

 
 

bug #56325: Make runtests Matlab compatible

Submitter:  Kai Torben Ohlhus <siko1056>
Submitted:  Wed 15 May 2019 02:48:48 PM UTC
   
 
Category:  Octave Function Severity:  1 - Wish
Priority:  3 - Low Item Group:  Feature Request
Status:  Wont Fix Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 5.1.0 Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 28 May 2019 07:19:28 AM UTC, comment #6: 

Thanks for the deprecation Rik.

Here are two pointers to discussions on the mailing list:

https://lists.gnu.org/archive/html/octave-maintainers/2019-05/msg00080.html
https://lists.gnu.org/archive/html/octave-maintainers/2019-05/msg00084.html

I think there is nothing more that can be done at the moment, except for reimplementing the whole function and Matlab testing framework, which is not likely to happen soon.

Kai Torben Ohlhus <siko1056>
Group Member
Sun 26 May 2019 01:41:12 AM UTC, comment #5: 

I deprecated 'runtests' for the upcoming version 6 of Octave in this changeset (https://hg.savannah.gnu.org/hgweb/octave/rev/8a04f5454350).  The functionality has been preserved in the new function 'oruntests' which does not clash with the Matlab function name 'runtests'.

Rik <rik5>
Group administrator
Sat 18 May 2019 12:07:44 AM UTC, comment #4: 

Matlab's runtests() is part of Matlab's xUnit-based Unit Testing framework. It's not just "not 100% compatible" with Octave's existing BIST testing framework; it's closer to 0% compatible, both in terms of how the tests are defined, and the functions and data structures used to run the tests and present results.

I think that Matlab compatibility is a strong reason for giving up the "runtests" function name and reserving that for a future Matlab-compatible test-running function. But the existing BIST framework is both extensive and valuable, so I think Octave should just rename its current "runtests" to "runbists" or something else that doesn't collide with the Matlab style "runtests", rather than drop it entirely or commit to porting its tests over to the new Matlab style.

> Testify should definitely drop the name, unless it wants to shadow the function... but that sounds like a bad idea.


Testify is intended to potentially be a replacement for Octave's current BIST/runtests(), so it uses a similarly-named "runtests2" function. (The "2" suffix is just to avoid shadowing Octave's own "runtests"; there's no real significance to it.) If Octave renames its "runtests", then Testify will follow suit.

Also, If there's interest in having an Octave implementation of a Matlab-compatible xUnit-style framework, I will add one to Testify. Comment here if you want it to happen: https://github.com/apjanke/octave-testify/issues/5.

I think both the BIST style tests and xUnit style tests (and doctest style tests, too!) could comfortably coexist in Octave. Ideally, probably the best end state is to have one unified "runalltests" function that can discover and run tests of both formats, so you can run all your tests with a single call, and get a unified results output. (You could probably even change the BIST runner implementation to do dymanically-constructed tests in terms of the xUnit style data/control structures, keeping the existing BIST definitions but running them under a new-style framework.) I don't know if that unified function could be called "runtests", though, because Matlab has claimed that part of the API namespace, along with its signature.

Andrew Janke <apjanke>
Fri 17 May 2019 08:06:28 PM UTC, comment #3: 

Thread on maintainers mailing list for discussion: https://lists.gnu.org/archive/html/octave-maintainers/2019-05/msg00080.html

Responses are welcome here as well.

I have no need for the Matlab runtests function, but if we are considering the name a reserved word from the Matlab API, then we should deprecate and delete our runtests function sooner rather than later. If no one cares strongly about keeping runtests in Octave, I would move it to deprecated status for the Octave 6 release.

Mike Miller <mtmiller>
Group Member
Thu 16 May 2019 07:38:10 AM UTC, comment #2: 

I think runtests should as of now be considered a missing function, and its name reserved.

Testify should definitely drop the name, unless it wants to shadow the function... but that sounds like a bad idea.

Juan Pablo Carbajal <juanpi>
Group Member
Wed 15 May 2019 09:28:22 PM UTC, comment #1: 

Is your suggestion here that Octave's 'runtests' function should be modified to be fully compatible, to operate on Matlab style test scripts, and the current behavior should be dropped?

If so, then I guess we need to define a new Octave specific test runner function to serve the current purposes.

The 'testify' package is currently working on major improvements to Octave's testing framework, so maybe it should give up the 'runtests' name as well.

Mike Miller <mtmiller>
Group Member
Wed 15 May 2019 02:48:48 PM UTC, original submission:  

Octave introduced the function runtests in 2010 and Matlab decided to create this function in R2013b [1], which is not 100% compatible and recently created problems with the matgeom package, that wants to have a compatible test suite for both.

Anyways there is an outstanding bug #44303 about duplicate test suite code, which can be regarded as well.

[1] https://www.mathworks.com/help/matlab/ref/runtests.html
[2] https://lists.gnu.org/archive/html/octave-maintainers/2019-05/msg00071.html

Kai Torben Ohlhus <siko1056>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by juanpi (Posted a comment)
  • -email is unavailable- added by siko1056 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-05-28 siko1056 StatusNone Wont Fix
        Open/ClosedOpen Closed
        Releasedev 5.1.0
    2019-05-17 mtmiller Priority5 - Normal 3 - Low
        Item GroupMatlab Compatibility Feature Request
    2019-05-15 siko1056 Dependencies- Depends on bugs #44303

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code