bugGNU Octave - Bugs: bug #56325, Make runtests Matlab compatible


bug #56325: Make runtests Matlab compatible

Submitted by:  Kai Torben Ohlhus <siko1056>
Submitted on:  Wed 15 May 2019 02:48:48 PM UTC  
Category:  Octave Function Severity:  1 - Wish
Priority:  3 - Low Item Group:  Feature Request
Status:  Wont Fix Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  5.1.0 Operating System:  Any

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Tue 28 May 2019 07:19:28 AM UTC, comment #6: 

Thanks for the deprecation Rik.

Here are two pointers to discussions on the mailing list:


I think there is nothing more that can be done at the moment, except for reimplementing the whole function and Matlab testing framework, which is not likely to happen soon.

Kai Torben Ohlhus <siko1056>
Project Member
Sun 26 May 2019 01:41:12 AM UTC, comment #5: 

I deprecated 'runtests' for the upcoming version 6 of Octave in this changeset (https://hg.savannah.gnu.org/hgweb/octave/rev/8a04f5454350).  The functionality has been preserved in the new function 'oruntests' which does not clash with the Matlab function name 'runtests'.

Rik <rik5>
Project Administrator
Sat 18 May 2019 12:07:44 AM UTC, comment #4: 

Matlab's runtests() is part of Matlab's xUnit-based Unit Testing framework. It's not just "not 100% compatible" with Octave's existing BIST testing framework; it's closer to 0% compatible, both in terms of how the tests are defined, and the functions and data structures used to run the tests and present results.

I think that Matlab compatibility is a strong reason for giving up the "runtests" function name and reserving that for a future Matlab-compatible test-running function. But the existing BIST framework is both extensive and valuable, so I think Octave should just rename its current "runtests" to "runbists" or something else that doesn't collide with the Matlab style "runtests", rather than drop it entirely or commit to porting its tests over to the new Matlab style.

> Testify should definitely drop the name, unless it wants to shadow the function... but that sounds like a bad idea.

Testify is intended to potentially be a replacement for Octave's current BIST/runtests(), so it uses a similarly-named "runtests2" function. (The "2" suffix is just to avoid shadowing Octave's own "runtests"; there's no real significance to it.) If Octave renames its "runtests", then Testify will follow suit.

Also, If there's interest in having an Octave implementation of a Matlab-compatible xUnit-style framework, I will add one to Testify. Comment here if you want it to happen: https://github.com/apjanke/octave-testify/issues/5.

I think both the BIST style tests and xUnit style tests (and doctest style tests, too!) could comfortably coexist in Octave. Ideally, probably the best end state is to have one unified "runalltests" function that can discover and run tests of both formats, so you can run all your tests with a single call, and get a unified results output. (You could probably even change the BIST runner implementation to do dymanically-constructed tests in terms of the xUnit style data/control structures, keeping the existing BIST definitions but running them under a new-style framework.) I don't know if that unified function could be called "runtests", though, because Matlab has claimed that part of the API namespace, along with its signature.

Andrew Janke <apjanke>
Fri 17 May 2019 08:06:28 PM UTC, comment #3: 

Thread on maintainers mailing list for discussion: https://lists.gnu.org/archive/html/octave-maintainers/2019-05/msg00080.html

Responses are welcome here as well.

I have no need for the Matlab runtests function, but if we are considering the name a reserved word from the Matlab API, then we should deprecate and delete our runtests function sooner rather than later. If no one cares strongly about keeping runtests in Octave, I would move it to deprecated status for the Octave 6 release.

Mike Miller <mtmiller>
Project Administrator
Thu 16 May 2019 07:38:10 AM UTC, comment #2: 

I think runtests should as of now be considered a missing function, and its name reserved.

Testify should definitely drop the name, unless it wants to shadow the function... but that sounds like a bad idea.

Juan Pablo Carbajal <juanpi>
Project Member
Wed 15 May 2019 09:28:22 PM UTC, comment #1: 

Is your suggestion here that Octave's 'runtests' function should be modified to be fully compatible, to operate on Matlab style test scripts, and the current behavior should be dropped?

If so, then I guess we need to define a new Octave specific test runner function to serve the current purposes.

The 'testify' package is currently working on major improvements to Octave's testing framework, so maybe it should give up the 'runtests' name as well.

Mike Miller <mtmiller>
Project Administrator
Wed 15 May 2019 02:48:48 PM UTC, original submission:  

Octave introduced the function runtests in 2010 and Matlab decided to create this function in R2013b [1], which is not 100% compatible and recently created problems with the matgeom package, that wants to have a compatible test suite for both.

Anyways there is an outstanding bug #44303 about duplicate test suite code, which can be regarded as well.

[1] https://www.mathworks.com/help/matlab/ref/runtests.html
[2] https://lists.gnu.org/archive/html/octave-maintainers/2019-05/msg00071.html

Kai Torben Ohlhus <siko1056>
Project Member


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


   bug dependencies.

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by juanpi (Posted a comment)
  • -email is unavailable- added by siko1056 (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-05-28 siko1056 StatusNone => Wont Fix
        Open/ClosedOpen => Closed
        Releasedev => 5.1.0
    2019-05-17 mtmiller Priority5 - Normal => 3 - Low
        Item GroupMatlab Compatibility => Feature Request
    2019-05-15 siko1056 Dependencies- => Depends on bugs #44303

    Back to the top

    Powered by Savane 3.6