bugGNU Octave - Bugs: bug #55522, Add failure summary to...


bug #55522: Add failure summary to __run_test_suite__ output?

Submitted by:  Andrew Janke <apjanke>
Submitted on:  Sun 20 Jan 2019 05:58:40 AM UTC  
Category:  Test Suite Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Feature Request
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Any

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Tue 22 Jan 2019 03:24:11 PM UTC, comment #3:

Implementing Matlab's OO test framework is a larger project; I see this patch as just an improvement to Octave's existing test output, and the class as just an internal convenience for _run_test_suite_. I'd rather just add a test summary here and look at doing their OO testing framework separately, probably as a package first. Matlab's Unit Test framework isn't going to be compatible with Octave's BISTs anyway (unless we extend it), because it doesn't have a notion of XFails, regressions, or feature/runtime-condition skips.

Do you want to use the +matlab namespace for internal classes and/or Octave classes that don't have corresponding Matlab ones, too?

Andrew Janke <apjanke>
Tue 22 Jan 2019 08:21:02 AM UTC, comment #2:

Thanks for your contribution. It is indeed a good idea to get faster feedback about failing tests, than scrolling back or looking at fntests.log.

In your changesets you introduce a new classdef:

Although it destroys creativity. I think if we want to overhaul Octave's testing framework, we should look early at Matlab: https://www.mathworks.com/help/matlab/class-based-unit-tests.html
Otherwise we later have the pain to become Matlab compatible and remove duplication.

There is already something similar (not 100% of course) to your "TestSuiteResult" class. And Octave 5.0 has introduced the "+matlab" namespace. So maybe you can try to implement and make use of

to get this overview done?

Kai Torben Ohlhus <siko1056>
Project Member
Sun 20 Jan 2019 08:41:26 AM UTC, comment #1:

And here's a refactoring to use an object-based test results, to avoid having to carry around 7 different variables to track the results. Includes nicer formatting for the failed file list, too.

file #46036: test-suite-summary-output-OOP-v2.patch

Andrew Janke <apjanke>
Sun 20 Jan 2019 05:58:40 AM UTC, original submission:

At the end of a run, _run_test_suite_ will output the number of failures, but it won't list which files had tests that failed, so you have to scroll back an manually scan through its long output.

How about adding a list of the files with failed tests to the results summary, so it's easier to identify where you need to go looking for problems?

Here's a patch to do so.


Andrew Janke <apjanke>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #46036:  test-suite-summary-output-OOP-v2.patch added by apjanke (16KiB - application/octet-stream)
file #46035:  test-suite-summary-output-OOP.patch added by apjanke (16KiB - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by apjanke (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-20 apjanke Attached File- => Added test-suite-summary-output-OOP-v2.patch, #46036
    2019-01-20 apjanke Attached File- => Added test-suite-summary-output-OOP.patch, #46035
    2019-01-20 apjanke Attached File- => Added test_suite_add_failure_summary_to_results_output.patch, #46034

    Back to the top

    Powered by Savane 3.4