bugGNU Octave - Bugs: bug #55522, Add failure summary to...

 
 

bug #55522: Add failure summary to __run_test_suite__ output?

Submitted by:  Andrew Janke <apjanke>
Submitted on:  Sun 20 Jan 2019 05:58:40 AM UTC  
 
Category:  Test Suite Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Feature Request
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Any

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 22 Jan 2019 03:24:11 PM UTC, comment #3: 

Implementing Matlab's OO test framework is a larger project; I see this patch as just an improvement to Octave's existing test output, and the class as just an internal convenience for _run_test_suite_. I'd rather just add a test summary here and look at doing their OO testing framework separately, probably as a package first. Matlab's Unit Test framework isn't going to be compatible with Octave's BISTs anyway (unless we extend it), because it doesn't have a notion of XFails, regressions, or feature/runtime-condition skips.

Do you want to use the +matlab namespace for internal classes and/or Octave classes that don't have corresponding Matlab ones, too?

Andrew Janke <apjanke>
Tue 22 Jan 2019 08:21:02 AM UTC, comment #2: 

Thanks for your contribution. It is indeed a good idea to get faster feedback about failing tests, than scrolling back or looking at fntests.log.

In your changesets you introduce a new classdef:

scripts/testfun/+octave/+test/+internal/TestSuiteResult

Although it destroys creativity.  I think if we want to overhaul Octave's testing framework, we should look early at Matlab: https://www.mathworks.com/help/matlab/class-based-unit-tests.html
Otherwise we later have the pain to become Matlab compatible and remove duplication.

There is already something similar (not 100% of course) to your "TestSuiteResult" class. And Octave 5.0 has introduced the "+matlab" namespace.  So maybe you can try to implement and make use of

matlab.unittest.TestResult
scripts/+matlab/+unittest/TestResult

https://www.mathworks.com/help/matlab/ref/matlab.unittest.testresult-class.html
to get this overview done?

Kai Torben Ohlhus <siko1056>
Project Member
Sun 20 Jan 2019 08:41:26 AM UTC, comment #1: 

And here's a refactoring to use an object-based test results, to avoid having to carry around 7 different variables to track the results. Includes nicer formatting for the failed file list, too.

file #46036:  test-suite-summary-output-OOP-v2.patch

  fixed/switch.tst ...............................................
  fixed/system.tst ...............................................
  fixed/transpose.tst ............................................
  fixed/try.tst ..................................................
  fixed/unwind.tst ...............................................
  fixed/while.tst ................................................

Summary:

  PASS                            14949
  FAIL                               16
  XFAIL (reported bug)               54
  SKIP (missing feature)             50
  SKIP (run-time condition)          29

  Failed tests:
     libinterp/corefcn/graphics.cc-tst
     libinterp/corefcn/help.cc-tst
     libinterp/corefcn/pr-output.cc-tst
     libinterp/corefcn/variables.cc-tst
     libinterp/parse-tree/oct-parse.in.yy-tst
     miscellaneous/inputname.m
     miscellaneous/unpack.m
     miscellaneous/zip.m
     ode/decic.m
     optimization/fminunc.m
     plot/util/copyobj.m
     plot/util/hgsave.m
     plot/util/printd.m
     set/ismember.m
     special-matrix/magic.m

See the file /Users/janke/Dropbox/computer/data/Documents/octave/fntests.log for additional details.

Andrew Janke <apjanke>
Sun 20 Jan 2019 05:58:40 AM UTC, original submission:  

At the end of a run, _run_test_suite_ will output the number of failures, but it won't list which files had tests that failed, so you have to scroll back an manually scan through its long output.

How about adding a list of the files with failed tests to the results summary, so it's easier to identify where you need to go looking for problems?

Here's a patch to do so.

After:

>> __run_test_suite__
[...]
  fixed/system.tst ............................................... PASS  103/103
  fixed/transpose.tst ............................................ PASS   16/16
  fixed/try.tst .................................................. PASS   18/18
  fixed/unwind.tst ............................................... PASS    2/2
  fixed/while.tst ................................................ PASS    6/6

Summary:

  PASS                            14948
  FAIL                               17
  XFAIL (reported bug)               54
  SKIP (missing feature)             50
  SKIP (run-time condition)          29

  Failed tests:
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/corefcn/help.cc-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/corefcn/pr-output.cc-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/corefcn/variables.cc-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/parse-tree/oct-parse.in.yy-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/image/getframe.m
[...]

Andrew Janke <apjanke>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #46036:  test-suite-summary-output-OOP-v2.patch added by apjanke (16KiB - application/octet-stream)
file #46035:  test-suite-summary-output-OOP.patch added by apjanke (16KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by apjanke (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-20 apjanke Attached File- => Added test-suite-summary-output-OOP-v2.patch, #46036
    2019-01-20 apjanke Attached File- => Added test-suite-summary-output-OOP.patch, #46035
    2019-01-20 apjanke Attached File- => Added test_suite_add_failure_summary_to_results_output.patch, #46034

    Back to the top


    Powered by Savane 3.5