bugGNU Octave - Bugs: bug #55522, Add failure summary to...

 
 

bug #55522: Add failure summary to __run_test_suite__ output?

Submitter:  Andrew Janke <apjanke>
Submitted:  Sun 20 Jan 2019 05:58:40 AM UTC
   
 
Category:  Test Suite Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Feature Request
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 22 Jan 2019 03:24:11 PM UTC, comment #3: 

Implementing Matlab's OO test framework is a larger project; I see this patch as just an improvement to Octave's existing test output, and the class as just an internal convenience for _run_test_suite_. I'd rather just add a test summary here and look at doing their OO testing framework separately, probably as a package first. Matlab's Unit Test framework isn't going to be compatible with Octave's BISTs anyway (unless we extend it), because it doesn't have a notion of XFails, regressions, or feature/runtime-condition skips.

Do you want to use the +matlab namespace for internal classes and/or Octave classes that don't have corresponding Matlab ones, too?

 Andrew Janke <apjanke>
Tue 22 Jan 2019 08:21:02 AM UTC, comment #2: 

Thanks for your contribution. It is indeed a good idea to get faster feedback about failing tests, than scrolling back or looking at fntests.log.

In your changesets you introduce a new classdef:


scripts/testfun/+octave/+test/+internal/TestSuiteResult


Although it destroys creativity.  I think if we want to overhaul Octave's testing framework, we should look early at Matlab: https://www.mathworks.com/help/matlab/class-based-unit-tests.html
Otherwise we later have the pain to become Matlab compatible and remove duplication.

There is already something similar (not 100% of course) to your "TestSuiteResult" class. And Octave 5.0 has introduced the "+matlab" namespace.  So maybe you can try to implement and make use of


matlab.unittest.TestResult
scripts/+matlab/+unittest/TestResult


https://www.mathworks.com/help/matlab/ref/matlab.unittest.testresult-class.html
to get this overview done?

 Kai Torben Ohlhus <siko1056>
Group Member
Sun 20 Jan 2019 08:41:26 AM UTC, comment #1: 

And here's a refactoring to use an object-based test results, to avoid having to carry around 7 different variables to track the results. Includes nicer formatting for the failed file list, too.

file #46036:  test-suite-summary-output-OOP-v2.patch


  fixed/switch.tst ...............................................
  fixed/system.tst ...............................................
  fixed/transpose.tst ............................................
  fixed/try.tst ..................................................
  fixed/unwind.tst ...............................................
  fixed/while.tst ................................................

Summary:

  PASS                            14949
  FAIL                               16
  XFAIL (reported bug)               54
  SKIP (missing feature)             50
  SKIP (run-time condition)          29

  Failed tests:
     libinterp/corefcn/graphics.cc-tst
     libinterp/corefcn/help.cc-tst
     libinterp/corefcn/pr-output.cc-tst
     libinterp/corefcn/variables.cc-tst
     libinterp/parse-tree/oct-parse.in.yy-tst
     miscellaneous/inputname.m
     miscellaneous/unpack.m
     miscellaneous/zip.m
     ode/decic.m
     optimization/fminunc.m
     plot/util/copyobj.m
     plot/util/hgsave.m
     plot/util/printd.m
     set/ismember.m
     special-matrix/magic.m

See the file /Users/janke/Dropbox/computer/data/Documents/octave/fntests.log for additional details.



 Andrew Janke <apjanke>
Sun 20 Jan 2019 05:58:40 AM UTC, original submission:  

At the end of a run, _run_test_suite_ will output the number of failures, but it won't list which files had tests that failed, so you have to scroll back an manually scan through its long output.

How about adding a list of the files with failed tests to the results summary, so it's easier to identify where you need to go looking for problems?

Here's a patch to do so.

After:


>> __run_test_suite__
[...]
  fixed/system.tst ............................................... PASS  103/103
  fixed/transpose.tst ............................................ PASS   16/16
  fixed/try.tst .................................................. PASS   18/18
  fixed/unwind.tst ............................................... PASS    2/2
  fixed/while.tst ................................................ PASS    6/6

Summary:

  PASS                            14948
  FAIL                               17
  XFAIL (reported bug)               54
  SKIP (missing feature)             50
  SKIP (run-time condition)          29

  Failed tests:
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/corefcn/help.cc-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/corefcn/pr-output.cc-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/corefcn/variables.cc-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/parse-tree/oct-parse.in.yy-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/image/getframe.m
[...]


 Andrew Janke <apjanke>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #46036:  test-suite-summary-output-OOP-v2.patch added by apjanke (16KiB - application/octet-stream)
file #46035:  test-suite-summary-output-OOP.patch added by apjanke (16KiB - application/octet-stream)
file #46034:  test_suite_add_failure_summary_to_results_output.patch added by apjanke (6KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by apjanke (Submitted the item)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-20 apjanke Attached File- Added test-suite-summary-output-OOP-v2.patch, #46036
    2019-01-20 apjanke Attached File- Added test-suite-summary-output-OOP.patch, #46035
    2019-01-20 apjanke Attached File- Added test_suite_add_failure_summary_to_results_output.patch, #46034

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.13-da57.
    Corresponding source code