bugmake - Bugs: bug #43414, Perl test script bugs/enhancements

 
 

bug #43414: Perl test script bugs/enhancements

Submitter:  John Malmberg <wb8tyw>
Submitted:  Tue 14 Oct 2014 01:01:53 PM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.1 Operating System:  Any
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 14 Oct 2014 01:01:53 PM UTC, original submission:  

1. The run_perl_tests.pl does not print out the number of tests attempted unless all tests pass.  It should always print the number of tests attempted.  This makes it harder to determine how good or bad results are until all tests are passing.

2. When a test script skips an individual test, the count of failed tests at the end of the run is changed by the number of tests skipped which is incorrect because failed tests are counted differently.

If you manually add up the number of tests failed from the report you will get a different count than on the summary if any scripts have individual tests skipped.  This is probably only noticed on non-Unix platforms.

3. It would be nice for tests to check the test working directory for files that were not present before the test after each test.  VMS also needs to check "/tmp" and "sys$scratch:" if different than "/tmp".  This is especially needed for VMS because make creates helper scripts in 'sys$scratch:' and their are still bugs in the VMS port where extra files are created or not cleaned up.

4. For each test in a script, the possible status values should be: Pass, Fail, xPass, xFail, and skipped.
xFail is for something known to be broken or unimplemented.
skipped is for a test that should never be run on a platform usually because it would never be applicable.
xPass is short for unexpected passes, where a test passed even though it was expected to fail.  This will catch if someone fixes a platform specific issue, but does not update the test scripts to match.

On the individual tests, it should report if any xFail tests actually passed, as it is possible that they could get fixed as a side effect of a different bugfix.

The summary should always report:
Total count of tests run.
Count of tests passed (includes tests unexpectedly passed)
Count of tests failed (includes tests unexpectedly failed)
Count of tests skipped.
It should then sanity check that Passed + failed + skipped == tests run.
It should then also report the number of expected test failures and the number of unexpected test passes.

John Malmberg <wb8tyw>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Updated the item)
  • -email is unavailable- added by wb8tyw (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-12-27 psmith Item GroupBug Enhancement

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code