bugmake - Bugs: bug #15942, test_driver.pl - run_each_test()...

 
 

bug #15942: test_driver.pl - run_each_test() fails to set $status when test dies, prints value of $status from prior test

Submitter:  greg keranen <gkeranen>
Submitted:  Wed 01 Mar 2006 11:26:46 AM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  4.0 Operating System:  Any
Fixed Release:  3.81 Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 10 Mar 2006 06:16:01 PM UTC, comment #5: 

I believe normal W32 was fixed in rc1, and cygwin builds will be fixed in the next release.

Perl provides a way to test whether the system supports symlinks and that's what I use originally.  For some reason, though, the Windows port of Perl reports that it DOES support symlinks even though it doesn't (not really).  So, I had to add an explicit check for W32.  I think that check was added in rc1.

Paul D. Smith <psmith>
Group administrator
Fri 10 Mar 2006 05:59:37 PM UTC, comment #4: 

Hi,
Has this just changed recently? I've seen the symlink tests fail every time I have ran the tests on MinGW/MSVC builds within MSYS. Admittedly I have not tried rc1, I only tried the betas.

Could the symlink tests be explicitly disabled if W32?
Kind regards
JG

J <now3d>
Fri 10 Mar 2006 05:22:19 PM UTC, comment #3: 

They are already disabled on any system where the $port_type is W32, and/or any system where symlinks don't work.  You can test this in Perl but the Windows port of Perl reports that they do work (odd).

If you find that this isn't working please let me know.

Paul D. Smith <psmith>
Group administrator
Fri 10 Mar 2006 11:51:51 AM UTC, comment #2: 

Can symlink checks be disabled on MinGW/MSVC builds when testing through MSYS (or Perl installed in a different environment on windows) please?
Kind regards
JG

J <now3d>
Fri 10 Mar 2006 01:02:01 AM UTC, comment #1: 

I implemented some of this.  I checked for $code being undefined and set $status properly if so, although I set it to be the same as if it were an error, which is how it's treated.  I left the string value as just "N/A"; this is printed after the test, like this:

   mytest ..................... $status

and I think just "N/A" works best here.

I changed the options/symlink test to return -1 for systems which don't support symlinks, like Windows.  Previously, as far as I can tell, that flag was not being used anywhere.

This is still a bit bogus because it only shows if a whole suite is not applicable, but...

Paul D. Smith <psmith>
Group administrator
Wed 01 Mar 2006 11:26:46 AM UTC, original submission:  

--- TEST ENVIRONMENT:
GNU Make 3.81rc1 on CYGWIN_NT-5.1 x2 1.5.19(0.150/4/2) i686

--- STEPS TO REPRODUCE
cd make-3.81rc1
make check

    Finding tests...
    [...]
    features/default_names ..................................
    * Test died (features/default_names): test_driver.pl: 453: abort at test_drive
    r.pl line 638.
   
    ok     (4 passed)

--- EXPECTED OUTPUT:
status should be set to "N/A"
   
--- ANALYSIS:
toplevel reports success 'ok (4 passed)' when default_names dies; should not say 'ok'.
On success, if $port_type eq 'UNIX', default_names should print:  'ok (3 passed)'
- this is the maximum number of tests that this script runs
On success, if $port_type eq 'W32', default_names should print:  'ok (2 passed)'

$status is not set at the top of the run_each_test() and neither is it set in the case where !defined($code)

--- CODE CAUSING THE PROBLEM:
tests/test_driver.pl function run_each_test()

[...]
    # How did it go?
    if (!defined($code))
    {
      $suite_passed = 0;

[...]
    }
   
    if ($suite_passed) {
      ++$categories_passed;
      $status = "ok     ($tests_passed passed)";
[...]
    }
    elsif ($code > 0) {
      $status = "FAILED ($tests_passed/$tests_run passed)";
    }
    elsif ($code < 0) {
      $status = "N/A";
      --$categories_run;
    }


--- PROPOSED FIX:
File attached: tests/test_driver.pl with modified function run_each_test()

NOTE:
    OLD:
        $status = "N/A";
    NEW:
        $status = "status = N/A";
    REASON:
        $status is only used for direct display to the user; simply printing "N/A" on a line by itself does not convey any clear meaning.

    # Run the actual test!
    $tests_run = 0;
    $tests_passed = 0;
    # 2006.02.28 -email is unavailable- added following line:
    $status = "status = N/A";
    $code = do $perl_testname;
[...]

    # 2006.02.28 -email is unavailable-
    # OLD: elsif ($code < 0) {
    elsif (($code < 0) || !defined($code) ) {
      $status = "status = N/A";
      --$categories_run;
    }

--- REGRESSION TEST
Proposed fix was applied and default_name set to die by run_make_tests.pl:

FILE: run_make_tests.pl set to force default_names to die:
   if ($osname =~ /Windows/i || $osname =~ /MINGW32/i ) {

OUTPUT from make check:
[...]
features/default_names .................................. Error running /cygdriv
e/c/greg/src/make-3.81rc1/tests/../make (expected 0; got 512): /cygdrive/c/greg/
src/make-3.81rc1/tests/../make  --no-print-directory -C /cygdrive/c/greg/src/mak
e-3.81rc1/tests/work/

* Test died (features/default_names): test_driver.pl: 455: compare_output fail
ed: Makefile at test_driver.pl line 643.

status = N/A
[...]

greg keranen <gkeranen>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #1759:  test_driver.pl added by gkeranen (30KiB - application/octet-stream - tests/test_driver.pl with modified function run_each_test())

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

Only logged-in users can vote.

 

Follow 6 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2006-04-01 psmith Fixed Release4.0 3.81
2006-03-10 psmith StatusNone Fixed
    Assigned toNone psmith
    Open/ClosedOpen Closed
    Fixed ReleaseNone 4.0
2006-03-01 gkeranen Attached File- Added test_driver.pl, #3458

Back to the top

Powered by Savane 3.13-f8d8.
Corresponding source code