bugmake - Bugs: bug #15942, test_driver.pl - run_each_test()...

 
 

bug #15942: test_driver.pl - run_each_test() fails to set $status when test dies, prints value of $status from prior test

Submitted by:  greg keranen <gkeranen>
Submitted on:  Wed 01 Mar 2006 11:26:46 AM UTC  
 
Severity:  3 - Normal Item Group:  Bug
Status:  Fixed Privacy:  Public
Assigned to:  Paul D. Smith <psmith> Open/Closed:  Closed
Component Version:  4.0 Operating System:  Any
Fixed Release:  3.81 Triage Status:  None

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 10 Mar 2006 06:16:01 PM UTC, comment #5: 

I believe normal W32 was fixed in rc1, and cygwin builds will be fixed in the next release.

Perl provides a way to test whether the system supports symlinks and that's what I use originally.  For some reason, though, the Windows port of Perl reports that it DOES support symlinks even though it doesn't (not really).  So, I had to add an explicit check for W32.  I think that check was added in rc1.

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Fri 10 Mar 2006 05:59:37 PM UTC, comment #4: 

Hi,
Has this just changed recently? I've seen the symlink tests fail every time I have ran the tests on MinGW/MSVC builds within MSYS. Admittedly I have not tried rc1, I only tried the betas.

Could the symlink tests be explicitly disabled if W32?
Kind regards
JG

J <now3d>
Fri 10 Mar 2006 05:22:19 PM UTC, comment #3: 

They are already disabled on any system where the $port_type is W32, and/or any system where symlinks don't work.  You can test this in Perl but the Windows port of Perl reports that they do work (odd).

If you find that this isn't working please let me know.

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Fri 10 Mar 2006 11:51:51 AM UTC, comment #2: 

Can symlink checks be disabled on MinGW/MSVC builds when testing through MSYS (or Perl installed in a different environment on windows) please?
Kind regards
JG

J <now3d>
Fri 10 Mar 2006 01:02:01 AM UTC, comment #1: 

I implemented some of this.  I checked for $code being undefined and set $status properly if so, although I set it to be the same as if it were an error, which is how it's treated.  I left the string value as just "N/A"; this is printed after the test, like this:

   mytest ..................... $status

and I think just "N/A" works best here.

I changed the options/symlink test to return -1 for systems which don't support symlinks, like Windows.  Previously, as far as I can tell, that flag was not being used anywhere.

This is still a bit bogus because it only shows if a whole suite is not applicable, but...

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Wed 01 Mar 2006 11:26:46 AM UTC, original submission:  

--- TEST ENVIRONMENT:
GNU Make 3.81rc1 on CYGWIN_NT-5.1 x2 1.5.19(0.150/4/2) i686

--- STEPS TO REPRODUCE
cd make-3.81rc1
make check

    Finding tests...
    [...]
    features/default_names ..................................
    *** Test died (features/default_names): test_driver.pl: 453: abort at test_drive
    r.pl line 638.

   

    ok     (4 passed)

--- EXPECTED OUTPUT:
status should be set to "N/A"

   

--- ANALYSIS:
toplevel reports success 'ok (4 passed)' when default_names dies; should not say 'ok'.
On success, if $port_type eq 'UNIX', default_names should print:  'ok (3 passed)'
- this is the maximum number of tests that this script runs
On success, if $port_type eq 'W32', default_names should print:  'ok (2 passed)'

$status is not set at the top of the run_each_test() and neither is it set in the case where !defined($code)

--- CODE CAUSING THE PROBLEM:
tests/test_driver.pl function run_each_test()

[...]
    # How did it go?
    if (!defined($code))
    {
      $suite_passed = 0;

[...]
    }

   

    if ($suite_passed) {
      ++$categories_passed;
      $status = "ok     ($tests_passed passed)";
[...]
    }
    elsif ($code > 0) {
      $status = "FAILED ($tests_passed/$tests_run passed)";
    }
    elsif ($code < 0) {
      $status = "N/A";
      --$categories_run;
    }

--- PROPOSED FIX:
File attached: tests/test_driver.pl with modified function run_each_test()

NOTE:
    OLD:
        $status = "N/A";
    NEW:
        $status = "status = N/A";
    REASON:
        $status is only used for direct display to the user; simply printing "N/A" on a line by itself does not convey any clear meaning.

    # Run the actual test!
    $tests_run = 0;
    $tests_passed = 0;
    # 2006.02.28 -email is unavailable- added following line:
    $status = "status = N/A";
    $code = do $perl_testname;
[...]

    # 2006.02.28 -email is unavailable-
    # OLD: elsif ($code < 0) {
    elsif (($code < 0) || !defined($code) ) {
      $status = "status = N/A";
      --$categories_run;
    }

--- REGRESSION TEST
Proposed fix was applied and default_name set to die by run_make_tests.pl:

FILE: run_make_tests.pl set to force default_names to die:
   if ($osname =~ /Windows/i || $osname =~ /MINGW32/i ) {

OUTPUT from make check:
[...]
features/default_names .................................. Error running /cygdriv
e/c/greg/src/make-3.81rc1/tests/../make (expected 0; got 512): /cygdrive/c/greg/
src/make-3.81rc1/tests/../make  --no-print-directory -C /cygdrive/c/greg/src/mak
e-3.81rc1/tests/work/

      • Test died (features/default_names): test_driver.pl: 455: compare_output fail

ed: Makefile at test_driver.pl line 643.

status = N/A
[...]

greg keranen <gkeranen>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #1759:  test_driver.pl added by gkeranen (30KiB - application/octet-stream - tests/test_driver.pl with modified function run_each_test())

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

Do you think this task is very important?
If so, you can add your encouragement to it.
This task has 0 encouragements so far.

Only logged-in users can vote.

 

 

 

Follow 6 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2006-04-01 psmith Fixed Release4.0 => 3.81
2006-03-10 psmith StatusNone => Fixed
    Assigned toNone => psmith
    Open/ClosedOpen => Closed
    Fixed ReleaseNone => 4.0
2006-03-01 gkeranen Attached File- => Added test_driver.pl, #3458

Back to the top


Powered by Savane 3.6