14127 |
@command{testsuite}. When run, @command{testsuite} executes each test |
@command{testsuite}. When run, @command{testsuite} executes each test |
14128 |
group in turn, producing only one summary line per test to say if that |
group in turn, producing only one summary line per test to say if that |
14129 |
particular test succeeded or failed. At end of all tests, summarizing |
particular test succeeded or failed. At end of all tests, summarizing |
14130 |
counters get printed. If any test failed, one debugging script gets |
counters get printed. One debugging directory is left for each test |
14131 |
automatically generated for each test group which failed. These |
group which failed, if any: such directories are named |
14132 |
debugging scripts are named @file{testsuite.@var{nn}}, where @var{nn} is |
@file{testsuite.dir/@var{nn}}, where @var{nn} is the sequence number of |
14133 |
the sequence number of the test group. In the ideal situation, none of |
the test group, and they include: |
14134 |
the tests fail, and consequently, no debugging script is generated out |
|
14135 |
of validation. |
@itemize @bullet |
14136 |
|
@item a debugging script named @file{run} which reruns the test in |
14137 |
|
@dfn{debug mode} (@pxref{testsuite Invocation}). The automatic generation |
14138 |
|
of debugging scripts has the purpose of easing the chase for bugs. |
14139 |
|
|
14140 |
The automatic generation of debugging scripts for failed test has the |
@item all the files created with @code{AT_DATA} |
14141 |
purpose of easing the chase for bugs. |
|
14142 |
|
@item a log of the run, named @file{testsuite.log} |
14143 |
|
@end itemize |
14144 |
|
|
14145 |
|
In the ideal situation, none of the tests fail, and consequently, no |
14146 |
|
debugging directory is left out of validation. |
14147 |
|
|
14148 |
It often happens in practice that individual tests in the validation |
It often happens in practice that individual tests in the validation |
14149 |
suite need to get information coming out of the configuration process. |
suite need to get information coming out of the configuration process. |
14190 |
\ / |
\ / |
14191 |
>-- testsuite* --< |
>-- testsuite* --< |
14192 |
/ \ |
/ \ |
14193 |
[atlocal] ->' `--> [testsuite.@var{nn}*] |
[atlocal] ->' `--> [testsuite.dir] |
14194 |
@end example |
@end example |
14195 |
|
|
14196 |
|
|
14209 |
setting environment variables before the command, such as in |
setting environment variables before the command, such as in |
14210 |
@samp{CC=my-home-grown-cc ./testsuite}. This results in the test suite |
@samp{CC=my-home-grown-cc ./testsuite}. This results in the test suite |
14211 |
not knowing this change, hence (i) it can't report it to you, and (ii) |
not knowing this change, hence (i) it can't report it to you, and (ii) |
14212 |
it cannot preserve the value of @code{CC} for subsequent runs@footnote{ |
it cannot preserve the value of @code{CC} for subsequent runs. |
14213 |
@c |
Autoconf faced exactly the same problem, and solved it by asking |
|
When a failure occurs, the test suite is rerun, verbosely, and the user |
|
|
is asked to ``play'' with this failure to provide better information. |
|
|
It is important to keep the same environment between the first run, and |
|
|
bug-tracking runs. |
|
|
@c |
|
|
}. Autoconf faced exactly the same problem, and solved it by asking |
|
14214 |
users to pass the variable definitions as command line arguments. |
users to pass the variable definitions as command line arguments. |
14215 |
Autotest requires this rule too, but has no means to enforce it; the log |
Autotest requires this rule too, but has no means to enforce it; the log |
14216 |
then contains a trace of the variables the user changed. |
then contains a trace of the variables the user changed. |
14350 |
|
|
14351 |
Autotest test suites support the following arguments: |
Autotest test suites support the following arguments: |
14352 |
|
|
14353 |
@table @samp |
@table @option |
14354 |
@item --help |
@item --help |
14355 |
@itemx -h |
@itemx -h |
14356 |
Display the list of options and exit successfully. |
Display the list of options and exit successfully. |
14432 |
still removed @emph{before}, therefore using this option is sane when |
still removed @emph{before}, therefore using this option is sane when |
14433 |
running several test groups. Do not create debugging scripts. Do not |
running several test groups. Do not create debugging scripts. Do not |
14434 |
log (in order to preserve supposedly existing full log file). This is |
log (in order to preserve supposedly existing full log file). This is |
14435 |
the default for debugging scripts. |
the default for debugging scripts, but it can also be useful to debug |
14436 |
|
the testsuite itself. |
14437 |
|
|
14438 |
@item --trace |
@item --trace |
14439 |
@itemx -x |
@itemx -x |