711 |
<chapter id=runningtests> |
<chapter id=runningtests> |
712 |
<title>Running Tests</title> |
<title>Running Tests</title> |
713 |
|
|
714 |
<para>There are two ways to execute a test suite. The most |
<para>There are two ways to execute a testsuite. The most |
715 |
common way is when there is existing support in the |
common way is when there is existing support in the |
716 |
<filename>Makefile</filename>. This support consists of a |
<filename>Makefile</filename>. This support consists of a |
717 |
<emphasis>check</emphasis> target. The other way is to execute the |
<emphasis>check</emphasis> target. The other way is to execute the |
812 |
<varlistentry> |
<varlistentry> |
813 |
<term>UNRESOLVED</term> |
<term>UNRESOLVED</term> |
814 |
<listitem><para>Output from a test requires manual inspection; the |
<listitem><para>Output from a test requires manual inspection; the |
815 |
test suite could not automatically determine the outcome. For |
testsuite could not automatically determine the outcome. For |
816 |
example, your tests can report this outcome is when a test does not |
example, your tests can report this outcome is when a test does not |
817 |
complete as expected.</para></listitem> |
complete as expected.</para></listitem> |
818 |
</varlistentry> |
</varlistentry> |
844 |
<listitem><para>Indicates a major problem (detected by the test case |
<listitem><para>Indicates a major problem (detected by the test case |
845 |
itself) in running the test. This is usually an unrecoverable error, |
itself) in running the test. This is usually an unrecoverable error, |
846 |
such as a missing file or loss of communication to the target. (POSIX |
such as a missing file or loss of communication to the target. (POSIX |
847 |
test suites should not emit this message; use |
testsuites should not emit this message; use |
848 |
<emphasis>UNSUPPORTED</emphasis>, <emphasis>UNTESTED</emphasis>, or |
<emphasis>UNSUPPORTED</emphasis>, <emphasis>UNTESTED</emphasis>, or |
849 |
<emphasis>UNRESOLVED</emphasis> instead, as |
<emphasis>UNRESOLVED</emphasis> instead, as |
850 |
appropriate.)</para></listitem> |
appropriate.)</para></listitem> |
1054 |
|
|
1055 |
<varlistentry id=tool-opt> |
<varlistentry id=tool-opt> |
1056 |
<term><option>--tool[name(s)]</option></term> |
<term><option>--tool[name(s)]</option></term> |
1057 |
<listitem><para>Specifies which test suite to run, and what |
<listitem><para>Specifies which testsuite to run, and what |
1058 |
initialization module to use. <option>--tool</option> is used |
initialization module to use. <option>--tool</option> is used |
1059 |
<emphasis>only</emphasis> for these two purposes. It is |
<emphasis>only</emphasis> for these two purposes. It is |
1060 |
<emphasis>not</emphasis> used to name the executable program to |
<emphasis>not</emphasis> used to name the executable program to |
1163 |
|
|
1164 |
<para>Typically, you don't need must to use any command-line options. |
<para>Typically, you don't need must to use any command-line options. |
1165 |
<option>--tool</option> used is only required when there are more than |
<option>--tool</option> used is only required when there are more than |
1166 |
one test suite in the same directory. The default options are in the |
one testsuite in the same directory. The default options are in the |
1167 |
local site.exp file, created by "make site.exp".</para> |
local site.exp file, created by "make site.exp".</para> |
1168 |
|
|
1169 |
<para>For example, if the directory <filename>gdb/testsuite</filename> |
<para>For example, if the directory <filename>gdb/testsuite</filename> |
1465 |
DejaGnu test environment using Tcl variables. This ties the |
DejaGnu test environment using Tcl variables. This ties the |
1466 |
DejaGnu test scripts into the <command>configure</command> and |
DejaGnu test scripts into the <command>configure</command> and |
1467 |
<command>make</command> programs. If this file is setup correctly, |
<command>make</command> programs. If this file is setup correctly, |
1468 |
it is possible to execute a test suite merely by typing |
it is possible to execute a testsuite merely by typing |
1469 |
<command>runtest</command>.</para> |
<command>runtest</command>.</para> |
1470 |
|
|
1471 |
<para>DejaGnu supports two <filename>site.exp</filename> |
<para>DejaGnu supports two <filename>site.exp</filename> |
1487 |
<symbol>DEJAGNU</symbol> to the name of the file. This is also |
<symbol>DEJAGNU</symbol> to the name of the file. This is also |
1488 |
refered to as the ``global'' config file.</para> |
refered to as the ``global'' config file.</para> |
1489 |
|
|
1490 |
<para>Any directory containing a configured test suite also has a |
<para>Any directory containing a configured testsuite also has a |
1491 |
local <filename>site.exp</filename>, capturing configuration values |
local <filename>site.exp</filename>, capturing configuration values |
1492 |
specific to the tool under test. Since <command>runtest</command> |
specific to the tool under test. Since <command>runtest</command> |
1493 |
loads these values last, the individual test configuration can |
loads these values last, the individual test configuration can |
1496 |
|
|
1497 |
<para>You can usually generate or update the testsuite's local |
<para>You can usually generate or update the testsuite's local |
1498 |
<filename>site.exp</filename> by typing <command>make |
<filename>site.exp</filename> by typing <command>make |
1499 |
site.exp</command> in the test suite directory, after the test |
site.exp</command> in the testsuite directory, after the test |
1500 |
suite is configured.</para> |
suite is configured.</para> |
1501 |
|
|
1502 |
<para>You can also have a file in your home directory called |
<para>You can also have a file in your home directory called |
1519 |
local to each test directory, rather than in the global |
local to each test directory, rather than in the global |
1520 |
<filename>site.exp</filename> in the installed DejaGnu |
<filename>site.exp</filename> in the installed DejaGnu |
1521 |
library. This file is mostly for supplying tool specific info |
library. This file is mostly for supplying tool specific info |
1522 |
that is required by the test suite.</para> |
that is required by the testsuite.</para> |
1523 |
|
|
1524 |
<para>All local <filename>site.exp</filename> files have |
<para>All local <filename>site.exp</filename> files have |
1525 |
two sections, separated by comment text. The first section is |
two sections, separated by comment text. The first section is |
1601 |
<para>This file defines the required fields for a local config |
<para>This file defines the required fields for a local config |
1602 |
file, namely the three config triplets, and the srcdir. It also |
file, namely the three config triplets, and the srcdir. It also |
1603 |
defines several other Tcl variables that are used exclusivly by |
defines several other Tcl variables that are used exclusivly by |
1604 |
the GCC test suite. For most test cases, the CXXFLAGS and LDFLAGS |
the GCC testsuite. For most test cases, the CXXFLAGS and LDFLAGS |
1605 |
are supplied by DejaGnu itself for cross testing, but to test a |
are supplied by DejaGnu itself for cross testing, but to test a |
1606 |
compiler, GCC needs to manipulate these itself.</para> |
compiler, GCC needs to manipulate these itself.</para> |
1607 |
|
|
2067 |
<chapter id=Extending xreflabel="Extending DejaGnu"> |
<chapter id=Extending xreflabel="Extending DejaGnu"> |
2068 |
<title>Extending DejaGnu</title> |
<title>Extending DejaGnu</title> |
2069 |
|
|
2070 |
<sect1 id=addsuite xreflabel="Adding a new Test Suite"> |
<sect1 id=addsuite xreflabel="Adding a new Testsuite"> |
2071 |
<title>Adding A New Test Suite</title> |
<title>Adding A New Testsuite</title> |
2072 |
|
|
2073 |
<para>The testsuite for a new tool should always be located in that tools |
<para>The testsuite for a new tool should always be located in that tools |
2074 |
source directory. DejaGnu require the directory be named |
source directory. DejaGnu require the directory be named |
2084 |
|
|
2085 |
<para>In general, the best way to learn how to write (code or even prose) |
<para>In general, the best way to learn how to write (code or even prose) |
2086 |
is to read something similar. This principle applies to test cases and |
is to read something similar. This principle applies to test cases and |
2087 |
to test suites. Unfortunately, well-established test suites have a way |
to testsuites. Unfortunately, well-established testsuites have a way |
2088 |
of developing their own conventions: as test writers become more |
of developing their own conventions: as test writers become more |
2089 |
experienced with DejaGnu and with Tcl, they accumulate more utilities, |
experienced with DejaGnu and with Tcl, they accumulate more utilities, |
2090 |
and take advantage of more and more features of |
and take advantage of more and more features of |
2091 |
<productname>Expect</productname> and <productname>Tcl</productname> in |
<productname>Expect</productname> and <productname>Tcl</productname> in |
2092 |
general.</para> |
general.</para> |
2093 |
|
|
2094 |
<para>Inspecting such established test suites may make the prospect of |
<para>Inspecting such established testsuites may make the prospect of |
2095 |
creating an entirely new test suite appear overwhelming. Nevertheless, |
creating an entirely new testsuite appear overwhelming. Nevertheless, |
2096 |
it is quite straightforward to get a new test suite going.</para> |
it is quite straightforward to get a new testsuite going.</para> |
2097 |
|
|
2098 |
<para>There is one test suite that is guaranteed not to grow more |
<para>There is one testsuite that is guaranteed not to grow more |
2099 |
elaborate over time: both it and the tool it tests were created expressly |
elaborate over time: both it and the tool it tests were created expressly |
2100 |
to illustrate what it takes to get started with DejaGnu. The |
to illustrate what it takes to get started with DejaGnu. The |
2101 |
<filename>example/</filename> directory of the DejaGnu distribution |
<filename>example/</filename> directory of the DejaGnu distribution |
2102 |
contains both an interactive tool called <command>calc</command>, and a |
contains both an interactive tool called <command>calc</command>, and a |
2103 |
test suite for it. Reading this test suite, and experimenting with it, |
testsuite for it. Reading this testsuite, and experimenting with it, |
2104 |
is a good way to supplement the information in this section. (Thanks to |
is a good way to supplement the information in this section. (Thanks to |
2105 |
Robert Lupton for creating calc and its test suite---and also the first |
Robert Lupton for creating calc and its testsuite---and also the first |
2106 |
version of this section of the manual!)</para> |
version of this section of the manual!)</para> |
2107 |
|
|
2108 |
<para>To help orient you further in this task, here is an outline of the |
<para>To help orient you further in this task, here is an outline of the |
2109 |
steps to begin building a test suite for a program example.</para> |
steps to begin building a testsuite for a program example.</para> |
2110 |
|
|
2111 |
<itemizedlist mark=bullet> |
<itemizedlist mark=bullet> |
2112 |
|
|
2897 |
|
|
2898 |
</sect1> |
</sect1> |
2899 |
|
|
2900 |
<sect1 id=adding xreflabel="Adding A Test Case To A Test Suite"> |
<sect1 id=adding xreflabel="Adding A Test Case To A Testsuite"> |
2901 |
<title>Adding A Test Case To A Test Suite.</title> |
<title>Adding A Test Case To A Testsuite.</title> |
2902 |
|
|
2903 |
<para>There are two slightly different ways to add a test |
<para>There are two slightly different ways to add a test |
2904 |
case. One is to add the test case to an existing directory. The |
case. One is to add the test case to an existing directory. The |
2937 |
|
|
2938 |
<listitem><para>Add the new directory name to the |
<listitem><para>Add the new directory name to the |
2939 |
<symbol>configdirs</symbol> definition in the |
<symbol>configdirs</symbol> definition in the |
2940 |
<filename>configure.in</filename> file for the test suite |
<filename>configure.in</filename> file for the testsuite |
2941 |
directory. This way when <command>make</command> and |
directory. This way when <command>make</command> and |
2942 |
<command>configure</command> next run, they include the new |
<command>configure</command> next run, they include the new |
2943 |
directory.</para></listitem> |
directory.</para></listitem> |