118 |
<preface id=preface> |
<preface id=preface> |
119 |
<title>Abstract</title> |
<title>Abstract</title> |
120 |
|
|
121 |
<para>This document attempts to describe the functionality of |
<para>This document describes the functionality of DejaGnu, the |
122 |
DejaGnu, the GNU Testing Framework. DejaGnu is entirely written in |
testing framework of the GNU project. DejaGnu is written in |
123 |
<productname>Expect</productname>, which uses |
<productname>Expect</productname>, which uses |
124 |
<productname>Tcl</productname> as a command |
<productname>Tcl</productname> as a command |
125 |
language. <productname>Expect</productname> serves as a very |
language. <productname>Expect</productname> acts as a very |
126 |
programmable shell; you can run any program, as with the usual |
programmable shell. As with other Unix command shells, you can |
127 |
Unix command shells---but once the program is started, your |
run any program, but once the program is started, your test script |
128 |
test script has fully programmable control of |
has programmable control over its input and output. This does not |
129 |
its input and output. This does not just apply to the programs |
just apply to the programs under test; <command>expect</command> |
130 |
under test; <command>expect</command> can also run any auxiliary |
can also run any auxiliary program, such as |
131 |
program, such as <command>diff</command> or <command>sh</command>, |
<command>diff</command> or <command>sh</command>, with full |
132 |
with full control over its input and output.</para> |
control over its input and output.</para> |
133 |
|
|
134 |
<para>DejaGnu itself is merely a framework for creation of a test |
<para>DejaGnu itself is merely a framework for the creation of |
135 |
suites. Test suites are distributed separately for each GNU |
testsuites. Testsuites are distributed with each |
136 |
tool.</para> |
application.</para> |
137 |
|
|
138 |
</preface> |
</preface> |
139 |
|
|
149 |
Tcl procedures crafted to support writing a test harness. A |
Tcl procedures crafted to support writing a test harness. A |
150 |
<emphasis>Test Harness</emphasis> is the testing |
<emphasis>Test Harness</emphasis> is the testing |
151 |
infrastructure that is created to support a specific program |
infrastructure that is created to support a specific program |
152 |
or tool. Each program can have multiple test suites, all |
or tool. Each program can have multiple testsuites, all |
153 |
supported by a single test harness. DejaGnu is written in |
supported by a single test harness. DejaGnu is written in |
154 |
<productname>Expect</productname>, which in turn uses |
<productname>Expect</productname>, which in turn uses |
155 |
<productname>Tcl</productname> -- Tool command |
<productname>Tcl</productname> -- Tool command |
156 |
language. There is more information on Tcl at the <ulink |
language. There is more information on Tcl at the <ulink |
157 |
URL="http://www.scriptics.com">Scriptics</ulink> web site, and the |
URL="http://www.scriptics.com">Scriptics</ulink> web site and the |
158 |
Expect web site is at <ulink |
Expect web site is at <ulink |
159 |
URL="http://expect.nist.gov">NIST</ulink>.</para> |
URL="http://expect.nist.gov">NIST</ulink>.</para> |
160 |
|
|
161 |
<para>DejaGnu offers several advantages for testing:</para> |
<para>Julia Menapace first coined the term ``DejaGnu'' to describe |
162 |
|
an earlier testing framework at Cygnus Support she had written |
163 |
|
for <command>GDB</command>. When we replaced it with the |
164 |
|
Expect-based framework, it was like DejaGnu all over again. |
165 |
|
More importantly, it was also named after my daughter, <ulink |
166 |
|
URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink> |
167 |
|
(now 13 years old as of September 2003), who was a toddler |
168 |
|
during DejaGnu's beginnings.</para> |
169 |
|
|
170 |
|
<para>DejaGnu offers several advantages for testing:</para> |
171 |
|
|
172 |
<itemizedlist mark="bullet" spacing="compact"> |
<itemizedlist mark="bullet" spacing="compact"> |
173 |
|
|
179 |
<listitem><para>DejaGnu provides a layer of abstraction which |
<listitem><para>DejaGnu provides a layer of abstraction which |
180 |
allows you to write tests that are portable to any host or |
allows you to write tests that are portable to any host or |
181 |
target where a program must be tested. For instance, a test |
target where a program must be tested. For instance, a test |
182 |
for <command>GDB</command> can run (from any Unix |
for <command>GDB</command> can run from any supported host |
183 |
based host) on any target architecture that DejaGnu |
system on any supported target system. DejaGnu runs tests on |
184 |
supports. Currently DejaGnu runs tests on many single board |
many single board computers, whose operating software ranges |
185 |
computers, whose operating software ranges from just a boot |
from a simple boot monitor to a real-time OS.</para> |
186 |
monitor to a full-fledged, Unix-like realtime OS.</para> |
</listitem> |
|
</listitem> |
|
187 |
|
|
188 |
<listitem><para>All tests have the same output format. This |
<listitem><para>All tests have the same output format. This |
189 |
makes it easy to integrate testing into other software |
makes it easy to integrate testing into other software |
190 |
development processes. DejaGnu's output is designed to be |
development processes. DejaGnu's output is designed to be |
191 |
parsed by other filtering script, and it is also human |
parsed by other filtering script and it is also human |
192 |
readable.</para> |
readable.</para> |
193 |
</listitem> |
</listitem> |
194 |
|
|
195 |
<listitem><para>Using Tcl and expect, it's easy to create wrappers |
<listitem><para>Using Tcl and Expect, it's easy to create wrappers |
196 |
for existing test suites. By incorporating existing tests under |
for existing testsuites. By incorporating existing tests under |
197 |
DejaGnu, it's easier to have a single set of report analyse |
DejaGnu, it's easier to have a single set of report analyse |
198 |
programs..</para> |
programs..</para> |
199 |
|
|
200 |
</listitem> |
</listitem> |
201 |
</itemizedlist> |
</itemizedlist> |
202 |
|
|
203 |
<para>Running tests requires two things: the testing framework, and |
<para>Running tests requires two things: the testing framework and |
204 |
the test suites themselves. Tests are usually written in |
the testsuites themselves. Tests are usually written in |
205 |
<productname>Expect</productname> using Tcl, but you can also use a |
<productname>Expect</productname> using Tcl, but you can also use |
206 |
Tcl script to run a test suite that is not based on |
a Tcl script to run a testsuite that is not based on |
207 |
<productname>Expect</productname>. |
<productname>Expect</productname>. <productname>Expect</productname> |
208 |
(<productname>expect</productname> script filenames conventionally |
script filenames conventionally use <emphasis>.exp</emphasis> as a |
209 |
use <emphasis>.exp</emphasis> as a suffix; for example, the main |
suffix; for example, the main implementation of the DejaGnu test |
210 |
implementation of the DejaGnu test driver is in the file |
driver is in the file |
211 |
<productname>runtest.exp</productname>.)</para> |
<productname>runtest.exp</productname>.)</para> |
212 |
|
|
|
<para>Julia Menapace first coined the term ``Deja Gnu'' to describe an |
|
|
earlier testing framework at Cygnus Support she had written for |
|
|
<command>GDB</command>. When we replaced it with the Expect-based |
|
|
framework, it was like DejaGnu all over again. More importantly, it |
|
|
was also named after my daughter,<ulink |
|
|
URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink> (now 12 |
|
|
years old in Sept of 2002), who was a toddler during DejaGnu's |
|
|
creation.</para> |
|
|
|
|
213 |
</sect1> |
</sect1> |
214 |
|
|
215 |
<sect1 id=new xreflabel="Release Notes"> |
<sect1 id=new xreflabel="Release Notes"> |
226 |
are:</para> |
are:</para> |
227 |
|
|
228 |
<itemizedlist> |
<itemizedlist> |
229 |
<listitem><para>More builtin support for building target binaries |
<listitem><para>More built-in support for building target binaries |
230 |
with the correct linker flags. Currently this only works with |
with the correct linker flags. Currently this only works with |
231 |
<productname>GCC</productname> as the cross compiler, |
<productname>GCC</productname> as the cross compiler, |
232 |
preferably with a target supported by |
preferably with a target supported by |
257 |
<para>To use DejaGnu on Windows, you need to first install the |
<para>To use DejaGnu on Windows, you need to first install the |
258 |
<ulink URL="http://www.cygwin.com/">Cygwin</ulink> |
<ulink URL="http://www.cygwin.com/">Cygwin</ulink> |
259 |
release. This works as of the B20.1 release. Cygwin is a POSIX |
release. This works as of the B20.1 release. Cygwin is a POSIX |
260 |
system for Windows. This covers both utility programs, and a libray |
system for Windows. This covers both utility programs and a library |
261 |
that adds POSIX system calls to Windows. Among them is pseudo tty |
that adds POSIX system calls to Windows. Among them is pseudo tty |
262 |
support for Windows that emulates the POSIX pty standard. The |
support for Windows that emulates the POSIX pty standard. The |
263 |
latest Cygwin is always available from <ulink |
latest Cygwin is always available from <ulink |
274 |
<sect1 id=designgoals xreflabel="Design Goals"> |
<sect1 id=designgoals xreflabel="Design Goals"> |
275 |
<title>Design Goals</title> |
<title>Design Goals</title> |
276 |
|
|
277 |
<para>DejaGnu grew out of the internal needs of Cygnus Solutions. (then |
<para>DejaGnu grew out of the internal needs of Cygnus Solutions, |
278 |
Cygnus Support). Cygnus maintained and enhanced a variety of free |
the company formerly known as Cygnus Support. Cygnus maintained |
279 |
programs in many different environments, and we needed a testing |
and enhanced a variety of free programs in many different |
280 |
tool that:</para> |
environments and we needed a testing tool that:</para> |
281 |
|
|
282 |
<itemizedlist mark="bullet"> |
<itemizedlist mark="bullet"> |
283 |
<listitem><para>was useful to developers while fixing |
<listitem><para>was useful to developers while fixing |
284 |
bugs.</para></listitem> |
bugs;</para></listitem> |
285 |
<listitem><para>automated running many tests during a software |
<listitem><para>automated running many tests during a software |
286 |
release process.</para></listitem> |
release process;</para></listitem> |
287 |
<listitem><para>was portable among a variety of host |
<listitem><para>was portable among a variety of host |
288 |
computers.</para></listitem> |
computers;</para></listitem> |
289 |
<listitem><para>supported cross-development |
<listitem><para>supported cross-development |
290 |
testing.</para></listitem> |
testing;</para></listitem> |
291 |
<listitem><para>permitted testing interactive programs, like |
<listitem><para>permitted testing interactive programs, like |
292 |
<command>GDB</command>; and </para></listitem> |
<command>GDB</command>; and </para></listitem> |
293 |
<listitem><para>permitted testing batch oriented programs, like |
<listitem><para>permitted testing batch oriented programs, like |
300 |
make sure that <command>GDB</command> works as well when cross-debugging |
make sure that <command>GDB</command> works as well when cross-debugging |
301 |
as it does in a native configuration. </para> |
as it does in a native configuration. </para> |
302 |
|
|
303 |
<para>Probably the greatest challenge was testing in a cross-development |
<para>Probably the greatest challenge was testing in a |
304 |
environment (which can be a real nightmare). Most cross-development |
cross-development environment. Most cross-development |
305 |
environments are customized by each developer. Even when buying packaged |
environments are customized by each developer. Even when buying |
306 |
boards from vendors there are many differences. The communication |
packaged boards from vendors there are many differences. The |
307 |
interfaces vary from a serial line to ethernet. DejaGnu was designed with |
communication interfaces vary from a serial line to Ethernet. |
308 |
a modular communication setup, so that each kind of communication can be |
DejaGnu was designed with a modular communication setup, so that |
309 |
added as required, and supported thereafter. Once a communication |
each kind of communication can be added as required and supported |
310 |
procedure is coded, any test can use it. Currently DejaGnu can use |
thereafter. Once a communication procedure is coded, any test can |
311 |
<command>rsh</command>, <command>rlogin</command>, |
use it. Currently DejaGnu can use <command>rsh</command>, |
312 |
<command>telnet</command>, <command>tip</command>, |
<command>rlogin</command>, <command>telnet</command>, |
313 |
<command>kermit</command>, and <command>mondfe</command> for remote |
<command>tip</command>, <command>kermit</command> and |
314 |
communications.</para> |
<command>mondfe</command> for remote communications.</para> |
315 |
|
|
316 |
</sect1> |
</sect1> |
317 |
|
|
319 |
<title>A POSIX conforming test framework</title> |
<title>A POSIX conforming test framework</title> |
320 |
|
|
321 |
<para>DejaGnu conforms to the POSIX 1003.3 standard for test |
<para>DejaGnu conforms to the POSIX 1003.3 standard for test |
322 |
frameworks. I was also a member of that committe.</para> |
frameworks. Rob Savoye was a member of that committee.</para> |
323 |
|
|
324 |
<para>The POSIX standard 1003.3 defines what a testing framework needs to |
<para>The POSIX standard 1003.3 defines what a testing framework needs to |
325 |
provide, in order to permit the creation of POSIX conformance test |
provide, in order to permit the creation of POSIX conformance test |
328 |
to POSIX conformance. POSIX 1003.3 does not specify a particular testing |
to POSIX conformance. POSIX 1003.3 does not specify a particular testing |
329 |
framework, but at this time there is only one other POSIX conforming test |
framework, but at this time there is only one other POSIX conforming test |
330 |
framework: TET. TET was created by Unisoft for a consortium comprised of |
framework: TET. TET was created by Unisoft for a consortium comprised of |
331 |
X/Open, Unix International, and the Open Software Foundation.</para> |
X/Open, Unix International and the Open Software Foundation.</para> |
332 |
|
|
333 |
<para>The POSIX documentation refers to <firstterm>assertions</firstterm>. |
<para>The POSIX documentation refers to <firstterm>assertions</firstterm>. |
334 |
An assertion is a description of behavior. For example, if a standard |
An assertion is a description of behavior. For example, if a standard |
335 |
says ``The sun shall shine'', a corresponding assertion might be ``The |
says ``The sun shall shine'', a corresponding assertion might be ``The |
336 |
sun is shining.'' A test based on this assertion would pass or fail |
sun is shining.'' A test based on this assertion would pass or fail |
337 |
depending on whether it is daytime or nighttime. It is important to note |
depending on whether it is day or night. It is important to note |
338 |
that the standard being tested is never 1003.3; the standard being tested |
that the standard being tested is never 1003.3; the standard being tested |
339 |
is some other standard, for which the assertions were written.</para> |
is some other standard, for which the assertions were written.</para> |
340 |
|
|
341 |
<para>As there is no test suite to test testing frameworks for POSIX |
<para>As there is no testsuite to test testing frameworks for POSIX |
342 |
1003.3 conformance, verifying conformance to this standard is done by |
1003.3 conformance, verifying conformance to this standard is done by |
343 |
repeatedly reading the standard and experimenting. One of the main |
repeatedly reading the standard and experimenting. One of the main |
344 |
things 1003.3 does specify is the set of allowed output messages, and |
things 1003.3 does specify is the set of allowed output messages and |
345 |
their definitions. Four messages are supported for a required feature of |
their definitions. Four messages are supported for a required feature of |
346 |
POSIX conforming systems, and a fifth for a conditional feature. DejaGnu |
POSIX conforming systems and a fifth for a conditional feature. DejaGnu |
347 |
supports the use of all five output messages; in this sense a test suite |
supports the use of all five output messages. In this sense a testsuite |
348 |
that uses exactly these messages can be considered POSIX conforming. |
that uses exactly these messages can be considered POSIX conforming. |
349 |
These definitions specify the output of a test |
These definitions specify the output of a test |
350 |
case:</para> |
case:</para> |
392 |
code. If the test isn't actually run, it must produce |
code. If the test isn't actually run, it must produce |
393 |
<emphasis>UNRESOLVED</emphasis> rather than just leaving that test |
<emphasis>UNRESOLVED</emphasis> rather than just leaving that test |
394 |
out of the output. This means that you have to be careful when |
out of the output. This means that you have to be careful when |
395 |
writing tests, to not carelessly use tcl statements like |
writing tests to not carelessly use Tcl commands like |
396 |
<emphasis>return</emphasis>---if you alter the flow of control of the |
<emphasis>return</emphasis>---if you alter the flow of control of the |
397 |
tcl code you must insure that every test still produces some result |
Tcl code you must insure that every test still produces some result |
398 |
code.</para> |
code.</para> |
399 |
|
|
400 |
<para>Here are some of the ways a test may wind up |
<para>Here are some of the ways a test may wind up |
428 |
<variablelist> |
<variablelist> |
429 |
<varlistentry> |
<varlistentry> |
430 |
<term>UNTESTED</term> |
<term>UNTESTED</term> |
431 |
<listitem><para>A test was not run. This is a placeholder, used |
<listitem><para>A test was not run. This is a place-holder, used |
432 |
when there is no real test case yet.</para></listitem> |
when there is no real test case yet.</para></listitem> |
433 |
</varlistentry> |
</varlistentry> |
434 |
</variablelist> |
</variablelist> |
452 |
</variablelist> |
</variablelist> |
453 |
|
|
454 |
<para>DejaGnu uses the same output procedures to produce these messages |
<para>DejaGnu uses the same output procedures to produce these messages |
455 |
for all test suites, and these procedures are already known to conform |
for all testsuites and these procedures are already known to conform |
456 |
to POSIX 1003.3. For a DejaGnu test suite to conform to POSIX 1003.3, |
to POSIX 1003.3. For a DejaGnu testsuite to conform to POSIX 1003.3, |
457 |
you must avoid the <emphasis>setup</emphasis>xfail} procedure as |
you must avoid the <emphasis>setup</emphasis>xfail} procedure as |
458 |
described in the <emphasis>PASS</emphasis> section above, and you must |
described in the <emphasis>PASS</emphasis> section above and you must |
459 |
be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate, |
be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate, |
460 |
as described in the <emphasis>UNRESOLVED</emphasis> section |
as described in the <emphasis>UNRESOLVED</emphasis> section |
461 |
above.</para> |
above.</para> |