/[dejagnu]/dejagnu/doc/overview.sgml
ViewVC logotype

Diff of /dejagnu/doc/overview.sgml

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.9 by bje, Tue Jul 29 11:57:09 2003 UTC revision 1.10 by bje, Sat Aug 16 06:31:50 2003 UTC
# Line 118  into another language, under the above c Line 118  into another language, under the above c
118    <preface id=preface>    <preface id=preface>
119      <title>Abstract</title>      <title>Abstract</title>
120    
121      <para>This document attempts to describe the functionality of      <para>This document describes the functionality of DejaGnu, the
122      DejaGnu, the GNU Testing Framework. DejaGnu is entirely written in      testing framework of the GNU project. DejaGnu is written in
123      <productname>Expect</productname>, which uses      <productname>Expect</productname>, which uses
124      <productname>Tcl</productname> as a command      <productname>Tcl</productname> as a command
125      language. <productname>Expect</productname> serves as a very      language. <productname>Expect</productname> acts as a very
126      programmable shell; you can run any program, as with the usual      programmable shell.  As with other Unix command shells, you can
127      Unix command shells---but once the program is started, your      run any program, but once the program is started, your test script
128      test script has fully programmable control of      has programmable control over its input and output.  This does not
129      its input and output.  This does not just apply to the programs      just apply to the programs under test; <command>expect</command>
130      under test; <command>expect</command> can also run any auxiliary      can also run any auxiliary program, such as
131      program, such as <command>diff</command> or <command>sh</command>,      <command>diff</command> or <command>sh</command>, with full
132      with full control over its input and output.</para>        control over its input and output.</para>
133    
134      <para>DejaGnu itself is merely a framework for creation of a test      <para>DejaGnu itself is merely a framework for the creation of
135      suites. Test suites are distributed separately for each GNU      testsuites. Testsuites are distributed with each
136      tool.</para>      application.</para>
137    
138    </preface>    </preface>
139    
# Line 149  into another language, under the above c Line 149  into another language, under the above c
149          Tcl procedures crafted to support writing a test harness. A          Tcl procedures crafted to support writing a test harness. A
150          <emphasis>Test Harness</emphasis> is the testing          <emphasis>Test Harness</emphasis> is the testing
151          infrastructure that is created to support a specific program          infrastructure that is created to support a specific program
152          or tool. Each program can have multiple test suites, all          or tool. Each program can have multiple testsuites, all
153          supported by a single test harness. DejaGnu is written in          supported by a single test harness. DejaGnu is written in
154          <productname>Expect</productname>, which in turn uses          <productname>Expect</productname>, which in turn uses
155          <productname>Tcl</productname> -- Tool command          <productname>Tcl</productname> -- Tool command
156          language. There is more information on Tcl at the <ulink          language. There is more information on Tcl at the <ulink
157          URL="http://www.scriptics.com">Scriptics</ulink> web site, and the          URL="http://www.scriptics.com">Scriptics</ulink> web site and the
158          Expect web site is at <ulink          Expect web site is at <ulink
159          URL="http://expect.nist.gov">NIST</ulink>.</para>          URL="http://expect.nist.gov">NIST</ulink>.</para>
160            
161          <para>DejaGnu offers several advantages for testing:</para>     <para>Julia Menapace first coined the term ``DejaGnu'' to describe
162            an earlier testing framework at Cygnus Support she had written
163            for <command>GDB</command>. When we replaced it with the
164            Expect-based framework, it was like DejaGnu all over again.
165            More importantly, it was also named after my daughter, <ulink
166            URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink>
167            (now 13 years old as of September 2003), who was a toddler
168            during DejaGnu's beginnings.</para>
169        
170        <para>DejaGnu offers several advantages for testing:</para>
171    
172      <itemizedlist mark="bullet" spacing="compact">      <itemizedlist mark="bullet" spacing="compact">
173    
# Line 170  into another language, under the above c Line 179  into another language, under the above c
179        <listitem><para>DejaGnu provides a layer of abstraction which        <listitem><para>DejaGnu provides a layer of abstraction which
180          allows you to write tests that are portable to any host or          allows you to write tests that are portable to any host or
181          target where a program must be tested. For instance, a test          target where a program must be tested. For instance, a test
182          for <command>GDB</command> can run (from any Unix          for <command>GDB</command> can run from any supported host
183          based host) on any target architecture that DejaGnu          system on any supported target system. DejaGnu runs tests on
184          supports. Currently DejaGnu runs tests on many single board          many single board computers, whose operating software ranges
185          computers, whose operating software ranges from just a boot          from a simple boot monitor to a real-time OS.</para>
186          monitor to a full-fledged, Unix-like realtime OS.</para>          </listitem>
       </listitem>  
187    
188        <listitem><para>All tests have the same output format. This        <listitem><para>All tests have the same output format. This
189          makes it easy to integrate testing into other software          makes it easy to integrate testing into other software
190          development processes. DejaGnu's output is designed to be          development processes. DejaGnu's output is designed to be
191          parsed by other filtering script, and it is also human          parsed by other filtering script and it is also human
192          readable.</para>          readable.</para>
193        </listitem>        </listitem>
194    
195        <listitem><para>Using Tcl and expect, it's easy to create wrappers        <listitem><para>Using Tcl and Expect, it's easy to create wrappers
196        for existing test suites. By incorporating existing tests under        for existing testsuites. By incorporating existing tests under
197        DejaGnu, it's easier to have a single set of report analyse        DejaGnu, it's easier to have a single set of report analyse
198        programs..</para>        programs..</para>
199    
200        </listitem>        </listitem>
201      </itemizedlist>      </itemizedlist>
202    
203      <para>Running tests requires two things: the testing framework, and      <para>Running tests requires two things: the testing framework and
204      the test suites themselves. Tests are usually written in      the testsuites themselves. Tests are usually written in
205      <productname>Expect</productname> using Tcl, but you can also use a      <productname>Expect</productname> using Tcl, but you can also use
206      Tcl script to run a test suite that is not based on      a Tcl script to run a testsuite that is not based on
207      <productname>Expect</productname>.      <productname>Expect</productname>. <productname>Expect</productname>
208      (<productname>expect</productname> script filenames conventionally      script filenames conventionally use <emphasis>.exp</emphasis> as a
209      use <emphasis>.exp</emphasis> as a suffix; for example, the main      suffix; for example, the main implementation of the DejaGnu test
210      implementation of the DejaGnu test driver is in the file      driver is in the file
211      <productname>runtest.exp</productname>.)</para>      <productname>runtest.exp</productname>.)</para>
212                    
     <para>Julia Menapace first coined the term ``Deja Gnu'' to describe an  
     earlier testing framework at Cygnus Support she had written for  
     <command>GDB</command>. When we replaced it with the Expect-based  
     framework, it was like DejaGnu all over again.  More importantly, it  
     was also named after my daughter,<ulink  
     URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink> (now 12  
     years old in Sept of 2002), who was a toddler during DejaGnu's  
     creation.</para>  
   
213    </sect1>    </sect1>
214    
215    <sect1 id=new xreflabel="Release Notes">    <sect1 id=new xreflabel="Release Notes">
# Line 227  into another language, under the above c Line 226  into another language, under the above c
226      are:</para>      are:</para>
227    
228      <itemizedlist>      <itemizedlist>
229        <listitem><para>More builtin support for building target binaries        <listitem><para>More built-in support for building target binaries
230          with the correct linker flags. Currently this only works with          with the correct linker flags. Currently this only works with
231          <productname>GCC</productname> as the cross compiler,          <productname>GCC</productname> as the cross compiler,
232          preferably with a target supported by          preferably with a target supported by
# Line 258  into another language, under the above c Line 257  into another language, under the above c
257        <para>To use DejaGnu on Windows, you need to first install the        <para>To use DejaGnu on Windows, you need to first install the
258          <ulink URL="http://www.cygwin.com/">Cygwin</ulink>          <ulink URL="http://www.cygwin.com/">Cygwin</ulink>
259          release. This works as of the B20.1 release. Cygwin is a POSIX          release. This works as of the B20.1 release. Cygwin is a POSIX
260          system for Windows. This covers both utility programs, and a libray          system for Windows. This covers both utility programs and a library
261          that adds POSIX system calls to Windows. Among them is pseudo tty          that adds POSIX system calls to Windows. Among them is pseudo tty
262          support for Windows that emulates the POSIX pty standard. The          support for Windows that emulates the POSIX pty standard. The
263          latest Cygwin is always available from <ulink          latest Cygwin is always available from <ulink
# Line 275  into another language, under the above c Line 274  into another language, under the above c
274    <sect1 id=designgoals xreflabel="Design Goals">    <sect1 id=designgoals xreflabel="Design Goals">
275      <title>Design Goals</title>      <title>Design Goals</title>
276    
277      <para>DejaGnu grew out of the internal needs of Cygnus Solutions. (then      <para>DejaGnu grew out of the internal needs of Cygnus Solutions,
278      Cygnus Support). Cygnus maintained and enhanced a variety of free      the company formerly known as Cygnus Support. Cygnus maintained
279      programs in many different environments, and we needed a testing      and enhanced a variety of free programs in many different
280      tool that:</para>      environments and we needed a testing tool that:</para>
281    
282      <itemizedlist mark="bullet">      <itemizedlist mark="bullet">
283        <listitem><para>was useful to developers while fixing        <listitem><para>was useful to developers while fixing
284        bugs.</para></listitem>        bugs;</para></listitem>
285        <listitem><para>automated running many tests during a software        <listitem><para>automated running many tests during a software
286        release process.</para></listitem>        release process;</para></listitem>
287        <listitem><para>was portable among a variety of host        <listitem><para>was portable among a variety of host
288        computers.</para></listitem>        computers;</para></listitem>
289        <listitem><para>supported cross-development        <listitem><para>supported cross-development
290        testing.</para></listitem>        testing;</para></listitem>
291        <listitem><para>permitted testing interactive programs, like        <listitem><para>permitted testing interactive programs, like
292        <command>GDB</command>; and </para></listitem>        <command>GDB</command>; and </para></listitem>
293        <listitem><para>permitted testing batch oriented programs, like        <listitem><para>permitted testing batch oriented programs, like
# Line 301  into another language, under the above c Line 300  into another language, under the above c
300      make sure that <command>GDB</command> works as well when cross-debugging      make sure that <command>GDB</command> works as well when cross-debugging
301      as it does in a native configuration. </para>      as it does in a native configuration. </para>
302    
303      <para>Probably the greatest challenge was testing in a cross-development      <para>Probably the greatest challenge was testing in a
304      environment (which can be a real nightmare).  Most cross-development      cross-development environment.  Most cross-development
305      environments are customized by each developer.  Even when buying packaged      environments are customized by each developer.  Even when buying
306      boards from vendors there are many differences.  The communication      packaged boards from vendors there are many differences.  The
307      interfaces vary from a serial line to ethernet.  DejaGnu was designed with      communication interfaces vary from a serial line to Ethernet.
308      a modular communication setup, so that each kind of communication can be      DejaGnu was designed with a modular communication setup, so that
309      added as required, and supported thereafter.  Once a communication      each kind of communication can be added as required and supported
310      procedure is coded, any test can use it.  Currently DejaGnu can use      thereafter.  Once a communication procedure is coded, any test can
311      <command>rsh</command>, <command>rlogin</command>,      use it.  Currently DejaGnu can use <command>rsh</command>,
312      <command>telnet</command>, <command>tip</command>,      <command>rlogin</command>, <command>telnet</command>,
313      <command>kermit</command>, and <command>mondfe</command> for remote      <command>tip</command>, <command>kermit</command> and
314      communications.</para>      <command>mondfe</command> for remote communications.</para>
315    
316      </sect1>      </sect1>
317    
# Line 320  into another language, under the above c Line 319  into another language, under the above c
319        <title>A POSIX conforming test framework</title>        <title>A POSIX conforming test framework</title>
320    
321        <para>DejaGnu conforms to the POSIX 1003.3 standard for test        <para>DejaGnu conforms to the POSIX 1003.3 standard for test
322        frameworks. I was also a member of that committe.</para>        frameworks. Rob Savoye was a member of that committee.</para>
323    
324        <para>The POSIX standard 1003.3 defines what a testing framework needs to        <para>The POSIX standard 1003.3 defines what a testing framework needs to
325        provide, in order to permit the creation of POSIX conformance test        provide, in order to permit the creation of POSIX conformance test
# Line 329  into another language, under the above c Line 328  into another language, under the above c
328        to POSIX conformance.  POSIX 1003.3 does not specify a particular testing        to POSIX conformance.  POSIX 1003.3 does not specify a particular testing
329        framework, but at this time there is only one other POSIX conforming test        framework, but at this time there is only one other POSIX conforming test
330        framework: TET. TET was created by Unisoft for a consortium comprised of        framework: TET. TET was created by Unisoft for a consortium comprised of
331        X/Open, Unix International, and the Open Software Foundation.</para>        X/Open, Unix International and the Open Software Foundation.</para>
332    
333        <para>The POSIX documentation refers to <firstterm>assertions</firstterm>.        <para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
334        An assertion is a description of behavior.  For example, if a standard        An assertion is a description of behavior.  For example, if a standard
335        says ``The sun shall shine'', a corresponding assertion might be ``The        says ``The sun shall shine'', a corresponding assertion might be ``The
336        sun is shining.''  A test based on this assertion would pass or fail        sun is shining.''  A test based on this assertion would pass or fail
337        depending on whether it is daytime or nighttime.  It is important to note        depending on whether it is day or night.  It is important to note
338        that the standard being tested is never 1003.3; the standard being tested        that the standard being tested is never 1003.3; the standard being tested
339        is some other standard, for which the assertions were written.</para>        is some other standard, for which the assertions were written.</para>
340    
341        <para>As there is no test suite to test testing frameworks for POSIX        <para>As there is no testsuite to test testing frameworks for POSIX
342        1003.3 conformance, verifying conformance to this standard is done by        1003.3 conformance, verifying conformance to this standard is done by
343        repeatedly reading the standard and experimenting.  One of the main        repeatedly reading the standard and experimenting.  One of the main
344        things 1003.3 does specify is the set of allowed output messages, and        things 1003.3 does specify is the set of allowed output messages and
345        their definitions.  Four messages are supported for a required feature of        their definitions.  Four messages are supported for a required feature of
346        POSIX conforming systems, and a fifth for a conditional feature. DejaGnu        POSIX conforming systems and a fifth for a conditional feature. DejaGnu
347        supports the use of all five output messages; in this sense a test suite        supports the use of all five output messages.  In this sense a testsuite
348        that uses exactly these messages can be considered POSIX conforming.        that uses exactly these messages can be considered POSIX conforming.
349        These definitions specify the output of a test        These definitions specify the output of a test
350        case:</para>        case:</para>
# Line 393  into another language, under the above c Line 392  into another language, under the above c
392            code.  If the test isn't actually run, it must produce            code.  If the test isn't actually run, it must produce
393            <emphasis>UNRESOLVED</emphasis> rather than just leaving that test            <emphasis>UNRESOLVED</emphasis> rather than just leaving that test
394            out of the output.  This means that you have to be careful when            out of the output.  This means that you have to be careful when
395            writing tests, to not carelessly use tcl statements like            writing tests to not carelessly use Tcl commands like
396            <emphasis>return</emphasis>---if you alter the flow of control of the            <emphasis>return</emphasis>---if you alter the flow of control of the
397            tcl code you must insure that every test still produces some result            Tcl code you must insure that every test still produces some result
398            code.</para>            code.</para>
399    
400            <para>Here are some of the ways a test may wind up            <para>Here are some of the ways a test may wind up
# Line 429  into another language, under the above c Line 428  into another language, under the above c
428          <variablelist>          <variablelist>
429            <varlistentry>                      <varlistentry>          
430              <term>UNTESTED</term>              <term>UNTESTED</term>
431              <listitem><para>A test was not run.  This is a placeholder, used              <listitem><para>A test was not run.  This is a place-holder, used
432              when there is no real test case yet.</para></listitem>              when there is no real test case yet.</para></listitem>
433            </varlistentry>            </varlistentry>
434          </variablelist>          </variablelist>
# Line 453  into another language, under the above c Line 452  into another language, under the above c
452          </variablelist>          </variablelist>
453        
454          <para>DejaGnu uses the same output procedures to produce these messages          <para>DejaGnu uses the same output procedures to produce these messages
455          for all test suites, and these procedures are already known to conform          for all testsuites and these procedures are already known to conform
456          to POSIX 1003.3.  For a DejaGnu test suite to conform to POSIX 1003.3,          to POSIX 1003.3.  For a DejaGnu testsuite to conform to POSIX 1003.3,
457          you must avoid the <emphasis>setup</emphasis>xfail} procedure as          you must avoid the <emphasis>setup</emphasis>xfail} procedure as
458          described in the <emphasis>PASS</emphasis> section above, and you must          described in the <emphasis>PASS</emphasis> section above and you must
459          be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,          be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
460          as described in the <emphasis>UNRESOLVED</emphasis> section          as described in the <emphasis>UNRESOLVED</emphasis> section
461          above.</para>          above.</para>

Legend:
Removed from v.1.9  
changed lines
  Added in v.1.10

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26