patchGNU Octave - Patches: patch #9633, Add test for correct exit status...

 
 

patch #9633: Add test for correct exit status behaviour

Submitter:  Karl Wette <kwwette>
Submitted:  Tue 08 May 2018 08:08:27 AM UTC
   
 
Category:  Core : other Priority:  7 - High
Status:  Need Info Privacy:  Public
Assigned to:  None Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 03 Jan 2020 04:39:25 PM UTC, comment #5: 

I guess I find writing portable shell scripts to be tedious to get right (See https://www.gnu.org/software/autoconf/manual/autoconf-2.64/html_node/Portable-Shell.html).  But, if someone can do it I have no objection.

The tests I was proposing could go in their own .tst file under the test/ directory or within their own directory under the test/ directory so I think it is easily extensible.

I hadn't been targeting running in existing installations, but it seems like it wouldn't be that hard to distinguish between run-octave and a regular octave run.

When using run-octave,


program_invocation_name
ans = /home/rik/wip/Projects_Mine/octave-dev/src/.libs/octave-gui


When using an installed version of Octave


program_invocation_name
ans = /usr/local/libexec/octave/5.1.0/exec/x86_64-pc-linux-gnu/octave-gui



Rik <rik5>
Group administrator
Fri 03 Jan 2020 03:24:05 AM UTC, comment #4: 

It's certainly possible to rewrite some commands in terms of the system function. However, it's a more awkward way to write these tests compared with shell scripts. It doesn't seem easily extensible to add more tests in the future.

The test would ideally work with both "run-octave" in the build directory and with a fully-installed Octave, just like the rest of the test suite, right?

For those interested, I am still maintaining and continuously running a more complete set of tests of Octave's command line interface than just a few exit status validations at https://gitlab.com/mtmiller/octave-test-suite.

Mike Miller <mtmiller>
Group Member
Thu 02 Jan 2020 11:01:16 PM UTC, comment #3: 

@Mike: Testing for the exit status seems like a useful behavior.  However, I'd rather avoid a shell script (which can be different between different Unix flavors and certainly different on Windows).  I wonder if the 4 tests can be re-written to occur within Octave by using the system() call?

For example, I re-wrote the third test (non-zero exit status) as


[st,out] = system ("./run-octave -f --eval 'exit(2)'")
st = 2
out =


What do you think?

Rik <rik5>
Group administrator
Tue 08 May 2018 04:55:59 PM UTC, comment #2: 

In the mean time, I have added equivalent tests to my external Octave test suite project

https://github.com/mtmiller/octave-test-suite/commit/8c43e64499c9bcb301320770bf5abc03e3d5e73e

which I hope can be merged into Octave at some point, but at the very least is continuously testing against Octave development to help prevent regressions.

Mike Miller <mtmiller>
Group Member
Tue 08 May 2018 08:22:25 AM UTC, comment #1: 

(Apologies: the original submission had some formatting problems.)

I use Octave primarily as a command-line script interpreter (i.e. I run executable scripts with #!/usr/bin/octave to invoke Octave) for scientific applications on computer clusters. It is important for this application that Octave conform to standard exit status behaviour:

  • zero exit status is returned on success, either through normal completion or by calling exit(0), and


  • non-zero exit status is returned on failure, either through raising an error() or by calling exit(n) with n != 0.


In Octave 4.2.2 the standard exit status behaviour was broken:

> $ octave --version
> GNU Octave, version 4.2.2
> ...
>
> $ echo '0;' > script.m; octave script.m; echo $?
> 0   # ok
>
> $ echo 'error("msg");' > script.m; octave script.m; echo $?
> error: msg
> error: called from
>     script at line 1 column 1
> 0   # should return not 0
>
> $ echo 'exit(0);' > script.m; octave script.m; echo $?
> terminate called after throwing an instance of 'octave::exit_exception'
> panic: Aborted -- stopping myself...
> Aborted
> 134   # should return 0
>
> $ echo 'exit(1);' > script.m; octave script.m; echo $?
> terminate called after throwing an instance of 'octave::exit_exception'
> panic: Aborted -- stopping myself...
> Aborted
> 134   # should return 1


However it seems to have been fixed recently in development:

> $ cat ./HG-ID
> 3d5f953e2ef6
>
> $ echo '0;' > script.m; ./run-octave script.m; echo $?
> 0   # ok
>
> $ echo 'error("msg");' > script.m; ./run-octave script.m; echo $?
> error: msg
> error: called from
>     script at line 1 column 1
> 1   # ok
>
> $ echo 'exit(0);' > script.m; ./run-octave script.m; echo $?
> 0   # ok
>
> $ echo 'exit(1);' > script.m; ./run-octave script.m; echo $?
> 1   # ok


I was unable to find a unit test for Octave's exit status behaviour, so the attached patch adds such a unit test to ensure the exit status behaviour is not broken in future. The patch adds a script ./test-octave-exit-status.sh which runs Octave code snippets either as a script or using --eval, and then tests Octave's exit status for the standard behaviour given above.

The script ./test-octave-exit-status.sh is called from "make check" before the rest of the Octave test suite; a failed test in ./test-octave-exit-status.sh will cause "make check" to fail; since the exit status tests are separate from the rest of the Octave test suite, a silent test failure might otherwise go unnoticed.

Karl Wette <kwwette>
Tue 08 May 2018 08:08:27 AM UTC, original submission:  

I use Octave primarily as a command-line script interpreter (i.e. I run
executable scripts with #!/usr/bin/octave to invoke Octave) for scientific
applications on computer clusters. It is important for this application that
Octave conform to standard exit status behaviour:

  • zero exit status is returned on success, either through normal completion or

  by calling exit(0), and

  • non-zero exit status is returned on failure, either through raising an error()

  or by calling exit(n) with n != 0.

In Octave 4.2.2 the standard exit status behaviour was broken:

----------------------------------------------------------------

$ octave --version
GNU Octave, version 4.2.2
...

$ echo '0;' > script.m; octave script.m; echo $?

  1.   # ok


$ echo 'error("msg");' > script.m; octave script.m; echo $?
error: msg
error: called from
    script at line 1 column 1

  1.   # should return not 0


$ echo 'exit(0);' > script.m; octave script.m; echo $?
terminate called after throwing an instance of 'octave::exit_exception'
panic: Aborted -- stopping myself...
Aborted
134   # should return 0

$ echo 'exit(1);' > script.m; octave script.m; echo $?
terminate called after throwing an instance of 'octave::exit_exception'
panic: Aborted -- stopping myself...
Aborted
134   # should return 1

----------------------------------------------------------------

However it seems to have been fixed recently in development:

----------------------------------------------------------------

$ cat ./HG-ID
3d5f953e2ef6

$ echo '0;' > script.m; ./run-octave script.m; echo $?

  1.   # ok


$ echo 'error("msg");' > script.m; ./run-octave script.m; echo $?
error: msg
error: called from
    script at line 1 column 1
1   # ok

$ echo 'exit(0);' > script.m; ./run-octave script.m; echo $?

  1.   # ok


$ echo 'exit(1);' > script.m; ./run-octave script.m; echo $?
1   # ok

----------------------------------------------------------------

I was unable to find a unit test for Octave's exit status behaviour, so the
attached patch adds such a unit test to ensure the exit status behaviour is not
broken in future. The patch adds a script ./test-octave-exit-status.sh which
runs Octave code snippets either as a script or using --eval, and then tests
Octave's exit status for the standard behaviour given above.

The script ./test-octave-exit-status.sh is called from "make check" before the
rest of the Octave test suite; a failed test in ./test-octave-exit-status.sh
will cause "make check" to fail; since the exit status tests are separate from
the rest of the Octave test suite, a silent test failure might otherwise go
unnoticed.

Karl Wette <kwwette>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44119:  test-octave-exit-status.patch added by kwwette (4KiB - text/x-patch - Fixed a minor bug in changes to test/module.mk)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by kwwette (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-01-02 rik5 StatusNone Need Info
    2019-03-07 mtmiller Carbon-CopyRemoved 80942 -
    2018-05-08 kwwette Attached File- Added test-octave-exit-status.patch, #44119
    2018-05-08 kwwette Attached File- Added test-octave-exit-status.patch, #44118

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code