bugGNU Octave - Bugs: bug #57584, spurious test "FAIL"s...

 
 

bug #57584: spurious test "FAIL"s when running test suite in Octave with "--traditional" startup parameter

Submitter:  Hartmut <hardy>
Submitted:  Sun 12 Jan 2020 08:24:22 PM UTC
   
 
Category:  Test Suite Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 16 Jan 2020 03:22:27 AM UTC, comment #13: 


> first checking to see whether there is some bug uncovered by the test when running with the option.


Sure, that's certainly worth doing for someone with the motivation. There are quite a few test failures with just disable_range set. For example, libinterp/dldfcn/amd.cc has one test failure because


>> disable_range (false)
>> zeros (1, 0)
ans = [](1x0)
>> disable_range (true)
>> zeros (1, 0)
ans = [](0x0)
>> test libinterp/dldfcn/amd.cc
***** testif HAVE_AMD
 assert (amd ([]), zeros (1,0))
!!!!! test failed
ASSERT errors for:  assert (amd ([]),zeros (1, 0))

  Location  |  Observed  |  Expected  |  Reason
     .          O(1x0)       E(0x0)      Dimensions don't match


Mike Miller <mtmiller>
Group Member
Thu 16 Jan 2020 01:55:24 AM UTC, comment #12: 

I agree we could disable tests if they don't make sense with --traditional (or some more specific option like disable_range).  But I think that should not be done without first checking to see whether there is some bug uncovered by the test when running with the option.  For example, in many cases, disable_range should only affect the amount of memory used, not the results.

John W. Eaton <jwe>
Group administrator
Wed 15 Jan 2020 08:32:59 PM UTC, comment #11: 

Looks like Hartmut and I posted within seconds, and mostly agree :)

Mike Miller <mtmiller>
Group Member
Wed 15 Jan 2020 08:30:50 PM UTC, comment #10: 

No actions have been taken yet.

My suggestions are

  • do nothing, or
  • add code to _run_test_suite_.m that forces certain settings to undo what --traditional does (for example, disable_range(false)), and restore them at the end of the function, or
  • add warnings to _run_test_suite_.m if any of the --traditional settings are set, that certain tests may fail and that should not be considered a bug.


Any changes to _run_test_suite_.m that change or check these settings further make this function specific to the Octave test suite, not a general purpose test runner as some people use it for.

My preference would be to do nothing, and just document somewhere or make it clear when it comes up that the test suite is not expected to work fully under --traditional behavior settings.

Mike Miller <mtmiller>
Group Member
Wed 15 Jan 2020 08:30:39 PM UTC, comment #9: 

@Markus: No, I think there hasn't been any action on Mike's comment #3.

I would summarize and divide up this bug report in three seperate issues:

  • A failure of a unit test in the publish function. This issue is already dealt with in bug #57439 (see comment #8).


  • A misbehavior of the run_test_suite command, reporting failing test different on the screen output than in the log file. This issue was my original intention of this bug report (see my comment #0 and comment #2). But this was an error on my side, I would now say we can close this issue as "works for me".


  • Actually failing unit tests, when Octave is running with the traditional startup option. There seem to be at least 125 of them. Mike had a look at them (see his comment #3 and comment #1). And they seem to all fail for a good reason, for example because they are using an Octave feature that was intentionally switched off by the traditional startup option. For this one there seem to be different ways "forward":
    • Just let it stay as it is now. (My personal favorite). Those tests fail for a good reason, because someone started Octave with non-standard options.
    • Change all (at least 125) unit tests, so that they will not run when the respective startup option was used that disabled an essential Octave feature that they need. Something like "missing feature" unit tests.
    • Change the run_test_suite command, so that it issues a generic message like "some tests mail fail to the the current setting of the parameters X and Y". (my second favorite, and also one of Mike's in commment #3.)
    • Change the run_test_suite command, so that always the standard options of Octave are used. (I think this would not be good, then the tests I was trying to do where not even possible.)


I will leave my "fingers away" from this bug report, now. My original intention is fulfilled. And for the (maybe) open questions I am not enough into the Octave code.

Hartmut <hardy>
Tue 14 Jan 2020 08:06:33 AM UTC, comment #8: 

The "publish.tst" error you are seeing is probably bug #57439.

I didn't follow this bug closely. Was there any action on Mike's comment #3?:

>> A sensible way forward might be to either set these options to expected values at the start of the test suite, or to error out with a message to the user that these --traditional options change the behavior of Octave sufficiently that the test suite isn't reliable.

Markus Mützel <mmuetzel>
Group administrator
Mon 13 Jan 2020 08:02:32 PM UTC, comment #7: 

Thanks for getting my "un-stuck". Yes, you are right. I now see the same as Mike, also the same as was written in his comment #4.

So what is left now of this bug report? Is it only about one failing test in publish.m when the traditional option is set? (I also see this.) Or are there more issues left that are worth keeping this bug report open?

Hartmut <hardy>
Mon 13 Jan 2020 07:30:23 PM UTC, comment #6: 

No, I think you are getting stuck on hgsave. I don't get any hgsave.m test failures in Octave 6, with or without --traditional. If you saw a test failure in hgsave.m with Octave 6, that wasn't due to --traditional.

Try "test blkdiag.m" instead. For me, it passes without --traditional, 1 test fails with --traditional (or with disable_range).

Mike Miller <mtmiller>
Group Member
Mon 13 Jan 2020 07:14:00 PM UTC, comment #5: 

Does this depend on the Octave version?

In Octave 5.1 release I get the behavior that Mike observes (his comment #4), which seems to be fine (at least in terms of behavior of the test command versus run_test_suite)


hartmut@hartmut-laptop:/opt/octave-5.1.0/bin$ ./octave --norc --traditional
GNU Octave, version 5.1.0

   [...snip...]

>> disable_diagonal_matrix
ans = 1
>> test hgsave.m
***** testif HAVE_MAGICK; any (strcmp ("gnuplot", available_graphics_toolkits ()))
 toolkit = graphics_toolkit ();

  [...snip...]

   graphics_toolkit (toolkit);
 end_unwind_protect
!!!!! test failed
'h2' undefined near line 40 column 11
>>


But in current dev Octave (hg id: f658d41003e4) I still see what I've seen before (my comment #2), which feels weired to me:


hartmut@hartmut-laptop:/opt/octave-dev/bin$ ./octave --norc --traditional
GNU Octave, version 6.0.0

   [...snip...]

>> disable_diagonal_matrix
ans = 1
>> test hgsave.m
PASSES 6 out of 6 tests
>>


Is this some "regression" in dev Octave, or am I doing something stupid, here?

Hartmut <hardy>
Sun 12 Jan 2020 09:49:57 PM UTC, comment #4: 

I can't replicate this behavior that you are describing, I can reproduce the test failures with _run_test_suite_ or with test, and the log file shows exactly what failed for me.

Mike Miller <mtmiller>
Group Member
Sun 12 Jan 2020 09:48:41 PM UTC, comment #3: 

I ran Octave with --traditional, and then reset disable_diagonal_matrix, disable_permutation_matrix, and disable_range all back to false, and ran the test suite. In this configuration, I only got 4 test failures, so most of these failures are due to the assumptions of the diagonal, permutation, and range special array types.

The 4 remaining failures:

  • savefig.m - test fails because --traditional sets 'save_default_options' to '-mat-binary', which breaks saving figures to a file
  • parser.tst - test fails because it expects the "Matlab-style short-circuit operation" warning, but that warning is disabled with --traditional
  • publish.tst - tests failed with "__imwrite__: invalid function handle", may not be related to --traditional at all, may be a plot system race condition


A sensible way forward might be to either set these options to expected values at the start of the test suite, or to error out with a message to the user that these --traditional options change the behavior of Octave sufficiently that the test suite isn't reliable.

Mike Miller <mtmiller>
Group Member
Sun 12 Jan 2020 09:38:00 PM UTC, comment #2: 

I still do not understand this! It is only run_test_suite that tells me there are test failures. When I look at the log file (from this very run) afterwards, I see not a single of these failures noted there! And when I start the test individually (still with the same startup options active) I also see no single test failure. ???

Hartmut <hardy>
Sun 12 Jan 2020 09:24:56 PM UTC, comment #1: 

I see the same, 125 test failures in my case.

I looked briefly at the log, and I think many of these test failures are due to behavior differences with the three options

  • disable_diagonal_matrix
  • disable_permutation_matrix
  • disable_range


These three are all set to true in --traditional mode, meaning diagonal, permutation, and range types are all disabled and become full matrices instead.

If you want to investigate this further, you might take a look at setting each of these options individually, run the test suite, see specifically why certain tests are failing, and then figure out if you think anything is worth fixing.

Here's one small example. When I run Octave normally and test the file lu.cc, I get 23 passed tests. When I set the option disable_diagonal_matrix, I get 1 test failure. When I set the option disable_permutation_matrix, I get 6 test failures.

Mike Miller <mtmiller>
Group Member
Sun 12 Jan 2020 08:24:22 PM UTC, original submission:  

I have observed this with a self compiled current dev Octave under linux. I don't think it's a big problem, but will report it here anyways.

Here is how to reproduce it:

  • start Octave (gui or cli) with the startup option "--traditional"
  • run the full test suit via run_test_suite (wait a bit...)
  • at the end it reports many FAILs of tests, 91 FAILS in my case.
  • search the text output of run_test_suite inside the console to pick one function with a FAIL, e.g. hgsave.m or copyobj.m.
  • manually run the tests of this function, e.g. "test hgsave.m" to see that there are actualla NO test failures.
  • have a look in fntests.log. There are also NO real test failures noted.


My conclusion is that the "problem" is somewhere in the routine of run_test_suite itself, since all tests I run manually eventually passed.

I don't know if this is a "legal" use case of run_test_suite at all. But it makes it hard to evaluate if some real tests fail when using this startup option. (I wanted to check this, because of the now solved bug #57528.)

Hartmut <hardy>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by hardy (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code