bugGNU Octave - Bugs: bug #49961, sign_test() returns incorrect...

 
 

bug #49961: sign_test() returns incorrect p-value

Submitted by:  Kirill Pushkaryov <kpushkaryov>
Submitted on:  Sat 31 Dec 2016 05:53:38 AM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: ConfirmedAssigned to: None
Originator Name: Open/Closed: Open
Release: 4.0.0Operating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 05 Jan 2017 05:15:45 PM UTC, comment #4:

Regarding comment #2: If those functions are going to be dropped, why aren't they deprecated right now? Maybe one can add a pointer to use the statistics package.

Regarding comment #3: That makes a blind copy&past to core less desirable, if the code contains errors. I tried some the tests from the Matlab documentation ([1] in comment #1) but those examples are working. Unless Octave has a Matlab compatible rng-function (bug #42557) those tests are ugly "hand-writing" of many numbers.

I suggest to first fix the issues in `signtest` from the statistics package and then copy&paste the code to a deprecated octave core function `sign_test`.

Maybe Arno Onken as package maintainer has more insight into that issue.

Kai Torben Ohlhus <siko1056>
Project Member
Thu 05 Jan 2017 04:24:36 AM UTC, comment #3:

As far as the sign test is not a general purpose algorithm, it sounds reasonable to leave it to the statistics package.

Unfortunately, it seems that signtest() is not entirely correct too, at least when tail = 'right':

Here we have p = 1 - binocdf(w, n, 0.5) = 1 - P(X <= w) = P(X > w), so X = w is omitted. But p-value is by definition the probability to observe the same or more extreme value under H0.

More problems I've noticed with signtest():
1) ties aren't excluded
2) incorrect results with tail = 'left':

We checked if a vector of a thousand -1's and one 1 has a median less than 0, and signtest() rejected this hypothesis! In the source we have p = binocdf(w, n, 0.5) = P(X <= sum(x < 0)). So, the more components of x are less than 0, the closer p is to 1, but it should be the other way around.

@Kai Torben Ohlhus: Alas, I don't have any tests. I've found a paper on the sign test (Dixon, Wilfrid J., and Alexander M. Mood. "The statistical sign test." Journal of the American Statistical Association 41.236 (1946): 557-566), and it includes a table of critical values, but that wouldn't be convenient for tests. I suppose, the needed values could be generated using some trustworthy statistics software.

Kirill Pushkaryov <kpushkaryov>
Tue 03 Jan 2017 04:17:34 PM UTC, comment #2:

At the next release of Octave the excess statistics functions are going to be dropped from core and moved to the statistics package. For the time being, we could copy "signtest" to "sign_test" on the stable branch. How does that sound?

Rik <rik5>
Project Administrator
Tue 03 Jan 2017 01:01:48 AM UTC, comment #1:

Thank you for the report. The function `sign_test` hasn't been worked on for years and is in contrast to `signtest` from the statistics package more Matlab compatible [1]. To improve the situation, I see the options:

a) Replace the current `sign_test` by `signtest` from the statistics package.

b) Fix `sign_test` by an expert of that area.

Personally, I favor option a), as b) might take a long time to happen.

@Kirill Pushkaryov: Can you add some references to known tests for that function?

[1]: https://www.mathworks.com/help/stats/signtest.html

Kai Torben Ohlhus <siko1056>
Project Member
Sat 31 Dec 2016 05:53:38 AM UTC, original submission:

The function sign_test() from core Octave (ver. 4.0.0) returns a seemingly incorrect p-value.
In Octave command window:

At the same time, signtest() function from the statistics package (ver. 1.2.4) gives a different result:

SciPy's scipy.stats.binom_test(3, 5) returns 1.0 too.

I inspected the source code of sign_test() and believe that it improperly handles the boundary points. In sign_test.m we have:
By default alt = "!=". So, mathematically cdf = F(b) = P(X <= b) and 1 - cdf = P(X > b), where X has binomial distribution with parameters n, 0.5. Depending on whether cdf < 1 - cdf (i.e. cdf < 0.5) or cdf > 0.5 we have pval = 2 * P(X <= b) or pval = 2 * P(X > b). The first case correctly includes X = b, but the second does not. A similar problem will occur with alt = ">".

sign_test.m in Octave 4.2.0 contains the same code.

Kirill Pushkaryov <kpushkaryov>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #39335:  sign_test_bug.m added by kpushkaryov (203B - text/x-objcsrc - Demonstration code)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by siko1056 (statistics package maintainer)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by siko1056 (Posted a comment)
  • -unavailable- added by kpushkaryov (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 05 Jan 2017 05:15:45 PM UTCsiko1056Carbon-Copy-=>Added -unavailable-
    Tue 03 Jan 2017 01:01:48 AM UTCsiko1056StatusNone=>Confirmed
    Sat 31 Dec 2016 05:53:38 AM UTCkpushkaryovAttached File-=>Added sign_test_bug.m, #39335

    Back to the top


    Powered by Savane 3.1-cleanup1