Thu 05 Jan 2017 05:15:45 PM UTC, comment #4:
Regarding comment #2: If those functions are going to be dropped, why aren't they deprecated right now? Maybe one can add a pointer to use the statistics package.
Regarding comment #3: That makes a blind copy&past to core less desirable, if the code contains errors. I tried some the tests from the Matlab documentation ([1] in comment #1) but those examples are working. Unless Octave has a Matlab compatible rng-function (bug #42557) those tests are ugly "hand-writing" of many numbers.
I suggest to first fix the issues in `signtest` from the statistics package and then copy&paste the code to a deprecated octave core function `sign_test`.
Maybe Arno Onken as package maintainer has more insight into that issue.
|
Thu 05 Jan 2017 04:24:36 AM UTC, comment #3:
As far as the sign test is not a general purpose algorithm, it sounds reasonable to leave it to the statistics package.
Unfortunately, it seems that signtest() is not entirely correct too, at least when tail = 'right':
Here we have p = 1 - binocdf(w, n, 0.5) = 1 - P(X <= w) = P(X > w), so X = w is omitted. But p-value is by definition the probability to observe the same or more extreme value under H0.
More problems I've noticed with signtest():
1) ties aren't excluded
2) incorrect results with tail = 'left':
We checked if a vector of a thousand -1's and one 1 has a median less than 0, and signtest() rejected this hypothesis! In the source we have p = binocdf(w, n, 0.5) = P(X <= sum(x < 0)). So, the more components of x are less than 0, the closer p is to 1, but it should be the other way around.
@Kai Torben Ohlhus: Alas, I don't have any tests. I've found a paper on the sign test (Dixon, Wilfrid J., and Alexander M. Mood. "The statistical sign test." Journal of the American Statistical Association 41.236 (1946): 557-566), and it includes a table of critical values, but that wouldn't be convenient for tests. I suppose, the needed values could be generated using some trustworthy statistics software.
|
Tue 03 Jan 2017 01:01:48 AM UTC, comment #1:
Thank you for the report. The function `sign_test` hasn't been worked on for years and is in contrast to `signtest` from the statistics package more Matlab compatible [1]. To improve the situation, I see the options:
a) Replace the current `sign_test` by `signtest` from the statistics package.
b) Fix `sign_test` by an expert of that area.
Personally, I favor option a), as b) might take a long time to happen.
@Kirill Pushkaryov: Can you add some references to known tests for that function?
[1]: https://www.mathworks.com/help/stats/signtest.html
|
Sat 31 Dec 2016 05:53:38 AM UTC, original submission:
The function sign_test() from core Octave (ver. 4.0.0) returns a seemingly incorrect p-value.
In Octave command window:
At the same time, signtest() function from the statistics package (ver. 1.2.4) gives a different result:
SciPy's scipy.stats.binom_test(3, 5) returns 1.0 too.
I inspected the source code of sign_test() and believe that it improperly handles the boundary points. In sign_test.m we have:
By default alt = "!=". So, mathematically cdf = F(b) = P(X <= b) and 1 - cdf = P(X > b), where X has binomial distribution with parameters n, 0.5. Depending on whether cdf < 1 - cdf (i.e. cdf < 0.5) or cdf > 0.5 we have pval = 2 * P(X <= b) or pval = 2 * P(X > b). The first case correctly includes X = b, but the second does not. A similar problem will occur with alt = ">".
sign_test.m in Octave 4.2.0 contains the same code.
|