Received  a bug resolution report from math works on the fillmissing inconsistencies. Probably worth reviewing bists against a recent Matlab version to see where things stand.

attached are updated ismissing and rmmissing functions that are compatible with Matlab 2022a, and also correct a corner case in ismissing where NaN is supplied as an indicator. Also attached fillmissing is complete, including all of the many corner cases I could find. Turs out the optional second output has many inconsistencies between methads that I wouldn't be surprised to see Matlab change in upcoming versions. If it matters, a future effort can pare down the BIST count to identify any redundant tests. The numerous method codepaths made it easier to duplicate tests for each path.

these updated files are pushed as:
https://sourceforge.net/p/octave/statistics/ci/b7d3493949a7

updated status to Ready for Test. also renamed to remove 'missing' from the patch description.  probably best to move 'missing' implementation to a separate feature request report if anything, as it may be a while before classdef is able to support a dynamic typing variable as described in comment #13.


(file #53210, file #53211, file #53212)

just uploading a partial implementation of fillmissing attached. let it sit for a while, but as suspected all of the different method options, including different handling of middle-versus-end missing values made this a bit more convoluted. Also, thought I could just mass most of the methods off to interp1 (since the methods seem drawn from interp1), only to find out interp1 fails when there's only 1 data point in that column for some methods while fillmissing doesn't. Oh, octave also still doesn't have a makima implementation. when it does this will work, but currently i just have interp1 issuing the error so nothing has to change when it gets added. that could probably be cleaned up.

anyway, the helptext is currently complete for eventhing that should get finished, and that don't rely on currently unimplemented classes. all of the property-value pairs are implemented, (samplepoints, endvalues, missinglocations, maxgap), all of the standard 'methods' are implemented. if i changed the error handling, the way I did it could expand the 'endvalues' handling to include all options independently of whats' done for the rest of the array, extending the function a bit beyond what matlab's does. may look at that at the end.

what's still missing is just the moving window methods and the fill-function method.

Once those are done i'll consider this complete (but far from optimized) for Octave's current class-set.  lots of code could probably move into subfunctions, but lately we've notice a lot of function call overhead I was trying to avoid, in addition to being lazy about looking for repeat code. lots of BISTs for all of the different option combinations. I kept finding exceptions so kept adding. some of the redundant ones could be trimmed later.

m-file attached for anyone needing this function in partial form before i get around to finishing it.

(file #53002)

Ok will do. Saw 1.4.3 was out, but news wasn’t bumped to 1.4.4 and didn’t see a bookmark like for 1.4.2. Didn’t know if there was any other maintenance items to be done.  I’ll push those and update news later today.

ok. noticed a typo in ismissing and the other in standardizeMissing. this patch includes the corrected ismissing, updates to rmmisssing, and adds the corrected standardizeMissing. (No changes made to the use of ismember yet, so the code is simple but it's not fast.).

fillmissing is pending but it'll take a bit longer with all of the options. will create a separate patch for that later that will depend on this one (for ismissing). at that point i'll also attached a zip with the four functions files in their fully working form.

(file #52522)

also, doing some time testing the loops in ismissing (some have faster alternatives, some are suprisingly fast), I'm noticing just how terribly slow the 'ismember' line is in standardizeMissing. It's nice and simple and clean, but might look for alternatives there

Shall we leave this open as "postponed" with low priority until ismissing/rmmissing also appear in core Octave?

Thanks, I changed the subdirectory from install-conditionally/distributions to /base and pushed it
https://sourceforge.net/p/octave/statistics/ci/7950d2715a5c55e822b2365b32e5fa90bd502d3a/

We can add these functions to the Statistics package if someone prepares a patch. Should they be in the install-conditionally subdirectory to allow for being eventually added to core Octave?

@Nicholas:
Even if they're accepted in core Octave (as I think they should be) they also need to be in the statistics package for some time to cater for Octave <= 6.4.0 if functions like e.g., pca.m in the statistics package are to be dependent on them.

Thanks.
It'll take a little time, unforeseen busy times at work.

After a quick glance I think the patches look good. However, in the scripts directory in help/__unimplemented__.m the stanzas about rmmissing and ismissing should be dropped as well.

Before you adapt that, let's first get the opinion of a core developer, in particular the one who some years ago move loads of the Octave core statistics functions to the statistics package (with good reason, many of those functions needed and still need overhaul and are too much of a maintenance burden for Octave).
IMO ismissing and rmmissing should be in core - they are quite useful outside the field of statistics as well.

>> ismissing
error: 'ismissing' undefined near line 1, column 1

The 'ismissing' function belongs to the statistics package from Octave
Forge which you have installed but not loaded.  To load the package, run
'pkg load statistics' from the Octave prompt.

Please read <https://www.octave.org/missing.html> to learn how you can
contribute missing functionality.

OK, I made a patch.

comment #1:

> I wonder why rmmissing.m contains a switch statement where an if/elseif block would suffice; is it to be able to extend to string classes, if ever implemented in core Octave?
>

That was the diea.

comment #1:

> Additionally I think there's no harm in extending beyond Matlab compatibility; I can see use cases for mixed-type cell arrays (e.g., output of csv2cell() or xlsread.m and the like).
>

Interesting, maybe later. For now I'd like to see 'pca' and its requirements implemented.

(file #51860)

These are the functions ismissing and rmmissing, for finding and removing missing data from a set (vector, matrix etc.).

I am not sure about the package they belong to.

Example:

octave:1> a = [1 2 3 NaN 4 5 NaN 7 8];

octave:2> ismissing(a)
ans =

  0  0  0  1  0  0  1  0  0

octave:3> rmmissing(a)
ans =

   1   2   3   4   5   7   8