patchGNU Octave - Patches: patch #10103, Add weighted standard deviation...

 
 

patch #10103: Add weighted standard deviation feature to 'std'

Submitter:  None
Submitted:  Mon 30 Aug 2021 09:42:10 AM UTC
   
 
Category:  Core : new feature Priority:  5 - Normal
Status:  Done Privacy:  Public
Assigned to:  None Originator Email:  -email is unavailable-
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 28 Mar 2022 07:47:29 PM UTC, comment #18: 

no issues reported in a few months, closing this as fixed.

Nicholas Jankowski <nrjank>
Group Member
Wed 01 Dec 2021 06:33:03 PM UTC, comment #17: 

found a few corner cases that needed adjustment. (for n=1, missed vector W size check, and needed to ignore any scalar value of W if n=1).  Added/updated BISTs accordingly, and made some adjustments to the docstring. Verified tests vs matlab.

Everything looks good over here, pushed to stable as
http://hg.savannah.gnu.org/hgweb/octave/rev/bc0de453fb6a

Since i'm new to this, attached a v2 patch here as well in case someone needs to back that out /fix it.

(file #52400)

Nicholas Jankowski <nrjank>
Group Member
Thu 25 Nov 2021 04:18:45 AM UTC, comment #16: 

@nrjank: If you think the patch is ready, can you push it to Octave 7 (stable) during the early development stage?

Kai Torben Ohlhus <siko1056>
Group Member
Wed 03 Nov 2021 04:00:09 AM UTC, comment #15: 

Ok.  was fairly straightforward as I mentioned below. I added it and a bunch of empty array tests. they pass all tests. attached is a single patch and the modified files. that incorporate the changes to std and var.

(file #52189, file #52190, file #52191)

Nicholas Jankowski <nrjank>
Group Member
Tue 02 Nov 2021 03:44:56 AM UTC, comment #14: 

Ok, I'm noticing a number of test failures.

before making a patch, always run 'test <function>' to be sure there aren't any surprises.

a few seem to be %!error tests in std that just needed variable names or error message text updated now that var is generating the messages:


Expected <OPT must be 0 or 1>, but got <var: W must be 0 or 1 or a vector of positive integers>

Expected <DIM must be an integer>, but got <var: DIM must be a positive integer scalar, vector, or 'all'>


Easy enough to fix. but more importantly, both var and std have issues with empty array handling:

While you force var([]) and std([]) to return NaN like matlab using an 'isempty', this fails the following test in std:


>> test std
***** assert (std (ones (1,3,0,2)), ones (1,3,0,2))
!!!!! test failed
ASSERT errors for:  assert (std (ones (1, 3, 0, 2)),ones (1, 3, 0, 2))

  Location  |  Observed  |  Expected  |  Reason
     .          O(1x1)     E(1x3x0x2)    Dimensions don't match


Empty arrays can have dimensionality, and that can matter if we want to maintain mathematical consistency:
https://octave.org/doc/latest/Empty-Matrices.html

it turns out that the test above is actually incorrect in matlab, and it should be:

assert (std (ones (1,3,0,2)), ones (1,1,0,2))


Maybe that was a matlab change as they not too long ago tried to bring consistency to their NaN and empty array handling throughout all of their functions. if dim=1 you get ones(1,3,0,2), but if dim=2, you don't.

There should be, but isn't a corresponding test in var.


So, noting that the same empty array can be created with NaN(1,0). i think in the past rather than just check isempty I did something like:

i did something like

if isempty(x)
  outsize = size(x);
  outsize(dim)=1;
  retval = NaN(outsize);
else
...


Now, that's not quite enough of course.  because in matlab the empty array processing seems to crash into the 2D vector handling algorithm.  see all the following:

Matlab 2021a:

>> var([])
ans =
   NaN
>> var([],[],1)
ans =
  1×0 empty double row vector
>> var([],[],2)
ans =
  0×1 empty double column vector
>> var([],[],3)
ans =
     []
>> std(ones(0,0))
ans =
   NaN
>> std(ones(1,0))
ans =
   NaN
>> std(ones(0,1))
ans =
   NaN
>> std(ones(1,0),[],1)
ans =
  1×0 empty double row vector
>> std(ones(1,0),[],2)
ans =
   NaN
>> std(ones(0,1),[],1)
ans =
   NaN
>> std(ones(0,1),[],2)
ans =
  0×1 empty double column vector



and adding in the nd array handling:

>> std(ones(1,3,0,2),[],1)
ans =
  1×3×0×2 empty double array
>> std(ones(1,3,0,2),[],2)
ans =
  1×1×0×2 empty double array
>> std(ones(1,3,0,2),[],3)
ans(:,:,1,1) =
   NaN   NaN   NaN
ans(:,:,1,2) =
   NaN   NaN   NaN
>> std(ones(1,3,0,2),[],4)
ans =
  1×3×0 empty double array


there's quite a few things to try to capture. again, let me look at how I handled something similar before and i'll see if it can just drop in here.


Nicholas Jankowski <nrjank>
Group Member
Mon 01 Nov 2021 03:49:14 PM UTC, comment #13: 

At first glance they apply clean to the latest tip. I'll give them a look.


Nicholas Jankowski <nrjank>
Group Member
Mon 01 Nov 2021 08:41:41 AM UTC, comment #12: 

Oops, Savannah swallowed by comment 😥

Thanks for the new patches Stefano Guidoni.  I did not review them in detail, but it looks good so far 👍

@nrjank: Are you still "in progress" on this?

Kai Torben Ohlhus <siko1056>
Group Member
Mon 01 Nov 2021 08:36:47 AM UTC, comment #11: 

Thanks for the new patches

Kai Torben Ohlhus <siko1056>
Group Member
Sun 31 Oct 2021 04:29:18 PM UTC, comment #10: 

Patch to make "std" the square root of "var". Every feature of "var" is automatically applied to "std" too.

(file #52177)

Anonymous
Sun 31 Oct 2021 12:43:06 PM UTC, comment #9: 

So I made two patches for "var", the first one allows to compute weighted variance, the second one allows to compute variance over an array splice (that is on multiple dimensions) or on "all" dimensions. When using multiple dimensions, including "all", weighted variance is not available, as it is for the commercial counterpart of Octave.


octave:1> help var

 -- var (X)
 -- var (X, W)
 -- var (X, W, DIM)
 -- var (X, W, "ALL")
     Compute the variance of the elements of the vector X.

     The variance is defined as

          var (X) = 1/(N-1) SUM_i (X(i) - mean(X))^2

     where N is the length of the X vector.

     If X is a matrix, compute the variance for each column and return
     them in a row vector.

     The argument W determines the weighting scheme to use.  Valid
     values are

     0:
          normalize with N-1, provides the square root of the best
          unbiased estimator of the variance [default]

     1:
          normalize with N, this provides the square root of the second
          moment around the mean

     a vector:
          compute the weighted variance with nonnegative scalar weights,
          the length of W must be equal to the size of X along dimension
          DIM

     If N is equal to 1 the value of OPT is ignored and normalization by
     N is used.

     The optional variable DIM can be used to force 'var' to operate
     over the specified dimension.  DIM can either be a scalar dimension
     or a vector of non-repeating dimensions over which to operate.
     Dimensions are positive integers.  When DIM is a vector, W must be
     either 0 or 1, and the variance is calculated over the array slice
     defined by DIM.

     Specifying dimension "ALL" will force 'var' to operate on all
     elements of X, when W is either 0 or 1, and is equivalent to 'var
     (X(:))'.

     See also: cov, std, skewness, kurtosis, moment.

Additional help for built-in functions and operators is
available in the online version of the manual.  Use the command
'doc <topic>' to search the manual index.

Help and information about Octave is also available on the WWW
at https://www.octave.org and via the help@octave.org
mailing list.


Usage of "all" and multiple dimensions:

octave:2> A = reshape ([1:8], 2, 2, 2);
octave:3> var(A, 0, [1 2])
ans =

ans(:,:,1) = 1.6667
ans(:,:,2) = 1.6667

octave:4> var(A, 0, [1 3])
ans =

   5.6667   5.6667

octave:5> var(A, 0 , 'all')
ans = 6
octave:6> var(A, 0, [1 2 3])
ans = 6


(file #52175, file #52176)

Anonymous
Thu 02 Sep 2021 02:00:49 PM UTC, comment #8: 


comment #7:

> what other functions does the weighting apply to?
>


It is used by mean, std, var, maybe others. On top of that it should apply to 'center' too.
However, to say the truth, I think that std in MATLAB is just:

retval = sqrt (var (varargin{:}));


So var.m should be something like:

den = sum (w);
mu = mean (x, dim, w);
retval = sum (w .* ((x .- mu) .^ 2)) / den;


or


den = sum (w);
cx = center (x, dim, w);
retval = sum (w .* (cx .^ 2)) / den;



And std.m should be just its square root, maybe with a bit of input checking before that.

Anonymous
Thu 02 Sep 2021 12:45:07 PM UTC, comment #7: 

there are a few outstanding features for a number of functions like std.  by all means if it's simple enough to add, that's fine, but it's not mandatory if you'd prefer to keep this patch tied to this feature. some have been addressed elsewhere, but not all solved yet:

- std and other functions need the "all" dim to work. easy, but things like weights make it slightly less simple than just adding x = x(:). (see bug #58116)

- nanflag - bug #50571, bug #50007

what other functions does the weighting apply to?


Nicholas Jankowski <nrjank>
Group Member
Thu 02 Sep 2021 10:43:46 AM UTC, comment #6: 

I'll post a new patch shortly.

comment #5:

> Are there any other corner cases that should be checked?  (matlab accepts inf and nan elements for W, is this output compatible?)
>


Yes, the answer is NaN when NaN or Inf values are there. I'll add a test case to show that. MATLAB also has got an option to omit NaN values, which is easy to add.

comment #5:

> Also, i didn't step through it but looking at line 131, should that be "./" ? is den guaranteed to be a scalar?
>


Yes, because the weight vector is, well, a vector, hence the sum over its elements is a scalar. I'll add a test case to show that matrices are not allowed.

As a side note, I'd like to point out that there is more work to do anyway. Weighted standard deviation depends on a weighted mean, which is not available at the moment (I compute the weighted mean, without using the mean function).

Anonymous
Wed 01 Sep 2021 07:53:50 PM UTC, comment #5: 

this looks pretty good. passes all tests and applies cleanly.

Are there any other corner cases that should be checked?  (matlab accepts inf and nan elements for W, is this output compatible?).  Also, i didn't step through it but looking at line 131, should that be "./" ? is den guaranteed to be a scalar?

A couple style points -

You have a few blank lines with whitespace.  Preferred to trim all trailing whitespace on lines with and without code. (supposedly that octave editor feature will be implemented in v7. at the moment I try to remember to edit the file in notepad++ and use the 'trim trailing whitespace' command.)

the patch should probably add a note to NEWS in the matlab compatibility section mentioning the extended functionality.

the commit header should be a bit more explicit for patches to core octave, including the patch or bug number, and a line for each file touched. see https://wiki.octave.org/Commit_message_guidelines

Nicholas Jankowski <nrjank>
Group Member
Wed 01 Sep 2021 04:54:07 PM UTC, comment #4: 

This should be a suitable patch.

I added three calculation tests.

(file #51861)

Anonymous
Mon 30 Aug 2021 05:47:51 PM UTC, comment #3: 

octave core uses mercurial, not git, for source management.  see https://wiki.octave.org/Mercurial for instructions on cloning the source and producing a changeset/patch. 


Nicholas Jankowski <nrjank>
Group Member
Mon 30 Aug 2021 04:56:13 PM UTC, comment #2: 

Do I need to use "git" or a "diff -cp" will suffice?

I'll add the tests for the calculations.

Anonymous
Mon 30 Aug 2021 03:09:55 PM UTC, comment #1: 

thanks, this is helpful, and also a boost to matlab compatibility.

since this is really a modification of std, these are best prepared as patches against the existing codebase. Are you set up to do that?

It could also use a couple tests at the end to verify correct calculation related to the vector weighting.


Nicholas Jankowski <nrjank>
Group Member
Mon 30 Aug 2021 09:42:10 AM UTC, original submission:  

This is the std function with weighted standard deviation.

It belongs to the core, but it could be added to the statistics package if you do not want it in the core.

Example:

octave:6> std(x)
ans =

   1.5239   1.5811   2.8067

octave:7> w = [1 7 8 5 19 3 4 27 1 2]
w =

    1    7    8    5   19    3    4   27    1    2

octave:8> std(x, w)
ans =

   1.5995   1.4361   2.2017

octave:9> help std
 -- std (X)
 -- std (X, W)
 -- std (X, W, DIM)
     Compute the standard deviation of the elements of the vector X.

     The standard deviation is defined as

          std (X) = sqrt ( 1/(N-1) SUM_i (X(i) - mean(X))^2 )

     where N is the number of elements of the X vector.

     If X is a matrix, compute the standard deviation for each column
     and return them in a row vector.

     The argument W determines the weighting scheme to use.  Valid
     values are

     0:
          normalize with N-1, provides the square root of the best
          unbiased estimator of the variance [default]

     1:
          normalize with N, this provides the square root of the second
          moment around the mean

     a vector:
          compute the weighted standard deviation with nonnegative
          scalar weights, the length of W must be equal to the size of X
          along dimension DIM

     If the optional argument DIM is given, operate along this
     dimension.

     See also: var, bounds, mad, range, iqr, mean, median.


Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #52400:  var_std_weightsdims_patch10103_v2.diff added by nrjank (16KiB - application/octet-stream)
file #52189:  var_std_weightsdims_patch10103.diff added by nrjank (15KiB - application/octet-stream - single patch for weights and dims + empty array handling, and patched std + var files.)
file #52190:  var.m added by nrjank (9KiB - text/plain - single patch for weights and dims + empty array handling, and patched std + var files.)
file #52191:  std.m added by nrjank (5KiB - text/plain - single patch for weights and dims + empty array handling, and patched std + var files.)
file #52177:  std.diff added by None (4KiB - text/x-patch - std = sqrt(var))
file #52175:  var.diff added by None (5KiB - text/x-patch - var.diff: weighted variance --- var_dims: variance over multiple dimensions )
file #52176:  var_dims.diff added by None (6KiB - text/x-patch - var.diff: weighted variance --- var_dims: variance over multiple dimensions )
file #51861:  std.diff added by None (4KiB - text/x-patch - hg export -r tip)
file #51838:  std.m added by None (5KiB - text/x-objcsrc)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by siko1056 (Interested)
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 18 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-03-28 nrjank Open/ClosedOpen Closed
    2022-03-28 nrjank StatusReady For Test Done
    2021-12-01 nrjank Attached File- Added var_std_weightsdims_patch10103_v2.diff, #52400
        StatusNeed Info Ready For Test
    2021-11-25 siko1056 StatusIn Progress Need Info
    2021-11-03 nrjank Attached File- Added var_std_weightsdims_patch10103.diff, #52189
        Attached File- Added var.m, #52190
        Attached File- Added std.m, #52191
    2021-11-02 nrjank StatusNeed Info In Progress
    2021-11-01 siko1056 StatusIn Progress Need Info
    2021-11-01 siko1056 Carbon-Copy- Added siko1056
    2021-10-31 None Attached File- Added std.diff, #52177
    2021-10-31 None Attached File- Added var.diff, #52175
        Attached File- Added var_dims.diff, #52176
    2021-09-01 nrjank Summary[octave] (statistics) Weighted standard deviation for 'std' Add weighted standard deviation feature to 'std'
    2021-09-01 None Attached File- Added std.diff, #51861
    2021-08-30 nrjank StatusNone In Progress
    2021-08-30 None Attached File- Added std.m, #51838

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code