bugGNU Octave - Bugs: bug #32168, [octave forge] (dataframe)...

 
 

bug #32168: [octave forge] (dataframe) skewness, kurtosis not properly handling data with stdev=0

Submitter:  Pascal Dupuis <cdemills>
Submitted:  Tue 18 Jan 2011 11:01:56 AM UTC
   
 
Category:  Octave Forge Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Confirmed Assigned to:  None
Originator Name:  Pascal Dupuis Open/Closed:  * Open
Release:  * other Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 08 Dec 2022 08:56:31 PM UTC, comment #12: 

testing on octave 8.0.1 with dataframe 1.2.0:


>> x = dataframe(rand(5,1))
warning: genvarname is obsolete; use matlab.lang.makeValidName or matlab.lang
.makeUniqueStrings instead
x = dataframe with 5 rows and 1 columns
_1      X
Nr double
 1 0.9377
 2 0.8110
 3 0.9436
 4 0.9106
 5 0.1947

>> skewness (x.data)
ans = -1.4022

>> skewness (x)
ans = dataframe with 1 rows and 1 columns
_1       X
Nr  double
 1 -1.4022

>> kurtosis (x.data)
ans = 3.1050

>> kurtosis (x)
ans = dataframe with 1 rows and 1 columns
_1      X
Nr double
 1 3.1050

(all still good there for first report problem)


   嬂᪁>> x = dataframe (zeros (5, 1))
x = dataframe with 5 rows and 1 columns
_1      X
Nr double
 1      0
 2      0
 3      0
 4      0
 5      0
>> skewness (x.data)
ans = NaN
>> skewness (x)
error: x(1): out of bound 0 (dimensions are 0x0)
error: called from
    df_mapper>@<anonymous> at line 28 column 30
    df_mapper at line 28 column 15
    power at line 22 column 8
    skewness at line 122 column 10
>> kurtosis (x.data)
ans = NaN
>> kurtosis (x)
error: x(1): out of bound 0 (dimensions are 0x0)
error: called from
    df_mapper>@<anonymous> at line 28 column 30
    df_mapper at line 28 column 15
    power at line 22 column 8
    kurtosis at line 125 column 10


so status unchanged from comment #9.  retitling to focus on one remaining issue, which was suggested in comment #2 should be handled better by overloading the functions in the dataframe class.

Nicholas Jankowski <nrjank>
Group Member
Wed 17 Nov 2021 10:27:27 PM UTC, comment #11: 

behavior in comment #9 is the same in Octave 6.4.0, dataframe pkg 1.2.0

Nicholas Jankowski <nrjank>
Group Member
Fri 01 Jun 2018 10:59:12 AM UTC, comment #10: 

Hello,
I will be away up to June 20th. I will look about this issue after this date.

Regards

Pascal

Pascal Dupuis <cdemills>
Group Member
Tue 29 May 2018 01:08:19 PM UTC, comment #9: 

Just tested with Octave 4.4 and dataframe 1.2.0:

```

>> pkg load dataframe


>> x = dataframe (rand (5, 1))

x = dataframe with 5 rows and 1 columns
_1        X
Nr   double
 1 0.471887
 2 0.431165
 3 0.268632
 4 0.573827
 5 0.031409

>> skewness (x.data)

ans = -0.65315

>> skewness (x)

ans = dataframe with 1 rows and 1 columns
_1        X
Nr   double
 1 -0.65315

>> kurtosis (x.data)

ans =  2.1050

>> kurtosis (x)

ans = dataframe with 1 rows and 1 columns
_1      X
Nr double
 1 2.1050
```

So, the problem with "zeros: invalid data type specified"  seems to be gone ;-)

The case of zero std is still not handled properly, however:

```

>> x = dataframe (zeros (5, 1))

x = dataframe with 5 rows and 1 columns
_1      X
Nr double
 1      0
 2      0
 3      0
 4      0
 5      0

>> skewness (x.data)

ans =  NaN

>> skewness (x)

error: x(1): out of bound 0
error: called from
    df_mapper>@<anonymous> at line 28 column 30
    df_mapper at line 28 column 15
    power at line 22 column 8
    skewness at line 120 column 10

>> kurtosis (x.data)

ans =  NaN

>> kurtosis (x)

error: x(1): out of bound 0
error: called from
    df_mapper>@<anonymous> at line 28 column 30
    df_mapper at line 28 column 15
    power at line 22 column 8
    kurtosis at line 123 column 10
```

Julien Bect <jbect>
Wed 16 Nov 2016 07:00:41 PM UTC, comment #8: 

@jbect can this bug report be closed?

Mike Miller <mtmiller>
Group Member
Fri 18 Oct 2013 05:23:28 PM UTC, comment #7: 

This bug report should be easy to close once I finish working on bug report #36718.

Julien Bect <jbect>
Fri 18 Oct 2013 02:52:15 PM UTC, comment #6: 

Ok, my question was indeed about the "closed" status of this bug report ;-)

Julien Bect <jbect>
Fri 18 Oct 2013 02:46:29 PM UTC, comment #5: 

No new bug report.  This report, since it had the history, was re-assigned to the Octave-Forge category and the Summary was re-written.  I accidentally closed it as well, but it is now open again with the new classification.

Rik <rik5>
Group administrator
Fri 18 Oct 2013 10:56:30 AM UTC, comment #4: 

Where is the new bug report ?

Julien Bect <jbect>
Thu 17 Oct 2013 06:12:18 PM UTC, comment #3: 

Pushing this bug on to the dataframe package rather than core Octave.

Rik <rik5>
Group administrator
Tue 24 Sep 2013 04:44:48 PM UTC, comment #2: 

Agree with Julien that fixing the problem within the dataframe package makes more sense in this case. Core functions can't be expected to know about arbitrary data types.

Nir Krakauer <nir_krakauer>
Wed 11 Sep 2013 10:40:13 AM UTC, comment #1: 

The problem is still there in Octave 3.6.4 and also in the latest development version. Moreover, the exact same problem exists with statistics/base/kurtosis.m.

Wouldn'it be a better idea, instead of modifying the base functions skewness and kurtosis, to to overload them for the @dataframe class?

As a bonus, this would allow to handle non-numeric columns gracefully (by returning NaN for them, for instance, plus perhaps a warning).


Julien Bect <jbect>
Tue 18 Jan 2011 11:01:56 AM UTC, original submission:  

In the file statistics/base/skewness.m, a test is performed to detect variables whose std is 0. As the skewness implies taking the inverse of the std, the skewness for those values is simply set to 0, to avoid division by 0. This is accomplished at lines 79-83. retval is initialised to be zeros of the class of the input. For a user-defined class, this fails with
"zeros: invalid data type specified"
Moreover, overloaded funcs for a dataframe object takes care about propagating meta-information. This way, the line
retval(ind) = x(ind) ./ (c * s(ind) .^ 3)
implies a lot of work, as we're dealing with 3 variables. I thus propose the enclosed patch. Ideas:

ind = (s > 0); <= keep tracks of thing to process some way/some other way
retval = sum (x .^ 3, dim); <= retval receive all its meta-inf from x

if !isobject(x),
    retval(~ind) = zeros (1, class (x)); <= fast way
else
    retval(~ind) = 0; %# let the class subsasgn handle the cast
endif   

retval(ind) = retval(ind) ./ (c * s(ind) .^ 3);  

Regards

Pascal


Pascal Dupuis <cdemills>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #22436:  skewness.pat added by cdemills (665B - image/x-coreldrawpattern)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by jwe (Updated the item)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by nir_krakauer (Posted a comment)
  • -email is unavailable- added by jbect (Posted a comment)
  • -email is unavailable- added by cdemills (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only project members can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-12-08 nrjank StatusNone Confirmed
        Operating SystemGNU/Linux Any
        Summary[octave forge] (dataframe) support for skewness, kurtosis [octave forge] (dataframe) skewness, kurtosis not properly handling data with stdev=0
    2017-08-13 jwe Summary@dataframe package support for skewness, kurtosis [octave forge] (dataframe) support for skewness, kurtosis
    2016-11-16 rik5 Carbon-CopyRemoved 72865 -
    2016-11-16 mtmiller Release3.3.54 other
    2013-10-18 rik5 Open/ClosedClosed Open
    2013-10-17 rik5 CategoryLibraries Octave Forge Package
        Open/ClosedOpen Closed
        Summaryskewness.m: initialise the output from the input @dataframe package support for skewness, kurtosis
    2011-01-18 cdemills Attached File- Added skewness.pat, #22436

    Back to the top

    Powered by Savane 3.12