bugGNU Octave - Bugs: bug #56874, corr > 1

 
 

bug #56874: corr > 1

Submitter:  Muhali <muhali>
Submitted:  Mon 09 Sep 2019 12:18:27 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Inaccurate Result
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * 5.1.0 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 01 Oct 2019 08:23:42 AM UTC, comment #5: 

I'm not sure if this may be helpful, but for the particular case, using the attached script the result is correct.
The idea is to write the correlation as

exp (log (corr))

and evaluate log (corr) using logarithm properties.

(file #47601)

Michele Ginesi <m_ginesi>
Tue 10 Sep 2019 04:26:54 PM UTC, comment #4: 

being quite illiterate in numerics, I wonder why


x'*y / sqrt(x'*x*y'*y) <= 1
corr(x, y) <= 1


gives a correct answer in the first and an incorrect one in the second case. So why not use the first formula as a definition of corr?

Muhali <muhali>
Tue 10 Sep 2019 12:29:00 PM UTC, comment #3: 

This has a very long history in numerical mathematics.
Simple application of the formula for cor(x,y) using
finite representation of x and y will result in cases
where cor(x,y) evaluates to a result outside [-1,1].

Therefore, the cor(x,y) should always return a value
on [-1,1], but it should also indicate a loss of
precision in any case where the most precisely evaluated
result is outside the bound.

A large literature on this is available.

Michael Godfrey <godfrey>
Group Member
Mon 09 Sep 2019 06:36:45 PM UTC, comment #2: 

yes it is numerical error, but somehow Matlab gets it right in this case. Correlation should never be outside [-1 1] as that produces uncontrollable results.

It's at least a compatibility issue, but probably more than that.

Muhali <muhali>
Mon 09 Sep 2019 06:11:56 PM UTC, comment #1: 


corr(x,y) - 1 = 2.2204e-16


numerical precision limit is hit in these lines:


    c = cov (x);
    s = sqrt (diag (c));
    retval = c ./ (s * s');


results could maybe be constrained to [-1,1]


    retval = min(max(c ./ (s * s'), -1), 1);


However the result could still be 1-eps. The user of corr needs to anticipate that the result is subject to numerical errors.

A.R. Burgers <arb>
Mon 09 Sep 2019 12:18:27 PM UTC, original submission:  

the attached file contains data for which corr > 1.


load foo.ot
corr(x, y) > 1


Muhali <muhali>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #47601:  test_corr.m added by m_ginesi (363B - text/x-objcsrc)
file #47469:  foo.ot added by muhali (336B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Updated the item)
  • -email is unavailable- added by m_ginesi (Updated the item)
  • -email is unavailable- added by godfrey (Posted a comment)
  • -email is unavailable- added by arb (Posted a comment)
  • -email is unavailable- added by muhali (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-09-01 jwe CategoryNone Octave Function
        Item GroupNone Inaccurate Result
        StatusNone Confirmed
    2019-10-01 m_ginesi Attached File- Added test_corr.m, #47601
    2019-09-09 muhali Attached File- Added foo.ot, #47469

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code