bugGNU Octave - Bugs: bug #56874, corr > 1

 
 

bug #56874: corr > 1

Submitted by:  Muhali <muhali>
Submitted on:  Mon 09 Sep 2019 12:18:27 PM UTC  
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  5.1.0 Operating System:  GNU/Linux

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 01 Oct 2019 08:23:42 AM UTC, comment #5: 

I'm not sure if this may be helpful, but for the particular case, using the attached script the result is correct.
The idea is to write the correlation as

exp (log (corr))

and evaluate log (corr) using logarithm properties.

(file #47601)

Michele Ginesi <m_ginesi>
Tue 10 Sep 2019 04:26:54 PM UTC, comment #4: 

being quite illiterate in numerics, I wonder why

x'*y / sqrt(x'*x*y'*y) <= 1
corr(x, y) <= 1

gives a correct answer in the first and an incorrect one in the second case. So why not use the first formula as a definition of corr?

Muhali <muhali>
Tue 10 Sep 2019 12:29:00 PM UTC, comment #3: 

This has a very long history in numerical mathematics.
Simple application of the formula for cor(x,y) using
finite representation of x and y will result in cases
where cor(x,y) evaluates to a result outside [-1,1].

Therefore, the cor(x,y) should always return a value
on [-1,1], but it should also indicate a loss of
precision in any case where the most precisely evaluated
result is outside the bound.

A large literature on this is available.

Michael Godfrey <godfrey>
Project Member
Mon 09 Sep 2019 06:36:45 PM UTC, comment #2: 

yes it is numerical error, but somehow Matlab gets it right in this case. Correlation should never be outside [-1 1] as that produces uncontrollable results.

It's at least a compatibility issue, but probably more than that.

Muhali <muhali>
Mon 09 Sep 2019 06:11:56 PM UTC, comment #1: 

corr(x,y) - 1 = 2.2204e-16

numerical precision limit is hit in these lines:

    c = cov (x);
    s = sqrt (diag (c));
    retval = c ./ (s * s');

results could maybe be constrained to [-1,1]

    retval = min(max(c ./ (s * s'), -1), 1);

However the result could still be 1-eps. The user of corr needs to anticipate that the result is subject to numerical errors.

A.R. Burgers <arb>
Mon 09 Sep 2019 12:18:27 PM UTC, original submission:  

the attached file contains data for which corr > 1.

load foo.ot
corr(x, y) > 1

Muhali <muhali>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #47601:  test_corr.m added by m_ginesi (363B - text/x-objcsrc)
file #47469:  foo.ot added by muhali (336B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by m_ginesi (Updated the item)
  • -email is unavailable- added by godfrey (Posted a comment)
  • -email is unavailable- added by arb (Posted a comment)
  • -email is unavailable- added by muhali (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-10-01 m_ginesi Attached File- => Added test_corr.m, #47601
    2019-09-09 muhali Attached File- => Added foo.ot, #47469

    Back to the top


    Powered by Savane 3.5