bugPSPP - Bugs: bug #35758, regression: incorrect results with...


bug #35758: regression: incorrect results with many missing values

Submitted by:  Ben Pfaff <blp>
Submitted on:  Thu 08 Mar 2012 06:31:08 AM UTC  
Category:  Numerical Errors Severity:  7 - Major
Status:  Fixed Assigned to:  None
Open/Closed:  Closed Release:  Before 0.8.0
Effort:  0.00

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Mon 04 Feb 2013 09:32:14 AM UTC, comment #2:

I had a look through the regression code.

It seems to be safe now with regard to missing values. However,
the /MISSING subcommand is absent. The behaviour is as if /MISSING=LISTWISE was specified, which is the Chicago default.

If somebody needs /MISSING=PAIRWISE or whatever, then we can deal with it as a feature request.

Closing this item.

John Darrington <jmd>
Project Administrator
Sun 11 Mar 2012 06:08:00 PM UTC, comment #1:

I pushed a fix for this particular problem. But I think the regression command would benefit from an audit of it's behaviour with respect to missing values. So I'm leaving this item open as a reminder.

John Darrington <jmd>
Project Administrator
Thu 08 Mar 2012 06:31:08 AM UTC, original submission:

Renan Levine writes on pspp-users (archived at http://lists.gnu.org/archive/html/pspp-users/2012-03/msg00015.html):

> This appears to be a bug in the PSPP regression routine with data with a
> large amount of missing values!
> I recently noticed some small discrepancies between simple bivariate
> regression results between IBM SPSS, STATA and PSPP. Until Prof.
> Shackman's email, I hadn't realized that the discrepancies only occur
> when there are many missing values. I was just confused...
> Sadly, I also find problems when running linear regressions using PSPP
> on data with missing values. I wish I knew what was causing the problem.
> So, using Dropbox, I wanted to make available some data which seems to
> illustrate the issue.
> Using psppire.exe 0.7.9-gab8ce2 on Windows AND psppire 0.7.8 on
> LinuxMint LXDE, PSPP calculates descriptive statistics just like SPSS
> and STATA on the same dataset, but does not calculate identical b
> coefficients when running bivariate or multivariate regressions.
> I created the following public opinion survey data files consisting of
> three variables from the 2004 Canadian Election Study which I recoded
> and declared certain values to be missing:
> http://dl.dropbox.com/u/35198072/ces2004-regtest.sav
> <http://www.queensu.ca/cora/ces.html> has many observations with missing
> values.
> http://dl.dropbox.com/u/35198072/ces2004-regtest2.sav has the same three
> variables, but I dropped all of the cases with missing values.
> This is the syntax file used to run descriptive statistics and three
> regression analyses.
> http://dl.dropbox.com/u/35198072/regression-tests.sps
> PSPP generates these regression results and descriptive statistics with
> missing values:
> http://dl.dropbox.com/u/35198072/regression-test-pspp1.html
> PSPP generates these regression results and descriptive statistics using
> the data without any missing values:
> http://dl.dropbox.com/u/35198072/regression-test-pspp2.html
> Here is the STATA output on the same output (.log is a text file - email
> me if you have a problem opening it). The first three regressions should
> match the output in regression-test-pspp1.html
> They are close, but not close enough... The bottom three regressions use
> the data with no missing values and these DO match PSPP's output (in
> regression-test-pspp2.html).
> http://dl.dropbox.com/u/35198072/regression-test-stata.log
> I also ran the data on SPSS and found results consistent with STATA.
> There did not seem to be any problems with Pearson's Chi-Square or
> Kendall's Tau-B when running a crosstab on the data with the missing
> values.
> I am sorry I don't know what has gone wrong, so I am making available
> this data in hopes someone might figure out where there is a mistake. I
> caution other users running regression on PSPP.

Ben Pfaff <blp>
Project Administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by jmd (Posted a comment)
  • -email is unavailable- added by jmd (Interested parties)
  • -email is unavailable- added by jmd (Interested parties)
  • -email is unavailable- added by blp (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-02-04 jmd Open/ClosedOpen => Closed
    2012-03-11 jmd StatusNone => Fixed
        ReleaseNone => Before 0.8.0
        Carbon-Copy- => Added gene shackman
        Carbon-Copy- => Added -email is unavailable-

    Back to the top

    Powered by Savane 3.3