bugPSPP - Bugs: bug #48040, GLM produces wrong output

 
 

bug #48040: GLM produces wrong output

Submitted by:  Alan Mead <amead>
Submitted on:  Fri 27 May 2016 02:57:26 PM UTC
 
Category:  Numerical Errors Severity:  5 - Average
Status:  Fixed Assigned to:  None
Open/Closed:  Closed Release:  None
Effort:  0.00
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       

 

( Jump to the original submission )

Wed 03 Nov 2021 07:10:59 PM UTC, comment #6: 

Sure, I can post that!

This is a design based on an L32 orthogonal array fractional factorial, modified to have one three-level and one four-level factor, a bunch of two-level factors plus some interactions. (That is what I mean by "balanced but fractional.") As a consequence of recoding two columns into a three-level factor, there is a column that is potentially sampling error (our nomenclature calls it idA since A was the three-level factor). As it is an unreplicated design, for our purposes we can exclude that column from the model to generate an error term and Fs. (In actuality these would be pseudo-Fs, but I am comfortable with this for a screening experiment...)

The data column is generated from a known formula with some random noise thrown in, so as opposed to real life, we know the "reality" of the model behind the analysis.

The SPSS syntax is:

UNIANOVA data BY a d e b m c h n i f k j l g
  /METHOD=SSTYPE(3)
  /INTERCEPT=INCLUDE
  /CRITERIA=ALPHA(0.05)
  /DESIGN=a b c d e f g h i j k l m n a*b a*c a*d c*d d*g e*f.

I know UNIANOVA isn't in PSPP yet, but GLM replacing UNIANOVA should work the same.

I am 90% sure the output in the .spv file is from those data, but not 100% sure. It could have been from a different run of the model and I don't have an SPSS license to check anymore. If you don't have the SPSS reader, I have copied that output to Excel. Regardless of the actual numbers, you can see the intercept is part of the ANOVA table.

In SPSS you will get a SS, df, MS, F and p-value for the intercept. Conceptually this makes sense to me as forcing the GLM through y=0 will muck around with everything. Since I created the mathematical model, I know in the "real" model the intercept is not zero, but in the real world, it would not generally be a good assumption to force it through y=0 anyway.

Please do let me know if you have questions or need more from me!

(file #52196, file #52197, file #52198)

Steve <steveo>
Wed 03 Nov 2021 06:34:02 PM UTC, comment #5: 

comment #3:

> Sorry to raise this from the dead, but this issue hit me today when looking at a student's problem and comparing SPSS output to PSPP.
>
> PSPP does not include the intercept for a balanced, but fractional design. I think that is a bug.
>
> Of a certainty, there is a real intercept (I put it there in the model that generates the data!) and it is knowable and testable, and not including it in the model greatly affects the analysis.


Sorry for the null comment.

Steve, can your student submit a small dataset and SPSS syntax that replicates the issue? That might be necessary to fix. Maybe John knows what you mean by "balanced but fractional" design but I don't.

Anonymous
Wed 03 Nov 2021 06:31:04 PM UTC, comment #4: 

comment #3:

> Sorry to raise this from the dead, but this issue hit me today when looking at a student's problem and comparing SPSS output to PSPP.
>
> PSPP does not include the intercept for a balanced, but fractional design. I think that is a bug.
>
> Of a certainty, there is a real intercept (I put it there in the model that generates the data!) and it is knowable and testable, and not including it in the model greatly affects the analysis.


comment #3:

> Sorry to raise this from the dead, but this issue hit me today when looking at a student's problem and comparing SPSS output to PSPP.
>
> PSPP does not include the intercept for a balanced, but fractional design. I think that is a bug.
>
> Of a certainty, there is a real intercept (I put it there in the model that generates the data!) and it is knowable and testable, and not including it in the model greatly affects the analysis.

Anonymous
Wed 03 Nov 2021 05:02:25 PM UTC, comment #3: 

Sorry to raise this from the dead, but this issue hit me today when looking at a student's problem and comparing SPSS output to PSPP.

PSPP does not include the intercept for a balanced, but fractional design. I think that is a bug.

Of a certainty, there is a real intercept (I put it there in the model that generates the data!) and it is knowable and testable, and not including it in the model greatly affects the analysis.

Steve <steveo>
Mon 15 Aug 2016 05:39:34 AM UTC, comment #2: 

I think this has been resolved now.  So I'm closing as Fixed.  If you think there are still issues outstanding, then please re-open it.

John Darrington <jmd>
Project Administrator
Fri 27 May 2016 04:24:10 PM UTC, comment #1: 

Thanks for the bug report.

PSPP (unlike SPSS) does not show the "intercept" if the model is
unbalanced, since it has no meaning (just what calculation SPSS does in this case is a mystery).

The negative F value (and consequently NaN df) are more of a concern, possibly also related to missing values.

Also the speed issues concern me too.

I suggest we discuss this further on pspp-dev.

John Darrington <jmd>
Project Administrator
Fri 27 May 2016 02:57:26 PM UTC, original submission:  

The attached data ("personality.sav") are three personality scores on a 50-item measure. They are moderately intercorrelated (0.35 - 0.51). There are a large number of missing values.

Using the attached data, I ran:
GLM agree_score BY  caution_score extra_score

The processing took 6 minutes (compared to SPSS < 1 sec) and produced attached "glm_output.txt" that doesn't match SPSS and which is nonsensical in many places.  For example, the tests for intercept are simply missing, the F values are all negative (which isn't possible), the p-values are all NaN, the error degrees of freedom are negative as are the error mean square.

I assume that a problem (not the only problem, see below) with these data are that there are many levels for the independent variables (about 50; they aren't really factors). This SHOULD NOT be a problem for GLM, but when I do a median split on the dependent variables:

recode caution_score (lo thru 35=1) (36 thru hi=2) (else=copy) into x1.
recode extra_score (lo thru 31=1) (32 thru hi=2) (else=copy) into x2.
execute.
GLM agree_score BY  x1 x2.

This GLM runs very quickly but still produces incorrect output. For example, the df for x1 and x2 should be 1 (they have 2 levels, 2-1=1). The only way PSPP could be calculating df=2 is if it senses 3 levels which probably means that it's treating missing as a level, which is obviously incorrect. the two independent variables should still have significant effects on the dep. variable but the SS is calculated as 0.00. Also, the p-value for one effect is NaN, which shouldn't happen.

If the internal mechanics of PSPP's GLM cannot handle multiple levels then the routine should count the levels and refuse to run when the levels are too many (or, better yet, use an algorithm that doesn't fail because the "general linear model" shouldn't choke when you feed it continuous variables... that's what "general" in GLM means...).

Clearly there are several other serious bugs in the routine: (1) df are calculated incorrectly. (2) Under some circumstances, error SS is wrong. (3) When F=0 or F=1, significance values should be printed as 1.000 and 0.000, not as "NaN"

I ran this using an old version of PSPP on Windows 7 but I have since installed GNU pspp 0.10.1-g1082b8 and verified that I have identical results.

Alan Mead <amead>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #52196:  spss_out.xlsx added by steveo (11KiB - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
file #52197:  Fractional@Exam@Problem.spv added by steveo (13KiB - application/octet-stream)
file #52198:  Fractional@Exam@Problem.sav added by steveo (2KiB - application/x-spss-sav)
file #37288:  glm_output2.odt added by amead (21KiB - application/vnd.oasis.opendocument.text - the .ODT file contains both PSPP and SPSS output (the only way to past ethe SPSS output was as formatted text) and the SAV file was created by SPSS 24.)
file #37289:  personality.sav added by amead (5KiB - application/x-spss-sav - the .ODT file contains both PSPP and SPSS output (the only way to past ethe SPSS output was as formatted text) and the SAV file was created by SPSS 24.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by steveo (Posted a comment)
  • -email is unavailable- added by jmd (Posted a comment)
  • -email is unavailable- added by amead (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-11-03 steveo Attached File- => Added spss_out.xlsx, #52196
        Attached File- => Added Fractional@Exam@Problem.spv, #52197
        Attached File- => Added Fractional@Exam@Problem.sav, #52198
    2016-08-15 jmd StatusNone => Fixed
        Open/ClosedOpen => Closed
    2016-05-27 amead Attached File- => Added glm_output2.odt, #37288
        Attached File- => Added personality.sav, #37289

    Back to the top


    Powered by Savane 3.9