bugPSPP - Bugs: bug #33036, System files with incorrectly...

 
 

bug #33036: System files with incorrectly encoded variable names cannot be opened.

Submitter:  None
Submitted:  Sat 09 Apr 2011 09:25:15 PM UTC
   
 
Category:  System/Portable File I/O Severity:  7 - Major
Status:  Fixed Assigned to:  None
Open/Closed:  Closed Release:  None
Effort:  0.00
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 26 Apr 2011 04:53:56 AM UTC, comment #10: 

I wrote up and pushed several commits that fix all of the issues that I was able to identify.  However, the attached files still cannot be opened, because one of the bugs that is fixed caused them to be written incorrectly.  Mindaugus, if you lost important data due to this bug, let me know and I will help you to recover it.

I'm confident that this bug is fixed, so I am closing it.  If there still seems to be a problem, please let me know so that I can reopen it (or feel free to file another bug).

Thanks,

Ben.

Ben Pfaff <blp>
Group administrator
Sun 24 Apr 2011 05:19:31 AM UTC, comment #9: 

There are actually several problems in this area.  I've fixed them on a branch named bug33036 at http://benpfaff.org/cgi-bin/gitweb.cgi?p=pspp;a=shortlog;h=refs/heads/bug33036.  I need to write some tests and improve the commit messages, and then I'll push these fixes.

Ben Pfaff <blp>
Group administrator
Sat 23 Apr 2011 06:14:37 PM UTC, comment #8: 

I think I see the problem.  The system file writer doesn't consistently translate strings, including variable short names, from UTF-8 to the proper encoding.  I'll work on a fix.

(This isn't GUI-specific--I can't read the file from the terminal either.)

Ben Pfaff <blp>
Group administrator
Sun 17 Apr 2011 01:40:33 PM UTC, comment #7: 

I don't remember.
But I try to produce. These steps maybe could lead to the same:

> LANG=C
> psppire

Insert new variable
Add some data for variable (1, 2)
Rename variable to ĖĮŲĄŽč
Save file

(file #23264)

Mindaugas <embar>
Sun 17 Apr 2011 10:24:53 AM UTC, comment #6: 

I can confirm that the crash has been fixed.

But file #23253 still cannot be opened.  It seems that this file
contains a variable name which UTF8 encoding would be appropriate.
However the file claims to be encoded in ISO-8859-13.

Hence pspp regards the variable name to be invalid.  So it should not have been possible to create this file.

I wonder thou if we should be a little more forgiving about "invalid" variable names.  If somehow such a file gets created, a user might not care too much if the variable names get mangled.  But if their data cannot be opened they'll care very much.

Mindaugas:  How exactly did you create this file?

John Darrington <jmd>
Group administrator
Sat 16 Apr 2011 06:43:36 PM UTC, comment #5: 


>Using commit ac5fc2c85d86f1a14cdea0a215666ab39b44a1dc the attached file loads and displays in the syntax editor fine (so long as the locale is appropriately set). However attempting to run the syntax crashes. Only happens in the GUI. The command line version is fine.


This is actually the same bug that I fixed with commit e63296a3c2bfc "gui: Fix crash in executor when inline data is missing."  I can reproduce it with the commit ac5fc2c85 that you mentioned but not current master.

I'll leave the bug open because it's possible that this bug was masking some underlying problem.  But if you or Mindaugas think that it's all fixed, please close the bug.

Ben Pfaff <blp>
Group administrator
Sat 16 Apr 2011 08:09:41 AM UTC, comment #4: 

I can confirm this problem on GNU/Linux.

Using commit ac5fc2c85d86f1a14cdea0a215666ab39b44a1dc  the attached file loads and displays in the syntax editor fine (so long as the locale is appropriately set).  However attempting to run the syntax crashes.   Only happens in the GUI.  The command line version is fine.

The attached file, when running with LC_ALL=de_DE.ISO88591 crashes when the syntax is executed. Stack trace attached.

I don't think it's related to the legality of the variable name. But rather its an character encoding issue. Updating the title of this bug accordingly.




(file #23258, file #23259)

John Darrington <jmd>
Group administrator
Fri 15 Apr 2011 07:59:49 AM UTC, comment #3: 

This don't lead to crash, file only can not be imported.

Mindaugas <embar>
Fri 15 Apr 2011 07:57:42 AM UTC, comment #2: 

I can reproduce it with 2011-04-11 build from http://pspp.awardspace.com/ in Windows XP SP3 and with Linux version 0.7.7-gfb5c91. Locale in my openSUSE: lt_LT.UTF-8. Locale in Windows: Lithuanian.

I get error:

error: `/home/vartotojas/PSPP-data.sav' near offset 0xb4: Invalid variable name
`ĄŽŠų'.


I attach file, that produce error in Windows and in Linux (it has variable "ĄŽŠų")

(file #23253)

Mindaugas <embar>
Sat 09 Apr 2011 10:42:28 PM UTC, comment #1: 

What version of PSPP did you use?  What locale is configured in Ubuntu?  What language/code page is configured on Windows?

Ben Pfaff <blp>
Group administrator
Sat 09 Apr 2011 09:25:15 PM UTC, original submission:  

I used PSPP on Linux, and by mistake used illegal characters in a variable name (i.e. äüö in my case). On Linux (Ubuntu 10.10), I could work with this file without problems. When opening the same file under Windows XP (SP3), PSPP crashes. The error message suggests that PSPP is trying to catch the illegal variable name, but somehow at this moment it crashes.

Opening the file on Linux, I could manually change the variable names, and everything works as well.

Two things probably need to be fixed. (1) the crash under Windows, (2) the fact that the Linux version can save variable names that are not allowed in SPSS.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #23264:  PSPP-data.sav added by embar (407B - application/x-spss-sav - Can not open because variable is question marks)
file #23258:  stacktrace.txt added by jmd (3KiB - text/plain)
file #23259:  umlaut-iso8859-1.sps added by jmd (73B - application/octet-stream)
file #23253:  PSPP-data.sav added by embar (414B - application/x-spss-sav - .sav file with ĄŽŠų variable)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jmd (Updated the item)
  • -email is unavailable- added by embar (Updated the item)
  • -email is unavailable- added by blp (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-04-26 blp StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2011-04-23 blp CategoryGraphical User Interface System/Portable File I/O
        StatusConfirmed In Progress
    2011-04-17 embar Attached File- Added PSPP-data.sav, #23264
    2011-04-17 jmd SummaryGUI crashes parsing inline data with non-ascii variable names. System files with incorrectly encoded variable names cannot be opened.
    2011-04-16 jmd Attached File- Added stacktrace.txt, #23258
        Attached File- Added umlaut-iso8859-1.sps, #23259
        Severity5 - Average 7 - Major
        StatusNeed Info Confirmed
        SummaryPSPP crashes if variable names are illegal GUI crashes parsing inline data with non-ascii variable names.
    2011-04-15 embar Attached File- Added PSPP-data.sav, #23253
    2011-04-09 blp StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code