bugPSPP - Bugs: bug #33036, System files with incorrectly...

 
 

bug #33036: System files with incorrectly encoded variable names cannot be opened.

Submitted by:  None
Submitted on:  Sat 09 Apr 2011 09:25:15 PM UTC  
 
Category: System/Portable File I/OSeverity: 7 - Major
Status: FixedAssigned to: None
Open/Closed: ClosedRelease: None
Effort: 0.00

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 26 Apr 2011 04:53:56 AM UTC, comment #10:

I wrote up and pushed several commits that fix all of the issues that I was able to identify. However, the attached files still cannot be opened, because one of the bugs that is fixed caused them to be written incorrectly. Mindaugus, if you lost important data due to this bug, let me know and I will help you to recover it.

I'm confident that this bug is fixed, so I am closing it. If there still seems to be a problem, please let me know so that I can reopen it (or feel free to file another bug).

Thanks,

Ben.

Ben Pfaff <blp>
Project Administrator
Sun 24 Apr 2011 05:19:31 AM UTC, comment #9:

There are actually several problems in this area. I've fixed them on a branch named bug33036 at http://benpfaff.org/cgi-bin/gitweb.cgi?p=pspp;a=shortlog;h=refs/heads/bug33036. I need to write some tests and improve the commit messages, and then I'll push these fixes.

Ben Pfaff <blp>
Project Administrator
Sat 23 Apr 2011 06:14:37 PM UTC, comment #8:

I think I see the problem. The system file writer doesn't consistently translate strings, including variable short names, from UTF-8 to the proper encoding. I'll work on a fix.

(This isn't GUI-specific--I can't read the file from the terminal either.)

Ben Pfaff <blp>
Project Administrator
Sun 17 Apr 2011 01:40:33 PM UTC, comment #7:

I don't remember.
But I try to produce. These steps maybe could lead to the same:

> LANG=C
> psppire

Insert new variable
Add some data for variable (1, 2)
Rename variable to ĖĮŲĄŽč
Save file

(file #23264)

Mindaugas <embar>
Sun 17 Apr 2011 10:24:53 AM UTC, comment #6:

I can confirm that the crash has been fixed.

But file #23253 still cannot be opened. It seems that this file
contains a variable name which UTF8 encoding would be appropriate.
However the file claims to be encoded in ISO-8859-13.

Hence pspp regards the variable name to be invalid. So it should not have been possible to create this file.

I wonder thou if we should be a little more forgiving about "invalid" variable names. If somehow such a file gets created, a user might not care too much if the variable names get mangled. But if their data cannot be opened they'll care very much.

Mindaugas: How exactly did you create this file?

John Darrington <jmd>
Project Administrator
Sat 16 Apr 2011 06:43:36 PM UTC, comment #5:

>Using commit ac5fc2c85d86f1a14cdea0a215666ab39b44a1dc the attached file loads and displays in the syntax editor fine (so long as the locale is appropriately set). However attempting to run the syntax crashes. Only happens in the GUI. The command line version is fine.


This is actually the same bug that I fixed with commit e63296a3c2bfc "gui: Fix crash in executor when inline data is missing." I can reproduce it with the commit ac5fc2c85 that you mentioned but not current master.

I'll leave the bug open because it's possible that this bug was masking some underlying problem. But if you or Mindaugas think that it's all fixed, please close the bug.

Ben Pfaff <blp>
Project Administrator
Sat 16 Apr 2011 08:09:41 AM UTC, comment #4:

I can confirm this problem on GNU/Linux.

Using commit ac5fc2c85d86f1a14cdea0a215666ab39b44a1dc the attached file loads and displays in the syntax editor fine (so long as the locale is appropriately set). However attempting to run the syntax crashes. Only happens in the GUI. The command line version is fine.

The attached file, when running with LC_ALL=de_DE.ISO88591 crashes when the syntax is executed. Stack trace attached.

I don't think it's related to the legality of the variable name. But rather its an character encoding issue. Updating the title of this bug accordingly.

(file #23258, file #23259)

John Darrington <jmd>
Project Administrator
Fri 15 Apr 2011 07:59:49 AM UTC, comment #3:

This don't lead to crash, file only can not be imported.

Mindaugas <embar>
Fri 15 Apr 2011 07:57:42 AM UTC, comment #2:

I can reproduce it with 2011-04-11 build from http://pspp.awardspace.com/ in Windows XP SP3 and with Linux version 0.7.7-gfb5c91. Locale in my openSUSE: lt_LT.UTF-8. Locale in Windows: Lithuanian.

I get error:

error: `/home/vartotojas/PSPP-data.sav' near offset 0xb4: Invalid variable name
`ĄŽŠų'.

I attach file, that produce error in Windows and in Linux (it has variable "ĄŽŠų")

(file #23253)

Mindaugas <embar>
Sat 09 Apr 2011 10:42:28 PM UTC, comment #1:

What version of PSPP did you use? What locale is configured in Ubuntu? What language/code page is configured on Windows?

Ben Pfaff <blp>
Project Administrator
Sat 09 Apr 2011 09:25:15 PM UTC, original submission:

I used PSPP on Linux, and by mistake used illegal characters in a variable name (i.e. äüö in my case). On Linux (Ubuntu 10.10), I could work with this file without problems. When opening the same file under Windows XP (SP3), PSPP crashes. The error message suggests that PSPP is trying to catch the illegal variable name, but somehow at this moment it crashes.

Opening the file on Linux, I could manually change the variable names, and everything works as well.

Two things probably need to be fixed. (1) the crash under Windows, (2) the fact that the Linux version can save variable names that are not allowed in SPSS.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #23264:  PSPP-data.sav added by embar (407B - application/x-spss-sav - Can not open because variable is question marks)
file #23258:  stacktrace.txt added by jmd (3KiB - text/plain)
file #23259:  umlaut-iso8859-1.sps added by jmd (73B - application/octet-stream)
file #23253:  PSPP-data.sav added by embar (414B - application/x-spss-sav - .sav file with ĄŽŠų variable)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jmd (Updated the item)
  • -unavailable- added by embar (Updated the item)
  • -unavailable- added by blp (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 13 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 26 Apr 2011 04:53:56 AM UTCblpStatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed
    Sat 23 Apr 2011 06:14:37 PM UTCblpCategoryGraphical User Interface=>System/Portable File I/O
      StatusConfirmed=>In Progress
    Sun 17 Apr 2011 01:40:33 PM UTCembarAttached File-=>Added PSPP-data.sav, #23264
    Sun 17 Apr 2011 10:24:53 AM UTCjmdSummaryGUI crashes parsing inline data with non-ascii variable names.=>System files with incorrectly encoded variable names cannot be opened.
    Sat 16 Apr 2011 08:09:41 AM UTCjmdAttached File-=>Added stacktrace.txt, #23258
      Attached File-=>Added umlaut-iso8859-1.sps, #23259
      Severity5 - Average=>7 - Major
      StatusNeed Info=>Confirmed
      SummaryPSPP crashes if variable names are illegal=>GUI crashes parsing inline data with non-ascii variable names.
    Fri 15 Apr 2011 07:57:42 AM UTCembarAttached File-=>Added PSPP-data.sav, #23253
    Sat 09 Apr 2011 10:42:28 PM UTCblpStatusNone=>Need Info

    Back to the top


    Powered by Savane 3.1-cleanup1