bugPSPP - Bugs: bug #33255, Windows operating systems cannot...


bug #33255: Windows operating systems cannot open files whose names contain non-ascii characters.

Submitted by:  John Darrington <jmd>
Submitted on:  Sat 07 May 2011 05:02:39 AM UTC  
Category:  Compilation/Portability Severity:  5 - Average
Status:  Fixed Assigned to:  None
Open/Closed:  Closed Release:  Future
Effort:  0.00

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Mon 14 Mar 2016 06:20:53 AM UTC, comment #11: 

I think this is finished now.  Bug #47331 describes a related but different problem.

John Darrington <jmd>
Project Administrator
Fri 23 Oct 2015 06:14:20 AM UTC, comment #10: 

As of commit 51ae576dcac25a9e196a2a7e8f6e692fa3c63d29 (Sat Oct 17 17:29:31 2015 +0200)
Windows operating systems (as well as others) can correctly save and rewrite filenames which contain non-ascii characters.

However, Windows will probably fail trying to read such files.

To fix this, the code in src/data/file-name.c will have to be rethought and a different abstraction used.

Updating the title of this item accordingly.

John Darrington <jmd>
Project Administrator
Mon 05 Oct 2015 09:01:46 PM UTC, comment #9: 

as an answer on comment 2, it is now possible to debug the MSWindows PSPP version. Maybe that helps with this issue.

Harry Thijssen <unknown_one>
Mon 28 Sep 2015 06:17:45 PM UTC, comment #8: 

Also reported in bug #46057

John Darrington <jmd>
Project Administrator
Tue 03 Dec 2013 05:02:18 PM UTC, comment #7: 

This seems to be the same, or very similar problem reported by  abraham laredo  sicsu <alsicsu@gmail.com>

John Darrington <jmd>
Project Administrator
Wed 24 Jul 2013 03:40:44 PM UTC, comment #6: 

Harry Thijssen <pspp@sjpaes.nl> writes in email:

> This is a really old bug and there are so many changes in all the software
> around. Maybe we can ask the list if anybody still encounters this problem
> with the latest MSWindows builds.
> I wouldn't be surprised if the problem has gone.

Ben Pfaff <blp>
Project Administrator
Wed 24 Jul 2013 05:28:19 AM UTC, comment #5: 

> Recent incarnations of windows use UTF-16LE encoding not UTF-8.

I'm pretty sure that Windows has had the "W" versions of library functions, that accept 16-bit Unicode encoding, practically forever.  Since Windows NT 3.x anyhow.

I've heard that there was a more recent change from using UCS-2 encoding (which only supports characters in the Unicode Basic Multilingual Plane) to UTF-16 encoding (which supports all Unicode characters via surrogate pairs).  But that wouldn't affect support for simple accented characters.

It might be worthwhile to bring this up on the gnulib mailing list (but Bruno, who would really know, has been MIA for a year or more).

Ben Pfaff <blp>
Project Administrator
Tue 14 Jun 2011 07:22:34 AM UTC, comment #4: 

I got this from a user with the latest PSPP 0.7.8. for MSWindows version. I guess this is the same problem:

-words accented: New thing I discovered, when the name of a directory
contains a word accented, then the pspp can't open a file from inside.
Before, this didn't happen.

The directory is named 'Ficheros estadísticos'

Harry Thijssen <unknown_one>
Sat 28 May 2011 04:45:34 PM UTC, comment #3: 

I had a chat with some of the mingw guys on irc and found the root of the problem: Recent incarnations of windows use UTF-16LE encoding not UTF-8.

The attached two patches (one against pspp, one against gnulib) demonstrate one way to circumvent the problem.  Obviously a proper fix will require more thought (especially if its to continue to work with older windoze).

(file #23460, file #23461)

John Darrington <jmd>
Project Administrator
Sun 15 May 2011 04:49:59 AM UTC, comment #2: 

I had another look.  Debugging on windows is not easy because it lacks even basic tools.  However I did ascertain that it is not a GUI related issue.  The problem exists on the terminal interface too.  I'm suspecting, in fact, that it is the open command which is at fault. (this is provided by gnulib?)
If a version with some diagnostic info can be hacked up it might help.

Something else I noticed (I don't think it's related to this problem, but one never knows): 

In file-handle-def.c the comment to fh_create_file says that the
ID must be UTF-8, but doesn't say anything about the FILE_NAME.
However since the implementation copies them both into the same string, then either we must insist that FILE_NAME is also UTF8 or we must convert it inside the function.

John Darrington <jmd>
Project Administrator
Fri 13 May 2011 04:34:14 AM UTC, comment #1: 

John, I don't yet see where the problem is occurring.  It looks to me that load_file() in psppire-data-window.c is converting from glib file name encoding into UTF-8 and using the UTF-8 version to compose the syntax string.  I'm having a lot of trouble seeing what's going wrong here; everything looks consistent.

Would you mind taking a closer look?



Ben Pfaff <blp>
Project Administrator
Sat 07 May 2011 05:02:39 AM UTC, original submission:  

Using 0.7.7-gfa1fed on w32, tried to save, then reopen a dataset, using a filename containing a non-ascii filename.

The file could be saved, but the name got corrupted.  It was then not possible to open this file.

The log from pspp.jnl show what is happening:

 SAVE OUTFILE="C:\Users\Martin.Wagner\öl.sav".

 GET FILE="C:\Users\Martin.Wagner\öl.sav".

 error: An error occurred while opening   `C:\Users\Martin.Wagner\öl.sav': No such file or directory.

John Darrington <jmd>
Project Administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #23460:  rename.patch added by jmd (1KiB - text/x-patch)
file #23461:  make-file.patch added by jmd (566B - text/x-patch)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by jmd
  • -email is unavailable- added by unknown_one (Posted a comment)
  • -email is unavailable- added by blp (Posted a comment)
  • -email is unavailable- added by jmd (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-03-14 jmd Open/ClosedOpen => Closed
    2016-03-14 jmd StatusIn Progress => Fixed
    2015-10-23 jmd CategoryInternationalization/Localization => Compilation/Portability
        StatusNeed Info => In Progress
        SummaryNon-ascii filenames not properly handled. => Windows operating systems cannot open files whose names contain non-ascii characters.
    2013-12-03 jmd Carbon-Copy- => Added sicsu <alsicsu@gmail.com>
    2012-02-23 jmd ReleaseBefore 0.8.0 => Future
    2011-05-28 jmd Attached File- => Added rename.patch, #23460
        Attached File- => Added make-file.patch, #23461
    2011-05-14 blp ReleaseNone => Before 0.8.0
    2011-05-13 blp StatusNone => Need Info

    Back to the top

    Powered by Savane 3.6