bugGNU Octave - Bugs: bug #59269, `get_ASCII_filename` cannot rely...

 
 

bug #59269: `get_ASCII_filename` cannot rely on short file names consisting of only ASCII characters

Submitter:  None
Submitted:  Wed 14 Oct 2020 12:15:31 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Fixed Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * Microsoft Windows Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 08 Nov 2020 11:09:20 AM UTC, comment #16: 

No new issues reported since this change.
Closing as fixed.

If it turns out that the issue isn't fixed, we could re-open this report or open a new one.

Markus Mützel <mmuetzel>
Group administrator
Sun 01 Nov 2020 12:42:10 PM UTC, comment #15: 

I pushed the following change to default:
https://hg.savannah.gnu.org/hgweb/octave/rev/3b90b3b00451

I have no way of verifying if this actually works.
At the moment, I suspect that it might be locale dependent. On my locale, short file names don't contain non-ASCII characters.

A possible test would be to create an image file that contains non-ASCII characters and try to read it with `imread`:

[file, pth] = uigetfile ();
a = imread (fullfile (pth, file));


Marking as ready for test. But I don't know if we have testers that have a Windows with CJK locale. So, I'll likely just close as fixed if no problems occur within a week or so...

Markus Mützel <mmuetzel>
Group administrator
Fri 30 Oct 2020 10:26:49 AM UTC, comment #14: 

I grafted that change to stable here:
https://hg.savannah.gnu.org/hgweb/octave/rev/139df373c107

@dillcn: It would be nice if you could confirm that this fixes the issue for you with a nightly build from tomorrow Oct 31 (or later):
https://buildbot.octave.space/#/download

Re-titling to reflect the remaining issue and moving release to "dev".

Markus Mützel <mmuetzel>
Group administrator
Wed 28 Oct 2020 09:10:16 AM UTC, comment #13: 

This particular error is already fixed for Octave 7 by using the `wchar_t` overload to open file streams on Windows:
https://hg.savannah.gnu.org/hgweb/octave/rev/159b6a1eb408

I'm considering grafting that change to stable (Octave 6).
Revisiting that changeset, I'd guess that there is only a low risk of error if we would graft it to stable (even if it touches a lot of files).
The work-around at the moment is to avoid non-ASCII characters in file names (at least in their first 6 bytes).

I'm tentatively setting the release to 6.0.92.

But on the long run, we'll probably need a more general fix for `octave::sys::get_ASCII_filename`. That change should probably be done on the default branch (Octave 7).

Markus Mützel <mmuetzel>
Group administrator
Mon 26 Oct 2020 09:34:39 AM UTC, comment #12: 

Thank you very much. This was very enlightening. I wasn't aware that short file names could still contain non-ASCII characters.
This isn't the case for me:

D:\SVN\Octave\abcäbc\录波文件>dir /X *.csv
 Datenträger in Laufwerk D: ist Volume
 Volumeseriennummer: 0EC1-7764

 Verzeichnis von D:\SVN\Octave\abcäbc\录波文件

20.10.2020  15:15                12 200807~1.CSV 2008年07月18日06时23分07秒.csv
               1 Datei(en),             12 Bytes
               0 Verzeichnis(se), 233.847.832.576 Bytes frei

That makes a big difference for how we are trying to access files on Windows.

I'll have a look at what we could change to make this work more generally.

Markus Mützel <mmuetzel>
Group administrator
Mon 26 Oct 2020 09:23:56 AM UTC, comment #11: 

These are the outputs:

>> system ("dir /X *.csv")

 Volume in drive C is Windows
 Volume Serial Number is D23F-611D

 Directory of C:\Users\lyd\Documents\projects\录录波波文文件件\wave_files

2020/09/21  20:22         5,497,700 131205~1.CSV 131205092548.csv
2019/11/01  23:25             1,398 2008-0~1.CSV -email is unavailable-
2020/09/21  23:20         3,777,918 2008年年~1.CSV  2008年年07月月18日日06时时23分分07秒秒.csv
               3 File(s)      9,277,016 bytes
               0 Dir(s)  167,830,794,240 bytes free
ans = 0

dillcn
Fri 23 Oct 2020 07:37:22 AM UTC, comment #10: 

That looks good to me.

Are short file names enabled on that drive? What do you see when you execute the following in the folder with these files?

system ("dir /X *.csv")


Markus Mützel <mmuetzel>
Group administrator
Fri 23 Oct 2020 02:56:41 AM UTC, comment #9: 

These are the output of "dir" and "icacls":
H 濜̀>> system ("dir /Q *.csv")
 Volume in drive C is Windows
 Volume Serial Number is D23F-611D

 Directory of C:\Users\lyd\Documents\projects\录录波波文文件件\wave_files

2020/09/21  20:22         5,497,700 DESKTOP-1BL1R3H\lyd    131205092548.csv
2019/11/01  23:25             1,398 DESKTOP-1BL1R3H\lyd    -email is unavailable-
2020/09/21  23:20         3,777,918 DESKTOP-1BL1R3H\lyd    2008年年07月月18日日06时时23分分07秒秒.csv
               3 File(s)      9,277,016 bytes
               0 Dir(s)  166,902,984,704 bytes free
ans = 0

>> system ("icacls *.csv")

131205092548.csv NT AUTHORITY\SYSTEM:(F)
                 BUILTIN\Administrators:(F)
                 DESKTOP-1BL1R3H\lyd:(F)

-email is unavailable- NT AUTHORITY\SYSTEM:(F)
                                    BUILTIN\Administrators:(F)
                                    DESKTOP-1BL1R3H\lyd:(F)

2008年年07月月18日日06时时23分分07秒秒.csv NT AUTHORITY\SYSTEM:(F)
                         BUILTIN\Administrators:(F)
                         DESKTOP-1BL1R3H\lyd:(F)

Successfully processed 3 files; Failed processing 0 files
ans = 0

dillcn
Tue 20 Oct 2020 02:33:55 PM UTC, comment #8: 

The markup messed up the commands.
Please, try the following in the directory with the files you want to read:

system ("dir /Q *.csv")
system ("icacls *.csv")


Markus Mützel <mmuetzel>
Group administrator
Tue 20 Oct 2020 02:32:19 PM UTC, comment #7: 

I tried to replicate with Octave 6.0.90 and Octave 5.2.0 on Windows 10 with the following commands:

>> [fname, fpath] = uigetfile ()

fname = 2008年年07月月18日日06时时23分分07秒秒.csv
fpath = D:\SVN\Octave\abcäbc\录录波波文文件件\
>> a = dlmread (fullfile (fpath, fname));


Apart from the doubled full width characters in the command window (bug #58637), this works correctly for me.

Do you have read permission for that file?
Could you please `cd` to the directory with the file you want to read and post what you see for `system ("dir /Q .csv")` and `system ("icacls .csv")`?

Markus Mützel <mmuetzel>
Group administrator
Tue 20 Oct 2020 02:59:58 AM UTC, comment #6: 

Thank you for your reply. I tested with version 6.0.90, and the same error still occurred.

dillcn
Mon 19 Oct 2020 07:58:42 AM UTC, comment #5: 

Also: You don't need to (and should not) open a file with `fopen` if you want to read its content with `importdata`. `fopen` is meant to be used for functions that expect a file identifier `fid` (e.g. `fgets`, `fprintf`, ...)

Markus Mützel <mmuetzel>
Group administrator
Mon 19 Oct 2020 07:54:49 AM UTC, comment #4: 

Ah. Thank you for explaining in more detail.

There have been some improvements wrt opening files with non-ASCII characters on Windows for Octave 6.
Could you please check if the same procedure still fails to read files (with `importdata`) with the current release candidate Octave 6.0.90a? You can download it from here:
https://alpha.gnu.org/gnu/octave/?C=M;O=D

The fullwidth CJK characters being displayed twice is bug #58637. (This is just a displaying issue in the command window. Internally, the file name and path should be fine.)

Markus Mützel <mmuetzel>
Group administrator
Mon 19 Oct 2020 06:05:00 AM UTC, comment #3: 

I'm sorry it wasn't described clearly, and I really don't know where the mistake was.
Windows version 10 with Octave 5.2.0.
I use "uigetfile('*.csv')" to get file path, and "filepath" got the correct file path.
But when I use "fopen(filepath)" to open the file, and use "importdata(filepath, ",")" to importdata, the following error occurred:
====error======
error: dlmread: unable to open file 'C:\Users\lyd\Documents\projects\录录波波文文件件\wave_files\2008年年07月月18日日06时时23分分07秒秒.csv'.
error: called from
    importdata>importdata_ascii at line 249 column 17
    importdata at line 128 column 40
===============
The real file path is "C:\Users\lyd\Documents\projects\录波文件\wave_files\2008年07月18日06时23分07秒.csv", and "filepath" does get the correct file path. But the Chinese characters in the error messages are repeated, and importdata does not work.

dillcn
Wed 14 Oct 2020 05:54:25 PM UTC, comment #2: 

I cannot reproduce on Windows 10 with Octave 5.2.0.

Which Windows version are you using?

I am also not sure why `uigetfile` would call `dlmread`.
Can you please show the exact steps to reproduce the error?

Markus Mützel <mmuetzel>
Group administrator
Wed 14 Oct 2020 01:00:14 PM UTC, comment #1: 

Unable to reproduce on Linux (all file names are accepted). Looks like it's a Windows-specific error?

Anonymous
Wed 14 Oct 2020 12:15:31 PM UTC, original submission:  

Use "uigetfile" to open a file. If the file path contains utf8 characters,such as Chinese characters, an "dlmread" error is reported:
error: dlmread: unable to open file '***'.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Updated the item)
  • -email is unavailable- added by dillcn (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-11-08 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2020-11-01 mmuetzel StatusConfirmed Ready For Test
    2020-10-30 mmuetzel StatusNeed Info Confirmed
        Release6.0.92 dev
        Summarydlmread error when file path have utf8 character `get_ASCII_filename` cannot rely on short file names consisting of only ASCII characters
    2020-10-29 siko1056 StatusConfirmed Need Info
    2020-10-28 mmuetzel Release5.2.0 6.0.92
    2020-10-26 mmuetzel StatusNeed Info Confirmed
    2020-10-14 mmuetzel StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code