bugGNU Octave - Bugs: bug #42036, Files and directories with...

 
 

bug #42036: Files and directories with non-ASCII characters not handled correctly on Windows

Submitter:  Armin Müller <arm_in>
Submitted:  Thu 03 Apr 2014 09:32:09 AM UTC
   
 
Category:  Libraries Severity:  1 - Wish
Priority:  5 - Normal Item Group:  Feature Request
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 22 Aug 2018 10:05:41 AM UTC, comment #26: 

The fixes have been applied on the default branch that will eventually be released as Octave 5.0 (probably by the end of this year).

Markus Mützel <mmuetzel>
Group administrator
Wed 22 Aug 2018 09:23:40 AM UTC, comment #25: 

I still have trouble witch point 3)

Octave 4.4.1, 64bit
Windows 10 Ver 1709, 64bit

In the Octave GUI "File Browser", double clicking on a folder that contains an Umlaut character:

  • File Browser will actually change into the indicated folder
  • Command Windows will show an "error: parse error" and remain in the present folder
  • Current Directory will also remain in the present folder
Armin Müller <arm_in>
Sat 14 Jul 2018 12:57:26 PM UTC, comment #24: 

All of the mentioned points should be fixed on the default branch except for point 4) and 5) which is tracked separately in bug #47571 and point 6) which is tracked in bug #50409.

Closing this bug as fixed since the bug in the title should no longer occur.

Markus Mützel <mmuetzel>
Group administrator
Thu 15 Mar 2018 01:48:05 PM UTC, comment #23: 

Retested running Octave 4.2.2 GUI under Windows 7 x64 German edition

Same behavior as illustrated in the original post.

1) + 2) + 3) + 5) = 100% identical

4) It is not possible anymore to enter non-ASCII characters. No bogus replacement of ANSI characters.

6) Time Zone is now shortened to "GMT"

Armin Müller <arm_in>
Mon 27 Mar 2017 08:42:58 PM UTC, comment #22: 

The GUI retains the settings.

Márton Marczell <marczellm>
Mon 27 Mar 2017 05:19:49 PM UTC, comment #21: 

@Márton: To clarify the severity of this bug: Is it possible for you to start the GUI at all? If it does start, does the GUI retain settings between sessions (e.g. can you change the GUI language)?


Markus Mützel <mmuetzel>
Group administrator
Sat 25 Mar 2017 09:35:06 AM UTC, comment #20: 

Please, when fixing this patch, don't forget to test the case where the Windows username contains an accent. I ended up with this username when I created my Windows account based on my Microsoft account and my real name, which is Márton. My user data directory is consequently C:\Users\Márton, and when I start Octave, either CLI or GUI, it says


error: C:/Users/Márton: No such file or directory


and I cannot access files and folders inside my user directory at all.

Márton Marczell <marczellm>
Sun 13 Nov 2016 01:50:56 PM UTC, comment #19: 

I just uploaded a patch (file #38945) to bug #49118 which might partially solve the issue here, too.

Markus Mützel <mmuetzel>
Group administrator
Tue 29 Mar 2016 06:51:10 AM UTC, comment #18: 

Workaroud to access the files in the directories with non-ASCII characters. (for files I do not find the work around)

From cmd prompt


subst W: (path to the directory which has non-ASCII characters)



C:\Users\Tatsu>subst /?
Associates a path with a drive letter.

Explanation of the subst command

SUBST [drive1: [drive2:]path]
SUBST drive1: /D

  drive1:        Specifies a virtual drive to which you want to assign a path.
  [drive2:]path  Specifies a physical drive and path you want to assign to
                 a virtual drive.
  /D             Deletes a substituted (virtual) drive.

Type SUBST with no parameters to display a list of current virtual drives.


For the "path" you can use UNC path so that you can access Network drive which does not have a drive letter.

Of course, make xxx.bat file is useful.

Tatsuro MATSUOKA <tmacchant>
Sat 16 May 2015 12:37:02 AM UTC, comment #17: 

Torsten

Thank you for your reply.
I have tried to build octave with the same patch as yours.

For my case, result was same as for octave-4.0.0-rc2
https://savannah.gnu.org/bugs/?44538

Tatsuro

Tatsuro MATSUOKA <tmacchant>
Thu 14 May 2015 07:43:29 AM UTC, comment #16: 

I have added the options into the main Makefile, here the diff:


diff -r e66c883008d4 Makefile.in
--- a/Makefile.in       Fri May 08 11:00:50 2015 -0400
+++ b/Makefile.in       Thu May 14 09:41:41 2015 +0200
@@ -297,7 +297,7 @@ ifeq ($(MXE_NATIVE_BUILD),yes)
     # different configuration tags.
     HOST_AND_BUILD_CONFIGURE_OPTIONS := \
            CC='$(MXE_CC)' CXX='$(MXE_CXX)' F77='$(MXE_F77)' DLLTOOL='$(MXE_DLLTOOL)' CCAS='$(MXE_CCAS)' \
-           CFLAGS='-O2' CXXFLAGS='-O2' ac_cv_f77_compiler_gnu=no RANLIB='$(MXE_RANLIB)'
+           CFLAGS='-O2 -DUNICODE -D_UNICODE' CXXFLAGS='-O2 -DUNICODE -D_UNICODE' ac_cv_f77_compiler_gnu=no RANLIB='$(MXE_RANLIB)'
     # CLGCC wrappers automatically adjust INCLUDE and LIB variables.
     CONFIGURE_CPPFLAGS :=
     CONFIGURE_LDFLAGS :=


Torsten Lilge <ttl>
Group Member
Mon 11 May 2015 09:30:53 AM UTC, comment #15: 

Torsten

How did you do that?
(I have build mxe-octave adding -DUNICODE and -D_UNICODE to the compile flags in the Makefile and changed the codepage to 65001 in the terminal window.)

I have tried to add  -DUNICODE and -D_UNICODE in CFLAGS and CXXFLAGS in src/octave-mk and built mxe-octave (4.0.0-rc4)  However, my results were different.

The ls command at codepage 932 (Japanese), ls command gives broken screen but they are different from octave-4.0.0-rc4 on ftp alpha site.

After system('chcp 65001'), for me, ls command works perfect.
The system('type nihongo.m') represent Japanese characters but nihongo.m did not work.

I could not handle path and file names which contain Japanese multibyte character are not able to be handled.

MinGW page suggests to use wmain or _tmain and use wchar or tchar.
However the use of it loses portability of the code.

I cannot figure out what is the good solution for this matter.
 



Tatsuro MATSUOKA <tmacchant>
Sun 10 May 2015 08:23:53 AM UTC, comment #14: 

I guess it is related since the mxe build seems to be an ansi, not an unicode build.

I have build mxe-octave adding -DUNICODE and -D_UNICODE to the compile flags in the Makefile and changed the codepage to 65001 in the terminal window. The result was that 'dir' shows the correct names (with German umlauts) whereas 'ls' does not. Double clicking a folder with umlauts in the file browser correctly changes the directory which can be checked by 'pwd' showing the correct name.

With codepage 850, 'ls' works correctly but 'dir' does not. A double click on a folder with umlauts still correctly changes the directory.

Torsten Lilge <ttl>
Group Member
Sat 09 May 2015 08:17:25 PM UTC, comment #13: 
Tatsuro MATSUOKA <tmacchant>
Sat 09 May 2015 11:05:00 AM UTC, comment #12: 

gnulib::readdir and opendir (for mingw) is a wrapper around Windows FindNextFile etc


John Donoghue <lostbard>
Group Member
Sat 09 May 2015 03:59:51 AM UTC, comment #11: 

octave uses gnulib dirent related functions.
However, mkdir, readdir, opendir seem not be used.
Can the situation be changed if gnulib alternatives are used ?

Tatsuro MATSUOKA <tmacchant>
Fri 08 May 2015 08:58:21 AM UTC, comment #10: 

I have test the mxe-octave that used gcc-5.1.0.
I cannot handle file names and path names which contain non-ascii character gcc-5.1.0 based mxe-octave.

The gcc-5.1.0 seems not to be solution for this issue.

Tatsuro MATSUOKA <tmacchant>
Sun 22 Mar 2015 02:15:32 PM UTC, comment #9: 

Interestingly, writing some code in Visual C++ 2008, using the WIn32 functions for directory traversal, and _tstat for the stat function, the Win32 code works in compiled as Unicode.

The code fails for the stat function is set to multibyte mode. The filenames at that point (multibyte mode) are the same as what mingw is getting from readdir.




John Donoghue <lostbard>
Group Member
Sun 08 Mar 2015 06:46:46 PM UTC, comment #8: 

I cannot speak for the original poster, but I guess bug #43099 is pretty similar.

Armin Müller <arm_in>
Sun 08 Mar 2015 05:19:07 PM UTC, comment #7: 

Maybe - I dont know if it is only that the system functions are getting/providing incorrect values, which then will be displayed incorrectly, or whether there are also display of the utf-8 string issues as well.

John Donoghue <lostbard>
Group Member
Sun 08 Mar 2015 02:44:39 PM UTC, comment #6: 

Is there any difference between this bug and bug #43099?

Mike Miller <mtmiller>
Group Member
Thu 05 Mar 2015 05:57:07 PM UTC, comment #5: 

Writing a standalone c program that uses open/read/close dir and stat functions in mingw, it appears that when providing the d_name  from readdir to the mingw stat function, it fails (only for the utf8 named files)


John Donoghue <lostbard>
Group Member
Wed 04 Mar 2015 06:50:38 PM UTC, comment #4: 

Single step through a = dir() shows:


>> a  =dir()
warning: dir: 'lstat (.\X+úbung.txt)' failed: No such file or directory
warning: called from
    dir at line 120 column 9
warning: dir: 'lstat (.\X+úbung)' failed: No such file or directory



John Donoghue <lostbard>
Group Member
Wed 04 Mar 2015 05:46:31 PM UTC, comment #3: 

On my machine, I have a folder and file that are using Unicode characters.

For both gui and cli (windows)....

Runing in octave, 'ls' and they show names (although names are mangled for the Unicode files).

Running in octave 'dir', and they do not seem to be shown at all.

If I run:

a = dir();
a.name


I see for those files:

ans = [](0x0)
ans = [](0x0)

 

John Donoghue <lostbard>
Group Member
Fri 04 Apr 2014 07:06:59 AM UTC, comment #2: 

You are right, there is the same issue with the Command Line only version of Octave.


octave:11> pwd
ans = d:\Octave
octave:12> cd M\374nchen/       <= autocomplete with TAB
octave:13> pwd
ans = d:\Octave\M³nchen
octave:14>


But it is more visible in the GUI, because the GUI elements have been nicely translated with Umlaut :-)

4) Right, I just noticed that the Grave and Circumflex are ASCII, the Degree and Acute are not.

6) Probably not only the file system, but the whole library functions.

Armin Müller <arm_in>
Fri 04 Apr 2014 03:41:09 AM UTC, comment #1: 

This works fine for me on a Linux system with 3.8.1, so likely specific to Windows filesystems or the library functions Octave is using to work with Windows files and directories.

Since you verified that the same commands in the command window do not work, this is not really a GUI issue at all.

Mike Miller <mtmiller>
Group Member
Thu 03 Apr 2014 09:32:09 AM UTC, original submission:  

There are some special issues when using "Umlaut" characters on a German keyboard. Running Octave 3.8.1 under Windows 7 x64 German edition
https://de.wikipedia.org/wiki/Umlaut

Let's say we are actually in the folder "D:\Octave" and there is a subfolder "D:\Octave\München"

1) In the file browser, "München" will be displayed correctly.

2) Some translated texts contain Umlaut characters. OK.

3) When double-clicking on the name "München", the message "error: D:/Octave/M**nchen: No such file or directory" will appear.

(** = garbled text, looks like Unicode)

According to the file browser, the directory has been successfully changed.

According to the "Current Directory" line, the current directory is still "D:\Octave"

Same with "pwd" answer.

4) When trying to enter "cd 'München'" in the command window by hand, the Umlaut "ü" will be replaced by a verical bar '|'.

Actually, a limited num

ä => d
Ä => D
ö => v
Ö => V
ü => |
Ü => \
° => 0 (degree symbol)
´ => 4 (acute accent)

5) Getting the Umlaut via copy & paste from an external ANSI editor is not pssible, either.

6) Messages in the command history get garbled. For example, the time stamp "Mitteleuropäische Zeit" after starting octave GUI is reading "Mitteleurop?ische Zeit".

Armin Müller <arm_in>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #31112:  AD4 - enter by hand.png added by arm_in (51KiB - image/png)
file #31113:  AD3 - double click.png added by arm_in (56KiB - image/png)
file #31114:  AD6 - history window.png added by arm_in (55KiB - image/png)

 

Carbon-Copy List
  • -email is unavailable- added by gwiora (I just posted [bug #53406])
  • -email is unavailable- added by marczellm (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by tiagoma
  • -email is unavailable- added by ttl (Posted a comment)
  • -email is unavailable- added by tmacchant (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by arm_in (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 16 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-07-14 mmuetzel StatusNone Fixed
        Open/ClosedOpen Closed
        Release4.2.1 dev
    2018-03-23 gwiora Carbon-Copy- Added gwiora
    2017-03-27 mmuetzel Release3.8.1 4.2.1
    2015-07-01 tiagoma Carbon-Copy- Added tiagoma
    2015-05-03 mtmiller Dependencies- bugs #44536 is dependent
    2015-04-25 rik5 Dependencies- bugs #44922 is dependent
    2014-04-04 mtmiller CategoryGUI Libraries
        Severity3 - Normal 1 - Wish
        Item GroupNone Feature Request
        SummaryUmlaut in file name or folder name not handled correctly Files and directories with non-ASCII characters not handled correctly on Windows
    2014-04-03 arm_in Attached File- Added AD1 - file browser - AD2 tooltip.png, #31111
        Attached File- Added AD4 - enter by hand.png, #31112
        Attached File- Added AD3 - double click.png, #31113
        Attached File- Added AD6 - history window.png, #31114

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code