bugGNU Octave - Bugs: bug #60306, Error running files in a folder...

 
 

bug #60306: Error running files in a folder containing letters that aren't encoded in system locale

Submitter:  None
Submitted:  Mon 29 Mar 2021 06:25:40 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Fixed Assigned to:  None
Originator Name:  t Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * Microsoft Windows Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 12 Apr 2021 11:36:48 AM UTC, comment #29: 

This work for me now with a build from the default branch with files in folders that contain "č" or the Chinese characters "字符" on Windows 10 with the locale encoding set to "CP1252".

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Sun 11 Apr 2021 11:42:07 AM UTC, comment #28: 

After the changes for bug #60014, we require a Windows API that corresponds to Windows 7 or newer. That means we are able to use `GetFinalPathNameByHandleW` now [1].

With that function, it is possible to significantly reduce the complexity of our implementation of `canonicalize_file_name` on Windows. At the same time, this also means we are able to solve the locale encoding issue without requiring C++17 STL functions.

I'd also expect that the new implementation will be more performant than the existing one because it requires to open a handle to the respective file only once per function call instead of several times (at least 3 times I would expect with the previous implementation).

I pushed the change here:
https://hg.savannah.gnu.org/hgweb/octave/rev/d7223245d26f

Marking as ready for test.

[1]: https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfinalpathnamebyhandlew

Markus Mützel <mmuetzel>
Group administrator
Fri 09 Apr 2021 04:34:10 PM UTC, comment #27: 

I opened a discussion about beginning to use C++17 here:

https://octave.discourse.group/t/using-c-17-features/1026

As for Boost, if I understand correctly, some parts are just header files and some require compiled library bits, so I guess the difficulty of adding Boost components as dependencies depends on exactly what features we use.

John W. Eaton <jwe>
Group administrator
Fri 09 Apr 2021 12:34:04 PM UTC, comment #26: 

Thanks. Afaics, the STL std::filesystem is (almost?) identical to boost::filesystem.
In particular, the boost::filesystem::canonical functions seems to be the same.

I'll bring up in the next meeting if it is ok to make Octave dependent on the boost libraries. That could be an alternative in case we'd prefer to wait a considerable time before adopting C++17.

Did you use the boost libraries before? Is it just like linking to any other library? Or is there something special that one might need to consider?

Markus Mützel <mmuetzel>
Group administrator
Fri 09 Apr 2021 10:31:24 AM UTC, comment #25: 
Dmitri A. Sergatskov <dasergatskov>
Fri 09 Apr 2021 09:42:55 AM UTC, comment #24: 

I pushed a partial fix here that afaict will allow to execute files in the current working directory even if it contains characters that cannot be encoded in the locale charset:
https://hg.savannah.gnu.org/hgweb/octave/rev/9f0649b4e912

The change will be part of Octave 7 which will probably be released some time next year.

This still won't allow to add paths containing non-encodable characters to the search path.

A "proper" fix will probably be to switch to using std::filesystem::canonical (and get rid of all the work-arounds we currently have for Windows).

Markus Mützel <mmuetzel>
Group administrator
Wed 07 Apr 2021 05:27:34 PM UTC, comment #23: 

comment #21:

> Yes, you could google how to fix it, change the folder name and what-not, but this shouldn't be happening like it isn't for any other program out there...
>
> Also the fact that it says not such file rather than what actually is the problem isn't very helpful.


I can understand that you would like to have a quick fix for this issue. But please let me add a few comments:

  • Nobody suggested that you should google how to fix it (or I have missed the related post)
  • Our suggestions to test without or with other special characters in the file path were required to find the cause of the bug you are seeing. Otherwise there is no way to fix this bug in future versions of octave.
  • You are completely right, this bug should not happen. That's why we are trying to find the cause and ways how to fix it.
  • A program that can not find a file under a given location can not determine why the file does not exist and can't give a more meaningful error message. In your case, the program does not know that it has misinterpreted the provided location due to encoding problems and is looking at the wrong location for the file.
Torsten Lilge <ttl>
Group Member
Wed 07 Apr 2021 04:07:47 PM UTC, comment #22: 

Unfortunately, that is a peculiarity of the Windows OS and due to a design choice MS made in the past (using a bunch of different locales instead of switching to Unicode/UTF-8). Octave (and its users) isn't the only program that has to live with the consequences of that decision. As I've shown in comment #14, Matlab has it's issues with this as well.
You've been presented with two different workarounds within just short of two days. First by Torsten less than an hour after your first submission who suggested to stick to ASCII characters in the file path. Second by me (selecting a locale that is "good" for you).

No one was claiming that this is something that "should be happening".
Like I pointed out, it took quite a while for (presumably) intelligent people to come up with a standard solution to the problem (the "std::filesystem" API in the C++17 STL). But it'll probably take some more time before support for that API is widespread enough so that we can use it without breaking support for some platforms.
Anyway, I'll ask on the next maintainers meeting if there are plans to allow the newer STLs.

Markus Mützel <mmuetzel>
Group administrator
Wed 07 Apr 2021 12:56:58 PM UTC, comment #21: 


> "CP1252" doesn't have a code point for "č". But it encodes "š" at code point 0x9A:
> https://en.wikipedia.org/wiki/Windows-1252
>

Try it with any other "special" character then. Doesn't work for any of them.

Yes, you could google how to fix it, change the folder name and what-not, but this shouldn't be happening like it isn't for any other program out there...

Also the fact that it says not such file rather than what actually is the problem isn't very helpful.

Thank god we don't do this at school anymore. Never using it again.

Tijana <tijana>
Wed 31 Mar 2021 08:26:27 AM UTC, comment #20: 

"CP1252" doesn't have a code point for "č". But it encodes "š" at code point 0x9A:
https://en.wikipedia.org/wiki/Windows-1252

I might work if you changed your system locale to one that encodes all (or most) characters you'd like to use. Maybe "CP1250" would better match your requirements:
https://en.wikipedia.org/wiki/Windows-1250

For instructions how to change the system locale on Windows, see e.g. here:
https://www.java.com/en/download/help/locale.html

A "real" fix in Octave will probably be to move away from gnulib's `canonicalize_file_name` replacements on Windows and use `std::filesystem::canonical` from C++17 STL instead. But we can't do that yet because we still support compilers that (only) know C++11.

Marking as postponed.


Markus Mützel <mmuetzel>
Group administrator
Tue 30 Mar 2021 07:52:41 PM UTC, comment #19: 


> Well, "umlaut a" isn't called "umlaut a" in German either (because it is a different letter with a distinct sound from "a"). But it's still called "umlaut a" internationally. Afaict, "č" is called "small Latin letter c with caron":
> https://www.compart.com/en/unicode/U+010D
>

As far as I know though (and I've been learning German for 7-ish years now), you still call them umlauts in German, which is not the case here... But it really doesn't matter does it... Anyhow...

>
> @Tijana: What does `__locale_charset__` return on your system?
>
>> canonicalize_file_name (".")

ans = C:\Users\tijan\Desktop\škola shit\primena racunara\Prvi domaci

(actual path is C:\Users\tijan\Desktop\škola shit\primena računara\Prvi domaci)

>> _locale_charset_

ans = CP1252

Tijana <tijana>
Tue 30 Mar 2021 06:14:24 PM UTC, comment #18: 

Well, "umlaut a" isn't called "umlaut a" in German either (because it is a different letter with a distinct sound from "a"). But it's still called "umlaut a" internationally. Afaict, "č" is called "small Latin letter c with caron":
https://www.compart.com/en/unicode/U+010D

Thanks again Dmitri for that information.

In the meantime, I believe I narrowed the issue down to `canonicalize_file_name`. With the current folder set to a path with these letters, I see the following:

>> canonicalize_file_name (".")
ans = D:\SVN\Octave\bug60306_non-ASCII\hü racunara\fgäööü škola racunara


The actual path is 'D:/SVN/Octave/bug60306_non-ASCII/hü računara/fgäööü škola računara'.

It looks like the result of gnulib's `canonicalize_file_name` replacements on Windows is wrong for me in this case.

We have the following note in the source code:

      // On Windows, convert to locale charset before passing to
      // canonicalize_file_name, and convert back to UTF-8 after that.

      // FIXME: This only allows non-ASCII characters in the file or path that
      // can be encoded in the locale charset.
      // Consider replacing this with std::filesystem::canonical once we allow
      // using C++17.


@Tijana: What does `__locale_charset__` return on your system?

Markus Mützel <mmuetzel>
Group administrator
Tue 30 Mar 2021 05:12:27 PM UTC, comment #17: 

but it's not only č it has a problem with... I tried removing the č and keeping the š (and vice versa), but neither worked...

Tijana <tijana>
Tue 30 Mar 2021 04:58:40 PM UTC, comment #16: 


> Does that letter have a special name (like "umlaut a" for the letter "ä")? That might make it easier searching for a solution.
>

Nope. Just a plain old Serbian č (ч in cyrilic)... Our letters don't really have names because every letter has its own vocal...
We've also got ć, š, ž and đ and no, these are not accented, they are just different letters...

Tijana <tijana>
Tue 30 Mar 2021 04:08:45 PM UTC, comment #15: 

Caron is the thing on the top of the letters:

https://en.wikipedia.org/wiki/Caron

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 30 Mar 2021 03:34:46 PM UTC, comment #14: 

Thanks Dmitri for transcribing the problematic letters from the image to plain text. With that, I can reproduce the issue with Octave 6.2. But it also doesn't work with Octave 6.1, Octave 5.2, or Octave 4.4.2 for me. So - at least for me - it's not a regression.

There seems to be something special going on on a lower level with the letter "č" in paths on Windows: When I save a file in a folder containing that letter and try to open it with a double click (default program for that file extension is Matlab R2016a on the system I tested with), I also get an error message from Matlab saying that the file couldn't be found. In that error message the letter "č" also is replaced with the letter "c". Similarly, the letter "š" (and fwiw some umlaut characters) are retained also in the Matlab error message.
Also in Matlab R2016a:

>> cd('D:\SVN\Octave\bug60306_non-ASCII\hü računara\fgäööü škola računara')
>> test_file
D:\SVN\Octave\bug60306_non-ASCII\hü raunara\fgäööü škola raunara\test_file.m: Can't open file.

>>


Who would have guessed that we are "compatible" to that level. ;-)

With some luck, someone already ran into the same issue and described a solution.

Does that letter have a special name (like "umlaut a" for the letter "ä")? That might make it easier searching for a solution.

Markus Mützel <mmuetzel>
Group administrator
Tue 30 Mar 2021 12:49:20 PM UTC, comment #13: 


comment #9:

> Could you please attach a file that doesn't run for you in Octave 6.2 but does work in Octave 6.1?
> What is the full path to the file at the location where it doesn't work?


yeah I can attach one of those that used to run fine in 6.1, but the 6.1 didn't work after I upgraded to 6.2

(file #51160)

Tijana <tijana>
Tue 30 Mar 2021 07:17:46 AM UTC, comment #12: 

I wonder if it is a utf-8 vs win-1250 vs iso-8859-2 mix-up

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 30 Mar 2021 06:55:40 AM UTC, comment #11: 

It is kind of peculiar that (č) got messed up in računara -> racunara, but (š) in škola got preserved.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 30 Mar 2021 06:28:14 AM UTC, comment #10: 


Kai Torben Ohlhus <siko1056>
Group Member
Tue 30 Mar 2021 06:00:29 AM UTC, comment #9: 

Could you please attach a file that doesn't run for you in Octave 6.2 but does work in Octave 6.1?
What is the full path to the file at the location where it doesn't work?

Markus Mützel <mmuetzel>
Group administrator
Mon 29 Mar 2021 07:38:10 PM UTC, comment #8: 

thanks! I hope it gets fixed fast so I can continue doing my homework!

Tijana <tijana>
Mon 29 Mar 2021 07:36:25 PM UTC, comment #7: 

I did not say that it is cool to have to avoid special characters. I just want to get information for being able to find the cause of the issue you are seeing.

cc-ing Markus, since this seems to be an encoding issue unter Windows

Torsten Lilge <ttl>
Group Member
Mon 29 Mar 2021 07:23:21 PM UTC, comment #6: 

Okay. It's fine when I put it on my desktop. Still not cool that I can't have a folder named in my language you know?

Tijana <tijana>
Mon 29 Mar 2021 07:20:21 PM UTC, comment #5: 

Sure. But 6.1 hasn't had any problems with those until I installed the 6.2

Tijana <tijana>
Mon 29 Mar 2021 07:17:58 PM UTC, comment #4: 

Could you try this with a path without special characters?

Torsten Lilge <ttl>
Group Member
Mon 29 Mar 2021 07:15:20 PM UTC, comment #3: 

Really?

It can't seem to find files when clicked on run and save as well.

And this leads to the same error message as if you want to open a file with octave? This was not clear to me.


Torsten Lilge <ttl>
Group Member
Mon 29 Mar 2021 07:05:35 PM UTC, comment #2: 

yes. I even wrote it in my submission.

Tijana <tijana>
Mon 29 Mar 2021 06:57:09 PM UTC, comment #1: 

Thanks for the report. Is there an error message when you select "Save and Run"?

Torsten Lilge <ttl>
Group Member
Mon 29 Mar 2021 06:25:40 PM UTC, original submission:  

The 6.1 worked completely fine, but after upgrading to 6.2 and trying to open any *.m files (octave is set as default), I would get 'error: no such file' even though it used to open completely fine before. It can't seem to find files when clicked on run and save as well. Have tried uninstalling, but it doesn't seem to fix the problem.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #51160:  Zadatak1.m added by tijana (286B - application/octet-stream)
file #51157:  Screenshot@2021-03-29@202228.png added by None (202KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by ttl
  • -email is unavailable- added by tijana (Posted a comment)
  • -email is unavailable- added by ttl (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-04-12 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2021-04-11 mmuetzel StatusPostponed Ready For Test
    2021-03-31 mmuetzel StatusConfirmed Postponed
        SummaryError running files in a folder containing letter &quot;č&quot; Error running files in a folder containing letters that aren't encoded in system locale
    2021-03-30 mmuetzel CategoryNone Interpreter
        Item GroupNone Unexpected Error or Warning
        StatusNone Confirmed
        Release6.2.0 dev
        Summaryno such file Error running files in a folder containing letter "č"
    2021-03-30 tijana Attached File- Added Zadatak1.m, #51160
    2021-03-29 ttl Carbon-Copy- Added mmuetzel
    2021-03-29 None Attached File- Added Screenshot@2021-03-29@202228.png, #51157

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code