bugGNU Octave - Bugs: bug #49118, GUI editor cannot run file in...

 
 

bug #49118: GUI editor cannot run file in folder name with umlaut

Submitted by:  Hartmut <hardy>
Submitted on:  Sat 17 Sep 2016 08:57:00 PM UTC  
 
Category:  GUI Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  dev Operating System:  Microsoft Windows

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Thu 12 Jul 2018 02:50:07 PM UTC, comment #74: 

Sorry, I tried to test this. But the patched linux build is not supposed to behave different than the current Octave release. Most improvements should show up on the mxe builds. And I haven't done mxe builds in a long time, now...

I'm fine with this. Let's keep this bug report closed until some trouble might become visible, probably with international uses of the mxe (Windows) built.

Hartmut <hardy>
Thu 12 Jul 2018 10:05:08 AM UTC, comment #73: 

It is probably OK to close the bug now. If some issues should arise, this bug report could be re-opened or new bug reports could be created.

Closing as fixed.

Markus Mützel <mmuetzel>
Project Member
Thu 12 Jul 2018 12:03:52 AM UTC, comment #72: 

@Markus: No response for one week from original poster.  When do you think this bug can just be closed?

Rik <rik5>
Project Administrator
Wed 04 Jul 2018 07:23:05 PM UTC, comment #71: 

Pushed a patch that passes the filename argument for HDF5 functions through get_ASCII_filename here:
http://hg.savannah.gnu.org/hgweb/octave/rev/64715551b515

This concludes the series of patches for this bug.
Marking as ready for test.

Markus Mützel <mmuetzel>
Project Member
Tue 03 Jul 2018 07:11:19 PM UTC, comment #70: 

Pushed a patch that passes the filename argument for std::ifstream through get_ASCII_filename here:
http://hg.savannah.gnu.org/hgweb/octave/rev/d6050ba12c0c

Markus Mützel <mmuetzel>
Project Member
Sat 30 Jun 2018 06:12:43 PM UTC, comment #69: 

Add function that returns an ASCII-only file name corresponding to the input file name here:
http://hg.savannah.gnu.org/hgweb/octave/rev/7dad5fa7e88e

TL;DR:
Any better ideas how to handle the situation on Windows?
Does it make sense to inline this function for Linux? How should that be done?

Some further explanations for the motivation:
At quite a few places in the code we are passing file names as char arrays to external library functions (or gnulib and standard functions fwiw).
When these functions try to locate the corresponding file on the disc, they would need to use the wide character API on Windows to correctly open files with non-ASCII characters.
But they have no way of knowing which encoding we are using for the passed string. So they have no way of reliably converting to a wchar_t array. (I.e. there is no possible fix for these functions with current C/C++.)

To solve the dilemma, the function "get_ASCII_filename" first checks whether there are any non-ASCII characters in the passed file name. If there are not, it returns the original name.
Otherwise, it tries to obtain the short file name (8.3 naming scheme) which only consist of ASCII characters and are save to pass. However, short file names can be disabled for performance reasons on the file system level with NTFS. So there is no guarantee that these exist.
If short file names are not stored, a hard link to the file is created. For this the path to the file is split at the deepest possible level that doesn't contain non-ASCII characters. At that level a hidden folder is created that holds the hard links. That means we need to have write access on that location. A path to that hard link is returned.

If the file system is FAT32, there are no hard links. But FAT32 always stores short file names. So we are save.

ExFAT that is occasionally used on USB sticks and SD cards stores neither short file names nor does it support hard links. So for exFAT with this function, there is (currently) no way to generate a file name that is stripped from non-ASCII characters but still is valid.

For Linux, this function essentially does nothing. It might be good idea if it could be inlined (and eventually optimized out by the compiler). But the Windows branch is much too large to be included in the header (I guess).

Markus Mützel <mmuetzel>
Project Member
Sat 30 Jun 2018 02:45:03 PM UTC, comment #68: 

Changed over to the corresponding wide character versions of the WinAPI functions used in sysdep.cc here:
http://hg.savannah.gnu.org/hgweb/octave/rev/197126863d95

Markus Mützel <mmuetzel>
Project Member
Fri 29 Jun 2018 08:36:23 PM UTC, comment #67: 

Pushed a patch to allow dynamic libraries to be loaded from paths with non-ASCII characters:
http://hg.savannah.gnu.org/hgweb/octave/rev/366d4b8f6668

This will also allow .mex and .oct files located there to be executed.

Markus Mützel <mmuetzel>
Project Member
Fri 29 Jun 2018 01:39:41 PM UTC, comment #66: 

Added a wrapper for fopen to support opening files at paths with non-ASCII characters and replaced some occurrences of std::fopen here:
http://hg.savannah.gnu.org/hgweb/octave/rev/7fbc39a46be8

Markus Mützel <mmuetzel>
Project Member
Fri 29 Jun 2018 10:42:47 AM UTC, comment #65: 

Pushed the part that adds support for Unicode names in directory listings here:
http://hg.savannah.gnu.org/hgweb/octave/rev/8945a6a6c0eb

Markus Mützel <mmuetzel>
Project Member
Thu 28 Jun 2018 12:42:56 PM UTC, comment #64: 

Pushed C++ functions for conversion between UTF-8 strings and wide character strings here:
http://hg.savannah.gnu.org/hgweb/octave/rev/c63f67d87b4a

Markus Mützel <mmuetzel>
Project Member
Wed 27 Jun 2018 07:04:06 PM UTC, comment #63: 

Basic support for non-ASCII characters in folders added here:
http://hg.savannah.gnu.org/hgweb/octave/rev/7fb40efda31f

Markus Mützel <mmuetzel>
Project Member
Wed 27 Jun 2018 06:58:58 PM UTC, comment #62: 

The patch in this bug grew enormous over time. I realize that rather than having one patch to rule them all, it might be better to split it up into smaller chunks and push over the next weeks.
I hope that also makes reviewing the changes easier.

I started with the first part that adds the "core" conversion functions between UTF-8 and wchar_t needed for the rest here:
http://hg.savannah.gnu.org/hgweb/octave/rev/7335d44f34b4

Markus Mützel <mmuetzel>
Project Member
Thu 01 Feb 2018 05:20:54 PM UTC, comment #61: 

While working on another bug I noticed that it is not possible to dynamically link libraries (.oct file functions) from directories with non-ASCII characters. The attached patch fixes that, too.

(file #43152)

Markus Mützel <mmuetzel>
Project Member
Sun 21 Jan 2018 04:27:21 PM UTC, comment #60: 

I wasn't taking into account that the length given from u8_conv_to_encoding was in bytes not sizeof (wchar_t). I am surprised that the GUI started at all with this massive out of bounds error.
The attached patch fixes this error.

(file #42999)

Markus Mützel <mmuetzel>
Project Member
Sun 21 Jan 2018 12:31:25 PM UTC, comment #59: 

Something is odd with the last patch. The GUI starts fine with it applied. But the CLI crashes on start. (Both on Windows 10.)
I will have a look at it again.

Markus Mützel <mmuetzel>
Project Member
Sat 20 Jan 2018 04:57:48 PM UTC, comment #58: 

I removed some of the C++ "wrappers" from "lo-sysdep" and moved the conversions and function calls inside the C wrappers.
Is it ok for the C conversion functions "u8_to_wchar" and "u8_from_wchar" to reside in "uniconv-wrappers.c"?
Please, let me know if that or anything else needs to change.

(file #42986)

Markus Mützel <mmuetzel>
Project Member
Wed 17 Jan 2018 10:27:12 PM UTC, comment #57: 

If it is preferred to move the #if defined (OCTAVE_USE_WINDOWS_API) conditionals to the functions in the "wrappers" folder I could definitely move them there. I was under the (false) impression that was not desired.
There is no hard reason why the conversion functions act on strings rather than char arrays.

Markus Mützel <mmuetzel>
Project Member
Wed 17 Jan 2018 10:06:13 PM UTC, comment #56: 

There's no need to create octave::sys::octave_mkdir since we already have octave::sys::mkdir that also accepts a std::string argument.

The existing octave::sys::mkdir function does this:

... = octave_mkdir_wrapper (name.c_str (), mode);

Could we just modify our octave_mkdir_wrapper to handle calling the proper function for wide characters?  If not, then I would just fix this octave::sys::mkdir to call the appropriate function.

To handle the file name argument, your patch was doing this instead

_wmkdir (u8_to_wstring (name).c_str ());

Does the conversion have to happen at the std::string level?  Is it not possible to apply the conversion function to the result of the name.c_str () function?

John W. Eaton <jwe>
Project Administrator
Wed 17 Jan 2018 09:48:51 PM UTC, comment #55: 

Sorry it looks like I misunderstood what you were asking for. And I am afraid I still don't understand.

To stay with the mkdir example: I didn't find a gnulib function which would handle the wide character API automatically. That doesn't necessarily mean that it doesn't exist. I just couldn't find it.
I thought I did "the next best thing" and created a wrapper function that does that for us. That wrapper is octave::sys::octave_mkdir. Should this be done differently?

Since there is no guarantee for wchar_t's size, it can change in the future. At the moment Windows uses UTF-16 encoded strings. But that might change in the future. (In fact, MS might consider using UTF-32 to ultimately achieve the original goal of "one wchar_t per character".) I figured the best thing would be to use gnulib's "u8_conv_to_encoding" which is able to convert from UTF-8 encoded char's to "wchar_t" (whatever its size is or will be). If anything should change we could rely on gnulib to do the right thing.
I created "u8_to_wstring" and "u8_from_wstring" just to make it easier to use gnulib's "u8_conv_to_encoding" and "u8_conv_from_encoding".

Markus Mützel <mmuetzel>
Project Member
Wed 17 Jan 2018 07:08:36 PM UTC, comment #54: 

Maybe I wasn't clear about what I wanted, but the current patch doesn't do what I meant to suggest.

For example, we already have octave::sys::mkdir in the file-ops.{h,cc}.  We don't  need to add octave:sys::octave_mkdir with the same signature in lo-sysdep.cc.  Ideally, the details of calling _wmkdir or whatever should be handled in gnulib and we would just use a wrapper around the gnulib function.  If that's not possible, then the next best thing would be to fix our wrapper function to do the right thing.  If that's not possible, then we could consider doing that job in the existing octave::sys::mkdir function.

Also, do we need to define our own unicode conversion function?  Doesn't gnulib have functions to do this job?

John W. Eaton <jwe>
Project Administrator
Wed 18 Oct 2017 11:18:33 AM UTC, comment #53: 

@jwe: the patch now fulfills the requirements you had in comment #44. Do you think it can be pushed now? As a french user, I can say that being able to operate on files or folders with non ascii characters is very important.

Pantxo Diribarne <pantxo>
Project Member
Mon 16 Oct 2017 06:15:04 PM UTC, comment #52: 

The attached version of the patch also adds support for opening a pipe with non-ASCII characters on Windows (to fix bug #52211).

(file #42167)

Markus Mützel <mmuetzel>
Project Member
Sat 26 Aug 2017 07:48:42 PM UTC, comment #51: 

Thank you for testing, Avinoam.

The attached patch also adds support for opening files with non-ASCII characters in functions that use std::ifstream. That includes dlmread (bug #51836).
I haven't found a portable solution for writing to a new file with std::ofstream (other than using the C++17 std::filesystem::path which is very new or Boost).

(file #41670)

Markus Mützel <mmuetzel>
Project Member
Tue 22 Aug 2017 01:29:14 PM UTC, comment #50: 

I have checked this patch on Win-10 machine with folder that contains Hebrew letters.
In 4.2.1, when trying to change directory to this directory, the result is "parse error", on 4.3.0+ with the patch it works fine.

Thanks!

Avinoam Kalma <avinoam>
Project Member
Thu 03 Aug 2017 02:04:50 PM UTC, comment #49: 

The last patch no longer applies again. The attached new patch also fixes a failing test with the last one.

(file #41401)

Markus Mützel <mmuetzel>
Project Member
Wed 19 Jul 2017 09:11:03 PM UTC, comment #48: 

Upps. :-)
Next attempt.

(file #41251)

Markus Mützel <mmuetzel>
Project Member
Wed 19 Jul 2017 09:04:12 PM UTC, comment #47: 

I don't see a new patch :-)

Maybe the patch could be in mxe-octave?

Philip Nienhuis <philipnienhuis>
Project Member
Wed 19 Jul 2017 06:09:14 PM UTC, comment #46: 

The patch in file #39829 no longer applies to a current tip. Attached is a refreshed patch that also adds support for NTFS variants that do not store short (8.3) filenames (like my new laptop with pre-installed Win10).

Is there any interest in taking this patch that only helps the user experience on Windows?
There are no new #if defined (OCTAVE_USE_WINDOWS_API) conditionals (other than in sysdep.cc and lo-sysdep.cc).

Markus Mützel <mmuetzel>
Project Member
Sat 25 Feb 2017 01:49:36 PM UTC, comment #45: 

As far as I can tell, gnulib won't use the wide character API on Windows making it impossible (or very un-reliable) to access files and folder with non-ASCII characters. One would have to convert the internal UTF-8 to the locale specific code page which would loose characters that cannot be represented in that codepage.
Using boost::Filesystem [1] could help since it seems to use the wide character API internally. But that would add a huge new dependency to Octave and need a (major?) re-write of the functions handling the file system access. Would that be acceptable and preferred?

Meanwhile, I found that uniconv accepts "wchar_t" as a code page string. This helps a lot when it comes to handling the different sizes of wchar_t on different platforms and its corresponding encoding (UTF-16 vs UTF-32). And hence eliminates the use of the Windows specific conversion functions.
I also moved all system dependent functions to the sysdep files.
Please, see the attached patch.

[1]: http://www.boost.org/doc/libs/1_42_0/libs/filesystem/doc/index.htm

(file #39829)

Markus Mützel <mmuetzel>
Project Member
Tue 31 Jan 2017 07:29:41 PM UTC, comment #44: 

I'd prefer to find a way to remove #if defined (OCTAVE_USE_WINDOWS_API) conditionals, not add more.

We already have system-dependent functions in gnulib and sysdep files in liboctave and libinterp.  I think we should just add system-dependent functions there if they are needed.

John W. Eaton <jwe>
Project Administrator
Mon 30 Jan 2017 10:40:46 PM UTC, comment #43: 

With the attached patch Octave also no longer crashes in Windows when calling dir in a directory with non-ASCII characters.

It also no longer needs a newer compiler since I went back to using the conversion function supplied by the WinAPI. I wrapped those calls in functions in a new header oct-winapi.h which must only be included for the Windows API. Is this conceptionally OK?

(file #39620)

Markus Mützel <mmuetzel>
Project Member
Tue 06 Dec 2016 08:15:45 PM UTC, comment #42: 

Attached is a patch in which I got rid of the pragma conditionals in the header files. I also added wrappers so that these conditionals are no longer needed in load-save.cc and the parser.

The conversion between byte strings (UTF-8, what Octave uses) and wstrings (UTF-16, what the Windows filesystem uses) is done in u8_to_wstring and wstring_to_u8 which converts between std::string and std::wstring with the respective encodings.

However, it now only compiles with gcc 5 or newer because it uses the C++11 function std::wstring_convert. I tried with gcc 5.4.0 in mxe-octave, and it works as expected for me on Windows 10.

I am still not sure which conditionals should be used. I stuck with OCTAVE_USE_WINDOWS_API. But I have also seen OCTAVE_HAVE_WINDOWS_FILESYSTEM and ! OCTAVE_HAVE_POSIX_FILESYSTEM in other places. I do not know the build system well enough to judge which should be used here...

(file #39166)

Markus Mützel <mmuetzel>
Project Member
Tue 29 Nov 2016 09:46:53 PM UTC, comment #41: 

No.  The version of gcc in mxe-octave was tied up with the 4.2.0 release and wanting to be a bit conservative.  I'm sure the next major version of Octave (4.4.0) will use a newer version of gcc.

Rik <rik5>
Project Administrator
Tue 29 Nov 2016 08:42:09 PM UTC, comment #40: 

"std::codecvt_utf8_utf16" and "std::wstring_convert" require at least version 5 of gcc. At the moment, we use gcc 4.9.4 in mxe-octave. Is there a reason why we don't use a newer version?

Markus Mützel <mmuetzel>
Project Member
Wed 23 Nov 2016 05:12:09 PM UTC, comment #39: 

Thank you very much for the feedback, Mike.

It was quite straight forward removing the C wrappers and introducing the abstaction layer for fopen (see attached patch).
However, I am at a loss where the functionality for getting a UTF-16 wchar_t from an Octave string should go. It would be nice to have something like "wc_str" that would return a pointer to a wchar_t string corresponding to the multi-byte string in std::string (like "c_str" does directly for char). Is that what you meant? Where should something like that go? In the octave_char_matrix class?
For the 8.3 paths: Is there already a class in Octave that is used for representing file paths?

The conditionals in the file-ops header (still present) won't be needed anymore when these two abstractions are implemented, I think.

I found that there is std::codecvt_utf8_utf16 in C++11. Thus, there is probably an easy way to not rely on the WinAPI for the conversion between UTF-8 (char, string) and UTF-16 (wchar_t, wstring). Is it OK to use features that are not present in earlier versions of Standard C++?

(file #39058)

Markus Mützel <mmuetzel>
Project Member
Thu 17 Nov 2016 04:38:41 AM UTC, comment #38: 

Retagging to 4.2.0 release.

I had a look at the patch, and even though I can't test or build with this, I suppose if this is the only way to get wide character file name support in Windows this is a reasonable approach. I think it might be cleaner if we could more clearly define the layer between the type that Octave uses for strings (now a std::string, but in the future it may become a std::wstring) from what the OS (or other libraries) expect to be given to represent the name of a file/directory/resource.

I don't think you need to add new wrapper functions for these Windows library calls in the liboctave/wrappers directory. The purpose of that directory is to encapsulate the workarounds between glibc and gnulib into a small C API that Octave's C++ libraries can use unconditionally. If you're going to make the code elsewhere dependent on whether it's Windows or not, and whether the function takes a char or a wchar_t, then you might as well just call the Windows function directly. For exmaple, instead of defining a new function octave_wmkdir_wrapper, just use an #if in place and call _wmkdir directly.

It would be nice if the workarounds that have to be done for HDF5 could be abstracted away a little more so the load-save.cc module doesn't have to know details about which OS it is running on, just about how to get a safe file name from an Octave string type.

Same thing with the parser, maybe we can add an fopen wrapper to liboctave/system/file-ops.cc to encapsulate calling either std::fopen or _wfopen.

Definitely avoid putting conditionals into public header files, the OCTAVE_USE_WINDOWS_API can't be put into the file-ops.h header file.

Mike Miller <mtmiller>
Project Administrator
Sun 13 Nov 2016 01:48:35 PM UTC, comment #37: 

Attached is a patch that should be working a lto better. The issues mentioned in comment #35 no longer occur.
It uses the wide character API in Windows where possible which enables the access to directories with non-ASCII characters. I had to resort to short 8.3 filenames for the H5F part because there is no wide character interface to it.

I am not really sure where the boundaries of Octave are. Please, let me know if something needs to be changed. Maybe move the conversion to wide characters (and back) into the wrappers?

This might also solve part of bug #42036 (directories with non-ASCII characters). CC'ing Tatsuro because he seems to have worked on that bug, too.

(file #38945)

Markus Mützel <mmuetzel>
Project Member
Mon 07 Nov 2016 07:14:27 PM UTC, comment #36: 

This bug won't stop the 4.2 release, but it is going to be a hard one to fix.  Usually one wants to convert encoding, if necessary, at the boundaries of the application and then use whatever encoding was chosen internally in a consistent manner.  But Octave's insides and outsides are all jumbled so that there are places in load/save, in the GUI, in the LOADPATH, etc. where the encoding matters.

Rik <rik5>
Project Administrator
Mon 07 Nov 2016 06:26:31 PM UTC, comment #35: 

There is still a lot more to do. Even if it is now possible to change to a directory with Umlauts, there are still problems executing files in there. Executing the un-modified file works. However, when I modify and save it in the editor, I get:

HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 0:
  #000: /yyy/mxe-octave/tmp-hdf5/hdf5-1.8.14/src/H5F.c line 439 in H5Fis_hdf5(): unable open file
    major: File accessibilty
    minor: Not an HDF5 file
  #001: /yyy/mxe-octave/tmp-hdf5/hdf5-1.8.14/src/H5Fint.c line 554 in H5F_is_hdf5(): unable to open file
    major: Low-level I/O
    minor: Unable to initialize object
  #002: /yyy/mxe-octave/tmp-hdf5/hdf5-1.8.14/src/H5FD.c line 992 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #003: /yyy/mxe-octave/tmp-hdf5/hdf5-1.8.14/src/H5FDsec2.c line 343 in H5FD_sec2_open(): unable to open file: name = 'D:/SVN/Octave/��ml����t/testing_utf8_2.m', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
    major: File accessibilty
    minor: Unable to open file

Where yyy is the path to the mxe-repo in Linux. The folder I was testing with was "D:/SVN/Octave/Ümläüt".

The change in the file is ignored (the old file is executed).
After I do a "path(path)" which normally helps when Octave does not pick up changed files, I get a dialog with:
"The file D:/SVN/Octave/Ümläüt/testing_utf8_2.m is shadowed by a file with the same name in the load path. To run or debug the function you are editing, change to the directory D:/SVN/Octave/Ümläüt."
When I click on "Change Directory", the same dialog pops up again.
Oddly, this happened after I ran the test file from the command window.

Additionally, the path is not displayed correctly in the "Current Directory:" dropdown list.

I guess the results from "dir" have to be converted from UTF-16 to UTF-8 as well?

Seeing all this, I'd like to propose that we don't block 4.2 on this issue (even if it is marked as a regression). Working with non-ASCII characters on Windows never really worked in Octave. Some aspects improved, some aspects got worse since the last release. But I think we are on the right track with accepting patches which move us towards "UTF-8 everywhere" even on Win32.

Having said that, I would like to continue working on that bug if someone could give me some help.

Markus Mützel <mmuetzel>
Project Member
Mon 07 Nov 2016 05:31:16 PM UTC, comment #34: 

Attached is a work in progress patch which enables changing to a directory with non-ASCII chars in Windows for me (feature A from comment #32).

I think this was working for Umlaut characters before file #38798 (the "cstrings-patch") was applied because our system codepage coincidentially was latin-1 and we only tried with Latin-1 characters.

The patch converts the UTF-8 encoded path to UTF-16 and calls the wchar_t variant of chdir (_wchdir) for Windows.

This is still "work in progress" because I definitely have the #if-conditions wrong. Can someone please help me figure out which constants should be checked (UNICODE, OCTAVE_USE_WINDOWS_API, ...)?
And more importantly: Is this the right way to do it? Most probably "mkdir", "rmdir" and other functions that access the file system would need special treatment for Win32 as well...

(file #38898)

Markus Mützel <mmuetzel>
Project Member
Wed 02 Nov 2016 06:50:24 PM UTC, comment #33: 

small addition to comment #32:
The feature BCD only works in Win7 with Octave 4.2.0-rc3 if you change the editor SETTING for text encoding to UTF-8. The default value here is "System" which lets the features BCD still fail.

side question: Would it be a good idea to change the default value of the GUI editor's text encoding to utf8? (Already existing m-files would still be opened in their existing text encoding. This would just be a change for newly created m-files.) This change might even be useful for linux octave, (maybe even necessary, I have never tested in linux with a different system setting than utf8.)

Hartmut <hardy>
Tue 01 Nov 2016 09:17:33 PM UTC, comment #32: 

I have done some (even more extensive) testing. (Maybe I should have done this before, but I wasn't expecting those side effects of my patch).

I have tested following FEATURES:

  • A) run a m-file script in a folder with an umlaut in its name
  • B) get the correct return value from inputdlg when typing an umlaut into its window
  • C) displaying the correct "question" text string in the inputdlg window, when this string contains an umlaut
  • D) display a string array correctly in the GUI variable explorer, if this string array contains an umlaut. Display at least some other UTF8-characters properly as well.

The features B+C+D only seem to work all or none. Feature A works or fails independently.

Here are my TEST RESULTS (all with qt4, Windows = Win7 and WinXP):

  • Octave 4.0.3 release:
    • Linux: A works, BCD works
    • Windows: A works, BCD fails
  • Octave 4.2.0-rc2:
    • Linux: A fails, BCD fails
    • Windows: A works, BCD fails
  • Octave 4.2.0-rc3:
    • Linux: A works, BCD works
    • Windows: A fails, BCD works
  • A have strong evidence that only applying the "cstrings-patch" (xx) to rc2
    • with Linux: swaps ABCD from "fails" to "works"
    • with Windows: swaps A from "works" to "fails", but also swaps BCD from "fails" to "works"
  • I have tested the "toLocal8BitPatch" (file #38861) and it does NOT help to make A work under Windows.

My CONCLUSION: So with the current rc3 the only failure is feature A under Windows. But none of the already discussed (text encoding related) patches helps. And undoing the "cstrings-patch" (comment #19) only for Windows would destroy the features BCD under Windows. (What is more important to have under Windows A or BCD?)
But this current (rc3) failure of feature A under Windows is still a regression, it used to work under Octave 4.0.3.

Does this (failing A = running scripts in umlaut folders) also fails for mxe-octave 4.2.0-rc3 built with the newly introduced qt5 option? Do those qt5 mxe-builts support features BCD?

Hartmut <hardy>
Tue 01 Nov 2016 12:34:18 PM UTC, comment #31: 

Ooops. We have improved this behavior under Linux (qt4) BUT instead we have now created the very same problems under Windows (where this aspect was fine with rc2 before).

I have tested Octave-4.2.0-rc3:

  • Under Linux (self built from official release tar-ball with qt4) both mentioned behaviors of this bug report are FINE now: I can run m-files in folders with umlaut. And I can type umlauts in a dialog window of inputdlg, they are afterwards returned and displayed correctly.
  • But on Win 7 things got worse with our changes between rc2 and rc3: The return value of inputdlg is NOT correct now if I type umlauts in this dialog windows. And I canNOT run m-files in folders with umlaut names any more.

So our changes between Octave 4.2.0-rc2 and rc3 seem to just have pushed this problem over from linux octave to windows octave :(

What do we do about this? Should we introduce even more compiler conditionals like "if Win32 then use file #38294"? But this patch was also relevant for bug #49386, and for those issues over there (mostly umlauts in Octave created windows) this patch still DOES help. Or shall we revert those changes because in linux there is at least an alternative (qt5) that is Windows does not yet exists?

Hartmut <hardy>
Thu 27 Oct 2016 05:53:36 PM UTC, comment #30: 

I've just compiled the current stable branch. And the original issue is now fixed for me on linux (I have only ever observed this specific issue on linux. Nevertheless this patch also improves the behavior of mxe-octave with non-english charcters on Windows as well.)

I suggest to close this bug report now as fixed. The discussion about improvements to the qt5 version of mxe-octave could be continued elsewhere (with an appropriate title). Other issues with non-english characters are dealt with in several seperate bug reports.

Hartmut <hardy>
Thu 27 Oct 2016 05:16:30 PM UTC, comment #29: 

I pushed your change to stable:

http://hg.savannah.gnu.org/hgweb/octave/rev/f88de8f27259

Can this bug report be closed as fixed?

John W. Eaton <jwe>
Project Administrator
Wed 26 Oct 2016 07:48:44 PM UTC, comment #28: 

I think that comment #1 and Rik's comment #8 are saying that linux builts with qt5 do NOT have this problem.

My tiny patch fully solves the original issue of this bug report (linux builts with qt4). Maybe the discussion of further improvements to qt5 builts in mxe-octave could be continued in a seperate bug report.

Hartmut <hardy>
Wed 26 Oct 2016 07:34:37 PM UTC, comment #27: 

Does this problem also happen with Qt5?

Your patch seems fine to me if it fixes the problem for Qt4.

John W. Eaton <jwe>
Project Administrator
Sun 23 Oct 2016 09:33:08 PM UTC, comment #26: 

I have compiled Octave with my proposed changes from comment #25 (change all QString::toLatin1 into QString::toLocal8Bit) and it does NOT help (with qt 4 under Linux). It seems that those encodings don't take effect "deep" enough in the GUI.

But I tried an even simpler patch, and it DOES help! If I re-add the line with "setCodecForCStrings" in octave-gui.cc for (only in qt4 builds), then all works fine:

  • running m-scripts in folders with Umlauts works fine from the GUI editor
  • using the inputdlg command to receive non-english characters, also works fine.

I will attach a tiny PATCH to show what I only did to solve all the observed behaviors reported in this bug report. I have tested this patch only for qt4 builds under Linux, so far. But I would nevertheless consider this patch very "low risk": It only changes the code for qt4 builts (qt5 builts will not change at all). And it only re-adds a single line of code, that was present in all former (GUI) releases as well.

If noone else finds any problems with this patch, then I would be happy to see it in the upcomaing Octave 4.2 release. It would make using the GUI a significant bit nicer for all non-english Octave users.

(file #38798)

Hartmut <hardy>
Sun 23 Oct 2016 01:01:45 PM UTC, comment #25: 

I have searched the Qt documentation a bit. Those are my sources:

  • For our GUI text encoding before cset 2fd4f1a3f4a8 (works only in qt4):
  • For our GUI text encoding after cset 2fd4f1a3f4a8 (works in qt4 and qt5):
    • http://doc.qt.io/qt-5/qtextcodec.html#setCodecForLocale
    • http://doc.qt.io/qt-5/qstring.html#toLatin1

      Returns a Latin-1 representation of the string as a QByteArray.
      The returned byte array is undefined if the string contains non-Latin1 characters. Those characters may be suppressed or replaced with a question mark.

    • http://doc.qt.io/qt-5/qstring.html#toLocal8Bit

      Returns the local 8-bit representation of the string as a QByteArray. The returned byte array is undefined if the string contains characters not supported by the local 8-bit encoding.
      QTextCodec::codecForLocale() is used to perform the conversion from Unicode. If the locale encoding could not be determined, this function does the same as toLatin1().
      If this string contains any characters that cannot be encoded in the locale, the returned byte array is undefined. Those characters may be suppressed or replaced by another.

If I understand this correctly, then our former combination like this

QTextCodec::setCodecForCStrings (QTextCodec::codecForName ("UTF-8"));
QByteArray text = _edit_area->text ().toAscii ();

was converting texts to a 8bit representation, according to the utf8 encoding.

Our new combination like this

QTextCodec::setCodecForLocale (QTextCodec::codecForName ("UTF-8"));
QByteArray text = _edit_area->text ().toLatin1 ();

is converting text to the latin1 8bit repesentation. This is not the same as the previous behavior. (If no locale is set, then this is a good replacement as mentioned in the qt4-qt5 transition document, but we had a locale set.)

Could the following be a better way to change the code to something that runs under qt4 and qt5?

QTextCodec::setCodecForLocale (QTextCodec::codecForName ("UTF-8"));
QByteArray text = _edit_area->text ().toLocal8Bit ();

I would technically be able to prepare a new cset for this, but I am hesitating to do such changes very deep in core Octave, because I have never used the qt libraries before. The files that might need a change like this, are Vt102Emulation.cpp, file-editor-tab.cc and workspace-model.cc.

Hartmut <hardy>
Sat 08 Oct 2016 02:23:24 PM UTC, comment #24: 

Related to earlier posts.  I did a complete clean and re-build of the MXE-octave version with Qt5 enabled.  There are no weird segfaults due to having both qt4 and qt5 libraries present.

It doesn't appear to solve this issue because I can't generate an umlaut character in the Command Window.  I can take an umlaut from another source, such as a web page or the notepad editor, and paste it into the Editor window.  When I try the same thing with the Command window the umlaut character is removed and the rest of the text is pasted correctly.

Rik <rik5>
Project Administrator
Sat 01 Oct 2016 02:47:32 PM UTC, comment #23: 

I'll have to let jwe take a look.  I do see a few instances where toAscii has been replaced by toLatin1.  Maybe that makes a difference.

-  QTextCodec *codec = QTextCodec::codecForName (_encoding.toAscii ());

+  QTextCodec *codec = QTextCodec::codecForName (_encoding.toLatin1 ());

Also, what strings are subject to UFT-8 may have changed.

 #if ! defined (Q_OS_WIN32)

-    QTextCodec::setCodecForCStrings (QTextCodec::codecForName ("UTF-8"));

+    QTextCodec::setCodecForLocale (QTextCodec::codecForName ("UTF-8"));

#endif

Rik <rik5>
Project Administrator
Sat 01 Oct 2016 11:53:06 AM UTC, comment #22: 

I've found the changeset that makes this happen:

http://hg.savannah.gnu.org/hgweb/octave/rev/2fd4f1a3f4a8
"allow building with Qt5 (bug #40252)" by jwe.

Using the repo version before this changeset makes both bugs (umlaut folders in GUI editor and inputdldg with umlauts) disapper in my Linux built (with qt4). Applying this single changeset makes both discussed bugs appear.

Hartmut <hardy>
Fri 30 Sep 2016 11:29:42 PM UTC, comment #21: 

See the attached file 'myconfig'.  It is written a tcsh script, but you can easily adapt it to bash or whatever.

For ccache, install that using your package manager.  Once that is done, add the directory '/usr/lib/ccache' to the start of your path.  The line from my .tcshrc file, for example, is

   set path = (. /usr/lib/ccache ~/local/bin $path ~/scripts)

After that, initialize the cache size with something like this which sets a 10 Gigabyte cache.

ccache -M 10G

Once that is done, go to the directory where you build, run 'myconfig' followed by 'make -jXX' where XX is the number of CPUS or sometimes CPUS+1.

You can verify that it is working by running

ccache -s

to see the cache statistics

(file #38634)

Rik <rik5>
Project Administrator
Fri 30 Sep 2016 09:58:51 PM UTC, comment #20: 

My builds take more than one hour currently. Could you copy-paste me some sample commands of "configure" and "make" with the options you normally use to speed things up? Especially those Cxx flags, I have never used before. Neither pipe nor cchache. (I'm still during the first build process, I needed to restart because of bug #49039, I will now re-apply the corresponding patch each time.)

Hartmut <hardy>
Fri 30 Sep 2016 09:45:30 PM UTC, comment #19: 

You might want to use configure to create a minimal octave build since you will be doing a full build quite frequently.

I would at least use '--disable-docs' to save some time.  Other things I would personally add include '--disable-jit' and '--disable-java'.

The compilation stage will go faster if you don't use optimization so you might want to explicitly set CFLAGS and CXXFLAGS to have -O0.  I also add '-pipe' so that you rely on memory for intermediate files rather than a slow disk.

I also use ccache which helps quite a lot.  If I'm just re-building an incremental difference it usually only takes about 2 minutes with ccache.

Rik <rik5>
Project Administrator
Fri 30 Sep 2016 09:14:23 PM UTC, comment #18: 

I have re-tested Octave-4.2.0-rc2 (from alpha.gnu.org, Qt4 if I understood this right) under Win7:

  • The original bug (running scripts in folders with umlaut) works FINE there.
  • Pantxo's inputdlg bug from comment #3 works FAILs.

So there really IS something to gain with changeing mxe Octave to Qt5. But we are now talking about two different bugs, to be precise.

I will start to "manually" bisect the Octave code between 4.0.3 and 4.2.0-rc2 (on my Ubuntu 14.04 machine with Qt4), and test for the two different mentioned bugs under Linux. This might take some days...

Hartmut <hardy>
Fri 30 Sep 2016 07:45:27 PM UTC, comment #17: 

I tried "hg log -b default -l 10 -k encoding -M libgui/" but all of those are related to the editor or Windows specific.

Pantxo Diribarne <pantxo>
Project Member
Fri 30 Sep 2016 07:13:11 PM UTC, comment #16: 

Are we talking about moving the mxe build to Qt5 prior to 4.2 release?
If so, considering the "build transparent" changes Qt brought between 4 and 5 (see e.g. bug #49053 on mac) and that only recently did we see Octave running with Qt5, I'd vote for accepting this regression and keep Qt4 stable build until 4.4 (or the 5??hours bisect but I am not proposing my self here :-)).

PS: I am very grateful to people who contributed the recent effort to be able to build Octave with Qt5.

Pantxo Diribarne <pantxo>
Project Member
Fri 30 Sep 2016 07:12:07 PM UTC, comment #15: 

I dont' have any objections against moving to Qt5.

But I don't see how changing something with MXE Octave -which is ment to generate binaries for WINDOWS OS- would help for this bug. The current Windows installer (4.2.0-rc2 from alpha.gnu.org) seems to already be fine and doesn't even show this bug.

This regression bug was observed under LINUX, and changing MXE Octave wouldn't help with the Linux binaries, would it?

So even after moving MXE Octave to Qt5, this bug will persist with Linux binaries that are built with Qt4. (Linux binaries with Qt5 are fine, already now.)

I seem to be missing something here.

Maybe here is our difference in thinking:

  • Rik has observed (in comment #8) that the inputdlg test from commmnt #3 FAILS with an MXE-Octave and Qt4.
  • I have observed (in comment #7) that the original problem (GUI editor not running scripts in folders with umlaut) works FINE with the precompiled installer of MXE-Octave 4.2.0-rc2.

Is there a difference between the original bug (folder name) and the indputdlg test of commment #3? Is there a difference between Rik's self compiled MXE Octave and the "official" rc2 installer?

Hartmut <hardy>
Fri 30 Sep 2016 06:29:22 PM UTC, comment #14: 

A bisection is a possibility, but it will be quite slow.  There are a lot of changesets between 4.0.3 and 4.2.0-rc2 and the test is interactive so it can't be automated to have bisect run on its own.  It's probably 5 hours of effort.

On the other hand, Qt4.8, the last version of the Qt4 branch, was released in 2011.  It's worth moving to Qt5 in any case, and it would fix this bug.

Rik <rik5>
Project Administrator
Fri 30 Sep 2016 05:58:07 PM UTC, comment #13: 

Since

  • my Octave 4.0.3 does not show this bug,
  • but Octave 4.2.0-rc2 does, and
  • both my Linux builts are built the same way (with qt4 when I understand this right)
  • and assuming that the qt4 libraries have not changed and not freshly introduced this bug

then I would like to ask: Isn't it very probably that we changed our Octave code somewhere between version 4.0.3 and 4.2.0-rc2 and introduced this behavior? Can we fix this by going back and look where it changed? Since it happens under Linux OS, and there is also a simple test script in comment #3, would a code bisection help here? (I don't have elaborated enough coding skills to do something like a bisection myself.)

Also, I thought this is a "Linux only" bug, at least does the official Windows installer 4.2.0-rc2 not show this behavior. So why would the introduction of a "qt5" option to MXE Octave be of any here in this case?

I do not want to be destructive or demotivating at all. I would just like to find a feasable way to remove this regression in the Octave Linux GUI with qt4. Or have I missed any relevant facts here?

Hartmut <hardy>
Fri 30 Sep 2016 10:10:48 AM UTC, comment #12: 

Yeah, I would suggest a clean build if changing from qt4 to qt5 to ensure everything uses qt5.

John Donoghue <lostbard>
Project Member
Fri 30 Sep 2016 03:45:16 AM UTC, comment #11: 

The Octave that was built installs and I'm able to run it.  There are a few issues.  When I try to switch to the Editor tab I get a segfault.  Also, when I try to open the Preferences menu I get a segfault where it tells me that the following module is at fault.

AppName: octave-gui.exe         AppVer: 0.0.0.0         ModName: qtcore4.dll
ModVer: 4.8.7.0         Offset: 000cf3a1

So it seems that some Qt4 components are still being linked in.  I suspect if they are all removed then it will work correctly.

Rik <rik5>
Project Administrator
Thu 29 Sep 2016 11:08:30 PM UTC, comment #10: 

I'm trying a self-built MXE with the new Qt5 option.  I'll let you know how things go in a few hours.

Rik <rik5>
Project Administrator
Thu 29 Sep 2016 07:25:07 PM UTC, comment #9: 

I pushed an initial changeset to mxe-octave for building a default-octave target that builds qt5 if --enable-qt5 is specified in the configure options.

John Donoghue <lostbard>
Project Member
Wed 28 Sep 2016 05:38:50 AM UTC, comment #8: 

I can confirm that a self-built version of Octave on Linux using Qt5 passes the inputdlg test in comment #3.  A self-built MXE version fails on Windows, but it is built with Qt4.  I rebuilt Octave on Linux using the configure option '--with-qt=4'.  This version now fails.

I'm adding jwe and JohnD to the CC list for this bug.  It seems like it would be useful to build the MXE version with Qt5.

Rik <rik5>
Project Administrator
Fri 23 Sep 2016 11:31:24 AM UTC, comment #7: 

I do NOT observe this bug in the current Windows pre-release 4.2.0-rc2 (official exe-installer from alpha.gnu.org). Is this maybe built with Qt5?
But I still DO observe this bug, with my (self compiled on Ubuntu 14.04) version of Octave 4.2.0-rc2 under Linux.

Hartmut <hardy>
Tue 20 Sep 2016 07:16:58 AM UTC, comment #6: 

@Dmitry: The Octave build from my original post was done with qt4, not with qt5. So this is compatible with your observations in comment #1.

Hartmut <hardy>
Tue 20 Sep 2016 06:57:36 AM UTC, comment #5: 

This bug has several distinct differences to the older bug #42036:

  • This new bug happens on linux systems (and probably as well on Windows systems), but bug 42036 was only present on Windows systems.
  • This new bug was not yet present in Octave 4.0.3 (neither Linux nor Windows), but I guess 42036 was. So this new bug seems to me to be a regression.
  • This new bug only concerns the GUI elements of Octave, where as 42036 was also present in the command line box of the GUI, as well as in the CLI-version of Octave.
Hartmut <hardy>
Mon 19 Sep 2016 11:07:46 PM UTC, comment #4: 

Is this the same bug as #42036?

Rik <rik5>
Project Administrator
Sat 17 Sep 2016 09:42:01 PM UTC, comment #3: 

end of the previous post:

inputdlg ("Non Ascii character please:");
# Type a non ascii char, e.g. "Arbeitsfläche", and press ok
# 4.0.3 -> { [1,1] = Arbeitsfläche }
# 4.2-rc1 -> { [1,1] = Arbeitsfl�che }

and the other way:

inputdlg ("Non Ascii character please:", "", 1, {"Arbeitsfläche"})
# default dialog text in 4.0.3 -> Arbeitsfläche
# default dialog text in 4.2-rc1 -> Arbeitsfläche

Pantxo Diribarne <pantxo>
Project Member
Sat 17 Sep 2016 09:39:50 PM UTC, comment #2: 

I can confirm the bug and this used to work in 4.0.3.
The folowing example using qt inputdlg shows an encoding mismatch which may be a clue:

Pantxo Diribarne <pantxo>
Project Member
Sat 17 Sep 2016 09:32:59 PM UTC, comment #1: 

openSUSE Tumbleweed: Qt5-based GUI works fine, Qt4-based GUI has problems with non-English characters in directory name.

Dmitry <rdmitry>
Sat 17 Sep 2016 08:57:00 PM UTC, original submission:  

This happes on my Ubuntu 14.04 computer (with german language settings) with a self compiled version of roughly Octave-4.2.0-rc1 (more precisely with hg id b0caa61934fe)

Here is how to reproduce the behavior:

  • save a little m-file script in a folder with an umlaut in it. I created "/home/myself/Arbeitsfläche/test_umlaut.m". My little script just contains "a=1".
  • start Octave in GUI mode
  • (1) change the working directory to this very folder
  • start the m-file script from the command line, in my case by typing "test_umlaut"
    • This runs fine.
  • (2) change the working directory back to something different
  • open the same m-file script in the GUI editor, and run it from there. For example press F5 to run the script.
    • Now the GUI will NOT change the working directory properly, even when you answer the question "Would you like to change the working directory" with yes. Instead I see an error message (text see below).
  • (3) manually change the working directory to the proper folder.
  • run the file again from within the GUI editor, for example by pressing F5.
    • I see again an error message (saying "file or folder not found"):

      error: /home/myself/Arbeitsfl�che: Datei oder Verzeichnis nicht gefunden
      error: parse error

The umlaut (ä) in the error message is also changed into a wrong character.

Note: Also the display in the top line of the GUI with the current folder name shows a wrong character for the umlaut.

In former versions like Octave 4.0.3 (also self compiled on the very same Ubuntu machine) this used to work fine. That's why I consider this to be a regression.

Hartmut <hardy>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #43152:  bug49118_win32_wchar_t_API_v11.patch added by mmuetzel (38KiB - application/octet-stream)
file #42999:  bug49118_win32_wchar_t_API_v10b.patch added by mmuetzel (36KiB - application/octet-stream)
file #42986:  bug49118_win32_wchar_t_API_v10.patch added by mmuetzel (36KiB - application/octet-stream)
file #42167:  bug49118_win32_wchar_t_API_v9.patch added by mmuetzel (37KiB - application/octet-stream)
file #41670:  bug49118_win32_wchar_t_API_v8.patch added by mmuetzel (37KiB - application/octet-stream)
file #41401:  bug49118_win32_wchar_t_API_v7.patch added by mmuetzel (31KiB - application/octet-stream)
file #41251:  bug49118_win32_wchar_t_API_v6.patch added by mmuetzel (30KiB - application/octet-stream)
file #39829:  bug49118_win32_wchar_t_API_v5.patch added by mmuetzel (27KiB - application/octet-stream)
file #39620:  bug49118_win32_wchar_t_API_v4.patch added by mmuetzel (19KiB - application/octet-stream)
file #39166:  bug49118_win32_wchar_t_API_v3.patch added by mmuetzel (10KiB - application/octet-stream)
file #39058:  bug49118_win32_wchar_t_API_v2.patch added by mmuetzel (11KiB - application/octet-stream)
file #38634:  myconfig added by rik5 (234B - application/octet-stream)

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by avinoam (Posted a comment)
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by rik5
  • -email is unavailable- added by rik5
  • -email is unavailable- added by pantxo (Posted a comment)
  • -email is unavailable- added by rdmitry (Posted a comment)
  • -email is unavailable- added by hardy (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 25 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-03-03 mtmiller Carbon-CopyRemoved 80942 => -
    2018-07-12 mmuetzel StatusReady For Test => Fixed
        Open/ClosedOpen => Closed
    2018-07-04 mmuetzel StatusIn Progress => Ready For Test
    2018-06-30 mmuetzel StatusPatch Submitted => In Progress
    2018-02-01 mmuetzel Attached File- => Added bug49118_win32_wchar_t_API_v11.patch, #43152
    2018-01-21 mmuetzel Attached File- => Added bug49118_win32_wchar_t_API_v10b.patch, #42999
    2018-01-20 mmuetzel Attached File- => Added bug49118_win32_wchar_t_API_v10.patch, #42986
    2018-01-16 mmuetzel Dependencies- => bugs #52905 is dependent
    2017-10-16 mmuetzel Attached File- => Added bug49118_win32_wchar_t_API_v9.patch, #42167
    2017-10-15 mmuetzel Dependencies- => bugs #52211 is dependent
    2017-08-26 mmuetzel Dependencies- => bugs #51836 is dependent
    2017-08-26 mmuetzel Attached File- => Added bug49118_win32_wchar_t_API_v8.patch, #41670
    2017-08-21 mmuetzel Dependencies- => bugs #40444 is dependent
    2017-08-03 mmuetzel Attached File- => Added bug49118_win32_wchar_t_API_v7.patch, #41401
        StatusConfirmed => Patch Submitted
    2017-07-19 mmuetzel Attached File- => Added bug49118_win32_wchar_t_API_v6.patch, #41251
    2017-02-25 mmuetzel Attached File- => Added bug49118_win32_wchar_t_API_v5.patch, #39829
    2017-01-31 rik5 Carbon-CopyRemoved 72865 => -
    2017-01-30 mmuetzel Attached File- => Added bug49118_win32_wchar_t_API_v4.patch, #39620
    2016-12-06 mmuetzel Attached File- => Added bug49118_win32_wchar_t_API_v3.patch, #39166
        Item GroupRegression => Incorrect Result
        Release4.2.0 => dev
        Operating SystemGNU/Linux => Microsoft Windows
    2016-12-06 mmuetzel Dependencies- => bugs #49768 is dependent

    Back to the top


    Powered by Savane 3.5