bugGNU Octave - Bugs: bug #58493, Accentued letter (french accent)...

 
 

bug #58493: Accentued letter (french accent) in file path problem and Magick++ exception

Submitter:  None
Submitted:  Wed 03 Jun 2020 05:31:55 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Fixed Assigned to:  None
Originator Name:  Loïc Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * Microsoft Windows Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 28 Jul 2020 08:14:46 AM UTC, comment #11: 

Works form me now.
Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Mon 27 Jul 2020 01:10:49 PM UTC, comment #10: 

OK to close this report as fixed?

John W. Eaton <jwe>
Group administrator
Sat 04 Jul 2020 02:21:54 PM UTC, comment #9: 

I pushed a change here that reverts reading the complete image into a blob. Instead it uses a combination of the two approaches outlined in comment #3:
https://hg.savannah.gnu.org/hgweb/octave/rev/09b60b423e6f

Marking as ready for test again.

Markus Mützel <mmuetzel>
Group administrator
Sun 21 Jun 2020 12:53:00 AM UTC, comment #8: 

@Markus: After some more reading, I agree with your assessment here, and am glad this is not on my to-do list. ;)

Andrew Janke <apjanke>
Tue 16 Jun 2020 09:15:36 PM UTC, comment #7: 

For the Windows installer, we are packaging GraphicsMagick not ImageMagick. They seem to use fopen (with char input) on all platforms.

The Windows API is a little weird when it comes to characters: There are basically to flavors of the API.
One flavor is the ANSI API that uses char. The name is misleading because that API really uses the locale encoding. (Well, it is "correctly" named if you are on a machine with US locale.)
The other one is the Unicode API using wchar_t.

In this case, it doesn't really matter how file names are stored in the file system (might as well be UTF-32 or a proprietary encoding) because the APIs should make the access transparent.

ImageMagick seems to have chosen to expect UTF-8 at there interfaces. On Windows, they convert these char arrays to wchar_t and use the Unicode API.

GraphicsMagick on the other hand has a more "native" interface. They don't do any conversion and just pass the input on to the ANSI API:
http://hg.code.sf.net/p/graphicsmagick/code/file/a42b29bea408/magick/blob.c#l2862

That really is a mess.

Good thing is that we do the packaging for Windows ourselves. So we know that we are dealing with GraphicsMagick. Otherwise, we would need to somehow detect on run-time if we were dealing with ImageMagick or GraphicsMagick, and pass the file names in the respective encoding.

Anyway, it looks like we are back to what I wrote in comment #3.

Maybe we could try both approaches I outlined there and only fail with a descriptive error if both failed.

I've also seen that "imread" fails if it is given an absolute path to a file.
But that is probably a different bug.

Markus Mützel <mmuetzel>
Group administrator
Tue 16 Jun 2020 02:04:14 PM UTC, comment #6: 

@Andrew: Thanks for the link to that forum post.
It really looks like Magick++'s file path interface is UTF-8:
https://imagemagick.org/api/MagickCore/utility-private_8h_source.html#l00128

That makes it much easier for us because "our native" encoding is also UTF-8.
In this case, the original problem might already be solved by this change on stable (which will eventually be released as Octave 6):
https://hg.savannah.gnu.org/hgweb/octave/rev/39c078e14824

So maybe, we can just revert the troublesome changeset:
https://hg.savannah.gnu.org/hgweb/octave/rev/04ee5e8694cb

I'll try that in my next cross-compile.

OT: C++20 is about to introduce a new type char8_t (additionally to char16_t and char32_t from C++11) that is exclusively for UTF-8 encoded strings. Maybe there is a little hope that library interfaces will become clearer in the future.

Markus Mützel <mmuetzel>
Group administrator
Tue 16 Jun 2020 12:38:36 AM UTC, comment #5: 

Looking at that upstream forum post a little more - it looks like Magick++ is expecting file names to be encoded in UTF-8, regardless of the locale/legacy code page it's running under on Windows. Is it possible that the current Octave code was passing a native-code-page-encoded string, which should have been transcoded to UTF-8, instead of the other way around?

Andrew Janke <apjanke>
Tue 16 Jun 2020 12:35:40 AM UTC, comment #4: 

Aren't file names in NTFS always stored as UTF-16 in the filesystem, regardless of the computer's locale settings? https://docs.microsoft.com/en-us/windows/win32/intl/character-sets-used-in-file-names (Though that doesn't help on removable media, which are still often formatted as FAT.)

This sounds like it might be an issue in Magick++ that should be reported upstream there? http://www.imagemagick.org/discourse-server/viewtopic.php?t=20795

Andrew Janke <apjanke>
Mon 15 Jun 2020 09:54:17 AM UTC, comment #3: 

Thanks for this hint.

Working around all the issues with different character encodings is a pita...

An alternative approach might be to convert the file name to the locale charset and revert to passing the file name to Magick++.
That would mean that file names are limited to characters that can be encoded in the locale charset. (So e.g. images with Cyrillic letters in the file name or path cannot be opened on a system with Western European locale.) But that limitation is probably something we might have to live with...
Or we could use "get_ASCII_filename". But that doesn't work on all file systems and might rely on having write access to any parent folder.

I'll have a look at it when I come around to it.

Markus Mützel <mmuetzel>
Group administrator
Mon 15 Jun 2020 09:19:20 AM UTC, comment #2: 

I took a look at the pushed fix:

+ verbatim+
  // [...] This is less efficient than passing
  // the file name to Magick::Image::ping because this requires to read the
  // entire file from the disk. So, do this on affected file systems only.
-verbatim-

Loading the entire file in memory can't be the solution because the file might be several GB in some fields of imaging (we routinely generate image files >500GB). To address this, imread has the option of reading single pages instead of the whole image file and this fix would defeat that completely.

Carnë Draug <carandraug>
Group Member
Sun 14 Jun 2020 11:56:28 AM UTC, comment #1: 

Thanks for your report. It also uncovered a few other issues and regressions with paths containing non-ASCII characters on Windows.
The reported issue should be fixed with this change on the default branch:
https://hg.savannah.gnu.org/hgweb/octave/rev/04ee5e8694cb

This will eventually be released as part of Octave 7 (probably in 2021).

In the meantime, you could work around this by mapping a drive letter to your user folder and access the files from there in Octave:

subst x: C:\Users\Loïc


Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Wed 03 Jun 2020 05:31:55 PM UTC, original submission:  

By running a program containing the following code:

-----------------------------------
X = imread('imagefile.bmp');
-----------------------------------

I receive the following error:

-----------------------------------
error: Magick++ exception: Magick: Unable to open file (C:\Users\Loïc\Documents\imagefile.bmp) reported
by magick/blob.c:3080 (OpenBlob)
error: called from
    _imread_ at line 80 column 10
    imageIO at line 118 column 28
    imread at line 106 column 33
    prog1 at line 3 column 8
-----------------------------------

This error is due to the umlaut in the name of the folder "Loïc", in the file path.
Indeed, the error does not appear when I launch my program from the root of my computer, before the "Loïc" folder.

So, I think it's a problem of french accent, and special characters.

Thank you guys

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-10-08 mmuetzel Dependencies- bugs #61307 is dependent
    2020-07-28 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2020-07-04 mmuetzel StatusIn Progress Ready For Test
    2020-06-15 mmuetzel StatusReady For Test In Progress
    2020-06-14 mmuetzel Item GroupMissed Error or Warning Unexpected Error or Warning
        StatusNone Ready For Test
        Release5.2.0 dev

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code