bugGNU TeXmacs - Bugs: bug #34209, Unicode characters in GUI interface


You are not allowed to post comments on this tracker with your current authentication level.

bug #34209: Unicode characters in GUI interface

Submitted by:  Joris van der Hoeven <vdhoeven>
Submitted on:  Mon 05 Sep 2011 07:11:02 AM UTC  
Category: Qt portPriority: 5 - Normal
Item Group: ErrorStatus: Confirmed
Privacy: PublicAssigned to: Miguel de Benito <mdbenito>
Originator Name: Open/Closed: Open
Release: Release: None
Fixed Release: NoneFixed Release: 

(Jump to the original submission Jump to the original submission)

Thu 02 Mar 2017 11:04:38 PM UTC, comment #7:

Well, reopening myself; I was slightly too optimistic...
There are still some instances where utf8 is misrendered in the GUI, for instance in tm-files.scm when giving informative message on the footer.
As soon as the windows version uses utf8 in urls (which is underway) this will be an easy fix.

Philippe Joyez <pjoyez>
Project Member
Wed 01 Mar 2017 08:54:45 PM UTC, comment #6:

I think most of the changes proposed by Miguel have been implemented since the last comment. I also fixed an issue with image filenames. Closing for clarity. Reopen and comment accordingly if this is still an issue.

Philippe Joyez <pjoyez>
Project Member
Tue 03 Dec 2013 10:00:41 AM UTC, comment #5:

Changes in url handling are extremely sensitive, so I would first need a minimal patch in order to study your proposal. For the moment, your patch contains various unessential changes, which makes it hard to focus on the main proposal.

Joris van der Hoeven <vdhoeven>
Project Administrator
Mon 02 Dec 2013 09:11:49 PM UTC, comment #4:

I've attached a proof-of-concept patch which adresses these issues in a more consistent way. The basic idea is to use the system's default encoding internally in the url class, then convert to and from cork (the encoding currently assumed everywhere else) whenever conversions to and from strings are required (constructors, concretize(), etc.)

To this end, we implement two functions, convert_from_filesystem_encoding() and convert_to_filesystem_encoding(), which should follow the guidelines explained in comment#3. In the patch they are little more than placeholders which assume filenames in UTF8.

Using them across url creation and conversion routines makes the use of heuristic detection unnecessary under Qt and fixes displaying of file names in menus under X11. There are several problems with the X11 version, though:

  • I'm not sure what encoding is appropriate for X11's XSetIconName, which is the function ultimately used to set the window name in x_window_rep::set_name. The man page seems to indicate that it is the "Host Portable Character Encoding", but this doesn't support most international charsets AFAIK.
  • Some WidKit widgets don't work well with the new internal representation using cork of urls converted to strings. As a result, saving a file with special chars in it (e.g. "hé.tm") creates a filename (e.g. "h%E9.tm") which under MacOS 10.9 crashes the Finder.

There are also some questions open:

  • Will these changes break support for some special encodings? For instance, the changes involve undoing Zou Hou's patch for chinese file names, so we should test with chinese again.
  • Using the encoding converter mechanism, which relies on urls to open its dictionaries, is a chicken and egg situation which results in many "convert-error, Couldn't open encoding dictionary..." upon TeXmacs startup.

(file #29761)

Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Mon 02 Dec 2013 11:33:51 AM UTC, comment #3:

This bug is not present under the Qt port thanks to François' heuristic detection of encodings.

The problem is that the encoding of file names is ignored by the url system and therefore file names are handled internally "as-is", whereas strings for widgets are expected to be in a specific encoding.

We need to convert all file names we read according to the OS / filesystem:

  • NTFS uses unicode16. Info for Windows filesystems
  • Extfs doesn't set any default encoding. Most applications use and assume utf8. QFile uses the user locale (this may be overriden)
  • MacOS uses a "decomposed" version of UTF-8 (some comments here)
Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Thu 07 Feb 2013 01:10:31 AM UTC, comment #2:

This has been "fixed" in svn r6254. Corrections are probably not robust and are only for Qt.

The reason is that encoding of file names is sometimes done as cork, sometimes as-is (utf8 in my system). It is not clear to me at which points in the url system internal encoding is expected, and at which it's the system's, so I think this is not a entirely a GUI issue.

Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Tue 04 Sep 2012 06:50:05 PM UTC, comment #1:

Note to self: The problem seems to be restricted to file names: the OS returns them as utf8 strings which TeXmacs' url system uses as-is. See SLOT_FILE in qt_tm_widget_rep.

Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Mon 05 Sep 2011 07:11:02 AM UTC, original submission:

In several cases, unicode characters are rendered incorrectly in GUI elements, such as title bars. Please check that corrections are robust and also work for the X11 version.

Joris van der Hoeven <vdhoeven>
Project Administrator


Attached Files
file #29761:  fix-unicode-in-gui-7984.patch added by mdbenito (16KiB - application/octet-stream - See comment #4)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by pjoyez (Posted a comment)
  • -unavailable- added by mdbenito (Posted a comment)
  • -unavailable- added by vdhoeven (Submitted the item)

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 02 Mar 2017 11:04:38 PM UTCpjoyezOpen/ClosedClosed=>Open
    Wed 01 Mar 2017 08:54:45 PM UTCpjoyezOpen/ClosedOpen=>Closed
    Mon 02 Dec 2013 09:11:49 PM UTCmdbenitoAttached File-=>Added fix-unicode-in-gui-7984.patch, #29761
    Tue 04 Sep 2012 06:50:05 PM UTCmdbenitoAssigned tomgubi=>mdbenito

    Back to the top

    Powered by Savane 3.1-cleanup1