bugUnifont - Bugs: bug #45538, Incorrent display of COMBINING...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #45538: Incorrent display of COMBINING ACUTE ACCENT, U+0301

Submitter:  Suntor <suntor>
Submitted:  Mon 13 Jul 2015 05:54:24 AM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  unifoundry
Open/Closed:  Closed

Jump to the original submission

Sat 01 Jul 2017 08:00:43 PM UTC, comment #14: 

Suntor,

Great, I will close this bug report since it is now working for you. After Unifont 10.0.03 has been released for a while (to be determined), I will create a Unifont 10.0.04 release that has this change and some other things in it. At that point I will no longer consider this to be experimental (unless someone else reports a different problem).


Paul

Paul Hardy <unifoundry>
Group administrator
Fri 30 Jun 2017 11:24:39 PM UTC, comment #13: 

Suntor,

Try the version in ftp://ftp.gnu.org/gnu/unifont/unifont-10.0.03/. I have modified the file src/hex2sfd and it appears to work on Debian GNU/Linux in LibreOffice. Since you also have LibreOffice available, we can at least try to get it working in the same application. Hopefully it works in other applications you use too.

I consider version 10.0.03 to be experimental for now, because altering the FontForge font generation is tricky. If there are problems, please report them and revert to version 10.0.02.

Good luck!


Paul Hardy

Paul Hardy <unifoundry>
Group administrator
Fri 30 Jun 2017 09:33:45 PM UTC, comment #12: 

I am going to look into this further, so I am leaving this bug open.

Paul Hardy <unifoundry>
Group administrator
Fri 30 Jun 2017 08:13:50 PM UTC, comment #11: 

Suntor,

I am attaching a UTF-8 version of the "xxx" sample and the C program I wrote to create it. It sends its output to standard output, so you should redirect its output to a file. The "x.txt" file that I am attaching is a UTF-8 file. You can experiment with the code.

I recommend just using UTF-8 in the future, because that is what web browsers and many other applications will expect by default if a file contains non-ASCII characters.


Paul


(file #41093, file #41094)

Paul Hardy <unifoundry>
Group administrator
Fri 30 Jun 2017 07:55:42 PM UTC, comment #10: 

Suntor,

The UTF-16 file that you just sent displays fine for me on Debian GNU/Linux. I do not have ready access to a Windows computer.

The U+0301.utf16le-bom.txt file that you sent is stored correctly according to the Unicode Standard.  In hexadecimal, little-endian UTF-16, it contains 0xFEFF, 0x0078, 0x0078, 0x0301, 0x0078. That is exactly what it should contain.

If you have a program that is not placing U+0301 on the middle "x", it is the program that has the bug. In this case the program appears to be Notepad.

Is there another text editor you can use on Windows, such as Notepad++? You can get it here: https://notepad-plus-plus.org/. I have never used it but I have heard about it.

Alternatively, you could download cygwin onto your Windows system. That will provide you with a Unix-like shell. Most of the system utilities in cygwin are from GNU, so they should act like they do in my copy of Debian GNU/Linux.

I recommend trying to work in UTF-8 if you have no other constraints. The W3 organization is recommending that people not use UTF-16 in the future, but switch to UTF-8 because it is byte-order independent.


Paul Hardy

Paul Hardy <unifoundry>
Group administrator
Fri 30 Jun 2017 04:59:49 PM UTC, comment #9: 

I consider this to be working properly with the latest Unifont release, for the reasons described below. I have tested this with Unifont 10.0.01, which was built on Debian 9 ("stretch"), released on 17 June 2017, using an updated version of FontForge. Unicode Standard version 10.0.0 was released three days after Debian 9. So this testing was done with a Unifont version that was built with very recent versions of Unicode, Debian, and FontForge.

Unicode Standard v. 10.0, section 2.1, Combining Characters, in the section on "Sequence of Base Characters and Diacritics", p. 56, states: "In the Unicode Standard, all combining characters are to be used in sequence following the base characters to which they apply." So in the text file, combining characters must appear after the character with which they combine. You can download that document (~12 MB) here:

http://www.unicode.org/versions/Unicode10.0.0/UnicodeStandard-10.0.pdf

So whatever input method editor (IME) you are using to enter characters might permit your typing combining characters before the character with which they combine, but the result must be stored in a file with the combining marks after the base character. If that is not happening, it is a bug in the IME, not in the font.

I tried the Unifont 10.0.01 build with the "xx́x" example that you gave and it rendered fine. So I decided to test it further with non-Latin vowels.

The U+03xx block contains the U+0301 glyph that you mentioned, as well as Greek letters, so Greek was a natural choice. I typed in something in Greek that would exercise U+0301 with some vowels: "Κύριε, ἐλέησον.  Ἀμήν." ["Lord, have mercy. Amen."] That places U+0301 over three Greek vowels. I then examined the output file with the "od" program and found that the IME had changed combinations of vowel + U+0301 into one glyph containing vowel + tonos (acute accent): U+03CD, U+03AD, and U+03AE.

That was not going to let me check the interaction of U+0301 with those vowels, so I wrote a short program to hard-code the UTF-8 code points and store the result in a file. I have attached the resulting file as "kyrie.txt". The file begins with the three-byte UTF-8 signature, and then the Greek in quotes above follows.  U+301 appears in UTF-8 as two bytes; in octal, they are "\314\201". You will see that sequence repeatedly in "kyrie.txt".

The resulting text appeared fine in: Debian's terminal window and LibreOffice Writer; Mac OS X El Capitan's TextEdit; Windows 7 Notepad and Word 2013. (I do not have access to a later version of Windows, so it will be difficult for me to support future Windows releases.) The only anomaly was because I ended "kyrie.txt" with just a line feed, not a carriage return/line feed pair, Notepad on Windows 7 printed the LF character. I can live with that. See the attached sample screen captures (I can only attach four files, so they are representative).

I also did not like the way the U+0301 accent was rendered as a vertical stroke in the Greek letters such as U+03CD, U+03AD, and U+03AE. I have touched up the entire U+03xx block, and am about to release that as Unifont 10.0.02.

I also checked the rendering of the "STARGATE" logo (also created by hand with a program so I knew that the byte ordering in the file complied with the Unicode Standard). It rendered correctly on Debian 9 in LibreOffice Writer when the selected font was Unifont--but not when the selected font was the default LibreOffice font, Liberation Serif!

The attached images show Unifont 10.0.02 (about to be released), but I did not touch hex2sfd so Unifont 10.0.01 would render the same (it did in the checks I made). I can only attach four files, so the attached files are a representative sample of the outcome.

In conclusion, as long as an editor/IME combination correctly stores Unicode combining characters after the base character as per the Unicode Standard, glyphs should appear properly with U+0301 on the systems that I tested.  Therefore, I am closing this bug report.


(file #41085,

Paul Hardy <unifoundry>
Group administrator
Sun 12 Mar 2017 11:17:33 PM UTC, comment #8: 

I will have to go through the code and see what the end difference is, and look through my notes to see why I made any changes. The one significant change I made was adding support for combining characters beyond Unicode Plane 0. I did test the change at the time and it worked on what I tried.

Paul Hardy <unifoundry>
Group administrator
Sun 12 Mar 2017 10:34:26 PM UTC, comment #7: 

It looks like the positioning information in Unifont did change. The 2008 fix added the variable $xoffset to src/hex2sfd (https://github.com/SamB/debian-unifont/commit/adcf6b0db27f2c4041bdda9ed8076f9e7caf38cd#diff-8bd048f6aa5ea68f794d52b794e9a9d2R88) but Unifont 9.0.06’s src/hex2sfd does not include that variable. Therefore the problem is in the font, not in the font rendering engine.

David Corbett <dscorbett>
Sun 12 Mar 2017 09:25:59 PM UTC, comment #6: 

David, I did get this working correctly in Debian in 2008, but that was almost 10 years ago and things could have changed. THe positioning information in Unifont should not have changed, but font rendering engines definitely have changed over time.

I am going to try this on different systems before the Unifont 10.0 release, which should come right after the Unicode 10.0.0 release. It sounds like different font rendering engines are treating combining characters differently depending on how much they think they know about a particular Unicode script. They sometimes do this instead of using positioning information that is embedded in a font, which is what they should be doing.  That positioning information is what I added to Unifont in 2008 for combining characters; before then Unifont did not support such positioning. I cannot do much about a font rendering engine making decisions independently of positioning information that the font encodes. But in case that is not the problem, I will try to see if this is something that I can fix before the next Unifont release.

Paul Hardy <unifoundry>
Group administrator
Wed 08 Mar 2017 11:26:43 PM UTC, comment #5: 

Didn’t you already fix this bug in the Debian version of Unifont in 2008?

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=497326#15

David Corbett <dscorbett>
Tue 21 Jul 2015 01:41:39 PM UTC, comment #4: 

ftstring indeed renders the combining character over the next glyph. It renders correctly under Pango, however (which is what gtk uses).

Wouldn't it suffice (at least for most cases) to reposition the glyphs for combining marks to the left?

Anonymous
Mon 20 Jul 2015 01:03:44 AM UTC, comment #3: 

Thank you for reporting this.

The Fontforge script that creates combining characters would have to be modified to change this behavior. Right now the combining characters are assigned zero width in the TTF version, but non-combining characters have non-zero width. Changing this might require assigning a negative width to combining characters, but then the position will be incorrect for the next character.

This will probably take more expertise with Fontforge than I have (writing a TrueType routine to modify the current position, display the combining character, then restore the position to what it was before the combining character was rendered). So if anyone thinks they can modify this, let me know.

On the BDF and PCF versions, there was never an attempt to get combining characters working with them because most bitmap font rendering engines did not properly display zero-width glyphs. I don't plan to add such support right now.

Paul Hardy <unifoundry>
Group administrator
Sat 18 Jul 2015 04:20:21 PM UTC, comment #2: 

Test last unifont-8.0.01.ttf version, and it does not correctly display this U+0301 combining symbol. See picture:


Suntor <suntor>
Thu 16 Jul 2015 05:52:00 PM UTC, comment #1: 

The 8.0.01 TTF renders correctly for me. The precompiled BDF and PCF don't show combining characters at all under gtk2 (which uses cairo), while ftstring displays them like spaced characters.

A simple adjustment to BBX and DWIDTH properties fixes the BDF display under ftstring, but under gtk2 they still don't render at all.

Anonymous
Mon 13 Jul 2015 05:54:24 AM UTC, original submission:  

COMBINING ACUTE ACCENT, U+0301 must be set AFTER target symbol as another combining symbols accoding Unicode standard, so the font must display combining glyph over previous symbol in text. But Unifont display combining glyph over next symbol in text. Such error was in Verdana font in Windows XP and had been fixed in Windows Vista and with “European Union Expansion Font Update” for early versions.

Suntor <suntor>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #41093:  x.c added by unifoundry (271B - application/octet-stream)
file #41094:  x.txt added by unifoundry (9B - text/plain)
file #41091:  Unifont-10.0.01 U+0301 error (WS2012R2).png added by suntor (511KiB - image/png - No changes... Still incorrect display U+0301 in Notepad in Windows... Error in latest unifont-10.0.01.ttf file.)
file #41092:  U+0301.utf16le-bom.txt added by suntor (10B - text/plain - No changes... Still incorrect display U+0301 in Notepad in Windows... Error in latest unifont-10.0.01.ttf file.)
file #41089:  Unifont-10.0.01 U+0301 error (WS2012R2).png added by suntor (511KiB - image/png - No changes... Still incorrect display U+0301 in Notepad in Windows... Error in latest unifont-10.0.01.ttf file.)
file #41090:  U+0301.utf16le-bom.txt added by suntor (10B - text/plain - No changes... Still incorrect display U+0301 in Notepad in Windows... Error in latest unifont-10.0.01.ttf file.)
file #41085:  kyrie.txt added by unifoundry (52B - text/plain)
file #34461:  U+0301 error.png added by suntor (14KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by unifoundry (Posted a comment)
  • -email is unavailable- added by suntor (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 19 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-07-01 unifoundry StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2017-06-30 unifoundry StatusWorks For Me In Progress
        Open/ClosedClosed Open
    2017-06-30 unifoundry Attached File- Added x.c, #41093
        Attached File- Added x.txt, #41094
    2017-06-30 suntor Attached File- Added Unifont-10.0.01 U+0301 error (WS2012R2).png, #41091
        Attached File- Added U+0301.utf16le-bom.txt, #41092
    2017-06-30 suntor Attached File- Added Unifont-10.0.01 U+0301 error (WS2012R2).png, #41089
        Attached File- Added U+0301.utf16le-bom.txt, #41090
    2017-06-30 unifoundry Attached File- Added kyrie.txt, #41085
        Attached File- Added Kyrie-eleison-Win7-Word2013.png, #41086
        Attached File- Added Kyrie-eleison-Debian-stretch-LibreOffice.png, #41087
        Attached File- Added Kyrie-eleison-Mac-TextEdit.png, #41088
        StatusPostponed Works For Me
        Open/ClosedOpen Closed
    2015-07-20 unifoundry StatusNone Postponed
        Assigned toNone unifoundry
    2015-07-18 suntor Attached File- Added U+0301 error.png, #34461

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code