bugFree UCS Outline Fonts - Bugs: bug #35409, Incorrect placement of khirik-yud...

 
 

bug #35409: Incorrect placement of khirik-yud for Yiddish in FreeSans

Submitter:  Ze'ev Clementson <zeev>
Submitted:  Mon 30 Jan 2012 11:33:58 PM UTC
   
 
Category:  character range Severity:  3 - Normal
Item Group:  letter positioning poor Status:  Fixed
Privacy:  Public Assigned to:  Stevan_White
Open/Closed:  Closed Release:  * 2010-09-19
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 01 Feb 2012 08:14:04 PM UTC, comment #21: 

I had a look through the WebKit's Bugzilla to see whether this issue had been already raised. It looks like there is something similar (but not exactly the same) here:
https://bugs.webkit.org/show_bug.cgi?id=10874

However, since it didn't exactly match, I submitted a new bug report: https://bugs.webkit.org/show_bug.cgi?id=77568

Ze'ev Clementson <zeev>
Wed 01 Feb 2012 06:47:37 PM UTC, comment #20: 

Hi Ze'ev,

Why don't you open the WebKit report?  
Just send me a link to it.
You explain your example, and I'll add in a Cyrillic one.

I'll write to Opera (they already hate me, no reason to spread the problem.)

Steve White <Stevan_White>
Group administrator
Wed 01 Feb 2012 06:10:29 PM UTC, comment #19: 

Steve,

That's terrific news! I'm pleased to hear that the lang attribute is now being correctly handled for OS X, Linux, and Windows. How do you want to handle the Webkit bug report? Do you want me to submit the bug report or will you? If you submit one, what can I do to add my support?

Ze'ev Clementson <zeev>
Wed 01 Feb 2012 05:52:29 PM UTC, comment #18: 

Ze'ev,

You may want to look at the last remarks on the Mozilla bug report,
   https://bugzilla.mozilla.org/show_bug.cgi?id=694205

Now shall we work on WebKit?

Steve White <Stevan_White>
Group administrator
Wed 01 Feb 2012 03:17:26 PM UTC, comment #17: 

The HTML file that I used was the modified version of yiddish-hebrew.html that you sent me (with the lang attributes moved down into the td elements). I haven't made any modifications to your version.

Ze'ev Clementson <zeev>
Wed 01 Feb 2012 01:57:47 PM UTC, comment #16: 

Hi Ze'ev,

I'm suspicious about my FireFox 10 tests.
Could you please post the HTML file you used to produce
Firefox001.jpg ?

Steve White <Stevan_White>
Group administrator
Wed 01 Feb 2012 11:36:27 AM UTC, comment #15: 

I don't quite understand your intent here. 

If you're going to distribute GNU FreeFont or a modification of it, you have to say it originated as GNU FreeFont, and where to find GNU FreeFont, and provide the GPLv3 (or later) with your distribution.

If you modify it, you ought to also outline what mods you made, then take responsibility for those modifications:
you should copyright the modifications yourself, and your docs should refer your users to your own project site
(we can't be responding to bugs in your modifications!)

But, let's continue such discussions off-line -- this has nothing to do with the bug report.

Steve White <Stevan_White>
Group administrator
Tue 31 Jan 2012 11:09:05 PM UTC, comment #14: 

Hi Steve,

Yes, I agree with your comment that it is probably better to use a font that is already present on Macs and which supports both Hebrew and Yiddish glyphs for these character combinations. Since Firefox 10.0 seems to differentiate them on Mac OS X, I could try a few fonts to see if any of them have Hebrew/Yiddish lang support and then submit a Safari bug report to Apple if I find one that can be used to illustrate the problem.

Re your comments on paying attention to the GPL - my understanding is that if I modify the FreeSans font and use it for displaying a document in an app, then I need to make the source of my modification to FreeSans available. I would definitely make my mods to FreeSans available under the same GPL+FE license that FreeSans is licensed under. Is that the only thing you were trying to highlight for me or are there other considerations?

Ze'ev Clementson <zeev>
Tue 31 Jan 2012 10:39:00 PM UTC, comment #13: 

Hi Ze'ev,

First, you asked about using FreeFont as an example for Apple.  This might not be as effective as using a commercial font. 

The issue here is font features being switched based on 'lang' attribute in a web page. 

This behaviour will be present only in fonts that support scripts used in multiple languages (Latin, Cyrillic, Arabic, Devanagari, and --now we see-- Hebrew).  I can see ArialUnicode does this in Arabic, but I'm sure there are others.  I'll look a little more, but they must be out there.  Probably newer fonts will be a better bet.

Most of the underlying font rendering libraries already support language-based feature switching.  We are seeing bugs in the browser software that calls those libraries.  In this case, the bug is that information of the 'lang' tag isn't being properly passed down to the rendering libraries.

I see no way to support this language-based switching in a browser with this particular problem.  If a solution can be found, it will be browser-specific.  The only "interim" solution for you is to produce two separate fonts, one for Yiddish and one for Hebrew.  (But with FreeFont, you can!  That's what free software is all about!!!)  (Again, pay attention to the GPL!)

The present case is not a legibility issue, just a stylistic convention.  I don't think this warrants the effort required to make it work for older browsers.

But it does warrant bug reports.  The issue is broader than the particular case, and in any case needs to be fixed.

Steve White <Stevan_White>
Group administrator
Tue 31 Jan 2012 08:39:46 PM UTC, comment #12: 

Steve, thank you for all of your help with this. I really appreciate the fast turn-around that you've provided and I think that your fix is the "correct" one.

Unfortunately, since no current browser provides 100% support for different font glyphs using the lang attribute, it will be some time before this will be generally useful. And, even when major browsers do support it, it will still be an issue with older browsers. I can't think of any "interim solution" to this, but it would be great if something could be done that worked for legacy browsers as well. I'm not expecting an answer from you on this but if you (or anyone else reading this) thinks of any alternative work-around for this issue, I would love to hear it.

Ze'ev Clementson <zeev>
Tue 31 Jan 2012 07:45:03 PM UTC, comment #11: 

Hi Ze'ev,

No I haven't posted a report to WebKit.  If you do so however, I would be glad to back up your report.  Would it be reasonable to report it first to WebKit then advise Apple too?

I just tried it under Windows 7.  No luck.  FireFox 10 has the same problem as FireFox 9.  And Explorer has never worked well on this system.  (Layer after layer of problem, and then either complete junk is displayed, or a completely black window!)

As to a release, I'm alread half a year behind, and there are several serious problems I need to overcome before I can justify it.  I don't know.  I've been saying "soon" for months.

Of course, you're welcome to hack the font.  However if you distribute it in any way, you have to follow the GPL rules.  If you're planning to use it as a Web font, there are instructions in the SVN, under notes/.

Cheers!

Steve White <Stevan_White>
Group administrator
Tue 31 Jan 2012 06:40:07 PM UTC, comment #10: 

I just tried the TeX test again and it works fine (see attached TeXShop001.jpg screenshot). Turns out it was "user error" - I was attempting to generate using TexShop/XeTeX and I should have been using TexShop/LaXeTeX (as I mentioned earlier, I rarely use TeX).

Safari is derived from WebKit. According to Wikipedia(http://en.wikipedia.org/wiki/WebKit), the Qt port of WebKit is also available to be used in Konqueror in KDE 4.1. Have you already submitted a similar bug report for any other WebKit-based browser (e.g. - Chrome)?

For my own application (for the time being), I will probably have to make do with my hacked version of the font so that it will display raised vowels in Yiddish in Safari.

In order to submit a bug report to Apple, I will probably need a "released" version of the FreeSans font to point them to. Do you have any idea when you will be posting a new release?


Ze'ev Clementson <zeev>
Tue 31 Jan 2012 06:22:17 PM UTC, comment #9: 

Your Safari results are not surprising.  My understanding is, Safari is derived from Konquerer.
In my Konqueror tests, it ignores the 'lang' attribute regarding mark positioning at least (but mark positioning is being applied for Hebrew text).

I didn't even know FireFox 10 was out.  I'll try it!

It occurred to me, I could implement the vowel positioning to yod a different way, using a different font feature.  But, in Safari, the mark feature is being applied; it is the switching by 'lang' tag that fails.  So that wouldn't help.

Did you try XeTeX again?

Steve White <Stevan_White>
Group administrator
Tue 31 Jan 2012 06:09:51 PM UTC, comment #8: 

I downloaded and installed the FreeSans.ttf file as well and retried the HTML test on Safari. Again, no raised vowels. I then changed the HTML so that both lang attributes were lang="yi". Again, no raised vowels. So, Safari is not recognizing the Yiddish positioning at all with your latest mod.

I then tried the test on Firefox 10.0 on my Mac. Firefox correctly recognized the lang tags and displayed the Hebrew vowels lowered and the Yiddish vowels raised! So, it looks like this bug is fixed in the latest version of Firefox.

Unfortunately, my application has to work in Safari as well (but this isn't your problem). I'll file a bug report with Apple.

I've attached screenshots of the HTML output in Safari & Firefox.


Ze'ev Clementson <zeev>
Tue 31 Jan 2012 05:49:07 PM UTC, comment #7: 

Hi,

Something went wrong with the sharing of the snapshot.  All four styles of FreeSans should be visible now.  Please try again.

BTW, it is "FreeSans.ttf" not "FreeSan.ttf".

If no Hebrew was when you an XeTeX on that file, it means the fonts were not available to the XeTeX system.  I have never used XeTeX on a Mac, so I can't really give advice there.  Maybe it's simply that the medium face isn't installed?

Regarding the HTML, as I explained, it should be enough to specify the language with a 'lang' attribute in a tag enclosing the text.

I have never seen a web browser that properly distinguishes languages using the same script.  This is not a bug with FreeFont, rather, bugs with the browsers.  They are slightly different:

FireFox: If Yiddish comes first, all is Yiddish.  If Hebrew, all is Hebrew.  I posted again to this report:
    https://bugzilla.mozilla.org/show_bug.cgi?id=694205

Konqueror (from which Safari is derived): seems to only do Hebrew (the default).  That is, it is ignoring the 'lang' attribute.  Interesting: it only displays marks properly if the 'dir' attribute is set to 'rtl'.

Opera:  Fails to do mark positioning in Hebrew altogether.  It's the worst.

Chromium: Only see Hebrew positioning (the default).

I'm personally very unhappy about this, but all we can do is file bug reports.

Steve White <Stevan_White>
Group administrator
Tue 31 Jan 2012 04:31:03 PM UTC, comment #6: 

I downloaded your latest font mods. Again, there were only 3 ttf files - no FreeSan.ttf file - was that deliberate?

I used TeXShop with XeTeX to attempt to generate the TeX file that you used. No Hebrew characters were output at all (see attached PDF and log). I also tried your HTML file. This time, both khirik and pasekh were displayed in the lower positions (e.g. - the "Hebrew" positions) for both Hebrew and Yiddish (see attached screenshot AfterMod2.jpg). How do I get the Yiddish positioning for khirik and pasekh to be activated?

BTW, I normally don't do any programming work with fonts or TeX so I may be doing something wrong with the TeX generation.

(file #24939, file #24941)

Ze'ev Clementson <zeev>
Tue 31 Jan 2012 12:05:52 PM UTC, comment #5: 

Find attached my latest attempt, along with the XeTeX source file.
I'll post the latest ttf files.

I'm also working on Serif, but I'm working on another range in those faces right now...

(file #24935, file #24936)

Steve White <Stevan_White>
Group administrator
Tue 31 Jan 2012 09:21:08 AM UTC, comment #4: 

Hi Ze'ev,

Your sample isn't FreeSans.  Look at the shape of the s in my sample and yours.

It looks as if your Hebrew text is in bold face.
FreeSansBold has no mark positioning for Hebrew, so the erratic positioning is to be expected.

Please try it again all in medium-weight face, and with FreeSans.

I'll also post snapshots of FreeSans with positioning in all faces (oblique takes more time, for obvious reasons).

Steve White <Stevan_White>
Group administrator
Tue 31 Jan 2012 12:49:45 AM UTC, comment #3: 

Hi Steve,

Yes, I think you're correct regarding the 'lang' attribute.

I deleted the FreeSans font from my system and downloaded the FreeSans Bold/Oblique/BoldOblique ttf files (there was no FreeSan.ttf file - was that deliberate?). Unfortunately, the khirik positioning is worse now. The khirik displays under and slightly to the left of the vav rather than under the yud (see attached AfterMod1.jpg screenshot).


Ze'ev Clementson <zeev>
Tue 31 Jan 2012 12:27:33 AM UTC, comment #2: 

Here is your text being displayed the latest SVN version of FreeSans using XeTeX.

Your HTML test file doesn't look right.
Tell me if I'm wrong: 'lang' is an element attribute, not a CSS property.
The reason is, CSS is just about presentation, but the 'lang' attribute affects the interpretation of the text.

I attached an improved version.

However, I can't get either my HTML tests or a fixed version of yours working right in Firefox.  I'm not surprised--I've been seeing this for a long time, and I filed a report with Mozilla.
It's like, Firefox associates a language with a script, and won't let go of the association.  So if it starts in Yiddish, Yiddish it remains.



(file #24929, file #24930, file #24931)

Steve White <Stevan_White>
Group administrator
Mon 30 Jan 2012 11:59:26 PM UTC, comment #1: 

Hi, Ze'ev,

Thanks for your very thorough report!

I have good news, and not so good news.

I think I "fixed" the problem, per se.  I'll send you a link to a snapshot.

Not so good:  Web browsers are notorious for fouling up subtle font feature details.  I'm aware of several cases where most other software will properly distinguish between text specified to be displayed in different languages, but all browsers I try fail.

I don't have a Mac handy, so you'll have to test it.

Steve White <Stevan_White>
Group administrator
Mon 30 Jan 2012 11:33:58 PM UTC, original submission:  

I documented this issue on the Freefont-bugs mailing list (http://lists.gnu.org/archive/html/freefont-bugs/2012-01/msg00000.html) and was told by Steve White that it is a bug and that I should file a bug report (http://lists.gnu.org/archive/html/freefont-bugs/2012-01/msg00001.html). For completeness, I have included the text of our emails below. Steve asked me to also indicate the following 3 things in the bug report:

   1) a bit of actual Unicode text showing the issue
The attached "yiddish-hebrew.html" file can be used to test correct display of the khirik-yud character combinations. For Yiddish, the khirik should be "raised" (see attached yiddish.jpg screen shot). For Hebrew, the khirik should not be raised.

   2) to know which OS and version you're using
I'm using Mac OS X 10.7.2 and have also tested this on iOS 5.0.1 - in both cases with the Apple Safari browser. However, this is reproducible on other OS/Browser combinations.

   3) to know which application software you're using.
The bug occurs when displaying Yiddish/Hebrew text in a browser.

Below is the text of my original email and Steve White's reply (from the 2 links referenced above).

- Ze'ev Clementson



On Mon, Jan 30, 2012 at 2:11 PM, Steve White wrote:

> Hi Ze'ev,
>
> This is a bug in FreeSans.
>
> The history of it, near as I can piece together:  I personally put in
> the feature tables
> for Hebrew, saw the need for positioning regarding these letters, made
> a release.
> Then sombody pointed out Hebrew wasn't working right, and I made a
> quick fix of disabling
> the Yiddish positioniing.  There it stands.
>
> It would be expeditious for you to open a proper bug report at
>   https://savannah.gnu.org/bugs/?group=freefont
> where we can effectively track progress.
>
> I also need:
>    1) a bit of actual Unicode text showing the issue
>    2) to know which OS and version you're using
>    3) to know which application software you're using.
>
> I'll take care of it as soon as I can, given this info.
>
> Thanks for writing!
>
> On Mon, Jan 30, 2012 at 6:23 PM, Ze'ev Clementson wrote:
>> Hello,
>>
>> I have been using the FreeSans font in an application that displays
>> Yiddish text using Unicode combining characters. In Yiddish, for the
>> "pasekh tsvey yudn" and "khirik yud" character combinations, the
>> convention is to "raise" the vowel. This is done for Yiddish only, not
>> Hebrew. With FreeSans, "pasekh tsvey yudn" is displayed properly;
>> however, "khirik yud" is not. I looked through the source code of
>> FreeSans.sfd to try to determine why this was happening and saw that
>> there was a yodhiriqhebrew ("khirik yud") definition set up; however,
>> it was not being displayed. The yodyodpatahhebrew ("pasekh tsvey
>> yudn") definition was being displayed properly so I looked through the
>> source to see why it was being displayed and the yodhiriqhebrew
>> combination was not.
>>
>> First, since "pasekh tsvey yudn" is displaying with a raised vowel, I
>> looked at the definition for it in the font source:
>>
>> StartChar: yodyodpatahhebrew
>> Encoding: 64287 64287 1299
>> Width: 400
>> Flags: HMW
>> LayerCount: 2
>> Fore
>> Refer: 853 1463 S 1 0 0 1 -99 280 2
>> Refer: 2821 1522 S 1 0 0 1 0 0 2
>> Validated: 32769
>> Ligature2: "'ccmp' Glyph Composition/Decomposition in Yiddish"
>> yodyodhebrew patahhebrew
>> LCarets2: 1 0
>> Ligature2: "'liga' Standard Ligatures in Hebrew subtable" yodyodhebrew
>> patahhebrew
>> EndChar
>>
>> I noted that the "Ligature2" element has two separate definitions for
>> ccmp & liga.
>>
>> Then, I looked for the definition of "khirik yud" in the source:
>>
>> StartChar: yodhiriqhebrew
>> Encoding: 64285 64285 1297
>> Width: 200
>> Flags: HMW
>> LayerCount: 2
>> Fore
>> Refer: 875 1497 N 1 0 0 1 0 0 2
>> Refer: 850 1460 N 1 0 0 1 -203 278 2
>> Validated: 32769
>> Ligature2: "'ccmp' Glyph Composition/Decomposition in Yiddish"
>> yodhebrew hiriqhebrew
>> EndChar
>>
>> The "Ligature2" element had only one definition for ccmp.
>>
>> So, I used FontForge to add the Ligature2 liga reference:
>>
>> StartChar: yodhiriqhebrew
>> Encoding: 64285 64285 1297
>> Width: 200
>> Flags: HMW
>> LayerCount: 2
>> Fore
>> Refer: 875 1497 N 1 0 0 1 0 0 2
>> Refer: 850 1460 N 1 0 0 1 -203 278 2
>> Validated: 32769
>> Ligature2: "'liga' Standard Ligatures in Hebrew subtable" yodhebrew hiriqhebrew
>> Ligature2: "'ccmp' Glyph Composition/Decomposition in Yiddish"
>> yodhebrew hiriqhebrew
>> EndChar
>>
>> Now, for both "pasekh tsvey yudn" and "khirik yud" combinations, the
>> vowels are raised. So, for Yiddish-only, this modification works fine
>> and the mod could potentially be incorporated into the font.
>> Unfortunately, the vowels are raised regardless of whether I display
>> Hebrew text (the vowels shouldn't be raised for Hebrew) with an HTML
>> xml:lang="he" lang="he" or Yiddish text (the vowels should be raised
>> for Yiddish) with an HTML  xml:lang="yi" lang="yi".
>>
>> So, my question is: how is it possible to specify that the "pasekh
>> tsvey yudn" and "khirik yud" combinations will be used with raised
>> vowels for Yiddish but without raised vowels for Hebrew? Is this a bug
>> in the font definition or do I need to do something else in my HTML
>> code to correctly indicate whether the font should use the 'liga'
>> lookup definitions?
>>
>> - Ze'ev
>>



Ze'ev Clementson <zeev>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #24944:  TeXShop001.jpg added by zeev (9KiB - image/jpeg)
file #24942:  Firefox001.jpg added by zeev (3KiB - image/jpeg)
file #24943:  Safari001.jpg added by zeev (3KiB - image/jpeg)
file #24939:  Hebrew-position.log added by zeev (12KiB - application/octet-stream)
file #24940:  AfterMod2.jpg added by zeev (3KiB - image/jpeg)
file #24941:  Hebrew-position.pdf added by zeev (9KiB - application/force-download)
file #24935:  Hebrew-position.pdf added by Stevan_White (24KiB - application/force-download)
file #24936:  Hebrew-position.tex added by Stevan_White (695B - application/x-tex)
file #24932:  AfterMod1.jpg added by zeev (3KiB - image/jpeg)
file #24930:  Hebrew-position.pdf added by Stevan_White (6KiB - application/force-download)
file #24931:  Hebrew-position.tex added by Stevan_White (443B - application/x-tex)
file #24926:  yiddish-hebrew.html added by zeev (831B - text/html - See bug description for explanation of these files.)
file #24927:  yiddish.jpg added by zeev (2KiB - image/jpeg - See bug description for explanation of these files.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by Stevan_White (Updated the item)
  • -email is unavailable- added by zeev (Submitted the item)
  • -email is unavailable- added by zeev (Yiddish FreeSans bug report requested by Steve White.)
  • -email is unavailable- added by zeev (Yiddish FreeSans bug report requested by Steve White.)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 23 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-05-03 Stevan_White StatusFix posted Fixed
        Open/ClosedOpen Closed
    2012-01-31 Stevan_White StatusProceeding Fix posted
    2012-01-31 zeev Attached File- Added TeXShop001.jpg, #24944
    2012-01-31 zeev Attached File- Added Firefox001.jpg, #24942
        Attached File- Added Safari001.jpg, #24943
    2012-01-31 zeev Attached File- Added Hebrew-position.log, #24939
        Attached File- Added AfterMod2.jpg, #24940
        Attached File- Added Hebrew-position.pdf, #24941
    2012-01-31 Stevan_White Attached File- Added Hebrew-position.pdf, #24935
        Attached File- Added Hebrew-position.tex, #24936
    2012-01-31 zeev Attached File- Added AfterMod1.jpg, #24932
    2012-01-31 Stevan_White Attached File- Added yiddish-hebrew.html, #24929
        Attached File- Added Hebrew-position.pdf, #24930
        Attached File- Added Hebrew-position.tex, #24931
    2012-01-30 Stevan_White Categoryoverall font problem character range
        Item Groupglyph(s) incorrect letter positioning poor
        StatusNone Proceeding
        Assigned toNone Stevan_White
    2012-01-30 zeev Attached File- Added yiddish-hebrew.html, #24926
        Attached File- Added yiddish.jpg, #24927
        Carbon-Copy- Added -email is unavailable-
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code