bugFree UCS Outline Fonts - Bugs: bug #38807, General features missing from...

 
 

bug #38807: General features missing from language tables

Submitter:  Georg Duffner <georgd>
Submitted:  Tue 23 Apr 2013 06:27:56 AM UTC
   
 
Category:  overall font problem Severity:  3 - Normal
Item Group:  character substitution issue Status:  Fix posted
Privacy:  Public Assigned to:  Stevan_White
Open/Closed:  Open Release:  * development
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 06 May 2013 09:50:27 AM UTC, comment #12: 

Hi Emmanual,

That's a very nice and informative article, but after reading through it, I don't find anything about the logic that was at the core of this problem.

The summary I wrote is empirical, based solely on my experiments.  I have seen suggestions of some funny behavior (the Adobe document for example uses some obscure terminology when discussing DFLT etc. -- but then it is a "file format" document).

I vaguely remember some other discussions of this -- it's been some years since I last looked into it. 

I have Haralambous' book here:
 ...rules defined for the "default" language are added to thosed defined for a specific language, whe cannot eliminate it: it will apply to all other languages that use the same script.

Well-- that sounds like an attempt at saying the same thing, but then it goes on to talk about the Adobe feature definition language, so it remains unclear whether the remark applies generally to OT fonts or just the language.  (The language also has an 'excludeDFLT' which would be awfully handy.)

Steve White <Stevan_White>
Group administrator
Sun 05 May 2013 10:37:44 AM UTC, comment #11: 

This article Windows Glyph Processing - Part Two: Glyph Processing in Detail (scroll a few pages to subheading Scripts and Language Systems) may explain more clearly how the script language selection works. It is still far from perfect, but more readable than technical specs.

Emmanuel Vallois <emmanuel_vallois>
Sat 04 May 2013 10:34:47 AM UTC, comment #10: 

Steve,
I know of no more documents.
But your summary is correct.

Georg Duffner <georgd>
Sat 04 May 2013 08:57:28 AM UTC, comment #9: 

George, besides the Adobe Font Feature Format and the MS documents you sent, do you know of any sources that detail the logic of 'dflt' and DFLT{'dflt'} ?  Because those sources are very poor.

Here is my summary:

The "dflt" language behaves as a wildcard, but only for those
which aren't listed anywhere among any of the tables of the current font.
Likewise, "DFLT{dflt}" behaves as a wildcard for script-language, but only for those which aren't listed anywhere among any of the tables of the font.

'dflt' example
==============
        table A to activate on: latn{'dflt'}
        table B to activate on: latn{'dflt', 'ESP '}

Then table A will not be activated for Spanish text, although it would have been without the presence of the specification for B.

DFLT{'dflt'} example
===================
        table A to activate on: DFLT{'dflt'}
        table B to activate on: grek{'dflt'}

Then table A will not be activated for Greek text, although it would have been without the presence of the specification for B.

Steve White <Stevan_White>
Group administrator
Fri 03 May 2013 06:26:10 PM UTC, comment #8: 

Hi Steve,

I’m glad I could help!

I know, it envolves some more work, but seeing the complexity with all the different scripts, it might be worth it to have a look into feature files instead of the FontForge interface, which, IMO, obfuscates more than it helps (and I only have had to deal with LGC until now).

Georg Duffner <georgd>
Fri 03 May 2013 06:17:54 PM UTC, comment #7: 

Hi Georg,

You'll see that I made a commit of FreeSerif* based on our discussion here, and the output of that script.

I'm still unconvinced of the CSS cases, they are different in some ways, but given my confusion over the role of 'dflt', I'm not so sure.

I'll consider your new suggestions.  The first one however refers to the creation of new tables, when the biggest headache here is maintenance of existing tables.

As I said this is really terrible: a case where local changes have surprising global effects.  I may be able to patch it up to some degree with scripts.

I've been in denial about this for a long time: of course I couldn't have accomplished what I did with Devanagari, for example, without having sort of understood how 'dflt' worked.  I remember thinking: I should formulate that clearly, and write it down -- but I never did.  Thanks for making me look at it hard again!

Steve White <Stevan_White>
Group administrator
Fri 03 May 2013 03:29:05 PM UTC, comment #6: 

Usually, fontforge fills the metadata with all available languagesystems.

I’m using this trick to simplify my life with special localized rules:

- in locl lookups replace the dangerous glyphs with a localized variant like sub [i f] by [i.TRK f.TRK];

- in liga no special treatment for turkic has to be taken you can even enable it without danger, you can list Turkish in the languages list (no more ccmpundo... needed)

- in smcp you simply add sub [i.TRK f.TRK] by [i.TRKsc f.TRKsc]; which you can savely enable for every language.

Also for german you can at least disable all triple ligatures except for ffi (and all f-ligatures with letters different from f, l and i if you had any) by adding a f.DEU which will only ligate in combinations f.DEU f.DEU, f.DEU i, f.DEU f.DEU i, f.DEU l; in your case it’s only ffl (for which there’s no possible occurence) that’s prevented, so I’m not sure if it’s worth it.

Georg Duffner <georgd>
Fri 03 May 2013 02:47:21 PM UTC, comment #5: 

Toward a more permanent resolution of the sorts of problems leading to the issue with ligatures, I wrote a script to find cases of 'dflt'-language tables which might be "shadowed" by an explicit language specified in another table.

tools/test/checkTableLangs.py

It's pretty rough, and takes no account of whether or not the font designer intends to disable a table.

The bigger problem here is to address the need of a font technician, to specify that this table should be activated for any language except these.  The TT/OT table specification do not lend themselves to this at all: innocent-looking modifications of one table can unintentionally disable another table.  No complaint will be raised in FontForge.  Ideally, a font editing user interface would spare the technician from this messy implementation detail.

Steve White <Stevan_White>
Group administrator
Thu 02 May 2013 02:56:16 PM UTC, comment #4: 

Georg,

Of course I was quite wrong regarding the activation of default ligatures.  As I think you were trying to say: the few languages I listed were not activated, because those languages appeared in the script-language lists of other GPOS features.  (I know about this, I had just conveniently forgotten it in recent modifications.)  I'm now fixing the issue.  THANKS!

However, regarding the activation of features by the experimental Mozilla properties, my position is unchanged.  This seems to be a different issue.  I'll try to provide examples.

Steve White <Stevan_White>
Group administrator
Wed 01 May 2013 08:31:19 PM UTC, comment #3: 

Georg,

I've spent some hours on this, done lots of tests, yet I can't claim to understand everything.  Some of what you say I really don't understand, but part of it is simply wrong (as best as I can understand it.)

Both of the documents you listed are familiar to me.  They are both very long and complex.  Again: which exact passage do you think has been violated?  You only specified a whole (page-long) section of one of them.

However, in particular, http://www.adobe.com/devnet/opentype/afdko/topic_feature_file_syntax.html#4.b, it states, referring to DFLT{dflt}
    "It is used if a program cannot find a match in the
     font to the current writing script and language."
But that would support my understanding, as well as the behaviour that I see everywhere (except in the experimental Mozilla CSS properties).

Freefont works quite well in the FontForge metrics window.  I don't see the connection between the logic of that application window (which makes sense for its users) and how a font renderer should activate features.

I'm not "bothered" by German ligature rules.  I only meant that they can't be properly implemented by font features alone.  I know in the past I tried, and at some point I also thought about disabling tham explicitly (although they aren't now...) However, it is interesting that you suggested that I try Dutch -- you might also have suggested Turkish or Spanish or Catalan. 
-- Because--
Harbuzz (and I think Windows too) specifically disables default ligatures for these few particular languages, but *not* for any other Latin-based languages I could find. 
(See attached modification of the test)
(I see the reasoning, but, frankly I hadn't realized it (maybe it's only in the new renderers.)
Again: default ligatures are activated by latn{dflt} except for German, Dutch, Spanish, Catalan, and Turkish, which are specifically disabled by the font renderer.
So, the example of the default ligatures supports my contention.

I must not understand your contention.

Let me ask, what do you think is the function of the feature activation languagesystem

    latn{dflt}

How do you think that makes ligatures work in English?  How is it that this makes it work for French and Romanian and Irish Gaelic?

I made a distinction between the experimental Mozilla OpenType features, and OpenType features that have previously been activated by language or CSS.  My point was, that your test mixes them all up.  If you separate the two, it is pretty clear that the experimental OpenType features follow a different logic regarding selection by language, than we have seen before.

With that all said, your last paragraph seems to be a moot point.  But there is another issue here, that somehow wasn't conveyed:  FreeFont supports hundreds of languages using Latin script, dozens using Cyrillic, Arabic, and Devanagari.  It would be a maintenance nightmare to list all of these languages for each feature that should be activated for all of them.
Fortunately, that's what defaults like

   latn{dflt}

accomplish.  (Except of course for the experimental Mozilla features, which seem to ignore them.)

I'm willing to keep working on this.  But so far, it looks to me that the font is correct.

(file #27990)

Steve White <Stevan_White>
Group administrator
Tue 23 Apr 2013 04:56:29 PM UTC, comment #2: 

Steve,

The documents I used when I started with EBG were the one referenced by Emmanuel Vallois in https://savannah.gnu.org/bugs/?38735http://www.microsoft.com/typography/otspec/chapter2.htm and the OpenType feature file specification that is much easier to understand. There, read especially http://www.adobe.com/devnet/opentype/afdko/topic_feature_file_syntax.html#4.b

My concerns are not about GPOS, as it seems legitimate to use a set of LangSystems different from the ones in GSUB. The troublesome features are the GSUB features themselves. Please try to make it work in Fontforge metrics window. Once you have that, many problems will be solved.

I’m well aware of German ligature rules. If that bothers you, just replace german with dutch in my testfile and try again. By the way, you won’t be able to do correct ligation in german by programmation. Not even if you include a complete dictionary.

The test file doesn’t confuse anything. It tests exactly the OT features in the font. The -moz-font-feature-settings do exactly what is expected from them. Internally, FF sets exactly this to true for features that are on by default (like liga, which is why you see ligatures in your test file).

You say you fear to have to list all the languages which you want to use the font with in the features. This is not true. You only have to list all the languages for which you have defined separate rules. Thus you have to add german, dutch, turkish, azeri, the tatars, catalan, spanish, the samis as well as serbian, bulgarian and macedonian to onum and the other general features. Else, they simply won’t work with these languages.

Georg Duffner <georgd>
Tue 23 Apr 2013 10:11:28 AM UTC, comment #1: 

Thanks, Georg,

I have to admit, Uniscribe does something different than I remember happening before.  I'll write more about this later.

You made pronouncements about how table activation is supposed to work.  Again, please:

  • if you're referring to a published standard, please

    indicate which one.

  • also indicate the passage in that standard

    that supports your claim.

Your test file confuses the new test properties of Mozilla with established properties with which the fonts have been tested extensively.  I attach a simpler subset, with just HTML and standard CSS. It doesn't show much, but:

  • positioning (kerning and mark placement) working in both

     German and English

  • ligatures working in French and English, not working in German


(the standard CSS small-caps feature is still unfortunately implemented by scaling, so it doesn't show anything about the font.)

Ligatures in German are problematic.  In words such as Tiefflug, the f's must not be ligated.  I do not know how to do ligation programmatically in German.  (See the Wikipedia article on ligatures for further discussion.)

I would like you to at least entertain the possiblility that some problems you are seeing reflect recent changes in rendering software, and that some of these might be regarded as bugs in that software.

I should also say: previously I had only seen 'onum' working in XeLaTeX.  I kept a PDF to prove it.  But today, with the new XeLaTeX which uses Harfbuzz, the 'onum' feature in the same file no longer works.
The XeLaTeX people have been complaining that recent changes have broken a lot of previously-functioning documents, and I know efforts are on to remedy that.

You know, FreeSerif supports hundreds of Latin-based languages, and has been to some degree tested in each of them.  If I read correctly your claim about script{dflt}, we must now explicitly list all those languages in each of the font's many features.  This was never the case in the past -- if it is the case now it should be fixed in software.

Clearly you and I have different points of view.  Let's continue work together and share information, respecting one another's expertise.

(file #27933)

Steve White <Stevan_White>
Group administrator
Tue 23 Apr 2013 06:27:56 AM UTC, original submission:  

This is a (more general) duplicate of https://savannah.gnu.org/bugs/?38735

Languagesystems that have their tables in a font, don’t inherit from their respective script dflt nor from DFLT{dflt}. Therefore, general features have to be registered with every single languagesystem to which they should be applied.

To see the problem, open FreeSerif.sfd in FontForge, open a metrics window and choose latn{DEU }. There you’ll find only a  hlig feature, containing only the lookup for t+z, but no kern, mark, liga, onum etc. features which might be desirable too. You can enter lette combinations which do have kern and mark definitions and will see that no GPOS gets applied (I wanted to attach a screenshot, but that tool doesn’ŧ work here at the moment).

Attached you find a test file for browsers. Harfbuzz (and thus Uniscribe probably too) make a difference between GSUB and GPOS features, so you don’t see an effect for kern and mark, but other engines may behave differently. I remember people having troubles when I forgot the languagesystems in EBGaramonds GPOS tables once.

Although this is demonstrated with german languagesystem it concerns all of them that have their tables in the font.

Georg Duffner <georgd>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #27990:  freeseriftests2.html added by Stevan_White (2KiB - text/html - modification of test showing ligatures in many latin-based languages)
file #27933:  freeseriftests2.html added by Stevan_White (860B - text/html - simpler set of tests using just HTML and standard CSS)
file #27932:  freeseriftests.html added by georgd (2KiB - text/html)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by emmanuel_vallois (Posted a comment)
  • -email is unavailable- added by Stevan_White (Updated the item)
  • -email is unavailable- added by georgd (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-05-04 Stevan_White StatusProceeding Fix posted
    2013-05-03 Stevan_White StatusNeed info Proceeding
    2013-05-01 Stevan_White Attached File- Added freeseriftests2.html, #27990
    2013-04-23 Stevan_White StatusNone Need info
        Assigned toNone Stevan_White
    2013-04-23 Stevan_White Attached File- Added freeseriftests2.html, #27933
    2013-04-23 georgd Attached File- Added freeseriftests.html, #27932

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code