bugFree UCS Outline Fonts - Bugs: bug #50857, Confusion between new and old...

 
 

bug #50857: Confusion between new and old Indic implementations

Submitter:  David Corbett <dscorbett>
Submitted:  Sun 23 Apr 2017 01:09:14 AM UTC
   
 
Category:  individual character(s) Severity:  3 - Normal
Item Group:  character substitution issue Status:  Proceeding
Privacy:  Public Assigned to:  Stevan_White
Open/Closed:  Open Release:  * development
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 15 Jul 2017 11:53:04 PM UTC, comment #12: 

I regretted posting these notes -- I had spent some time on it, but the next day I found I had overlooked several things -- even about the tables I was looking directly at.
On the other hand, your responses have clarified a couple of things for me.
(that's your thanks in advance!)

I'm only afraid I may have wasted your time a little...

> The 'akhn' 'blwf' 'pstf' tags reverse their canonical input order between the old and new shaping rules.


The only features where the order differs are 'pref', 'blwf', and 'pstf'. 'akhn' is the same in both implementations.

I had confused the issue with 'akhn'.  As you say, whereever it is documented in OT, the expected input is consistent.
However, I think this is a case where an old shaping engine got the expected input wrong.  Therefore mos indic ranges in FreeFont have the same kind of work around for this lookup

>  The docs for bng2 mention a 'pref' tag... but it isn't used. What was it for?


Sorry-- these were my notes for myself, not really for general consumption. 
I know Bengali doesn't use pre-base forms as such.  But the OT docs for Bengli did mention 'pref' without elaboration.
I wondered why...if they used the lookup for some other purpose.

The verious OT docs have been pretty spotty over the years, all the way to being non-existent for some ranges.
But frankly I haven't looked at them for a couple of years, they may have improved.

> Q: how does deva form rakar forms?


Again, these were questions for myself, regarding FreeFont. 
I meant, how does FreeFont form the rakar forms in the 'deva' shaping engines?  I meant to investigate that myself.

> 'cjct'
> dev2 only, forms many simple conjuncts, many for Sanskrit
> Q: how does deva do this?


Same as the previous question.

> The ory2 docs mention half tables repeatedly, but gives no examples.


Just as with 'pref' in 'bng2', you can ignore it.

> But 'half' not used -- Gurmukhi really has nothing like this.


'half' means two things: true half forms, and halant forms which act like half forms for the purpose of placing pre-base vowel signs. The 'gur2' documentation explains this further; see the part beginning “While there are no half-forms in Gurmukhi”.

> The gujr and gjr2 docs mention half tables repeatedly, but give no examples.


You wrote: "Maybe you are reading some old documentation."
These notes were from some time ago, when I was looking at current documentation.

Steve White <Stevan_White>
Group administrator
Sat 15 Jul 2017 03:41:05 PM UTC, comment #11: 

Your last comment has a few questions I will answer and a few misunderstandings I will try to clear up.

> The 'akhn' 'blwf' 'pstf' tags reverse their canonical input order between the old and new shaping rules.

The only features where the order differs are 'pref', 'blwf', and 'pstf'. 'akhn' is the same in both implementations.

>  The docs for bng2 mention a 'pref' tag... but it isn't used. What was it for?

It is for pre-base forms of consonants that logically follow the base consonant. Bengali does not use this feature. All the OpenType Indic scripts use the same features, even though not all features are used in all scripts. They are just there to make the specs similar and in case a font developer wants to do something unusual.

> Q: how does deva form rakar forms?

First, 'blwf' substitutes ⟨ra halant⟩ with ⟨ra.blwf⟩. Then 'vatu' substitutes ⟨consonant ra.blwf⟩ with ⟨consonant_ra.vatu⟩ and it substitutes ⟨consonant.half ra.blwf⟩ with ⟨consonant_ra.half.vatu⟩. “consonant_ra.vatu” means a full consonant with a rakaar, and “consonant_ra.half.vatu” means a half consonant with a rakaar.

> 'cjct'
> dev2 only, forms many simple conjuncts, many for Sanskrit
> Q: how does deva do this?

'akhn' and 'pres' seem appropriate, depending on when the conjunct should be formed relative to reordering.

> The ory2 docs mention half tables repeatedly, but gives no examples.

Just as with 'pref' in 'bng2', you can ignore it.

> But 'half' not used -- Gurmukhi really has nothing like this.

'half' means two things: true half forms, and halant forms which act like half forms for the purpose of placing pre-base vowel signs. The 'gur2' documentation explains this further; see the part beginning “While there are no half-forms in Gurmukhi”.

> The gujr and gjr2 docs mention half tables repeatedly, but give no examples.

Actually, it does give an example right in the first section. Maybe you are reading some old documentation. The current documentation is at https://www.microsoft.com/typography/OpenTypeDev/gujarati/intro.htm.

David Corbett <dscorbett>
Fri 14 Jul 2017 11:33:39 AM UTC, comment #10: 

Hi David,

I haven't tried Pango in a while.  I think of it as the shaping engine of older Linux, which I still intend to support in the next release (but it should probably be dropped after that.)

If recent versions of Pango behave like Harfbuzz, it should be fine.

I will look into the issue you mentioned.

In the mean time, I did a survey of the questionable tables in FreeSerif (there are others in FreeSans).
The intent is to make tests that I can run on the different systems, to see of the fragile things are broken,
and to produce some sort of document explaining how it all works.

But I may have turned up some further problems.

This is a really complicted thing, besides its intrinsic complications:

  • the OpenType "shaping engine" specs changed between Windows XP and Windows 7. Some tables were added, others tragically changed their expected input order.


  • The quality of the OpenType documentation varies a lot from one script to another.

It has gotten better overall, but there are still big ommissions.

  • Harfbuzz was meant to emulate Windows Uniscribe, but it isn't perfect.


  • there were political problems -- nevermind.


  • Most of the scripts were already in FreeFont, and were tested and working right in the 2012 release for XP and Pango.

A few others were added since.  All of them have had their tables re-worked for Harfbuzz, but I don't have good records of how well each was tested for Pango.

  • There has also been a gradual degradation -- some of these I haven't deliberately touched in years.



===============================================================
Tables in FreeSerif:

The 'akhn' 'blwf' 'pstf' tags reverse their canonical input order between
the old and new shaping rules.

The 'half' tag is spottily documented.  Half forms are important in some
scripts, non-existent in others, and thought of differently in a few.

The 'pref' tag is very spottily documented.  Only a few scripts
really use pre-base consonant forms as such -- and there it's unclear
how they're supposed to work.

The 'cjct' and 'rkrf' tags were introduced in the new shaping rules.
Both are really nice to have, but it means to support the old rules you
have to include othr tables, which are unfortunately clumsier.

Bengali
-------
Bengali doesn't have rakar forms as such.
The docs for bng2 mention a 'pref' tag... but it isn't used.  What was it for?

'blwf' makes baphala, raphala, and laphala forms from ba, ra, la and asimara
bng2: hasanta + cons
beng: cons + hasanta

'pstf' makes yaphala forms
beng: ya + hasanta
bng2: hasanta + ya
!!! note these tables are in order beng bng2 !!!

'cjct' various conjuncts
!!! this table only exists in bng2 yet it has a beng script tag !!!


Malayalam
---------
Harfbuzz and Uniscribe behave differently regarding language tags in Malayalam.
The very use of language tags became politically controversial, and caused
practical problems in the development of this range.

The 'cjct' and 'rkrf' tags are not used and not mentioned in docs.

'akhn' is used to make lhlh and vhvh conjuncts in both versions,
but the set is very different.
mlym: cons + cons + candrakala
where cons is only lla or va
mlm2: cons + candrakala + cons
for some consonants that have doubled forms plus a few others.
A separate table takes care of chillu forms.
(where are these dealt with for mlym?)
'blwf' phala forms for la, lla
mlm2: candrakala + cons
mlym,mlm2: cons + candrakala  (using contextual chain)
!!! This doesn't make sense, either.
both of these tables are set for both mlym,mlm2
maybe the contextual chain resolves the conflict? !!!

'pstf' makes va, ya, ra forms
mlm2,mlym: candrakala + cons
mlym: cons + candrakala (contepxtual chain)
mlm2,mlym: cons + candrakala ( ra only, contextual chain)
!!!  this too is questionable !!!


Sinhala
-------
There is only the new tag 'sinh' for shaping.
Doesn't use 'blwf' 'cjct', but 'rkrf' is used.

My notes indicate that I knew little of the standards for this language
when I was working on it -- I think the OT page was missing or skeletal.

Devanagari
----------
'akhn' ja-nya and ka-ssa akhands.  Also eyelash-ra with ZWNJ
dev2,deva: cons + virama + cons
dev2,deva: cons + virama + cons
!!! I wonder if this isn't broken. !!!

'rkrf' rakar forms for all consonants
dev2: cons + virama + ra
Q: how does deva form rakar forms?

'blwf' rakaar
deva,dev2: virama + ra
plus similar vocalic forms for Sanskrit
deva: ra + virama

'cjct'
dev2 only, forms many simple conjuncts, many for Sanskrit
Q: how does deva do this?

Oriya
-----
There was no OT documentation for the orya script tag.
The ory2 docs mention half tables repeatedly, but gives no examples.

'akhn' two lookups, both
ory2,orya:
ory2,orya:
!!! Oh dear this is really weird.
Both tables take one cons + holonto + cons
and one cons + cons + holonto!!!
They are at least labelled wrong!

'blwf'
ory2: holonto + cons

'cjct'
ory2: cons + holonto + cons
Q: does it work for orya?

Tamil
-----
There was no OT documentation for the taml script tag.
The tam2 OT doc explicitly says 'half' forms are not used.

'akhn' ka-ssa ahand
taml: cons + cons + pulli
tam2: cons + pulli + cons

'pstf' only replaces some vowel signs with post-base forms
taml,tam2: single substitution


Gurmukhi
--------
The docs for guru were skeletal, mention blwf, pstf, and half.
The guru and gur2 docs mention half tables repeatedly, but give no examples.
But 'half' not used -- Gurmukhi really has nothing like this.

'pstf'
gur2,guru: ya + virama
gur2: virama + ya

'blwf' below-base forms of ha,ra, va
gur2: cons + virama
gur2: virama + cons
!!! something wrong here !!!

Gujarati
--------
The gujr and gjr2 docs mention half tables repeatedly, but give no examples.

'akhn' ja-nya ka-ssa
gjr2,gujr: cons + cons + virama
gjr2,gujr: cons + virama + cons
'blwf'
gjr2,gujr: ra + virama
gjr2: virama + ra

'half' used


Kannada
-------
No OT docs for knda.
I marked blwf as questionable regarding docs.

'akhn' ja-nya ka-ssa
knd2,knda: cons + cons + halanta
knd2: cons + halanta + cons

'blwf' three tables
knda: cons + halanta
knd2: a decomposition for jnya akahan!
knd2,knda: contextual chain operating on
halanta + cons
nukta + cons


Steve White <Stevan_White>
Group administrator
Sat 08 Jul 2017 03:12:50 PM UTC, comment #9: 

Pango uses HarfBuzz for its shaping now; do recent versions of Pango still have the bug that necessitated this work-around?

The recent reversions have broken ⟨ল্দ⟩ in HarfBuzz again. Here is why. The input glyph sequence is ⟨la_bng hasanta_bng da_bng⟩. 'bng2' takes precedence over 'beng', so the shaper uses "'blwf' ba,ra below-base forms in Bengali2". It does not find any matches: there aren’t any. So far, so good. But it is not done yet: it continues to the next 'bng2' lookup, which is "'blwf' ba,ra below-base forms in Bengali". In that table it does find a match, ⟨la_bng hasanta_bng⟩, and the substitution applies. The final glyph output sequence is ⟨bn_laphala da_bng⟩.

Uniscribe renders the same string just fine.

David Corbett <dscorbett>
Fri 07 Jul 2017 11:04:59 PM UTC, comment #8: 

OK for posterity ( I will try to put this together with the main documentation later.)

up to revision 3380 of FreeSerif, had 2 'blwf' lookups for Bengali in FreeSerif.
'blwf' ba,ra below-base forms in Bengali2
bng2{dflt}
entries hasanta + consonant
'blwf' ba,ra below-base forms in Bengali
bng2{dflt} beng{dflt}
entries consonant + hasanta
(consonant being ba, la, asimira, or ra)

The existence of the second table acting on bng2 was the subject of the original posting.
But it's a work-around for a bug.

How the trick works is:

In Win XP, there is no bng2, so consonant+hasanta falls to the second table in the second lookup.
Which is fine.

In Win 7 and Harfbuzz, 'bng2' takes precedence, so the first table is active.
The consonant+hasanta pairs are re-orderd to hasanta+consonant.
The table makes its replacement; after that there is no combination for the second lookup to act on.

In Pango, the bug is that 'bng2' is active, but it has to act on consonant+hasanta, because it doesn't do that re-ordering.
So the combination isn't recognized by the first table, but falls through to the second table, where (yay) it's taken care of.

Steve White <Stevan_White>
Group administrator
Thu 06 Jul 2017 08:03:53 PM UTC, comment #7: 

Sorry for thte long delay.  I knew I had made notes on this -- they were just carefully saved somewhere I didsn't expect.  In a long note from the year after the last commit:

===============================

Input order is reversed for consonants in the static classes, at least for
these tables
          old        new
'blwf'    cons+hal   hal+cons
'pstf'    cons+hal   hal+cons
That is, for the old rules, to act on a text combinaton of form
"halant consonant", the entry in the table must be "consonant halant".
Note this applies only to consonants in the static classes pertaining to the
table and known to the engine.  Other consonants are taken in their original order.

I typically have two sets of tables one for old and one for new rules, for 'blwf' and 'pstf', but it's messier than that: Pango has a bug, whereby it takes the consonant and halant in the new order for the old scripts.  This makes it very difficult to make a font work for both Windows and Pango (not impossible, difficult).
===============================
Plan


1) un-do the changes I made regarding this report.
2) make some tests that illustrate this fix in each script, to work in Pango and Harfbuzz (as well as Win XP and 10)
3) brush these notes up and put them someplace less secure.

Steve White <Stevan_White>
Group administrator
Mon 24 Apr 2017 09:24:31 AM UTC, comment #6: 

This is a systematic action.  Why?

Now, a memory is coming back.  I think there was a renderer implementation on some system that required values in some of these tables in a non-standard order.  What I did here is a cludge to get around that.  I'll look to see if I have notes on it, but -- things have gotten scattered.

It was supposed to work something like this.  If a renderer finds the character group in the wrong order, the entries will be ignored.  I found that two lookups were arranged in the right order, with one of them activating on both old and new scripts, the font could be made to work properly.

Of course this is very delicate: changing the order of the lookups breaks it.

It is important to know for what renderer I did this.  If it was a single release of an old renderer, I will gladly remove all these.  But if it was something less transitory, well, it might be better to just document how the thing was meant to work, and why.

Pity FontForge doesn't provide per-lookup annotations.

Steve White <Stevan_White>
Group administrator
Mon 24 Apr 2017 09:06:15 AM UTC, comment #5: 

You are completely right, David.  I ran your grep.  After some cleanup, the results are:

FreeMono: "'blwf' below-base forms in Devanagari"  
FreeSansBold: "'blwf' Below Base Forms in Devanagari"  
FreeSansBold: "'blwf' below-base forms in Devanagari for Sanskrit-decomp.vocalic-reph"  
FreeSansBold: "'blwf' Below Base Forms in Gurmukhi"  
FreeSansBold: "'pstf' Post Base Forms in Gurmukhi"  
FreeSansBold: "'pref' pre-base ra in Malay+AQEA-lam2-Reformed"  
FreeSansBold: "'blwf' below-base la in Malayalam2"  
FreeSansBold: "'blwf' below-base la in Malay+AQEA-lam"  
FreeSansBold: "'pstf' post-base forms in Malayalam2"  
FreeSansBold: "'pstf' ra in Malay+AQEA-lam"  
FreeSans: "'pref' pre-base ra in Malay+AQEA-lam2-Reformed"  
FreeSans: "'blwf' below-base la in Malayalam2"  
FreeSans: "'blwf' below-base la in Malay+AQEA-lam"  
FreeSans: "'pstf' post-base forms in Malayalam2"  
FreeSans: "'pstf' ra in Malay+AQEA-lam"  
FreeSans: "'blwf' Below Base Forms in Devanagari"  
FreeSans: "'blwf' below-base forms in Devanagari for Sanskrit-decomp.vocalic-reph"  
FreeSans: "'blwf' Below Base Forms in Gurmukhi"  
FreeSans: "'pstf' Post Base Forms in Gurmukhi"  
FreeSerifBold: "'pref' pre-base ra in Malayalam2-Reformed"  
FreeSerifBold: "'blwf' below-base la in Malayalam2"  
FreeSerifBold: "'blwf' below-base la in Malayalam"  
FreeSerifBold: "'pstf' post-base forms in Malayalam2"  
FreeSerifBold: "'pstf' ra in Malayalam"  
FreeSerifBold: "'blwf' subscript cons on ra via ZWJ in Kannada"  
FreeSerifBold: "'blwf' below-base forms in Devanagari for Sanskrit-decomp.vocalic-reph"  
FreeSerif: "'pref' pre-base ra in Malay+AQEA-lam2-Reformed"  
FreeSerif: "'blwf' below-base la in Malayalam2"  
FreeSerif: "'blwf' below-base la in Malay+AQEA-lam"  
FreeSerif: "'blwf' below-base cons in Malay+AQEA-lam-clscl"  
FreeSerif: "'pstf' post-base forms in Malayalam2"  
FreeSerif: "'pstf' ra in Malay+AQEA-lam"  
FreeSerif: "'blwf' below base forms in Davanagari2"  
FreeSerif: "'blwf' below-base forms in Devanagari for Sanskrit-decomp.vocalic-reph"  
FreeSerif: "'pstf' post-base forms in Oriya"  
FreeSerif: "'pstf' post-base forms in Oriya2"  
FreeSerif: "'pstf' ya post-base form in Bengali"  
FreeSerif: "'pstf' post-base forms in Tamil2"  
FreeSerif: "'pstf' post-base forms in Gurmukhi"  
FreeSerif: "'blwf' below-base forms in Gujarati"  
FreeSerif: "'blwf' subscript cons in Kannada2" 

As you said, none of these can be right, even if they somehow don't cause overt rendering bugs.

Steve White <Stevan_White>
Group administrator
Sun 23 Apr 2017 11:22:31 PM UTC, comment #4: 

The Bengali cluster was just one example. There are still plenty of broken Indic lookup tables. Did you run the command I posted?

David Corbett <dscorbett>
Sun 23 Apr 2017 09:10:51 AM UTC, comment #3: 

Nevermind my kneejerk reactions, please.

I guess I found the problem, and maybe misunderstood your remark about mixing.  There was a specific lookup, whose name indicated it  was for Bengali 2 (my notation for bng2), but which was in fact activated for the old shaper as well.  This would have affected certain other conjucts as well.  I removed the incorrect script tag, and now the conjuct renders correctly.  In SVN.

I also checked the other faces supporting Bengali -- they didn't seem to have this problem.

Thanks!

Steve White <Stevan_White>
Group administrator
Sun 23 Apr 2017 08:37:38 AM UTC, comment #2: 

Hi,

There is indeed some confusion, but it isn't clear to me what its origin is.  I see the incorrect effect, I see there is only one 'half' table for both versions.  But I find in my notes that Benglali 'half' table takes consonant + halant for both 'beng' and 'bng2'.  There are some other notes about 'la' being handeled in a peculi

These notes were the product of some time-consuming research and testing, and I remember this particular point about 'half' as being surprising and rather annoying.  So I don't thingk what I did was wrong at the time.

But I don't have time to sort out the history of the thing.  I'll just have to re-test, see what the behavior is now, and make it right.

Steve White <Stevan_White>
Group administrator
Sun 23 Apr 2017 08:02:05 AM UTC, comment #1: 

The mixed implementation was deliberate, whether it was a good idea  or not.  When I made these tables, I was actively testing them im Windows XP and Harfbuzz.

The intent was to have one release that supported the older font shapers.  The next release was to be the last that supported the old ones.  But unfortunately a lot of time has passed -- it's no longer clear that it's worth the effort.

To re-do this approach will be a job for another day.

I will regard this (ল্দ) as the subject of this bug report.  I assume you mean it misrenders in newer systems.

Let me know if you see other such mis-renderings!

Steve White <Stevan_White>
Group administrator
Sun 23 Apr 2017 01:09:14 AM UTC, original submission:  

Many Indic scripts have two OpenType implementations; e.g. Devanagari has both 'deva' and 'dev2'. One difference is the order of halant and consonant as inputs to 'pref', 'blwf', and 'pstf': the old implementation of these features applies to consonant + halant sequences, whereas the new implementation applies to halant + consonant sequences. A font should not assign conflicting implementations to the same lookup for these three features.

FreeMono, FreeSans, FreeSans Bold, FreeSerif, and FreeSerif Bold mix the implementations. For example, "'blwf' ba,ra below-base forms in Bengali" applies to both 'bng2' and 'beng'. Therefore ⟨ল্দ⟩ misrenders. You can list the broken lookups thus:


grep "\['\(pre\|blw\|pst\)f' .*'\(\(bng\|dev\|gjr\|gur\|knd\|mlm\|mym\|ory\|tml\|tel\)2\)" *.sfd | grep "'\(beng\|deva\|gujr\|guru\|knda\|mlym\|mymr\|orya\|taml\|telu\)'"


David Corbett <dscorbett>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by Stevan_White (Posted a comment)
  • -email is unavailable- added by dscorbett (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-04-24 Stevan_White StatusFix posted Proceeding
    2017-04-23 Stevan_White StatusConfirmed Fix posted
    2017-04-23 Stevan_White StatusNone Confirmed
    2017-04-23 Stevan_White Categoryoverall font problem individual character(s)
        Assigned toNone Stevan_White

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code