bugFree UCS Outline Fonts - Bugs: bug #56746, Malayalam: secondary glyphs formed...

 
 

bug #56746: Malayalam: secondary glyphs formed by prebase substitution with some matras are broken

Submitter:  Aiswarya Kaitheri Kandoth <aiswarya>
Submitted:  Sun 11 Aug 2019 02:07:52 PM UTC
   
 
Category:  character range Severity:  3 - Normal
Item Group:  character substitution issue Status:  Fixed
Privacy:  Public Assigned to:  Stevan_White
Open/Closed:  Open Release:  * development
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 07 Jan 2020 02:56:46 PM UTC, comment #13: 

Tested. Reported bug is fixed. I didn't see any new bugs while using traditional FreeSans.

Aiswarya Kaitheri Kandoth <aiswarya>
Mon 06 Jan 2020 02:10:49 PM UTC, comment #12: 

[comment #3:]

> Your PDF file was made with a font GayathriRegular.  I searched for this, and all I found was a bunch of non-functional fonts by the same name, whose glyphs are different from those in your PDF file.  Where did you get this font?


Here is the source: https://gitlab.com/smc/fonts/gayathri

[comment #5:]

> But... how is it that I have sent proofs of rather long texts to native speakers, and nobody mentioned this??? Do such letter combinations happen so rarely?  Do native speakers just overlook the problems?


I didn't notice in UDHR. But while doing gnu.org translations I noticed it because of the word 'program'.


Aiswarya Kaitheri Kandoth <aiswarya>
Mon 30 Dec 2019 10:17:59 AM UTC, comment #11: 

To correct my post of the 29th:

There are several complications, and several overall policies that can be taken.

To implement the reformed orthography in the old 'mlym' shaping engine, would require at least a temporary glyph for each consonant conjunct with and without ra, I think.  The reason is that the only mechanism for putting the pre-base ra glyph in front of a consonant conjunct is to first combine the consonants and the following ra into a single glyph, then perform a multiple replacement ra+conjunct.  The difficulty in doing this is exactly why the 'pref' lookup tag was added in the 'mlm2' engine.

But in FreeFont, for several reasons, I chose to support only a reduced subset of consonant ligatures, and I still think that's important.  So I'm going to drop support for reformed orthography in the old shaping engine.  Nobody cares but me. Before the new shaping engine, I don't think any font properly supported the reformed orthography anyway.  But for me, I can relax my efforts by giving up on that.  When the conditions are right, reformed can be supported in the new 'mlm2' shaping engine.

As to the problems: in what I wrote before, I had thought to use the 'pref' tag to signal that a consonant conjucnt should be formed.  But this doesn't work for classical orthography in the FreeFont approach to Malayalam, because it is the final consonatn of the cluster that receives the ra "tail".

Instead, the 'pref' lookup will be used only for the reformed orthography, and for the classical orthography, a temporary post-base ra glyph is produced by a 'pstf' lookup instead.  Later, consonants that are followed by that glyph may be decorated with the ra-tail.  But after any ra-tail ligatures have been formed, any leftover temporary ra glyphs have to be recovered as candrakala-ra.

Provided the font shaping engine is working properly, lookups like these should support the classical orthography in both 'mlym' and 'mlm2', and the reformed orthography in 'mlm2':

        'pref' candrakala + ra -> pre-base rakar ('mlm2', reformed only)
        'pstf' ra + candrakala -> post-base rakar (classical only)

        'psts' rakar + cons -> ligature ('mlm2', reformed only)
        'psts' cons + rakar -> ligature (classical only)

        'psts' post-base rakar -> candrakala + ra (classical only)

Steve White <Stevan_White>
Group administrator
Sun 29 Dec 2019 07:35:18 PM UTC, comment #10: 

I just made another commit, which seems to have fixed the problem I mentioned.  The tests I've looked at seem good.  I'll look again tomorrow.

The summary of the lookups I mentioned doesn't work as well as I had imagined.  In particular, I know of now way to support reformed orthography with the old shaping engine without an expanded set of conjunct ligatues.   I'll write more on that tomorrow.

See how it looks!

Steve White <Stevan_White>
Group administrator
Sun 29 Dec 2019 02:31:31 PM UTC, comment #9: 

I introduced a new bug, that happens when ra follows conjoined consonants.  Example
&#x0d38;&#x0d4d;&#x0d24;&#x0d4d;&#x0d30;
The ra tail should be applied to the final consonant, but it's being applied to the first one.

Steve White <Stevan_White>
Group administrator
Sun 29 Dec 2019 01:35:13 PM UTC, comment #8: 

That is a messy issue for several reasons.  It took me a long time to remember what the issues were, and to sort out a solution for it.  I'll record all that here for posterity.

This version of FreeFont is still intended to support the older 'mlym' font shaper.  And the fomation of the rakar forms has changed very much with the newer 'mlm2' font shaper. 

Also, my intent has been to support both the classical orthography as well as reformed.  This almost worked, but for a bug in external Linux software.  For now that has been shelved, but I want to retain the option.

Mostly in deference to the creators of the original font faces, Rachana and Meera, I have chosen to support only the classical script for now.

-----------------------------------------------

Classically, the ra-sound is attached to a preceding consonant or cluster by a long tail that curls from the end of the letters beneath them to emerge to the left of the letters.  It is a ligature.  In the reformed orthography, this ligature is dropped; instead, a separate sign is place before the consonant or cluster.

That behaviour is pretty easy to implement in the new font shaper with its 'pref' lookup tag, and rather harder in the old one.  If everything is working right however, it is possible to implement both in one font, allowing the user to pick which orthography is desired by means of a TrueType switch.

With all that in mind, here is how it can be implementd:
-----------------------------------------------
strategies:

mlm2:
-----
Form the preceding rakar form of ra with a 'pref' table.
        candrakala+ra -> rakar
then form the rakar conjuncts by matching with that glyph.
        rakar+cons
This has one advantage that, even if the rakar conjunct isn't supported for the consonant or cluster, the rakar sign is still in the right place.

mlym:
-----
Convert the ra to an intermediate form with a pstf table.  form later rakar ligature with a 'pstf' lookup for
           cons+rakar
To support reformed orthography, finally, another table will decompose these ligatures into
           rakar cons
This will work, however, only when the rakar ligature is supported
-----------------------------------------------
lookup tables:

Idea is to convert ra to a rakar form in all cases, and form conjuct ligatures in all cases, but to de-compose the conjunts for reformed orthography.  This has the advantage of performing re-ordering even for the old 'mlym' font shaping engines.

        'pref' candrakala + ra -> pre-base rakar ('mlm2' only)
        'pstf' ra + candrakala -> post-base rakar ('mlym' only)

        'psts' rakar + cons -> ligature ('mlm2')
        'psts' cons + rakar -> ligature ('mlym')

        'psts' rakar + cons decomposition ('mlym', reformed)

If only classical orthography is supported, the decomposition lookup should be omitted.

If 'mlm2' is supprted, only the 'mlm2' 'pref' and the 'mlm2' 'psts' lookups are needed.  A language tag should turn the ligature formation off for most combinations in the reformed orthography.

Steve White <Stevan_White>
Group administrator
Sat 28 Dec 2019 01:24:16 PM UTC, comment #7: 

I have found the origin of the problem, by building various past revisions of FreeSans,
and looking at the attached HTML test:

r3708 was still doing reformed orthography -- this changed in a later revision.
r3737 the classical rakar form was being appied correctly.
r3738 the rakar form surrounded by vowels is no longer working

=======================
The log entry for r3738 reads:

r3738 | Stevan_White | 2017-11-06 13:32:19 +0100 (Mon, 06 Nov 2017) | 4 lines

Malayalam: made combinations with candrakala+ra to display the candrakala
        and ra, unless they form a ligature.
        Adjusted vowel anchors on many subscript consonants.
=======================

About the other fonts which are showing reformed orthography: my guess was wrong.
It is not due to the bug in the font shaping software -- the fonts I checked only do reformed orthography.

So now it's much clearer what is going on.  Now I need to understand what I thought I was doing
in commit 3738, and devise a better means of addressing it.

(file #48128)

Steve White <Stevan_White>
Group administrator
Sat 28 Dec 2019 01:58:52 AM UTC, comment #6: 

About the other free fonts in the test: I just started out with a poor selection.  I installed the latest package of fonts for Linux Mint.  From these, most of the fonts behave correctly in your example.

These include Chilanka, Manjari, Uroob, Meera, Dyuthi, AnjaliOldLipi, Keraleeyam, Karumbi, Rachana v. 7,.0.0

RaghuMalayalam may be working correctly, but it seems to be for reformed orthography.  Oh, I see.  Lohit Malayalam is also doing reformed.  And so is Noto Serif.

Samyak is trying to do reformed, but it doesn't work right (the reduced ra glyph should appear to the left of the consonant, but it shows up on the right.  I think.)

OK, so only one of them is actually wrong.  I'm surprised by all the reformed orthography fonts.  Unless... I need to check something (tomorrow).   I know of a bug in the font shaping engine in Linux.  I'll try to contact you directly about this.

Steve White <Stevan_White>
Group administrator
Sat 28 Dec 2019 01:32:18 AM UTC, comment #5: 

Further thoughts on this.  Your examples make perfectly good sense to me, from what I understand of the orthography.  The consonant conjunct should be formed first, then the vowels should be applied to the conjunct.


As far as the TrueType process is concerned, that observation means that the consonant conjunct must be formed very early in the lookup tables, before the shaping engine re-orders the vowels.  (The font is not really responsible for that reordering.)

I doubt that it ever occurred to me to test that.  I doubt that the consonants were ever formed early enough for this to work correctly.  And now I see (or I think I see) that other free Malayalam fonts suffer from similar problems.

There seems to be nothing wrong with the HTML file.

But... how is it that I have sent proofs of rather long texts to native speakers, and nobody mentioned this??? Do such letter combinations happen so rarely?  Do native speakers just overlook the problems?

It is likely that I am misreading something here.

Steve White <Stevan_White>
Group administrator
Fri 27 Dec 2019 11:55:46 PM UTC, comment #4: 

A lot of conjuncts are broken.  I had big test pages of conjuncts that were at some point mostly looking right (to my eye) and now large numbers of them are not being formed.  I will have to go back to the last time I made changes to the range, 2017 or early 2018, and see what I did.  It will be slow detective work.

I am paerplexd however, that in my HTML text file, none of the free Malayalam fonts I have here is working properly.  This seems very strange.  Is my test file messed up?  I just copied the text out of your OTF file.

Steve White <Stevan_White>
Group administrator
Fri 27 Dec 2019 12:53:13 PM UTC, comment #3: 

Your PDF file was made with a font GayathriRegular.  I searched for this, and all I found was a bunch of non-functional fonts by the same name, whose glyphs are different from those in your PDF file.  Where did you get this font?

Steve White <Stevan_White>
Group administrator
Fri 27 Dec 2019 11:31:20 AM UTC, comment #2: 

I see it.  I knew I had left this range in a bad state a few years back...

Steve White <Stevan_White>
Group administrator
Sun 11 Aug 2019 05:41:18 PM UTC, comment #1: 


FYI: I tried hb-shape, here are the details.

Rendering of പ + ് + ര + ൈ = പ്രൈ:
In FreeSans.otf
[pa_mal=0+855|candrakala=0+0|ai_mal=2+1404|ra_mal=2+716]

In Gayathri-Regular.otf
[ai1=0+2575|p1r3=0+1930]


Aiswarya Kaitheri Kandoth <aiswarya>
Sun 11 Aug 2019 02:07:52 PM UTC, original submission:  


The secondary glyphs formed by prebase substitution appears broken when it comes with some matras, which are െ, േ, ൈ, ൊ, ോ, ൌ (these are matras correspond to the vowels എ, ഏ, ഐ, ഒ, ഓ, ഔ respectively). Attached .odt file with details. A .pdf file is attached to show the correct form (it is in Gayathri Malayalam font).

Tested with LibreOffice Writer 6.2.3.2, Scribus 1.5.4 and Inkscape 0.92.4.
OS: Mageia 7.
 
The problem was noted in 2012 release. Then I build using fontforge 20190413 from trunk using svn to test in the development version.

Aiswarya Kaitheri Kandoth <aiswarya>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #47324:  FreeSans_pres_with_vow.odt added by aiswarya (10KiB - application/vnd.oasis.opendocument.text)
file #47325:  FreeSans_pres_with_vowel.pdf added by aiswarya (7KiB - application/pdf)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by Stevan_White (Posted a comment)
  • -email is unavailable- added by aiswarya (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-01-09 Stevan_White StatusFix posted Fixed
    2019-12-29 Stevan_White StatusProceeding Fix posted
    2019-12-29 Stevan_White StatusFix posted Proceeding
    2019-12-29 Stevan_White StatusProceeding Fix posted
        SummaryMalayalam in Sans: secondary glyphs formed by prebase substitution with some matras are broken Malayalam: secondary glyphs formed by prebase substitution with some matras are broken
    2019-12-28 Stevan_White Attached File- Added FreeFont_pres_with_Vowel.html, #48128
        StatusConfirmed Proceeding
    2019-12-27 Stevan_White Categoryoverall font problem character range
        Item Groupletter positioning poor character substitution issue
        StatusNone Confirmed
        Assigned toNone Stevan_White
    2019-08-11 aiswarya Attached File- Added FreeSans_pres_with_vow.odt, #47324
        Attached File- Added FreeSans_pres_with_vowel.pdf, #47325

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code