bugFree UCS Outline Fonts - Bugs: bug #51125, Quranic annotation signs overlap...

 
 

bug #51125: Quranic annotation signs overlap vowel signs

Submitted by:  David Corbett <dscorbett>
Submitted on:  Fri 26 May 2017 08:23:49 PM UTC  
 
Category: individual character(s)Severity: 3 - Normal
Item Group: letter positioning poorStatus: Proceeding
Privacy: PublicAssigned to: Steve White <Stevan_White>
Open/Closed: OpenRelease: development

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 03 Jun 2017 10:53:11 PM UTC, comment #20:

This comment has three parts.

Part 1

Here is one way to make shaddah + kasrah + annotation sign work without a shaddah + kasrah ligature.

Create a mark set (or mark class) containing all the marks that can attach to shaddah even when there is a kasrah between them, i.e. the above-letter annotation signs like U+06DA. Set the mark-to-mark lookup that should attach the annotation sign to the shaddah to only use that mark set.

That would usually be enough but there is an extra step required in FreeFont. "'mkmk' harakat in Arabic" (the relevant table) controls both "arab-harakat-blw" and "arab-harakat-abv". That table would have to be split since the "arab-harakat-blw" lookup should not skip kasrahs like "arab-harakat-abv" should.

Now, I am not saying you definitely should do this. But I think it is good to understand LookupFlags. It is not hard to make this work, though of course using a ligature is simpler.

Part 2

This is why shaddah + kasrah + annotation sign made shaddah and annotation sign overlap in the pre-ligature FreeFont of a few days ago.

Let’s say the renderer has already processed a few glyphs and the current glyph is the annotation sign. "arab-harakat-abv" applies to this glyph. It is a mark-to-mark attachment so the renderer searches backwards until it finds a mark. It finds the kasrah. "arab-harakat-abv" does not apply to kasrah, so the renderer gives up. The annotation sign stays at its default position, which happens to overlap the shaddah.

The technique in part 1 fixes this by changing “searches backwards until it finds a mark” to “searches backwards until it finds a mark in the given mark set”, which would find the shaddah.

Part 3

> no standard has been cited that says these characters should be reversed

No standard says these characters should be reversed. Unicode says they may be reversed (http://www.unicode.org/faq/normalization.html#8 and the next FAQ entry too).

Shaddah + kasrah and kasrah + shaddah are canonically equivalent to each other. That means that a font should handle both sequences identically. This is not specific to Arabic: for example, a font should render x + acute + cedilla identically to x + cedilla + acute, because they are canonically equivalent.

To make things easier for fonts, some renderers canonicalize the input before passing it to the font, but they are not obligated to so. There is no standard. It is completely up to each renderer.

Although canonicalization is completely optional for a renderer, it turns out to be common in practice. So you as a font maker do not actually need to handle both orders. For compatibility with all possible Unicode-compliant renderers, you would need to, but I don’t suggest spending time on that.

David Corbett <dscorbett>
Sat 03 Jun 2017 06:48:55 AM UTC, comment #19:

The problem is, so far no standard has been cited that says these characters should be reversed, yet the font software reverses them. I would like to see some standards documentation describing this re-encoding, and its scope.

Since the marks are reversed, the font must explicitly do something about it, or the letters will not be rendered right. If words with these marks are to be rendered, doing something special is not "optional".

The reversal is intended to facilitate the style of the mark under the shaddah -- one need only make a ligature as I've done. To support the other style, I'm not sure what to do -- it would at least take some thought.

As to what is right and wrong -- the standard says this started happening with metal typesetting. I wonder what on earth spurred them to move a sign like that. I wonder if a style-war ensued.

Steve White <Stevan_White>
Project AdministratorIn charge of this item.
Sat 03 Jun 2017 12:06:22 AM UTC, comment #18:

All of those examples come from https://github.com/khaledhosny/quran-data.

There are two ways to render shaddah and kasrah on the same letter. The shaddah always goes above the letter, but the kasrah can go either below the letter or below the shaddah. They are equivalent; it is a matter of font design and neither is wrong. The same goes for kasratan as for kasrah. You should change FreeFont’s style only if you think it looks better, not as a work-around.

Given a text with a kasratan then a shaddah, HarfBuzz will reorder the shaddah before the kasratan. Here is an example:

The angle-bracketed input code point sequences are different but the square-bracketed output glyph sequences are the same. Therefore, when writing lookups, it is safe to assume that shaddah will always precede kasrah, at least with HarfBuzz.

> Are there more cases of that kind of behavior?

Which behavior? Optionally moving a diacritic to below the shaddah? That’s just kasrah and kasratan, and maybe some of the kasrah-based marks in Arabic Extended-A.

David Corbett <dscorbett>
Fri 02 Jun 2017 09:47:21 PM UTC, comment #17:

Hi David,

Well there are several things going on here, some mysterious.

First, I had given up on the idea of positioning annotation marks to one another, because I have no idea how they should interact, so I removed all the "base" anchors from them. Well OK so you want to make them interact, and OK so they shouldn't collide. Fine -- I put anchors on them so they stack (ridiculously).

I will be gladly surprised if you can tell me these are from real texts.

Beside that, the worst thing is the combinations with shadda. Something very strange is going on here. Different fonts render them very differently --

&#x0642;&#x064d;&#x0651;&#x06da;
qaf - kasratan - shadda - high jeem sign

Now... I would have expected the kasratan to go beneath the qaf and the other letters to be stacked. But it doesn't work -- and not just in FreeFont. Several other Arabic fonts freak out here.
But Scheherazade-kerned puts the kasratan under the shadda What?
Well, the standard says something about this (Fig 9.5) but it is wonderfully vague as to what is right. "Some computer fonts" do it this way. nice!

The only way I can explain the behavior is that something has reversed the kasratan and shadda in this case. This makes it difficult for a font to handle the combination any way other than the way "some computer fonts" handle it. Either that or freak out, which is what most of them do.

Could you please enlighten us, as to where you got these examples, and what you think they're supposed to illustrate, and what the heck is going on with kasratan and shaddah? Are there more cases of that kind of behavior?

Thanks

Steve White <Stevan_White>
Project AdministratorIn charge of this item.
Fri 02 Jun 2017 03:22:33 PM UTC, comment #16:

These have overlaps:

اۜۗ ا۟ۖ ا۟ۗ ا۟ۘ ا۟ۚ ا۟ۛ اۤۖ اۤۗ اۤۚ بِّۚ بۡۚ ةَۢۖ تُۢۚ تۡۖ تۡۗ تۡۚ ثۡۗ جِّۗ حۡۖ رِّۗ رࣲّۖ رࣲّۚ رۡۖ رۡۚ صۜۡ صۤۚ عۡۚ فُۢۚ فۡۖ قٍّۚ قِّۖ قِّۗ قِّۚ كࣲّۗ لٍّۗ لِّۖ لۡۜ مِّۖ مِّۚ مۡۖ مۡۗ مۡۘ مۡۚ مۡۛ نِّۚ نۡۜ نۤۚ هۡۖ هۡۗ هۡۜ وُّۢۚ ىٰۤۖ ىٰۤۗ ىٰۤۚ یٍّۚ یِّۖ یِّۚ یࣲّۗ یۤۖ یۤۚ

David Corbett <dscorbett>
Fri 02 Jun 2017 01:33:42 PM UTC, comment #15:

Find attached a simple test of stacking 2-high many of the combining marks of the Arabic ranges.

Note that some of the marks aren't implemented in the Mono face yet.

I don't think any mark ought to modify any of the annotation marks.

It looks pretty good to me, in the latest commit.

(file #40846)

Steve White <Stevan_White>
Project AdministratorIn charge of this item.
Wed 31 May 2017 07:56:31 PM UTC, comment #14:

NFC is described in http://www.unicode.org/reports/tr15/tr15-44.html. The third line of figure 5 gives an example of reordering dots. NFC puts the dot below before the dot above. This is because the dot below has Canonical_Combining_Class=220 and the dot above has Canonical_Combining_Class=230 and 220 < 230.

> What do you mean by "top of the line"?

I mean the font height.

> First somehow align to the right at some (unclear) hight, and then if there are intervening diacritics, raise to avoid collision.

Yes, I believe this is accurate.

David Corbett <dscorbett>
Wed 31 May 2017 06:52:01 PM UTC, comment #13:

Hi David,

about the NFC. I poked around on the Unicode site and found this
http://www.unicode.org/charts/normalization/chart_Letter-Other.html
which shows normalized forms for a few mark combinations.

But that's very few -- it doesn't suffice to explain the effects I'm seeing, where the orders of combinations whole classes of marks are evidently reversed.

Am I not looking at the right thing?

Steve White <Stevan_White>
Project AdministratorIn charge of this item.
Wed 31 May 2017 06:33:21 PM UTC, comment #12:

Hi,

Just to acknowledge that I see what you see: in the attached PNG taken from your Quran Medina example are at least two annotaions, on the right side of the first line, and in the middle of the third line.

It does seem that the annotations in this document are aligned to the right of wide letters -- especially see the one in the third line.

The bottom of the annotation on the first line is substantially lower than some diacritics on the same line however. This illustrates that the logic is complicated: First somehow align to the right at some (unclear) hight, and then if there are intervening diacritics, raise to avoid collision.

I suspect there is a way to do something like what we see in this manuscript. It will be a great deal of work and time to get it right though, even after some further questions are answered. It will have to be another day.

(file #40828)

Steve White <Stevan_White>
Project AdministratorIn charge of this item.
Tue 30 May 2017 09:47:49 PM UTC, comment #11:

Hi,

About the line of annotatiosn:

What do you mean by "top of the line"? That really isn't well defined, unless it's the font height. To leave enough room for tall letters and diacritics so they won't collide with a line of annotiations would involve changing the font height. That won't happen.

I could just force all those annotations to be some fixed distance above the baseline, and blame any collisions on the text. This doesn't seem like a good solution to me.

I broke down and looked at a couple of commercial fonts that implement these marks. They use relative positioning to letters similiar to those I made. These are also general purpose fonts.

If there's a viable technical way to achieve this, I might consider it. But it would be a lower-priortiy project.

Steve White <Stevan_White>
Project AdministratorIn charge of this item.
Tue 30 May 2017 04:24:48 PM UTC, comment #10:

> This NFC thing is the kind thing I've been talking about. This is the first I've heard of it.

HarfBuzz canonicalizes all text using NFC, except that some characters have their combining character classes modified (https://github.com/behdad/harfbuzz/blob/92e2c4baafa0401cb8d7dd2bbd70acfaeaf2aabf/src/hb-unicode-private.hh#L105). I think Uniscribe is similar.

> As to the exact positionings, the business of positioning annotations "at the top of the line" is probably untenable in this family of fonts, given the technical constraints.

What are the technical constraints?

> Give it another couple of days to stabilize.

Sure, no pressure.

David Corbett <dscorbett>
Tue 30 May 2017 03:24:45 PM UTC, comment #9:

Hi,

This NFC thing is the kind thing I've been talking about. This is the first I've heard of it. (Thanks! And it might explain some trouble I had in Hebrew too.)

Some orderings of marks don't make sense, as I said. This has a practical effect on the person trying to implement positionings.
Without it, I was struggling to position marks in orders that wern't going to happen. Really, I lost many hours completely befuddled, befored it dawned on my what I was seeing.

Yes, I'm aware of some other problems with mark combinations that probably are necessary (esp. with hamza.) Give it another couple of days to stabilize.

As to the exact positionings, the business of positioning annotations "at the top of the line" is probably untenable in this family of fonts, given the technical constraints.

Generally, I want to achieve a good representation of these marks, but none of the FreeFonts is intended for typesetting the Qur'an. The Serif range is meant to reflect styles used in newspapers. I'll see how much I can do, given that direction.

Thanks greatly for your help, as always!

Steve White <Stevan_White>
Project AdministratorIn charge of this item.
Tue 30 May 2017 03:03:52 PM UTC, comment #8:

> It turns out the renderer is re-ordering some of the marks automatically -- so not only should it work either way, it often makes no difference how the marks are encoded. I don't understand this very well though, and it really confused me.

Since both orders are canonically equivalent, the renderer is free to reorder the marks. HarfBuzz, for example, uses a variant of NFC. This is, incidentally, why some renderers compose U+05D9 U+05B4 to the Yiddish U+FB1D even in Hebrew.

> Today's commit should fix most of the positioning issues.

It’s a good first draft, but your well-founded insistence on documentation and not guessing led me to find many sources (linked in previous comments) which indicate that this is not done yet. Specifially:

  • Annotation signs should stay at a constant height near the top of the line (except that they may go higher if necessary to avoid overlapping an ascender or another mark) but FreeFont treats them like harakat. For example, the heights of U+06D6 in ⟨ءۖ⟩ and ⟨ءَۖ⟩ should be the same.
  • Annotation signs following shaddah and kasrah overlap the shaddah: ⟨سِّۚ⟩.
  • Annotation signs should go above the left part of base letters: ⟨فۡۖ⟩.
David Corbett <dscorbett>
Mon 29 May 2017 12:16:24 PM UTC, comment #7:

Hi,

This is a complicated issue -- the character classes show only one aspect of it. It turns out the renderer is re-ordering some of the marks automatically -- so not only should it work either way, it often makes no difference how the marks are encoded. I don't understand this very well though, and it really confused me.

Today's commit should fix most of the positioning issues. Lightly tested -- really I should make a table showing all 2-mark combinations that make sense.

Steve White <Stevan_White>
Project AdministratorIn charge of this item.
Sun 28 May 2017 03:00:35 PM UTC, comment #6:

> I see that in FreeFont the anchors for the Koranic marks in particular allow them to be put on top of other harakat, but not the other way around. Your example shows the marks the other way around.

Unicode gives Arabic vowel signs low combining character classes and Quranic annotation signs high CCCs, so it does not matter which order they are in in an encoded character sequence. For example, U+0643 U+064E U+06D6 is canonically equivalent to U+0643 U+06D6 U+064E.

I have not been able to find a standards document describing the exact positioning of annotation signs: they are probably all in Arabic, which I can’t read. But I have found more examples and I am confident that the following is the proper behavior. An annotation sign may appear with or without a vowel sign. An annotation sign always appears high, rather than staying close to the base letter as vowel signs do. An annotation sign appears above the left of wide letters, rather than centered as most diacritics are.

https://archive.org/stream/AlQuranMadinah/Al%20Quran%20Madinah#page/n69/mode/1up
line 3, word 3:
The annotation sign U+06D6 is as high as possible, although there is blank space below it.
line 4, word -1:
The annotation sign is above the left of the kaf, which is a wide letter here.

https://archive.org/stream/QuranMushafQatar/Quran%20Mushaf%20Qatar#page/n299/mode/1up
line 6, word 1:
Another annotation sign which is placed high even though there is plenty of blank space below it.
line -4, word -1:
Another left-aligned annotation sign.

https://archive.org/stream/QuranSindhi/Quran-With-Sindhi#page/n988/mode/1up
line 8, word 3:
Left-aligned.

http://www.islamic-awareness.org/Quran/Text/Scribal/haleem.html
“Six pausal signs, `alāmāt al-waqf - are used in the mus̲h̲af, placed higher than all other signs”
This is referring to the annotation signs.

https://www.arabeyes.org/Qtbugs#Text_Rendering_bugs
“The expected behavior is that all "Waqf" signs are drawn on the highest possible position regardless of the height of the base character and/or the number of the harakat applied to it.”

http://www.tug.org/tugboat/tb27-2/tb87fahmy.pdf#page=5
section 5.1
See especially the paragraph beginning “Human calligraphers”.

David Corbett <dscorbett>
Sun 28 May 2017 11:57:55 AM UTC, comment #5:

There are two documents that touch on these questions, at least to describe groupings of marks that behave differently.

L2/10-455 2010-11-01
Of hamza and other harakat

L2/11-069 2011-02-09
Letter-making Arabic harakat

both by Roozbeh Pournader.

The latter describes groupings of "letter-making" and "non-letter-making" marks. The Koranic marks belong to the second grouping.

Another resource is the Wikipedia page on "Arabic Diacritcs", which discusses also "ijam", which it describes as "short vowel marks".

Now in FreeFont, I had distinguished ijam (marks that are properly part of the letter) from hamza and harakat. The idea was that harakat could be anchored onto ijam, but not vice-versa, and I think some harakat could go on other harakat.

I see that in FreeFont the anchors for the Koranic marks in particular allow them to be put on top of other harakat, but not the other way around. Your example shows the marks the other way around.

I can fix the Koranic marks. That should resolve the issue of this report, but the main question of which marks can go on other marks and how, is still open.

Steve White <Stevan_White>
Project AdministratorIn charge of this item.
Sat 27 May 2017 12:20:59 PM UTC, comment #4:

What we need is a standards document, that says which encodings are meant to produce Arabic text, and how they ought to be rendered.

Otherwise we're relying on heresay. With all due respect (!) even your telling me something is the case is heresay.

I mean, if one mark is to "follow" another mark, is it to be placed over the mark, or to its right? Or what? Can the marks be reversed? Then what happens?

This is the first question: do some of the marks "modify" others, always following the others, or can the marks come in any order? Which marks are which?

The example you give is a good start. But I'm not going to infer from this, for example, that some marks are right-aligned. That could easily be a calligraphers fancy.

Steve White <Stevan_White>
Project AdministratorIn charge of this item.
Sat 27 May 2017 02:57:01 AM UTC, comment #3:

https://archive.org/stream/Quran-Arabic-Hamariweb.com/standard1-quran#page/n456/mode/1up is a page from a Quran. In the fourth line from the bottom is the cluster ⟨كَۖ⟩, showing how annotation signs and vowel signs interact. In the first line is the cluster ⟨دِۖ⟩, where the annotation sign is quite high, leaving the space blank where a vowel sign above would go if there were one. At the end of the fifth line is the cluster ⟨فۡۖ⟩, showing that annotation signs above wide letters are left-aligned; for another example, see ⟨صۤۚ⟩ in the first line of the preceding page.

I’m sure there are other styles of annotation marks but this is a start.

David Corbett <dscorbett>
Sat 27 May 2017 02:15:15 AM UTC, comment #2:

I got these diacritic sequences (among many others) from https://github.com/khaledhosny/quran-data. I used U+0647 ARABIC LETTER HEH as a base for my examples because it is a simple letter without any ascender or descender.

Quranic annotation signs are just like other combining marks. They go above or below a base letter. If there is also a vowel sign, they are positioned using 'mkmk' or something equivalent. An alternative, used by the font Amiri, is for the annotation marks to always be quite high or quite low, leaving room for vowel marks even if there aren’t any.

I expect the precise details are a matter of the font-designer’s taste, but I am certain that FreeFont’s current behavior is illegible.

David Corbett <dscorbett>
Fri 26 May 2017 10:03:48 PM UTC, comment #1:

Hi David,

as usual, what is lacking here is guidance concerning interactions of these marks. Without documentaion, we are guessing.

I just skimmed through the standard, ch 9, again. I didn't find any mention there (but it is pretty long, maybe I missed something) and I did a web search for "unicode 06d6 placement". Lots of hits, nothing useful.

What makes you think these sequences are valid Arabic?

Steve White <Stevan_White>
Project AdministratorIn charge of this item.
Fri 26 May 2017 08:23:49 PM UTC, original submission:

In FreeMono and FreeSerif, Quranic annotation signs overlap vowel signs, as in ⟨هًۖ⟩ and ⟨هِۭ⟩.

David Corbett <dscorbett>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by Stevan_White (Posted a comment)
  • -unavailable- added by dscorbett (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 12 Aug 2017 04:05:02 PM UTCStevan_WhiteStatusFix posted=>Proceeding
    Fri 02 Jun 2017 01:33:42 PM UTCStevan_WhiteAttached File-=>Added diacritics-stacked.html, #40846
    Wed 31 May 2017 06:33:21 PM UTCStevan_WhiteAttached File-=>Added Quran_Madinah_annotations.png, #40828
    Mon 29 May 2017 12:16:24 PM UTCStevan_WhiteStatusNeed info=>Fix posted
    Fri 26 May 2017 10:03:48 PM UTCStevan_WhiteStatusNone=>Need info
      Assigned toNone=>Stevan_White

    Back to the top


    Powered by Savane 3.1-cleanup1