bug #36440: Ligature of अ and virâma in Devanagari

 Submitted by: Guillaume Jacques Submitted on: Wed 09 May 2012 09:17:00 AM UTC Category: individual character(s) Severity: 3 - Normal Item Group: letter positioning poor Status: Need info Privacy: Public Assigned to: Steve White Open/Closed: Open Release: 2012-05-03

You are not logged in

Sun 13 May 2012 09:44:07 PM UTC, comment #11:

Thanks, Zdenek!

There are several layers of issues here.

1) a new othography for Khaling, vs conventional use of virama
2) shaping software enforcing a usage rule on mark characters
2a) this peculiarity of XeTeX, and Guillaume's trick
3) lookup tables in the font

My view is, if in the font I specify that the virama (which is a mark) should be anchored to certain base glyphs in the character range, I expect that specification to be honored. I was surprised and displeased that the shaping software refused to acknowledge any tables regarding a vowel/virama combination. I have never seen this kind of behaviour in other scripts, and I find it overprotective.

The Unicode and OpenType documentation says virama should be applied to consonants, it does not say it must not be applied to other glyphs (whether it makes traditional sense is beside the point). I have to wonder what other undocumented "help" to expect from this software.

Guillaume, given this unexpected problem, do you think it is worth considering a different orthographic solution to your problem? There are other accent marks and such in the Devanagari Unicode range, for instance the grave and acute. What sound is this construct meant to represent, anyway?

Steve White <Stevan_White>
Sun 13 May 2012 09:00:45 PM UTC, comment #10:

The first note: \setmainfont and \setromanfont are synonyms.

Virāma is technically speaking a dependent vowel (matra). If it is not preceded by a consonant, it should be displayed with the dotted circle. Why the dotted circle disappears if the font is forced is a kind of mystery. From the LaTeX sense the first case differs from the following. When tracing is switched on, it can be seen that a group is open and the font selection machinery is running. I ran xelatex -no-pdf + xdvipdfmx -z 0 so that I could look into the PDF file by a text editor. I have deleted everything and kept just the object containing the text so that I could compare the files with diff. It showed the following difference:

\$ diff -c khaling-serif.pdf khaling-serif.PDF

• khaling-serif.pdf 2012-05-13 22:27:56.000000000 +0200

--- khaling-serif.PDF 2012-05-13 22:28:05.000000000 +0200
***************

• 2,11 ****

%\uffff
6 0 obj
<<
! /Length 198
>>
stream
! q 1 0 0 1 72 769.89 cm 0 G 0 g 0 G 0 g 0 G 0 g BT /F1 9.963 Tf 76.71 -62.76 Td[<004b0057001e>-333<06f2073a>-250<004900520048>]TJ ET 0 G 0 g BT /F1 9.963 Tf 231.13 -630.64 Td[<0015>]TJ ET 0 G 0 g Q

endstream
endobj
--- 2,11 ----
%\uffff
6 0 obj
<<
! /Length 202
>>
stream
! q 1 0 0 1 72 769.89 cm 0 G 0 g 0 G 0 g 0 G 0 g BT /F1 9.963 Tf 76.71 -62.76 Td[<002b0057001e>-333<06f21354073a>-250<004900520048>]TJ ET 0 G 0 g BT /F1 9.963 Tf 231.13 -630.64 Td[<0015>]TJ ET 0 G 0 g Q

endstream
endobj

The difference is just <06f2073a> in the working code vs <06f21354073a> in the nonworking code, ie just the dotted circle is inserted, xdvipdfmx does not select the font for the second time. The only explanation is that the font selectin machinery somehow interacts with the lookup tables and the dotted circle is not inserted. It is definitely a nonstandard behavior, maybe even an unnoticed bug in ICU. I would not be surprised if this trick ceased to work in a future version of XeTeX.

One more comment: virāma should never follow a vowel, it is an incorrect input. It is not a surprise that different font tables and different rendering engines copes with an incorrect input in different ways. However, if such incorrect input is useful for linguists, I agree that it might be implemented but users should be warned that it will work with one font and one engine, it cannot be guaranteed to work in other environments.

Zdenek Wagner <icebearsoft>
Sun 13 May 2012 06:55:50 PM UTC, comment #9:

I have experimented a bit with the LaTeX, and I don't understand why forcing the font on the virama should have the effect it does, and why not forcing it doesn't work. I thought it might be the \setromanfont, but changing to \setmainfont had no effect. Then I tried making the language to be default. No change.

\setmainfont[Script=Devanagari]{FreeSerif}
\newfontfamily\forceit{FreeSerif}
\newcommand{\mg}[1]{{\forceit #1}}
\newcommand{\gs}{अ\mg{्}} % this results in the mark being applied
%\newcommand{\gs}{अ्} % this doesn't work
%\newcommand{\gs}{अ{्}} % this doesn't work

Steve White <Stevan_White>
Sat 12 May 2012 08:05:46 PM UTC, comment #8:

I should explain my terminology.

You asked for a "ligature". Technically, the a-virama isn't a ligature, because the parts aren't directly connected. You could replace two characters with one glyph, but that isn't the natural thing to do here. This is a job for mark placement.

The tables for mark placement are already there in the latest release of FreeSerif. That is what must be neatly placing the virama beneath the 'a'.

The mystery is: why the other font rendering software (besides the ICU software used in XeTex) ignores these tables...

Cheers!

Steve White <Stevan_White>
Sat 12 May 2012 08:00:28 PM UTC, comment #7:

So...
Your pdf file was mangled, and I didn't know how to run the tex snippets you provided.
So I made my own XeLaTeX file, and made a PDF with it.
Find both attached.

I don't use the MS font, I just use FreeFont.

I also see other things I don't understand. Please let me know what is wrong and what is right.

Now in principle this should work, because the virama is a combining mark, and the font has a mark positioning "anchor" for the combination of the 'a' and the virama. The rendering software for XeTeX is using this, whereas the software for my Linux system and for FireFox seems not to use it. Why? Don't know.

Steve White <Stevan_White>
Sat 12 May 2012 07:46:34 PM UTC, comment #6:

Hi Guillaume,

First, the pdf file you sent is all mangled on my end. I see letters, but their positions are scrambled. Weird.

There are multiple problems with this combination, as I see it.

Let me explain a bit. Fonts don't really do anything: they are data. Besides glyph outlines, they contain tables providing further information, such as specifying that certain combinations of characters should be replaced with a certain glyph for text in a certain script (substitution), or to move certain combinations of glyphs with relation to one another (positioning). These get terribly complicated, and the standards are very poor as to just when much of it should happen.

Font rendering software draws the glyphs, and executes the substitution and positioning specified in the tables. Your OS has some such software, but some applications carry their own.

The problem I'm seeing is (a surprise to me), for Indic scripts, the rendering software I'm using (on Linux) does not like execute substitutions on vowels. Or maybe it's certain substitutions. Unfortunately, this is mostly un-documented. If you ask the people who wrote the software, they will just say to read the code yourself, to see what it does. And if you suggest to change anything... well I have several times, and so far only one change was ever made, and that took 2 years, lots of repeatedly making a case, and a fork in development of a project.

They will not do anything unless they can refer to a standard of some sort.

Now, in this case, I would have thought as a last resort to use the "ccmp" table, which as I read the standard, ought to be available in any script -- it's the basic character composition/decomposition table, supposed to be executed first. Unfortunately, not everybody agrees, evidently. I tried the old Indic tables too, no luck. And I know the new tables (from 2005) are not implemented in Linux.

I'm not sure about XeTeX. It uses a different font rendering engine than Linux or FireFox... maybe it has ccmp. I'll give it another try.

Steve White <Stevan_White>
Thu 10 May 2012 09:23:39 AM UTC, comment #5:

Let me post a larger sample of verb conjugations, using the little trick presented in the previous message. I have also uploaded the PDF generated by XeTeX. The font is Sanskrit 2003.

\caption{अकर्मक क्रिया \gs{}यत् भन्‍नु" }
\begin{tabular}{l|l|l|l|l|l|l|l|l|l|l|l|l} \toprule
&अभूत & भूत & आज्ञार्थक \\
उ:ङ् &\gs{}या:इङ &\gs{}यास्त \\
इऽचि &\gs{}यचि &\gs{}यास्ति \\
ओऽचु &\gs{}यचु &\gs{}यास्तु \\
इक् &\gs{}याइह्कि &\gs{}याइह्तिकि \\
ओक् &\gs{}याइह्क &\gs{}याइह्तक \\
इन् & इ\gs{}य:इ & इ\gs{}यास्त्य &\gs{}यचे \\
एऽचि & इ\gs{}यचि & इ\gs{}यास्ति &\gs{}यचिये \\
ए:न् & इ\gs{}या:न्‍नि & इ\gs{}यास्त्यनु &\gs{}यास्नुये \\
अम् & \gs{}य:इ & \gs{}यास्त्य \\
अम्सु & \gs{}यचि & \gs{}यास्ति \\
अम्ह्यम् & \gs{}या:न्‍नु / & \gs{}यास्त्यनु \\
\bottomrule
\end{tabular}
\end{table}

\begin{table}[H]
\caption{अकर्मक क्रिया ओम्त् पाक्‍नु" }
\begin{tabular}{l|l|l|l|l|l|l|l|l|l|l|l|l} \toprule
&अभूत & भूत & आज्ञार्थक \\
अम् & \gs{}वाम्द्यु & \gs{}वा:म्त्य \\
\bottomrule
\end{tabular}
\end{table}

Guillaume Jacques <rgyalrong>
Thu 10 May 2012 09:09:17 AM UTC, comment #4:

Dear Steve,

There is no mention indeed of Khaling, because this is an new orthography which we are devising, and my collaborators insist on using this ligature. Here is an example of a string of Unicode characters including the combination (see also the uploaded file sample2.jpg)

इअ्यास्त्य

Actually the Mangal font does allow the combination of अ and ्, but does not work well with all other ligatures. I don't know much about how fonts work, but maybe there is a feature there that could be used to fulfill the purpose.
I made a command \gs{} (glottal stop) to write this ligature, using the following code:
\newfontfamily\mangalfont{Mangal}
\newcommand{\mg}[1]{{\mangalfont #1}}
\newcommand{\gs}{अ\mg{्}}

Of course the result is not optimal and it would be much nicer if this ligature were allowed in a good-looking font.

Guillaume Jacques <rgyalrong>
Thu 10 May 2012 08:39:46 AM UTC, comment #3:

Hi again Guillaume,

I really need further information to proceed with this, as explained in comment #1.

It isn't as easy as I had thought -- it is much harder than I would have guessed. It seems the font rendering engines do not want to make any ligatures with vowel marks. I have never seen such behaviour outside of Indic ranges.

Thanks!

Steve White <Stevan_White>
Wed 09 May 2012 08:18:43 PM UTC, comment #2:

For the record:

1) there is no mention of this construct (or the Khaling language) in Unicode Chapter 9.

2) that document only talks about viramas after consonants, not vowels.

3) Likewise the OpenType docs for Devanagari only talk about viramas affecting consonants.

4) For unknown reasons, all my attempts at using Indic ligatures to make this work have failed. Tried several different feature tags. No luck. I'm missing something, or the rendering software is blocking it.

Steve White <Stevan_White>
Wed 09 May 2012 07:45:22 PM UTC, comment #1:

Hi Guillaume,

I would gladly do this, but have run into technical problems.

Please, could you provide a whole Unicode word in which this construction appears?

Is the combination just two characters, or is it possible that some other character, such as a zero-width joiner (ZWJ) should be coded between the characters, to force them to join?

The dotted circle is the "dotted circle" unicode character, which is taken from the font by the font rendering software, to indicate that a combining mark hasn't combined with anything. That is, the font doesn't do this. However, perhaps there's some way the font can convince the rendering software to make a ligature (haven't yet figured this out.)

Thanks!

Steve White <Stevan_White>
Wed 09 May 2012 09:17:00 AM UTC, original submission:

For a new orthography for the Khaling language, we need to display the ligature of the अ with the virâma, as in the attached .jpg file. Now अ and ् appear separated, and the virâma is displayed with a little circle.
We would be most grateful if you could implement this ligature.

Guillaume Jacques <rgyalrong>

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

Comment:

Attached Files
file #25848:  khaling-essai.tex added by Stevan_White (2KiB - application/x-tex)
file #25849:  khaling-essai.pdf added by Stevan_White (9KiB - application/octetstream)
file #25828:  khaling-essai.pdf added by rgyalrong (16KiB - application/pdf)
file #25827:  sample2.jpg added by rgyalrong (2KiB - image/jpeg)
file #25823:  virama.jpg added by rgyalrong (1KiB - image/jpeg)

Depends on the following items: None found

Items that depend on this one: None found

Carbon-Copy List
• -unavailable- added by icebearsoft (Posted a comment)
• -unavailable- added by Stevan_White (Posted a comment)
• -unavailable- added by rgyalrong (Submitted the item)
•

Do you think this task is very important?
This task has 0 encouragements so far.

Only logged-in users can vote.

Please enter the title of George Orwell's famous dystopian book (it's a date):