bugUnifont - Bugs: bug #52630, Unifont Expansion and Contraction

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #52630: Unifont Expansion and Contraction

Submitter:  Paul Hardy <unifoundry>
Submitted:  Sun 10 Dec 2017 02:01:59 AM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  unifoundry
Open/Closed:  Closed

Discussion locked!

Jump to the original submission

Mon 30 Jul 2018 03:40:40 AM UTC, comment #10: 

I am closing this because I have gone over all I intend to go over in this somewhat narrative bug entry, and have settled on changes such as leaving combining character code points in combining.dat files.  Rendering is somewhat at the mercy of font rendering engines anyways, no matter what Unifont does.

Thank you,


Paul Hardy

Paul Hardy <unifoundry>
Group administrator
Wed 06 Jun 2018 12:14:34 AM UTC, comment #9: 

If a glyph is a combining character, I intend to leave it in a *combining.txt file.

Incidentally, I am the one who drew the Gujarati glyphs. I generally drew Indic consonants with four blank columns on the left and on the right to leave room for vowel marks, not because I thought that Gujarati should be single-width. I drew them at a time when I could read Gujarati so I was familiar with the script.

Paul Hardy <unifoundry>
Group administrator
Thu 28 Dec 2017 03:45:52 PM UTC, comment #8: 

Now that the Miao spacing marks are back in plane01-combining.txt, the Miao consonants should be two cells wide again to compensate.

David Corbett <dscorbett>
Thu 28 Dec 2017 03:23:57 AM UTC, comment #7: 

I have decided to leave all combining characters in *combining.txt files.  Doing otherwise gets more complicated, and at least for now HarfBuzz will treat them as combining characters so for now they should be handled as such.  The ones I removed for Miao are back in the Unifont 10.0.07 release.

I am also planning to add in all glyphs marked as non-spacing marks for the next release.


Paul Hardy

Paul Hardy <unifoundry>
Group administrator
Wed 20 Dec 2017 03:31:12 PM UTC, comment #6: 

Okay, I understand.

David Corbett <dscorbett>
Wed 20 Dec 2017 07:02:32 AM UTC, comment #5: 

David,

The last concrete example I provided to you about the uniformity of Indic glyph widths is that someone redrawing glyphs would be able to do so without having to know how to read a computer program (unibmp2hex.c in this case) to ferret out what the exceptions are. I have had people make contributions who are able to use the graphical template but who do not understand how to read computer source code. It simplifies things for someone drawing those glyphs knowing that the glyph widths are consistent within scripts where it is possible. I do not want to keep asking you to do this and have you keep insisting otherwise, so please do not submit any more Indic glyphs.

Also, please realize that there are tens of thousands of glyphs in Unifont, and if I were to entertain every request to make cosmetic changes based on a personal preference, there would be no end to it. If a glyph has a real bug (for example, the one missing pixel you reported in U+010A89--that is a real bug) that is a different situation.


Paul Hardy

Paul Hardy <unifoundry>
Group administrator
Tue 19 Dec 2017 06:46:53 PM UTC, comment #4: 

I know that Latin’s width is not hard-coded. I was making an analogy: just as Latin’s width is not hard-coded, Indic’s width should not be hard-coded.

I don’t understand your second paragraph. I did not suggest making all of Indic single-width or making all of Unifont double-width. I suggested keeping Indic as mostly double-width, but without as much hard-coding, and with some single-width exceptions.

I also did not suggest adding a GPOS table. You probably have the same objection to a GSUB table, though, but I don’t see it as a slippery slope. All the stubbed GSUB table would do is make OpenType renderers display Unifont the same way TrueType renderers do, effectively disabling OpenType. If there has been no flood of bug reports requesting OpenType support, I don’t think this would be the catalyst. It would be so antithetical to Unifont’s 1∶1 character–glyph ratio, it just doesn’t seem plausible.

Postponement makes sense. I note that Indic changes have been postponed since August and I feel like we’re going in circles. After the coming release, I would find it helpful if you could provide a specific concrete example of a problem my Indic suggestions would cause.

David Corbett <dscorbett>
Mon 18 Dec 2017 03:42:43 AM UTC, comment #3: 

David,

Scripts like Latin and Cyrillic are not hard-coded to be any width. Most of those glyphs are single-width, but someone is able to render a glyph as double-width if they feel they must.

The Unicode Standard does not specify widths for charcell-type fonts. It does include suggestions of glyphs being normal or wide, but these are not requirements in the Unicode Standard. And the Indic glyphs are pretty much all marked as being normal width, which would mean that glyphs have to fit within 8 pixels in Unifont (including, for example, Devanagari consonants and any combining marks)...unless glyphs in Unifont became a lot larger or Unifont became a vector font. Those are the two cases where making all the Indic fonts single-width would be workable. Unifont right now is over 17 Megabytes; I do not want to make any change such as doubling glyph dimensions that would substantially increase the font size. Unifont also is not going to become a vector font.

The font contains full spacing information given the horizontal-only spacing of Unifont glyphs. If I add a GPOS table, then Unifont is no longer just a plain TrueType font; it becomes an OpenType font. And once that happens, I am concerned it could open the floodgates and bugs will start getting filed against Unifont's OpenType conformance (of which, currently, there is none whatsoever apart from its TrueType conformance). That would be going too far. I think it would be a case of the tail wagging the dog.

I am going to hold off for now on making any changes to any Indic fonts, rather than accepting or rejecting anything off the bat. I do need to put together another release, so postponing changes to the Indic fonts will allow me to proceed with other requested changes and come back to this afterwards.

And I am not opposed to adjusting scripts such as Devanagari and Bengali; I just do not want to discard the basic strokes that those fonts contain. They were some of the first contributions to Unifont, and I think they are beautifully drawn.

Unifont is the fallback font for FontForge, so that is one case where at least having a glyph that looks acceptable as a standalone glyph actually serves a purpose. But I agree that Devanagari, Bengali, and possibly other scripts do not render well with combining characters in the current form. Bengali has even more contractions than Hindi though, so I cannot see someone who knows Bengali (or Hindi, Sanskrit, etc.) wanting to use Unifont for any document or web page. That is another case where Unifont could potentially become more complicated by containing glyph substitutions for such contractions, but I think it is going too far. I would like to try not to make Unifont any more complicated than it has become so far.


Paul Hardy

Paul Hardy <unifoundry>
Group administrator
Mon 18 Dec 2017 01:08:07 AM UTC, comment #2: 

Even in plane 0, not all scripts have a single width in Unifont: examples include Latin, Cyrillic, and Arabic. Hard-coding some ranges to be double-width actually makes it harder to contribute for those with artistic talent who can’t read programs, because they may assume they may draw glyphs of whatever width is appropriate, not realizing their design choices are overridden.

This is a real problem in the current version of the font. The pixels of U+0ABD GUJARATI SIGN AVAGRAHA are all in the left half of the double-width glyph: it was obviously drawn with the intent that it be single-width. The artist might have drawn it differently had they known about unibmp2hex, but that would have required reading a computer program.

It is easy for you, qua maintainer, to work around that by (say) recentering the glyph; but in some cases, there is no easy work-around. U+0F05 TIBETAN MARK CLOSING YIG MGO SGAB MA should be single-width to join with a preceding U+0F04 and following U+0F05 without looking stretched. However, unibmp2hex makes it double-width. There are many such cases where forcing double-width is wrong or at least counterproductive.

I believe I have a solution. Hard-code characters to be double-width based not just on their scripts but on their Unicode character properties, and provide a simple format like combining.txt for overriding the widths. (I suggest a width override file just because there are always exceptions when it comes to Unicode.) The three kinds of character we can be pretty sure are double-width are fullwidth CJK characters, Indic base glyphs that can take left-side matras, and marks that go on those Indic base glyphs. Here is a rough draft list:

https://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%5B%5B%3Aea%3DF%3A%5D%5B%3Aea%3DW%3A%5D-%5B%5B%3AEmoji_Presentation%3A%5D%5Cu1F93B%5Cu1F946%5D%5D+%5B%5B%3AInSC%3DConsonant%3A%5D%5B%3AInSC%3DConsonant_Head_Letter%3A%5D%5B%3AInSC%3DVowel_Independent%3A%5D%5B%3AInSC%3DVowel_Dependent%3A%5D+%26+%5B%5B%3ADeva%3A%5D%5B%3ABeng%3A%5D%5B%3AGuru%3A%5D%5B%3AGujr%3A%5D%5B%3AOrya%3A%5D%5B%3ATaml%3A%5D%5B%3AMlym%3A%5D%5B%3ASinh%3A%5D%5B%3AMymr%3A%5D%5B%3AKhmr%3A%5D%5B%3ABugi%3A%5D%5B%3ALana%3A%5D%5B%3ABali%3A%5D%5B%3ASund%3A%5D%5B%3ALepc%3A%5D%5B%3AJava%3A%5D%5B%3ACham%3A%5D%5B%3AMtei%3A%5D%5B%3AKthi%3A%5D%5B%3ACakm%3A%5D%5B%3AShrd%3A%5D%5B%3ASind%3A%5D%5B%3AGran%3A%5D%5B%3ANewa%3A%5D%5B%3ATirh%3A%5D%5B%3ASidd%3A%5D%5B%3ATakr%3A%5D%5B%3AAhom%3A%5D%5D%5D%5D&abb=on&g=&i=

Unifont will never be a beautiful Indic font, of course, but if you implement my suggestions, Indic will both be legible and not appear to have extraneous U+0020s scattered throughout text. I don’t think this is difficult and it would make the font so much better. I am not just speaking for myself when I say it is less difficult: it would be easier for any artist to contribute a glyph and have that glyph appear in the font as intended.




Unifont’s Devanagari looks good in a code chart, but in text, the marks overlap the bases. I think it is possible to fix it with minimal disruption to the original maintainers’ legacy. By making the left and right matras thinner, shifting the top matras higher, shrinking the candrabindus, and shaving a row or two off the stem of most consonants, it should look okay.

This goes for Bengali, Newa, and Tirhuta too, more or less.

The suggestions I made for Tibetan were not to redraw everything, but to change the block to be single-width. The glyphs would mostly look the same but include less white space.




OpenType is too complex for Unifont, I agree. It may yet be worth making it into an OpenType font. (I know this sounds contradictory.)

When HarfBuzz processes Unifont, it treats it not as a TrueType font, but as an OpenType font that happens to be missing all its OpenType tables. It therefore does Indic reordering, which breaks Unifont, since its vowel signs work by overstriking. The only way to convince HarfBuzz not to second-guess the font is to add a stubbed OpenType GSUB table with the 'DFLT' script.

Similarly, some Thai clusters (e.g. ฟิ) are garbled in Uniscribe. I suspect a stubbed table would work around that.

Adding this table is a ≅1-line change to the SFD preamble hard-coded in hex2sfd. I think it is worth that to make Unifont work with OpenType shapers.

David Corbett <dscorbett>
Sun 10 Dec 2017 05:05:12 AM UTC, comment #1: 


> ...because basic TrueType only contains information about vertical positioning.


I meant to say that basic TrueType only contains information about horizontal positioning.


Paul Hardy

Paul Hardy <unifoundry>
Group administrator
Sun 10 Dec 2017 02:01:59 AM UTC, original submission:  


I learned that HarfBuzz is what renders glyphs in Pango, which is used to display text in GNOME and other environments.  The HarfBuzz maintainer has fixed the situation with double diacritics; see

https://bugzilla.gnome.org/show_bug.cgi?id=787284

I also learned from that exchange that HarfBuzz handles combining characters by its own set of heuristics if a font does not contain the GPOS OpenType extension to TrueType.  This is actually necessary when it comes to vertical stacking of combining characters, because basic TrueType only contains information about vertical positioning.

With that being the case, I do not think it is worthwhile to specify combining characters with anything other than single- or double-width.  I will leave the Russian glyphs that were assigned a 12 pixel width the way they are though; there is no solution for them that is a good solution.


In addition to being able to draw glyphs at the correct aspect ratio with the unibmp2hex program, one other goal I had was to make it easy for others to draw glyphs who had artistic talent that I lack, but did not know how to read computer programs.  To that end, I kept the precedent that Unifont established of applying a constant width to most scripts in the Basic Multilingual Plane.  In general, Latin glyphs were single width, and Indic and CJK scripts were double width.  The CJK scripts carry this to an extreme by even containing double-width Latin letters for consistency.  An exception was symbol scripts.  I have thought about this a great deal, and realized that if I start adding exceptions to script widths in unibmp2hex, it will require that someone know how to understand computer programs to know what the width of each individual glyph in a script will be.  So I would like to stick with Unifont’s original design of a single width for most scripts, at least the Plane 0 scripts.

I also do not wish to throw away the hard work of Unifont pioneers, who created glyphs for scripts such as Tibetan and Devanagari before the graphics tools that I developed existed.  The Devanagari script with its serifs might not match the sans serif nature of most of Unifont, but it is also one of the best looking scripts in all of Unifont—if not the best looking script.  There is an issue with its combining characters overlapping consonants that it would combine with.  I would modify the glyphs that exist though, not replace them wholesale.

Unifont was originally just designed to produce a BDF font.  Many BDF rendering engines did not handle combining characters at all.  So the original form of Devanagari and similar scripts was something that would render glyphs that someone could understand, even though they would not appear in the combined way they would be written.  The BDF font to this day does not do anything special for combining characters.  That is something that should probably change.  It would take testing in various windowing systems that I have not taken the time to do.

On the TrueType font, I actually do not know why anybody would use Unifont for Indic scripts if an alternative full OpenType font exists.  And for common Indic scripts, there are plenty of free OpenType fonts available.  For Vedic Sanskrit, there is nothing that compares in completeness to the font from Omkarananda Ashram.  Indic fonts for the most part have consonant contractions that every native speaker will expect, but that Unifont’s simple plain TrueType format does not support.  So I think tinkering with them quickly reaches a point of diminishing returns.  I think it would be better to start considering contributions to a free OpenType font that does support such contractions.

Another emerging issue is that FontForge developers have begun work on TruFont, an alternative to FontForge:

http://trufont.github.io/

Dave Crossland, who has headed FontForge development for several years, has shifted his focus to supporting TruFont development.  This article with follow-on comments provides a lot of insight into the reasons for developers moving from FontForge maintenance to TruFont development:

http://typedrawers.com/discussion/1165/introducing-trufont-world-s-first-native-ufo3-font-editor-fully-cross-platform

So I have in the back of my mind been thinking of a future with an alternative to FontForge.  At some point in the future, it is possible that FontForge maintenance will dwindle to an unworkable state and that will necessitate a change.


As a full OpenType vector-based, nicer looking alternative to Unifont, GNU FreeFont would be a natural choice for contributions.  I have been too busy with Unifont to approach the FreeFont maintainers about contributing to their efforts so far.


In closing, I do not think “but this is the way we’ve always done it” is reasonable justification for never changing.  With Unifont though, its origins as a BDF font and its plain TrueType format do not support features that many of the world’s scripts require.  I think it serves a valuable place as a fallback font, a font of last resort.  I do not think it is worthwhile to take such a simple font format and add complications like GPOS table support or other OpenType extensions.

What are the thoughts of others on these topics?


Paul Hardy <unifoundry>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by unifoundry (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-07-30 unifoundry StatusIn Progress None
        Open/ClosedOpen Closed
        Discussion LockNone Locked

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code