bugFree UCS Outline Fonts - Bugs: bug #34190, Persian characters in FreeSerif...

 
 

bug #34190: Persian characters in FreeSerif not connecting right

Submitter:  sina <sineau>
Submitted:  Fri 02 Sep 2011 06:18:11 AM UTC
   
 
Category:  character range Severity:  3 - Normal
Item Group:  glyph(s) incorrect Status:  Fixed
Privacy:  Public Assigned to:  Stevan_White
Open/Closed:  Closed Release:  * 2010-09-19
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 04 Apr 2012 12:32:59 PM UTC, comment #35: 

Oh wow, Masoud!  Thanks!

Please let me know of any problems you see (I am also quite willing to consider your corrections to the current SVN files!)

I know the letters are not of professional quality.  Let me say what the goals were:

1) correctness for several languages that use the Arabic script
2) style appropriate for daily publications, e.g. newspapers, reports, etc.  That is, easy to read, similar in style to compact faces as used in newsprint.
3) weight and metrics similar to the existing FreeSerif font, so that text of various languages can be intermixed.  (Making the basic letters fit in the existing line was a major problem.)

The letters were roughly based on the previous set in FreeSerif which were made by a native speaker, but had many technical problems (as you noticed).  Most of the changes were made by me.  I referred to a couple of calligraphy books and some paper newsprint.  Then some changes were made based on comments by native speakers.

But again, I can't claim to have an eye for Arabic typography: it is strange to me -- although I see patterns, I don't know what looks "right".

Let us know!

Steve White <Stevan_White>
Group administrator
Wed 04 Apr 2012 10:27:03 AM UTC, comment #34: 

Sorry, I was wrong. The connection problems are solved in the latest snapshot. I can see many other improvements as well. Now I am starting to love this font. Thank you.

Masoud Abkenar <amp3030>
Sun 01 Apr 2012 10:58:19 PM UTC, comment #33: 

Hi Steve!

I got the latest revision (2195) and compiled it. It looks like that Arabic has improved a lot (I see some blue marks in FontForge that seem to be some kerning features etc.) But for Persian, apart from addition of Persian digits (copied from Arabic digits which have almost the same shape), I couldn't find any improvement. None of the connection problems of Persian characters has been fixed. Furthermore, I can still see some general problems affecting both Persian and Arabic, that I have fixed in my modifications.

I think I can copy some of my modifications to the new font. But I don't know if I should use some version controlling system, or I can change the latest revision and send it back to you. Which way is better?

Cheers,
Masoud.

Masoud Abkenar <amp3030>
Sun 01 Apr 2012 09:20:45 PM UTC, comment #32: 

Hi Masoud!

I think your alterations were made against the 2010 release of FreeFont.  As I mentioned in the Ubuntu report, the entire range has been reworked, replaced, greatly expanded, and checked by native speakers, since the 2010 release.

It would be good to have a list of the particular alterations you made, but perhaps it would be better for you to look at the development version of the font first.  We would be very interested to hear what you think!

I'll give you a link to some snapshots, but you can also download the SFD files directly here:
http://svn.savannah.gnu.org/viewvc/trunk/freefont/?root=freefont

Cheers!

Steve White <Stevan_White>
Group administrator
Sun 01 Apr 2012 07:16:39 PM UTC, comment #31: 

Hi,
I'm coming from Launchpad:
https://bugs.launchpad.net/ubuntu/+source/ttf-freefont/+bug/730770

I have worked on Persian/Arabic range of FreeSerif. I will attach my modified version of FreeSerif here.

If necessary, I can make a list of all glyphs that I have changed/added.

Masoud Abkenar <amp3030>
Fri 04 Nov 2011 03:27:38 PM UTC, comment #30: 

About the FreeMono: in several cases, a bug was present in FreeSerif but not visible, because of a style choice on the part of the original submitter of the letters was carried over by me in several cases.  The substitution tables were the same in the two fonts, but the bug was only visible in monospace.

Steve White <Stevan_White>
Group administrator
Fri 04 Nov 2011 03:21:20 PM UTC, comment #29: 

Great, Emmanuel!

I fixed most of these.  In SVN.

I'm surprised you picked up on the FreeMono so quickly.  I hadn't told anybody, I think, and I wasn't sure it was a good idea.  But this very exercise illustrated one way in which it can be a good idea: the different connecting forms are necessarily more distinct in monospace.  So for text where the proper form is very important, such as code, the monospace can be helpful in identifying mistakes.

Most of the problems were due to my reluctance to make new character names for those not hard-coded into FontForge.  General principle: it's good to give human-readable names to characters, ESPECIALLY when they're referred to from elsewhere, such as in lookup tables.

As to your question about making all the letters join correctly: this is a rather big project, and how to proceed is unclear to me.
If one were to do it, it might be best to make a script that generates the glyphs somehow.  On the other hand, many characters never appear in certain positions, or perhaps are not traditionally joined like similiarly-shaped glyphs -- which, I don't know. 
For now, the two Arabic Presentation Forms ranges are there because previous standards had considered their sets of intermediate forms most important.  So until I get specific requests, I'll stick with the forms in those ranges.


Only FreeMono
624 final != (correct)FE86 *TABLE BUG -- fixed (both mono and serif)
634 initial != (correct)FEB7 * TABLE BUG -- fixed. mystery.
647 isolated != FEE9 diffrent sizes * GOOD POINT  -- not yet
64C not impl. != (correct)FE72 * -- implemented
652 not impl. != (correct)FE7E * -- implemented
691 final != (correct)FB8D * TABLE BUG -- fixed (both mono and serif)
6BA final != (correct)FB9F TABLE BUG -- fixed (in both mono and serif)
6CC isolated != (unimpl)FBFC (compatibility characters for other positional forms are implemented) * -- implemented
6D3 isolated != (correct)FBB0 * GLYPH BUG -- fixed

(You missed 622 final != FE81  -- fixed)

In both FreeMono and FreeSerif Bold
6AD isolated != FBD3 (dots placed differently, don’t know which is better)
Yeah, well, I need more info on this.
 The placement is due to space constraints, and the desire to draw complete new letter forms as little as possible.  But I don't know the "correct" form.

In all fonts (Mono, Serif and Serif Bold)
671 final != (correct)FB51 * TABLE BUG -- fixed
6AD initial != FBD5 } 6AD initial & 6AD
6AD final != FBD4 } final are swapped * GLYPH+TABLE BUGS -- fixed
6BB medial != (correct)FBA3 * TABLE BUG -- fixed
6C1 isolated (incorrect) != (correct)FBA6 * MULIPLE TABLE,GLYPH BUGS -- fixed NOTE: the unicode sample glyphs of the two characters you mention are not the same
6D0 final != (correct)FBE5 * TABLE BUG -- fixed

Thanks bunches!

Steve White <Stevan_White>
Group administrator
Fri 04 Nov 2011 09:38:39 AM UTC, comment #28: 

Hi there !

I have updated my page with a version that has a dropdown list at the top to choose the font to test, and so I checked the new range in FreeMono. I also cross-checked and a few of them are also in FreeSerif.

Only FreeMono
624 final != (correct)FE86
634 initial != (correct)FEB7
647 isolated != FEE9 diffrent sizes
64C not impl. != (correct)FE72
652 not impl. != (correct)FE7E
691 final != (correct)FB8D
6BA final != (correct)FB9F
6CC isolated != (unimpl)FBFC (compatibility characters for other positional forms are implemented)
6D3 isolated != (correct)FBB0

In both FreeMono and FreeSerif Bold
6AD isolated != FBD3 (dots placed differently, don’t know which is better)

In all fonts (Mono, Serif and Serif Bold)
671 final != (correct)FB51
6AD initial != FBD5 } 6AD initial & 6AD
6AD final != FBD4   } final are swapped
6BB medial != (correct)FBA3
6C1 isolated (incorrect) != (correct)FBA6
6D0 final != (correct)FBE5

For checking that every character in each joining group indeed have similar positional forms, I made another test page for all Unicode characters that take positional forms, grouped by joining group. For detailed explanation about how the characters are supposed to join, refer to the section on the Arabic script in chapter 8 of the Unicode Standard
    Steve, do you plan to make all Arabic characters join correctly for the next release (this is not a request)?

I really appreciate your efforts!

(file #24299, file #24300)

Emmanuel Vallois <emmanuel_vallois>
Sat 29 Oct 2011 12:18:40 PM UTC, comment #27: 

Hi Sineau!

1) Be aware that features such as the FC6A ligature can be applied on a langage-by-language basis.

I will mark FC6A etc as Arabic-only, for now: in SVN.

It would be helpful to know, which ligatures are most important for Persian.  (Before you try to do it yourself, you might look for a web page where this is already discussed.  I'm not sure what I'm looking for...)

2) What do you mean "beh won't be levelled"?  You mean, the beh will take the normal medial form, starting on the baseline, making a spike, then back down to the baseline?

3) As to how to do it...  Just take some notes and try to arrange them so I have a chance to understand.  (And open a separate bug report). 

Really keep it to just the top few serious problems -- certainly under 10, so I have a chance to deal with them.  Then we'll do another iteration.

My experience is the improvement in each iteration is very significant; after 3 or so such iterations, we'll be down to very minor details.

Cheers!

Steve White <Stevan_White>
Group administrator
Sat 29 Oct 2011 05:15:21 AM UTC, comment #26: 

Emmanuel,
Thanks. Both version helped me a lot in detecting errors.

Steve,
I'll do that. But I'm not sure how to do it, since Persian is a little different from Arabic in what considers readable. For example while FC6A considers readable in Arabic, I should mark it as "Annoying" for a Persian user. In a persian standard handwriting "Beh" and "Reh" will connect normally and "Beh" won't be levelled.
There are some other instances as well. Note that I tried to put my subjective aesthetics aside and just focus on readablity.
So how this is done?

sina <sineau>
Fri 28 Oct 2011 10:41:12 PM UTC, comment #25: 

I’m sorry, there was a bug in the script I use to generate the page page, I should not work on such things at 2am! The code pints of compatibility characters were wrong, and some other details.

Here is the corrected version.

(file #24235)

Emmanuel Vallois <emmanuel_vallois>
Fri 28 Oct 2011 08:20:42 PM UTC, comment #24: 

Sineau,

I usually close bug reports with a new release, (which I hope will come soon).

Yes, please post complaints about different problems separately.

Please, if you can possibly spare the time, look at some text samples and report to me any problems you find.

First look for outright errors -- stuff that any native reader would consider an incorrect representation of the text.

After that, sort of a sorting by degrees: "very distracting", then "annoying", then "needing adjustment".

I really want to polish this up!

Steve White <Stevan_White>
Group administrator
Fri 28 Oct 2011 07:47:03 PM UTC, comment #23: 

At last I passed the cache hell and FontForge bugs in ubuntu package. And I can confidently state that the bug's fixed! 
I think some kernings should be fixed maybe, but that's another bug.
Thank you Steve. Persian pages are a lot better now.

PS: should we close the bug?

sina <sineau>
Fri 28 Oct 2011 05:51:53 PM UTC, comment #22: 

Hi Sineau!

The main version is in trunk, but the font files are dated very recently, some of them as of today! 
   http://svn.savannah.gnu.org/viewvc/trunk/freefont/sfd/?root=freefont
What are you looking at?

Mind you, this is a work-in-progress.  It is likely there are many outright mistakes!

Remember, once you build it, the tricky bit is getting rid of older cached versions.
I know of 3 different places

1) system font cache
2) ~/.fonconfig/
3) your browser's cache, for web fonts (such as downloaded from the FreeFont site.)

I sent you instructions before on how to remove older versions in (1) and (2).
If you're using a browser to view things, you'll also want to clear its cache.

Cheers!

Steve White <Stevan_White>
Group administrator
Fri 28 Oct 2011 03:47:35 PM UTC, comment #21: 

I had a look and the one in trunk is dated 2010-09-19. I used svn to get it. Is it correct?

sina <sineau>
Fri 28 Oct 2011 03:35:44 PM UTC, comment #20: 

Hi again Steve,
I'm going to have a go on this issue again. Should I use the sfd files on trunk?

sina <sineau>
Fri 28 Oct 2011 12:02:09 PM UTC, comment #19: 

Wow Emmanuel!
That's a great page!
All these goofs were mine, and all easily fixed.
I also copied the changes to bold.  All in current SVN.
Please let me know of any further errors!

Steve White <Stevan_White>
Group administrator
Thu 27 Oct 2011 11:56:54 PM UTC, comment #18: 

Hi all !

To fully test positional forms and that compatibility characters render the same as the normal one, I have made an HTML file to display all Arabic characters of Unicode.

I hope it wil be useful, with it I already listed the following problems:
- FBD7-FBD8 letter u incorrect, should be like 6C7
- 67A incorrectly has the same final/middle/initial forms as 679
- 6B7 final
- 6BB medial
- 6D0 final
- 6D3 final vs FBB1

(file #24213)

Emmanuel Vallois <emmanuel_vallois>
Fri 30 Sep 2011 06:26:24 PM UTC, comment #17: 

Hi again Sineau!

Still lots more changes in SVN, many toward other letter-connecting issues I found.

There is a nice test of letter positions here:
http://en.wikipedia.org/wiki/Arabic_alphabet#Table_of_basic_letters

Cheers!

Steve White <Stevan_White>
Group administrator
Sat 24 Sep 2011 04:52:07 PM UTC, comment #16: 

Hi, Sineau!

The messages you refer to age coming out of some versions of FontForge. There has been some discussion about this on the mailing lists.
It is a problem with FontForge, not FreeFont, and it doesn't affect the performance of the TrueType fonts.

Do you mean by "the same errors", the same letter placement problems you reported before, please send me a screenshot.
My guess is, you're not seeing the newer versions.

It is very likely that you have multiple versions of the font installed, or caches of the fonts.

Do a
    locate FreeSerif
to see them.

However removing the older copies isn't usually enough. 

First as root, then as your normal user, do
     fc-cache -rv
to re-generate the font cache.

(I know it's a mess, but I didn't invent this system!)

You may want to simply (back up and) delete older copies.

If you remove the system-wide copies, you should re-start X windows.  For local copies, you only need to re-start applications that use them.

Steve White <Stevan_White>
Group administrator
Sat 24 Sep 2011 01:34:45 PM UTC, comment #15: 

Sorry for delay but at last I did it today. First of all there's an error when I generate the fonts with Font Forge that goes like this: "The font contains errors.
Contains anchor points for some, but not all, classes in a subtable." And when I click on review it lists some of the characters with this error. But then I go on and save the font anyway. I installed the FreeSerif I generated myself, and it's still got the same errors.  Maybe I didn't install it correctly (I did it via Font viewer of ubuntu).

sina <sineau>
Thu 22 Sep 2011 04:04:16 PM UTC, comment #14: 

Sineau,

Just to let you know... There are now a lot of changes to FreeSerif in SVN.  Except for most of the ligatures in Arabic Presentation Forms-B, the Arabic range is now in principle complete.  Furthermore, having done that, I'm working on porting the range to the bold face.

I have pointed it at web pages in Arabic, Farsi, Urdu, etc, and done some visual checks.  Also, I put effort into making  Koran passages to display diacritcals basically.

I would say, it's at least close to what other fonts display. 

There are a lot of remaining issues with letter shapes and spacing, and for all I know, some may be serious.

Looking forward to your assessment!

Steve White <Stevan_White>
Group administrator
Wed 14 Sep 2011 06:20:51 PM UTC, comment #13: 

Sorry Steve, I was heavily busy in the past couple of days, I downloaded last revisions but didn't have time to do anything, I will do it this weekend. That's Friday.

sina <sineau>
Wed 14 Sep 2011 09:49:46 AM UTC, comment #12: 

Hi again Sineau,

I made a lot of changes to the Arabic range lately, many having to do with connecting letters, but also just overall clean-up of the letters.

The changes are in SVN.

It would be great to hear your impressions!

Steve White <Stevan_White>
Group administrator
Mon 12 Sep 2011 01:12:52 PM UTC, comment #11: 

Hi Sineau,

First, note that we've changed the version control system from CVS to SVN.  If you're using Linux, you now get FreeFont using
   svn co svn://svn.savannah.gnu.org/freefont/modulename

In my copy, the letter Gaf seems to be working correctly.
Could you send me a screen shot of what you're seeing?

The letter Keheh was not supported at all in FreeFont, neither in U+FB8E ~ U+FB91 nor in U+06A9.  So that's a different kind of problem--whatever you were seeing, was letters from some different font.

I have now added Keheh with placed forms, and lookup tables for that.

Have a look!

Steve White <Stevan_White>
Group administrator
Wed 07 Sep 2011 05:47:01 AM UTC, comment #10: 

Here گ in همگی and همدیگر got a problem. The other problem is with another character ک (range U+FB8E ~ U+FB91) which has the same problem in connecting to other words.

sina <sineau>
Tue 06 Sep 2011 10:55:39 AM UTC, comment #9: 

Ah of course.  I'll attach another picture of Arabic and Farsi samples.

If building is a real problem for you, I could build a special copy.  (Better if you do it yourself though.)

If you see other problems, please do post separate reports!

Thanks!


Steve White <Stevan_White>
Group administrator
Tue 06 Sep 2011 07:14:02 AM UTC, comment #8: 

Stevan,
The statement is correct but I couldn't confirm my bug is solved since I haven't had the time to build it.

There is also two new problems which caught my eye and I think I should file two new bugs.

sina <sineau>
Mon 05 Sep 2011 06:38:30 PM UTC, comment #7: 

Hey Sineau,

Could I get you to do us a favor?

Have a look at the range sample page for the Arabic alphabet.
I added a sample for Farsi.

http://www.gnu.org/software/freefont/ranges/Arabic.html

Steve White <Stevan_White>
Group administrator
Mon 05 Sep 2011 10:02:14 AM UTC, comment #6: 

Hi Sineau,

Great!

The latest versions are available by CVS

cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/sources/freefont co freefont

You will need FontForge installed in order to build the font.

Please note that the Latin range in CVS is messed up: the kerning got destroyed some time ago and I haven't gotten around to fixing it.

Cheers!

Steve White <Stevan_White>
Group administrator
Mon 05 Sep 2011 09:28:11 AM UTC, comment #5: 

Yeah that looks okay. How can I get this update? Is there a launchpad ppa available or I should use svn?

sina <sineau>
Mon 05 Sep 2011 09:21:15 AM UTC, comment #4: 

Sorry, Sineau,

The Sinhala was a brain-slip.  I had just been discussing the Sinhala range with somebody else.

I opened your (OpenOffice) text file with LibreOffice Writer, on my system where the latest update to FreeSerif is installed.

The letters you mentioned seem to be connecting.

Let me know how it looks to you.  See attachment
FreeSerif-corrected-Persian.png

To get it quite right, I had to increase the line spacing of the last line, otherwise the deep descender of the tcheh would be cur off.


Steve White <Stevan_White>
Group administrator
Mon 05 Sep 2011 08:32:04 AM UTC, comment #3: 

Stevan,
I'm not sure what do you mean by Sinhala. But as I told you this is Persian and Unicode characters are categorised under Arabic.

Here are those Unicodes. Please note there are more than one code for each character: the first is for individual form and the others are contextual forms.
for پ: U+FB56, U+FB57, U+FB58 & U+FB59

for چ: U+FB7A, U+FB7B, U+FB7C & U+FB7D

for گ: U+FB92, U+FB93, U+FB94 & U+FB95

To compare use a font which is designed for Farsi (persian):

http://farsiweb.ir/wiki/Products/PersianFonts



(file #23940)

sina <sineau>
Fri 02 Sep 2011 07:30:54 AM UTC, comment #2: 

Hi Sineau!  Welcome!

Please send me Unicode text of some Sinhala that displays the problem you are referring to.  Especially, send me the text of any screenshots you posted.

Cheers!

Steve White <Stevan_White>
Group administrator
Fri 02 Sep 2011 06:22:18 AM UTC, comment #1: 

Stevan, I think I just needed a push to do this.

I am willing to help on this issue. I think it best resolves if you know the language. However I don't know how to do it and I'm not sure if that's okay with you. If you are, give me some basic instructions and I'll find my way.

sina <sineau>
Fri 02 Sep 2011 06:18:11 AM UTC, original submission:  

There are certain characters in Arabic range which are exclusive to Persian (پ ژ گ چ) and not used in Arabic writing. Contextual form for these characters are not correct, and the individual form prints instead.


more info: https://secure.wikimedia.org/wikipedia/en/wiki/Arabic_writing#Table_of_basic_letters

https://secure.wikimedia.org/wikipedia/en/wiki/Perso-Arabic_script#Letters

https://secure.wikimedia.org/wikipedia/en/wiki/Perso-Arabic_script#Changes_from_the_Arabic_writing_system (item 5)


sina <sineau>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #25542:  FreeSerifFa11.ttf added by amp3030 (1MiB - application/x-font-ttf - FreeSerifFa: modified FreeSerif with corrected/enhanced Arabic/Persian glyphs.)
file #23940:  Persian characters.odt added by sineau (13KiB - application/vnd.oasis.opendocument.text)
file #23922:  freeserif-persian.png added by sineau (14KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by amp3030 (Posted a comment)
  • -email is unavailable- added by emmanuel_vallois (Updated the item)
  • -email is unavailable- added by Stevan_White (Posted a comment)
  • -email is unavailable- added by sineau (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 14 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-05-03 Stevan_White StatusFix posted Fixed
        Open/ClosedOpen Closed
    2012-04-01 amp3030 Attached File- Added FreeSerifFa11.ttf, #25542
    2011-11-04 emmanuel_vallois Attached File- Added Arabic test page.html, #24299
        Attached File- Added Unicode joining test page.html, #24300
    2011-10-28 emmanuel_vallois Attached File- Added Arabic test page.html, #24235
    2011-10-27 emmanuel_vallois Attached File- Added Arabic testing page.html, #24213
    2011-09-06 Stevan_White Attached File- Added FreeSerif-Arabic-Persian-samples.png, #23952
    2011-09-05 Stevan_White StatusProceeding Fix posted
    2011-09-05 Stevan_White Attached File- Added FreeSerif-corrected-Persian.png, #23942
    2011-09-05 sineau Attached File- Added Persian characters.odt, #23940
    2011-09-02 Stevan_White StatusNone Proceeding
        Assigned toNone Stevan_White
    2011-09-02 sineau Attached File- Added freeserif-persian.png, #23922

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code