bugGNU roff - Bugs: bug #62830, [PATCH] [grops] support CJK fonts...

 
 

bug #62830: [PATCH] [grops] support CJK fonts encoded in UTF16

Submitter:  TANAKA Takuji <takuji>
Submitted:  Thu 28 Jul 2022 12:56:27 PM UTC
   
 
Category:  Driver grops Severity:  1 - Wish
Item Group:  Feature change Status:  Need Info
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Open Planned Release:  1.24.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 01 Jan 2024 01:49:42 AM UTC, comment #13: 

Oliver posted the following to the bug-groff list; apparently the GNU Savannah web site was not permitting him to comment on this ticket, which is pretty weird.


Hi,

thank you very much for considering my previous comment on the CJK font naming issue.

It is perfectly fine with me to build the font names by terms of primary language, i.e. starting with C for Chinese, K for Korean, J for Japanese.

In my humble opinion and according to decades of personal experience in this area, a majority of documents in these languages will be composed and perused by users of one language alone, and just by following prior practice, using a language-based naming scheme will allow the majority of users to navigate in familiar waters.

The minory of users who --- from the beginning --- compose documents with the need for equal typeface across several languages will use those fonts which provide a full coverage of CJK characters, and for them, the first letter doesn't really matter. They'll pick the font first, then the language, in contrast to the users mentioned before.

Best regards,
Oliver.


G. Branden Robinson <gbranden>
Group administrator
Fri 29 Dec 2023 10:39:52 PM UTC, comment #12: 

Just a note to point out that gropdf (in the deri-gropdf-ng git branch) already handles CJK documents. The attached pdf is an example, you should see CJK in the pdf bookmarks as well.

(file #55489)

Deri James <deri>
Group Member
Fri 29 Dec 2023 07:50:57 PM UTC, comment #11: 

comment #9:

> It would be helpful to me if Messrs. Tanaka and Corff could
> reach a consensus on a schema for naming CJK font families.  :)


To assist in that, I'm adding Oliver to this bug's cc.  Oliver, do you have any response to comment #10 here?  (You can post a comment to the bug report without having to create a savannah account.)

Dave <barx>
Group Member
Fri 29 Dec 2023 01:56:37 PM UTC, comment #10: 

I updated my patch to adjust to the latest upstream source code.

> While Mincho and Gothic loosely compare to serif and sans serif styles
> I think the font family classification terminology should be language-agnostic


Thank Oliver for kind comments.
I agree that "M (stand for Mincho)",  "G (stand for Goshikku or Gothic)" are not langage-agnostic.
However CJK language users require language specific fonts for each language.
In fact, Adobe have provided free CJK fonts "Source Han" and prepared 4 kinds of localized font files
Simplified Chinese, Traditional Chinese, Japanese and Korean to support regional variations.

https://source.typekit.com/source-han-serif/

I think 8 kinds of definitions for CJK (4 resional variations and 2 styles) is minimum as basic fonts.
Since we need 4 resional variants, I do not think language-specific naming is unappropreate for standard name.

"Source Han" has localized font names and they are consistent of my proposal.

Source Han Sans Simplified Chinese    : 思源黑体 sīyuán hēitǐ     : CSH ; H stand for hēitǐ
Source Han Serif Simplified Chinese   : 思源宋体 sīyuán sòngtǐ    : CSS ; S stand for sòngtǐ

Source Han Sans Traditional Chinese   : 思源黑體 sīyuán hēitǐ     : CTH ; H stand for hēitǐ
Source Han Serif Traditional Chinese  : 思源宋體 sīyuán sòngtǐ    : CTS ; S stand for sòngtǐ

Source Han Sans Japanese              : 源ノ角ゴシック Gen-no-kaku-goshikku : JPG ; G stand for Goshikku
Source Han Serif Japanese             : 源ノ明朝       Gen-no-mincho        : JPM ; M stand for Mincho

Source Han Sans Korean                : 본고딕   Bon GoTik       : KRG ; G stand for Gotik
Source Han Serif Korean               : 본명조   Bon MyeongJo    : KRM ; M stand fo Myeongjo


If you still do not agree the langage-specific name,
langage-agnostic names based on English could be possible
for example CSSS, CSSR, CTSS, CTSR, JPSS, JPSR, KRSS, KRSR as followings:

Source Han Sans Simplified Chinese    :: CSSS  SS stand for Sans-serif
Source Han Serif Simplified Chinese   :: CSSR  SR stand for Serif

Source Han Sans Traditional Chinese   :: CTSS
Source Han Serif Traditional Chinese  :: CTSR

Source Han Sans Japanese              :: JPSS
Source Han Serif Japanese             :: JPSR

Source Han Sans Korean                :: KRSS
Source Han Serif Korean               :: KRSR

Actually, I still recommend the formaer langage-specific ones.
But I accept the latter langage-agnostic ones if you do not agree my opinion.

Best,

(file #55487)

TANAKA Takuji <takuji>
Thu 10 Aug 2023 01:27:44 PM UTC, comment #9: 

Quoting Oliver Corff from the mailing list post cited in comment #7:


I had a look at https://savannah.gnu.org/bugs/?62830 and Tanaka Takuji
mentions the work of UKAI Fumitoshi et al., defining M for Japanese
Mincho and G for Japanese Gothic style.

While Mincho and Gothic loosely compare to serif and sans serif styles
(let's say, Times Roman and Helvetica for the first idea), the problem
with those Japanese fonts which are publicly available or free (as in
free to use) is that they are heavily leaning towards Japanese usage of
the CJK character set. Many of these fonts are derived from earlier work
which covered only Japanese and still do not fully cover the CJK Unified
Character set.

I attached three screenshots of my character map, showing the region
where the characters grouped under the radical 人 (rén, "(hu)man") are
assembled. IPAMincho is visibly defective in its coverage, the bold sans
serif characters are substituted from the Droid Sans font.
WenQuanYi_Zen_Hei is a Chinese-Made sans serif CJK Unified Ideograph
font which covers everything but is a good approximation to "Gothic"
typeface, and ZYSong18030 is the same in Serif-style.

This unintentional mixup of character representations from different
fonts, until recently, was a headache when writing e.g. presentations
for a PR China audience using Microsoft's PowerPoint. At least on
machines until Windows 7 (don't know the current state) an
out-of-the-box slide full of Chinese text would show a horrible mix of
typeface because the "CJK" preset of the OS and of the Office software
suite used to be a Mincho variant, with the result that many frequent
CJK characters which are only used in Mainland China were displayed in a
different style. Setting the document default East Asian font to
something different would solve the problem.

A word on names:   The Japanese loanword "gothic" corresponds to what is
historically known as "blackletter". The Chinese term for this is simply
a translation: "hei" (black), as in "heiti" ("ti", pronounced "tee",
stands for "body"). The "serif" equivalent Mincho name implies the Ming
Dynasty, however the more common usage in China is Song (who were two
centuries before the Ming), universally known for their cultural
refinement, also expressed in calligraphy and typeface.  Ming (or
Japanese "Mincho", "-cho" [long vowel] simply means "dynasty") in Japan
or Song (in China) is the most commonly used typeface for books, serious
newspapers etc.

So I think the font family classification terminology should be
language-agnostic (i.e., avoiding terms like "Mincho" or "heiti" because
fonts explicitely labelled as such might be designed for the needs of
one particular language only (e.g., Chinese (traditional or
simplified?), Japanese (even here: old style vs. new style), Korean),
and by design, might be severely limited for any CJK display outside
that language.

Visible and fundamental differences between different CJK presentations
are shown on the Wikipedia page "List of CJK fonts":
https://en.wikipedia.org/wiki/List_of_CJK_fonts

Figure 1 shows the differences between different typeface styles (NOT
language-specific), while figure 2 shows the graphematical differences
of the same characters in different languages. Notable are columns 1 and
3. NB: This image is not composed of characters taken from a font but
was created as a SVG file.

Please note that CJK Unified Ideographs contain most but not all of the
language-specific differences in stroke order etc. So, for demanding
typographical work one will always use a font which is specifically
designed to match a certain style. For the vast majority of generic text
in modern everyday life (be the genre sciences, humanities, non-fiction,
fiction, newspapers) it is possible to use a single pan CJK font to
cover several CJK languages; think of a Japanese text book for Chinese
learners, of a Chinese-Japanese dictionary, of a Japanese analysis of a
Chinese newspaper article with portions of original text in quotes. Yet,
there are exceptional situations where the usage of a generic CJK font
is such a no-go that it has the potential to create a diplomatic
incident; simply because minutiae in stroke order have a high impact on
the (language/national) identity perception of the reader.

Unfortunately, I cannot say for certain what a typical standard font in
a typical off-the-shelf Japanese Linux version is. I'll have to download
a representative version and run it, at least in a virtual machine or
from a USB live stick, but I'll certainly explore that and share my
findings.


It would be helpful to me if Messrs. Tanaka and Corff could reach a consensus on a schema for naming CJK font families.  :)

G. Branden Robinson <gbranden>
Group administrator
Fri 05 May 2023 12:19:41 AM UTC, comment #8: 

Oliver Corff posted a lengthy commentary about CJK fonts: http://lists.gnu.org/r/groff/2023-05/msg00008.html

Dave <barx>
Group Member
Sat 15 Apr 2023 07:18:20 AM UTC, comment #7: 

I updated my patch.

1. Font description files

There is a precedent of font description file for Japanese support of groff by Japanese developers (Fumitoshi UKAI et al.)

https://answers.launchpad.net/ubuntu/+source/groff/1.18.1.1-12

They defined font description named "M", "G" for Japanese.
M : Japanese Mincho style
G : Japanese Gothic style

"M", "G" are possible candidates.
But I wonder if Chinese/Korean people might feel uncomfortable.
It is the reason that I proposed font description "JPM", "JPG" and CK fonts.


2. src/devices/grohtml/post-html.cpp:

2a & 2e. Encoding US-ASCII or UTF-8, -U option

I tried three step of option setting:
 -U0 : US-ASCII : use named character references or numerical character references
 -U1 : UTF-8 (partial) : use named character references for known characters, UTF-8 literals for unknown characters (default)
 -U2 : UTF-8 (full) : use UTF-8 literals


2b. `to_utf8_string`.

I have moved it to libgroff/font.cpp for trial.

2c. switching text styling properties.

I have removed the function from my patch.

2d. `to_unicode`.

I have renamed it to_unicode() to to_numerical_char_ref().


3. src/devices/grops/ps.cpp

3a. I have renamed is_utf16 to  is_utf16be

3b. I have replaced wchar_t by uint16_t.

3c. postscript name and encoding.

For CJK fonts, encoding is always explicitly shown in PostScript font name
by the structure of (Specific font name)-(style)(-(character set))-(encoding)-(direction).
For example:

/Ryumin-Light-Identity-H
/Ryumin-Light-UniJIS-UTF16-H
/Ryumin-Light-UniJIS-UTF8-H
/Ryumin-Light-EUC-H
/Ryumin-Light-RKSJ-H
/GothicBBB-Medium-Identity-H
/GothicBBB-Medium-UniJIS-UTF16-H
/GothicBBB-Medium-UniJIS-UTF8-H
/GothicBBB-Medium-EUC-H
/GothicBBB-Medium-RKSJ-H

This is a sample PostScript file:
https://github.com/t-tk/PostScript-CJK-samples/blob/master/box-multi.eps

Therefore, I think it is reasonable to get encoding information from PostScript font names.
I guess most of PostScript interpreters do so.


4. src/include/font.h, src/libs/libgroff/font.cpp

I removed "ENABLE_UCSRANGE" macro from my patch.


5. smoke tests.

I replaced UTF-8 literal by octal code expression.


(file #54631)

TANAKA Takuji <takuji>
Fri 31 Mar 2023 11:14:13 AM UTC, comment #6: 

Updating fields to reflect my intention to work on this once groff 1.23.0 is kicked out of the nest.

G. Branden Robinson <gbranden>
Group administrator
Sun 04 Dec 2022 12:23:26 AM UTC, comment #5: 

Thank you for the update.

I don't think I am going to have time to properly consider this patch in the depth it requires before the groff 1.23.0 release, which I am hoping to get at least a release candidate out for before the end of the calendar year.  Only 5 open Savannah tickets remain for that.

I would however like to take a fresh look at this issue (and Russian localization, bug #63076) early in the groff 1.24 development cycle.  How soon that begins will depend on how many urgent bug reports we get against 1.23.0.

On the bright side you can expect relatively few changes to occur between now and that might make your patches difficult to apply or maintain out of the tree.

Here are some thoughts I have for when I can return to this work (or for another groff developer to step up and consider discussing or addressing).

1. I was uncertain about the wisdom of shipping more font description files, but it's not like there isn't a precedent; except for the FreeEuro font, we don't ship any fonts proper--just descriptions of fonts that the user must obtain elsewhere.  So  shipping CSH, CSS, CTH, CTS, JPG, JPM, KOG, and KOM font descriptions for the "dvi", "html", "ps", and "utf8" output devices is not without precedent.

2. src/devices/grohtml/post-html.cpp:

2a. I wonder if defaulting to ASCII for the html output device is   necessary.  Apparently UTF-8 is overwhelmingly the encoding used by most web pages in the world.  https://w3techs.com/technologies/details/en-utf8

2b. The new `to_utf8_string` function might be better housed in libgroff or libdriver, if in fact there is not already a suitable function present in one of those libraries.  Another possibility is that there is some gnulib module we could use here, and not have to carry our own implementation at all.

2c. I am uneasy with switching text styling properties (bold, italic) off based on the groff font name in use.  I think it might be better to have a new font description file directive (see groff_font(5)) that tags a font as being unstyled.  Any font with this property would cause the disablement of bold and italic flags.

2d. Maybe the existing `to_unicode` function should be renamed; from the name along, it's not obvious how it is distinct from `to_utf8_string`.

2e. The `-U` option seems like a good idea, and perhaps is a flag letter we can re-use elsewhere in groff as we improve its Unicode support.

3. src/devices/grops/ps.cpp

3a. `is_utf16` should be renamed to reflect whether it uses UTF-16BE or UTF-16LE.

3b. I'm uneasy with the use of wchar_t.  I think maybe we want to use int32_t, or if that can't be assumed to be available in C++98 (check this), then we should have a type alias ("typedef" [sic]) and use an int, which must be at least 32 bits on any GNU system.

3c. Again we're inferring properties from font names, it looks like:


+  const char *psname = f->get_internal_name();
+
+  if (psname && strstr(psname,"-UTF16-")) {


And again I think I'd prefer a font description file property to communicate this information.

4. src/include/font.h, src/libs/libgroff/font.cpp

I wouldn't have a preprocessor-based feature gate like this "ENABLE_UCSRANGE" macro.  I would enable the feature for all builds.  This will give it exercise and help uncover bugs.

5. Thank you for the 'dvi' and 'ps' device smoke tests!  It might be necessary to rewrite the UTF-8-encoded literals for CJK glyphs as octal escape sequences to the printf(1) command for portability, sadly.  Surprising things go wrong on *BSD and macOS systems.

I emphasize that I don't require any changes to be made at this time to address the above points; they are for consideration and discussion by developers (including the patch author!) before any revision occurs.  I simply wanted to get these points down while they were fresh in my mind.

G. Branden Robinson <gbranden>
Group administrator
Wed 23 Nov 2022 01:28:17 AM UTC, comment #4: 

I updated my patch.

- sync with master of https://git.savannah.gnu.org/cgit/groff.git

I hope the maintainers to consider accepting my request.

Thanks,
Takuji TANAKA

(file #53990)

TANAKA Takuji <takuji>
Sat 03 Sep 2022 08:47:02 AM UTC, comment #3: 

I have updated my patch (cjk-ps-html_20220902.patch).

I followed up the latest repository.
I added CJK support for html, dvi, tty(utf8).
I added CJK font settings for html, dvi, tty(utf8).

In post-grohtml, I added option "-U" to enable output in charset="UTF-8".

I do not need to patch on grodvi for dvi output and patch on grotty for tty(utf8) output.

More test results are shown at https://github.com/t-tk/groff-CJK-font.

TANAKA Takuji <takuji>
Thu 04 Aug 2022 12:26:17 AM UTC, comment #2: 

An information-rich discussion thread is at < https://lists.gnu.org/archive/html/groff/2022-07/msg00003.html >.

G. Branden Robinson <gbranden>
Group administrator
Fri 29 Jul 2022 06:16:56 AM UTC, comment #1: 

(removed a duplicate attachment)

Dave <barx>
Group Member
Thu 28 Jul 2022 12:56:27 PM UTC, original submission:  

I wrote a set of patches in order to support CJK fonts in PostScript outputs.
Attached you will find the patches.

- It enables to support CJK non-embedded fonts in UTF16 encoding by grops.
- It enables to define charset by a range of Unicode code points in font definition.
  It is useful for CJK fonts because the number of glyphs with the same metric are very large.


More tests are shown in GitHub:
https://github.com/t-tk/groff-CJK-font


I hope the maintainers to consider accepting my request.

Thanks,
Takuji TANAKA

TANAKA Takuji <takuji>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #55489:  tanaka.pdf added by deri (33KiB - application/pdf)
file #55487:  cjk-ps-html_20231223.patch added by takuji (87KiB - application/octet-stream)
file #54631:  cjk-ps-html_20230415.patch added by takuji (87KiB - application/octet-stream)
file #53990:  cjk-ps-html_20221118.patch added by takuji (86KiB - application/octet-stream)
file #53654:  cjk-ps-html_20220902.patch added by takuji (86KiB - application/octet-stream - I have updated my patch. I added support CJK for html, dvi and tty (utf8).)
file #53481:  cjk-ps.patch added by takuji (33KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by deri (Updated the item)
  • -email is unavailable- added by barx (has knowledge of the issue)
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by takuji (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 16 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-01-01 gbranden Dependencies- bugs #65099 is dependent
    2023-12-29 deri Attached File- Added tanaka.pdf, #55489
    2023-12-29 barx Carbon-Copy- Added -email is unavailable-
    2023-12-29 takuji Attached File- Added cjk-ps-html_20231223.patch, #55487
    2023-08-10 gbranden StatusPostponed Need Info
    2023-04-15 takuji Attached File- Added cjk-ps-html_20230415.patch, #54631
    2023-03-31 gbranden Assigned toNone gbranden
        Planned ReleaseNone 1.24.0
    2023-03-31 gbranden StatusNone Postponed
    2022-11-23 takuji Attached File- Added cjk-ps-html_20221118.patch, #53990
    2022-09-03 barx Summary[grops] support CJK fonts encoded in UTF16 [PATCH] [grops] support CJK fonts encoded in UTF16
    2022-09-03 takuji Attached File- Added cjk-ps-html_20220902.patch, #53654
    2022-07-29 barx Severity3 - Normal 1 - Wish
    2022-07-29 barx Attached File#53482 Removed
    2022-07-28 takuji Attached File- Added cjk-ps.patch, #53482
    2022-07-28 takuji Attached File- Added cjk-ps.patch, #53481

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code