bugGNU roff - Bugs: bug #62830, [PATCH] [grops] support CJK fonts...

 
 

bug #62830: [PATCH] [grops] support CJK fonts encoded in UTF16

Submitter:  TANAKA Takuji <takuji>
Submitted:  Thu 28 Jul 2022 12:56:27 PM UTC
   
 
Category:  Driver grops Severity:  1 - Wish
Item Group:  Feature change Status:  Fixed
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  1.24.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 21 Nov 2024 07:52:51 PM UTC, comment #15: 


commit 78d1ef7c37edeb8cc39ae15bec3020eb31472bd8
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (1/6).

    * src/include/unicode.h (to_utf8_string): Declare new function.

    * src/libs/libgroff/unicode.cpp (to_utf8_string): New function converts
      input integer into UTF-8 sequence (or an HTML character entity in
      hexadecimal if the integer is out of range).

commit 6692471f0a31f00b052cec9b223ed963a130edc1
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (2/6).

    * src/include/font.h (class font): Declare private member variable
      `wch`, a pointer to an existing list type `font_char_metric`.  Declare
      private member function `get_font_wchar_metric()` to access it.

    * src/libs/libgroff/font.cpp (struct font_char_metric): Add members
      `next` (a pointer to the struct's own type) and `end_code` of type
      `int`.

      (glyph_to_ucs_codepoint): New function returns UCS code point from a
      (non-composite) `glyph` object, or -1 if invalid.

      (font::font): Constructor initializes `wch` member variable to null
      pointer.

      (font::~font): Destructor frees storage allocated in `font::load()`
      for `special_device_coding` member of `wcp` struct, and that of `wcp`
      itself.

      (font::contains): If `glyph_to_ucs_codepoint()` returns a valid value
      for the glyph, populate its wide character metrics and return true.

      (font::get_font_wchar_metric): New function obtains font metrics of
      input character by Unicode code point.

      (font::get_width, font::get_height, font::get_depth)
      (font::get_italic_correction, font::get_left_italic_correction)
      (font::get_subscript_correction, font::get_character_type)
      (font::get_code, font::get_special_device_encoding): If
      `glyph_to_ucs_codepoint()` returns a valid value for the glyph,
      populate its wide character metrics and return the appropriate
      parameter based on them.

      (font::get_width): Add conditional guard when computing width for a
      glyph from a "Unicode font"; use the computation only if the device
      description file ("DESC") didn't declare "unscaled_charwidths".

      (font::load): Recognize new directive in font description files:
      "charset-range", which works like the existing "charset" directive
      except that the glyph descriptions use a `name` of the form
      "uFFFF..uFFFF" (where "FFFF" is a hexadecimal digit sequence), and
      apply the metrics identically to all glyphs in the designated range.

      (font::load): When processing glyph descriptions in "charset" section
      and the device has declared the "unicode" directive, stop scaling the
      width of the glyph by what `wcwidth()` returns for it.  (Does this fix
      Savannah #44018?)

commit 64e5f5c687160592d1a47a8dff83d8088fdcc39b
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (3/6).

    * src/preproc/html/pre-html.cpp (scanArguments): Recognize but ignore
      new option `-U`, used by `grohtml` postprocessor.

    * src/devices/grohtml/post-html.cpp: Declare new constant integer
      objects `CHARSET_ASCII`, `CHARSET_MIXED`, and `CHARSET_UTF8` to
      configure representation of character entities in output.

      (main): New option `-U` takes argument configuring the means of
      encoding character entities.  If the argument is `0` or `-`, select
      `CHARSET_ASCII`; if `1`, select `CHARSET_MIXED`, and if `2` or `+`,
      select `CHARSET_UTF8`, which is also the default.

      (to_unicode): Replace this function with...  (to_numerical_char_ref):
      ...this, which generates a hexadecimal HTML character entity.

      (html_printer::add_to_sbuf): Write out UTF-8 sequence if
      `charset_encoding` is not `CHARSET_ASCII`, otherwise a numerical
      character reference.

      (get_html_entity): Return UTF-8 sequence if `charset_encoding` is
      `CHARSET_UTF8`.  Otherise, Return UTF-8 sequence if `charset_encoding`
      is not `CHARSET_ASCII`, otherwise a numerical character reference.

      (html_printer::writeHeadMetaStyle): Describe document {XHTML: encoding
      and} content as UTF-8 if `charset_encoding` is not `CHARSET_ASCII`,
      otherwise as US-ASCII.

commit 7d91bcb4c29a8ba882149422f8e227edef4678ac
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (4/6).

    * src/devices/grops/ps.h:
    * src/devices/grops/ps.cpp: Include C99 "stdint.h" header for desired
      `unit16_t` data type.

      (class ps_output): Change type of `put_string` member function's first
      argument from `const char *` to `const uint16_t *`.  Add third
      argument of Boolean type, `is_utf16le`.

    * src/devices/grops/ps.cpp (ps_output::put_string): Adjust computations
      of `len` and `col` locals if the font in use is UTF-16LE-encoding, and
      write out 4-digit instead of 2-digit hexadecimal numeric literals when
      that is the case.

      (class ps_printer): Change type of `sbuf` member variable from `char`
      to `uint16_t`.  Change type of third argument to `set_subencoding`
      member function from `unsigned char *` to `uint16_t *`.

      (ps_printer::set_subencoding): Rename third argument from `codep` to
      `code`--it's no longer an indirect reference to a single `char`, but a
      2-element `uint16_t` array.  If the font's "internalname" directive
      contains the substring "-UTF16-", populate `code` argument with
      little-endian 16-bit value.

      (ps_printer::set_char): Declare `code` as above: a 2-element
      `uint16_t` array instead of an unsigned char.  Handle case of `code`
      using surrogate pairs (`code[1] > 0`).

      (ps_printer::flush_sbuf): Conditionalize form of output on font
      encoding.  Set the Boolean argument to `ps::put_string()` per the
      font's "internalname" directive matching the substring "-UTF16-".

commit 76c81423da32cf5eb1451bd69fd4ec7da5ad12c3
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (5/6).

    Ship font description files.  These are intended as abstractions of
    faces to permit consistent naming while permitting customization, just
    as with the 12 text typefaces supported across output devices for Latin
    scripts in groff (three families of four styles each).  These CJK font
    descriptions are not organized into groff font families, but are
    similar.

            CSH: Simplified Chinese, Hei style
            CSS: Simplified Chinese, Song style
            CTH: Traditional Chinese, Hei style
            CTS: Traditional Chinese, Song style
            JPG: Japanese, Gothic style
            JPM: Japanese, Mincho style
            KOG: Korean, Gothic style
            KOM: Korean, Mincho style

    * font/devdvi/CSH:
    * font/devdvi/CSS:
    * font/devdvi/CTH:
    * font/devdvi/CTS:
    * font/devdvi/JPG:
    * font/devdvi/JPM:
    * font/devdvi/KOG:
    * font/devdvi/KOM:
    * font/devhtml/CSH:
    * font/devhtml/CSS:
    * font/devhtml/CTH:
    * font/devhtml/CTS:
    * font/devhtml/JPG:
    * font/devhtml/JPM:
    * font/devhtml/KOG:
    * font/devhtml/KOM:
    * font/devps/CSH:
    * font/devps/CSS:
    * font/devps/CTH:
    * font/devps/CTS:
    * font/devps/JPG:
    * font/devps/JPM:
    * font/devps/KOG:
    * font/devps/KOM:
    * font/devutf8/CSH:
    * font/devutf8/CSS:
    * font/devutf8/CTH:
    * font/devutf8/CTS:
    * font/devutf8/JPG:
    * font/devutf8/JPM:
    * font/devutf8/KOG:
    * font/devutf8/KOM: Ship font descriptions.

    * font/devdvi/devdvi.am (DEVDVIFONTFILES):
    * font/devhtml/devhtml.am (DEVHTMLFONTS, DEVHTMLFONTFILES):
    * font/devdvi/devps.am (DEVPSFONTFILES):
    * font/devutf8/devutf8.am (DEVUTF8FONTS, DEVUTF8FONTFILES): Add them.

    The test "contrib/hdtbl/examples/test-hdtbl.sh" fails at this commit.

commit f7ca7ae9dd65e8747b3b23a53d50010720a0c711
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Wed Nov 20 19:51:12 2024 -0600

    [hdtbl]: Update test expectations WRT new fonts.

    * contrib/hdtbl/examples/test-hdtbl.sh.in: Update test expectations to
      reflect addition of 8 font descriptions for CJK support.

commit 26dbf12ff0808169fedcc01a573280cdfba7c4ba
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (6/6).

    * src/roff/groff/tests/dvi-device-smoke-test.sh:
    * src/roff/groff/tests/ps-device-smoke-test.sh: New tests exercise
      output drivers and their encodings of CJK characters.

    * src/roff/groff/groff.am (groff_TESTS): Run tests.

    Fixes <https://savannah.gnu.org/bugs/?62830>.


G. Branden Robinson <gbranden>
Group administrator
Thu 21 Nov 2024 04:49:56 AM UTC, comment #14: 

Things look pretty good with this.  I made some modest changes to the patch.  Here's an interdiff to show them clearly.


diff -u b/font/devdvi/KOG b/font/devdvi/KOG
--- b/font/devdvi/KOG
+++ b/font/devdvi/KOG
@@ -1,5 +1,5 @@
 #
-#  Korean, GoThic style
+#  Korean, Gothic style
 #     with upTeX tfm font
 #

diff -u b/font/devhtml/KOG b/font/devhtml/KOG
--- b/font/devhtml/KOG
+++ b/font/devhtml/KOG
@@ -1,5 +1,5 @@
 #
-#  Korean, GoThic style
+#  Korean, Gothic style
 #

 name KOG
diff -u b/font/devps/KOG b/font/devps/KOG
--- b/font/devps/KOG
+++ b/font/devps/KOG
@@ -1,5 +1,5 @@
 #
-#  Korean, GoThic style
+#  Korean, Gothic style
 #  Adobe-Korea1
 #

diff -u b/font/devutf8/KOG b/font/devutf8/KOG
--- b/font/devutf8/KOG
+++ b/font/devutf8/KOG
@@ -1,5 +1,5 @@
 #
-#  Korean, GoThic style
+#  Korean, Gothic style
 #

 name KOG
diff -u b/src/devices/grohtml/post-html.cpp b/src/devices/grohtml/post-html.cpp
--- b/src/devices/grohtml/post-html.cpp
+++ b/src/devices/grohtml/post-html.cpp
@@ -95,13 +95,11 @@
                                             /* end of each page?                        */
 static int groff_sig = FALSE;               /* "This document was produced using"       */
 html_dialect dialect = html4;               /* which html dialect should grohtml output */
-#define PARTIAL 1
-#define FULL    2
-static int charset_utf8 = PARTIAL;          /* charset partially in "UTF-8" or in       */
-                                            /* character entity references by default.  */
-                                            /* If false then "US-ASCII".                */
-                                            /* If FULL then all Unicode characters are  */
-                                            /* written in UTF-8 and do not use character*/
+static const int CHARSET_ASCII = 0;
+static const int CHARSET_MIXED = 1;
+static const int CHARSET_UTF8  = 2;
+static int charset_encoding = CHARSET_MIXED;/* The character set may be plain ASCII,    */
+                                            /* pure UTF-8, or a mixture of character    */
                                             /* entity references.                       */


@@ -1408,13 +1406,15 @@
 }

 /*
- *  to_numerical_char_ref - returns a numerical character reference of unicode ch.
+ *  to_numerical_char_ref - returns a numerical character reference of
+ *                          unicode character code `ch`.
  */

 static char *to_numerical_char_ref (unsigned int ch)
 {
-  static char buf[16];
-
+  // Make static buffer large enough for a 64-bit `int` type in
+  // hexadecimal (8 bytes) plus '&#x;' plus null terminator.
+  static char buf[8 + 4 + 1];
   sprintf(buf, "&#x%X;", ch);
   return buf;
 }
@@ -4425,7 +4425,9 @@
       html_glyph = 0;

     if ((0 /* nullptr */ == html_glyph) && (code >= UNICODE_DESC_START))
-      html_glyph = charset_utf8 ? to_utf8_string(code) : to_numerical_char_ref(code);
+      html_glyph = static_cast<bool>(charset_encoding)
+                    ? to_utf8_string(code)
+                    : to_numerical_char_ref(code);
   } else
     html_glyph = get_html_translation(sbuf_style.f, s);

@@ -4506,7 +4508,7 @@
       case 0x003E: return "&gt;";
       default: return 0;
     }
-  } else if (charset_utf8==FULL) {
+  } else if (CHARSET_UTF8 == charset_encoding) {
       return to_utf8_string(code);
   } else {
     switch (code) {
@@ -4747,7 +4749,9 @@
       case 0x2666: return "&diams;";
       case 0x27E8: return "&lang;";
       case 0x27E9: return "&rang;";
-      default: return (charset_utf8 ? to_utf8_string(code) : to_numerical_char_ref(code));
+      default: return (static_cast<bool>(charset_encoding)
+                        ? to_utf8_string(code)
+                        : to_numerical_char_ref(code));
     }
   }
 }
@@ -5182,7 +5186,8 @@
          "content=\"groff -Thtml, see www.gnu.org\">\n", stdout);
     fputs("<meta http-equiv=\"Content-Type\" "
          "content=\"text/html; charset=", stdout);
-    fputs(charset_utf8 ? "UTF-8" : "US-ASCII", stdout);
+    fputs(static_cast<bool>(charset_encoding)
+           ? "UTF-8" : "US-ASCII", stdout);
     fputs("\">\n", stdout);
     fputs("<meta name=\"Content-Style\" content=\"text/css\">\n",
          stdout);
@@ -5190,7 +5195,8 @@
   }
   else {
     fputs("<?xml version=\"1.0\" encoding=\"", stdout);
-    fputs(charset_utf8 ? "UTF-8" : "us-ascii", stdout);
+    fputs(static_cast<bool>(charset_encoding)
+           ? "UTF-8" : "us-ascii", stdout);
     fputs("\"?>\n", stdout);
     fputs("<!DOCTYPE html PUBLIC \"-//W3C//"
          "DTD XHTML 1.1 plus MathML 2.0//EN\"\n", stdout);
@@ -5206,7 +5212,8 @@
          "content=\"groff -Txhtml, see www.gnu.org\"/>\n", stdout);
     fputs("<meta http-equiv=\"Content-Type\" "
          "content=\"text/html; charset=", stdout);
-    fputs(charset_utf8 ? "UTF-8" : "US-ASCII", stdout);
+    fputs(static_cast<bool>(charset_encoding)
+           ? "UTF-8" : "US-ASCII", stdout);
     fputs("\"/>\n", stdout);
     fputs("<meta name=\"Content-Style\" content=\"text/css\"/>\n",
          stdout);
@@ -5568,6 +5575,8 @@
     { NULL, 0, 0, 0 }
   };
   opterr = 0;
+  // TODO: Rename `U` option, which generally means "unsafe mode" in
+  // groff, to `u`.
   while ((c = getopt_long(argc, argv,
          "a:bCdD:eF:g:Ghi:I:j:lno:prs:S:U::vVx:y", long_options, NULL))
         != EOF)
@@ -5639,18 +5648,20 @@
       split_level = atoi(optarg) + 1;
       break;
     case 'U':
-      /* default: PARTIAL */
       if (optarg) {
+       // TODO: This argument semantic scheme seems unergonomic to GBR;
+       // come up with an alternative.
         if ((strcmp(optarg, "0") == 0 || strcmp(optarg, "-") == 0))
-          charset_utf8 = FALSE;
+          charset_encoding = CHARSET_ASCII;
         else if ((strcmp(optarg, "1") == 0))
-          charset_utf8 = PARTIAL;
-        else if (optarg && (strcmp(optarg, "2") == 0 || strcmp(optarg, "+") == 0))
-          charset_utf8 = FULL;
+          charset_encoding = CHARSET_MIXED;
+        else if (optarg && ((strcmp(optarg, "2") == 0)
+                            || strcmp(optarg, "+") == 0))
+          charset_encoding = CHARSET_UTF8;
         else
-          charset_utf8 = FULL;
+          charset_encoding = CHARSET_UTF8;
       } else
-        charset_utf8 = FULL;
+        charset_encoding = CHARSET_UTF8;
       break;
     case 'v':
       printf("GNU post-grohtml (groff) version %s\n", Version_string);


Interdiff rejected one hunk because it depended on code style cleanups to "ps.cpp" that I inserted into my working copy before applying your patches.


--- interdiff-1.Qm0i6k
+++ interdiff-1.Qm0i6k
@@ -219,7 +221,7 @@ ps_output &ps_output::put_string(const char *s, size_t n)
     else
       len += 4;
   }
-  if (len > n*2) {
+  if (len > n*2 || is_utf16be) {
     if (col + n*2 + 2 > max_line_length && n*2 + 2 <= max_line_length) {
       putc('\n', fp);
       col = 0;


I also wrote changelog entries for the patch.


commit a02d03dc271dc85315e9f439692b2b2bfe7dc596
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (1/6).

    * src/include/unicode.h (to_utf8_string): Declare new function.

    * src/libs/libgroff/unicode.cpp (to_utf8_string): New function converts
      input integer into UTF-8 sequence (or an HTML character entity in
      hexadecimal if the integer is out of range).

commit 0f5b9b63a7ee943e9c1f62d020ed9877e7bfef9b
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (2/6).

    * src/include/font.h (class font): Declare private member variable
      `wch`, a pointer to an existing list type `font_char_metric`.  Declare
      private member function `get_font_wchar_metric()` to access it.

    * src/libs/libgroff/font.cpp (struct font_char_metric): Add members
      `next` (a pointer to the struct's own type) and `end_code` of type
      `int`.

      (glyph_to_ucs_codepoint): New function returns UCS code point from a
      (non-composite) `glyph` object, or -1 if invalid.

      (font::font): Constructor initializes `wch` member variable to null
      pointer.

      (font::~font): Destructor frees storage allocated in `font::load()`
      for `special_device_coding` member of `wcp` struct, and that of `wcp`
      itself.

      (font::contains): If `glyph_to_ucs_codepoint()` returns a valid value
      for the glyph, populate its wide character metrics and return true.

      (font::get_font_wchar_metric): New function obtains font metrics of
      input character by Unicode code point.

      (font::get_width, font::get_height, font::get_depth)
      (font::get_italic_correction, font::get_left_italic_correction)
      (font::get_subscript_correction, font::get_character_type)
      (font::get_code, font::get_special_device_encoding): If
      `glyph_to_ucs_codepoint()` returns a valid value for the glyph,
      populate its wide character metrics and return the appropriate
      parameter based on them.

      (font::get_width): Add conditional guard when computing width for a
      glyph from a "Unicode font"; use the computation only if the device
      description file ("DESC") didn't declare "unscaled_charwidths".

      (font::load): Recognize new directive in font description files:
      "charset-range", which works like the existing "charset" directive
      except that the glyph descriptions use a `name` of the form
      "uFFFF..uFFFF" (where "FFFF" is a hexadecimal digit sequence), and
      apply the metrics identically to all glyphs in the designated range.

      (font::load): When processing glyph descriptions in "charset" section
      and the device has declared the "unicode" directive, stop scaling the
      width of the glyph by what `wcwidth()` returns for it.  (Does this fix
      Savannah #44018?)

commit 525f6215348305de608c9ac65fe299f3e2076e4c
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (3/6).

    * src/preproc/html/pre-html.cpp (scanArguments): Recognize but ignore
      new option `-U`, used by `grohtml` postprocessor.

    * src/devices/grohtml/post-html.cpp: Declare new constant integer
      objects `CHARSET_ASCII`, `CHARSET_MIXED`, and `CHARSET_UTF8` to
      configure representation of character entities in output.

      (main): New option `-U` takes argument configuring the means of
      encoding character entities.  If the argument is `0` or `-`, select
      `CHARSET_ASCII`; if `1`, select `CHARSET_MIXED`, and if `2` or `+`,
      select `CHARSET_UTF8`, which is also the default.

      (to_unicode): Replace this function with...  (to_numerical_char_ref):
      ...this, which generates a hexadecimal HTML character entity.

      (html_printer::add_to_sbuf): Write out UTF-8 sequence if
      `charset_encoding` is not `CHARSET_ASCII`, otherwise a numerical
      character reference.

      (get_html_entity): Return UTF-8 sequence if `charset_encoding` is
      `CHARSET_UTF8`.  Otherise, Return UTF-8 sequence if `charset_encoding`
      is not `CHARSET_ASCII`, otherwise a numerical character reference.

      (html_printer::writeHeadMetaStyle): Describe document {XHTML: encoding
      and} content as UTF-8 if `charset_encoding` is not `CHARSET_ASCII`,
      otherwise as US-ASCII.

commit fcd0113a296a91f32bc9f61b93604daa80b41fb3
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (4/6).

    * src/devices/grops/ps.h:
    * src/devices/grops/ps.cpp: Include C99 "stdint.h" header for desired
      `unit16_t` data type.

      (class ps_output): Change type of `put_string` member function's first
      argument from `const char *` to `const uint16_t *`.  Add third
      argument of Boolean type, `is_utf16le`.

    * src/devices/grops/ps.cpp (ps_output::put_string): Adjust computations
      of `len` and `col` locals if the font in use is UTF-16LE-encoding, and
      write out 4-digit instead of 2-digit hexadecimal numeric literals when
      that is the case.

      (class ps_printer): Change type of `sbuf` member variable from `char`
      to `uint16_t`.  Change type of third argument to `set_subencoding`
      member function from `unsigned char *` to `uint16_t *`.

      (ps_printer::set_subencoding): Rename third argument from `codep` to
      `code`--it's no longer an indirect reference to a single `char`, but a
      2-element `uint16_t` array.  If the font's "internalname" directive
      contains the substring "-UTF16-", populate `code` argument with
      little-endian 16-bit value.

      (ps_printer::set_char): Declare `code` as above: a 2-element
      `uint16_t` array instead of an unsigned char.  Handle case of `code`
      using surrogate pairs (`code[1] > 0`).

      (ps_printer::flush_sbuf): Conditionalize form of output on font
      encoding.  Set the Boolean argument to `ps::put_string()` per the
      font's "internalname" directive matching the substring "-UTF16-".

commit d471e4fb84861d23f5c0b5a064db57896e544aaa
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (5/6).

    Ship font description files.  These are intended as abstractions of
    faces to permit consistent naming while permitting customization, just
    as with the 12 text typefaces supported across output devices for Latin
    scripts in groff (three families of four styles each).  These CJK font
    descriptions are not organized into groff font families, but are
    similar.

            CSH: Simplified Chinese, Hei style
            CSS: Simplified Chinese, Song style
            CTH: Traditional Chinese, Hei style
            CTS: Traditional Chinese, Song style
            JPG: Japanese, Gothic style
            JPM: Japanese, Mincho style
            KOG: Korean, Gothic style
            KOM: Korean, Mincho style

    * font/devdvi/CSH:
    * font/devdvi/CSS:
    * font/devdvi/CTH:
    * font/devdvi/CTS:
    * font/devdvi/JPG:
    * font/devdvi/JPM:
    * font/devdvi/KOG:
    * font/devdvi/KOM:
    * font/devhtml/CSH:
    * font/devhtml/CSS:
    * font/devhtml/CTH:
    * font/devhtml/CTS:
    * font/devhtml/JPG:
    * font/devhtml/JPM:
    * font/devhtml/KOG:
    * font/devhtml/KOM:
    * font/devps/CSH:
    * font/devps/CSS:
    * font/devps/CTH:
    * font/devps/CTS:
    * font/devps/JPG:
    * font/devps/JPM:
    * font/devps/KOG:
    * font/devps/KOM:
    * font/devutf8/CSH:
    * font/devutf8/CSS:
    * font/devutf8/CTH:
    * font/devutf8/CTS:
    * font/devutf8/JPG:
    * font/devutf8/JPM:
    * font/devutf8/KOG:
    * font/devutf8/KOM: Ship font descriptions.

    * font/devdvi/devdvi.am (DEVDVIFONTFILES):
    * font/devhtml/devhtml.am (DEVHTMLFONTS, DEVHTMLFONTFILES):
    * font/devdvi/devps.am (DEVPSFONTFILES):
    * font/devutf8/devutf8.am (DEVUTF8FONTS, DEVUTF8FONTFILES): Add them.

    Note: the test "contrib/hdtbl/examples/test-hdtbl.sh" fails at this
    commit.

commit 3cc1ed649c09f739b8df0aafb3a461f9b030082b
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Wed Nov 20 19:51:12 2024 -0600

    [hdtbl]: Update test expectations WRT new fonts.

    * contrib/hdtbl/examples/test-hdtbl.sh.in: Update test expectations to
      reflect addition of 8 font descriptions for CJK support.

commit 500a5c98fc8258de3b69b846304e56fafb196797 (HEAD -> master)
Author: TANAKA Takuji <ttk@t-lab.opal.ne.jp>
Date:   Fri Dec 29 13:56:37 2023 +0000

    Support CJK fonts encoded in UTF-16 (6/6).

    * src/roff/groff/tests/dvi-device-smoke-test.sh:
    * src/roff/groff/tests/ps-device-smoke-test.sh: New tests exercise
      output drivers and their encodings of CJK characters.

    * src/roff/groff/groff.am (groff_TESTS): Run tests.

    Fixes <https://savannah.gnu.org/bugs/?62830>.


There is a little work left to do; see the TODO items in the interdiff, and I also need to document the new font description file directive in groff_font(5) and our Texinfo manual, and document the newly available font names in grodvi(1), grohtml(1), grops(1), and grotty(1).

Thank you for your immense patience, Tanaka-sama.

G. Branden Robinson <gbranden>
Group administrator
Mon 01 Jan 2024 01:49:42 AM UTC, comment #13: 

Oliver posted the following to the bug-groff list; apparently the GNU Savannah web site was not permitting him to comment on this ticket, which is pretty weird.


Hi,

thank you very much for considering my previous comment on the CJK font naming issue.

It is perfectly fine with me to build the font names by terms of primary language, i.e. starting with C for Chinese, K for Korean, J for Japanese.

In my humble opinion and according to decades of personal experience in this area, a majority of documents in these languages will be composed and perused by users of one language alone, and just by following prior practice, using a language-based naming scheme will allow the majority of users to navigate in familiar waters.

The minory of users who --- from the beginning --- compose documents with the need for equal typeface across several languages will use those fonts which provide a full coverage of CJK characters, and for them, the first letter doesn't really matter. They'll pick the font first, then the language, in contrast to the users mentioned before.

Best regards,
Oliver.


G. Branden Robinson <gbranden>
Group administrator
Fri 29 Dec 2023 10:39:52 PM UTC, comment #12: 

Just a note to point out that gropdf (in the deri-gropdf-ng git branch) already handles CJK documents. The attached pdf is an example, you should see CJK in the pdf bookmarks as well.

(file #55489)

Deri James <deri>
Group Member
Fri 29 Dec 2023 07:50:57 PM UTC, comment #11: 

comment #9:

> It would be helpful to me if Messrs. Tanaka and Corff could
> reach a consensus on a schema for naming CJK font families.  :)


To assist in that, I'm adding Oliver to this bug's cc.  Oliver, do you have any response to comment #10 here?  (You can post a comment to the bug report without having to create a savannah account.)

Dave <barx>
Group Member
Fri 29 Dec 2023 01:56:37 PM UTC, comment #10: 

I updated my patch to adjust to the latest upstream source code.

> While Mincho and Gothic loosely compare to serif and sans serif styles
> I think the font family classification terminology should be language-agnostic


Thank Oliver for kind comments.
I agree that "M (stand for Mincho)",  "G (stand for Goshikku or Gothic)" are not langage-agnostic.
However CJK language users require language specific fonts for each language.
In fact, Adobe have provided free CJK fonts "Source Han" and prepared 4 kinds of localized font files
Simplified Chinese, Traditional Chinese, Japanese and Korean to support regional variations.

https://source.typekit.com/source-han-serif/

I think 8 kinds of definitions for CJK (4 resional variations and 2 styles) is minimum as basic fonts.
Since we need 4 resional variants, I do not think language-specific naming is unappropreate for standard name.

"Source Han" has localized font names and they are consistent of my proposal.

Source Han Sans Simplified Chinese    : 思源黑体 sīyuán hēitǐ     : CSH ; H stand for hēitǐ
Source Han Serif Simplified Chinese   : 思源宋体 sīyuán sòngtǐ    : CSS ; S stand for sòngtǐ

Source Han Sans Traditional Chinese   : 思源黑體 sīyuán hēitǐ     : CTH ; H stand for hēitǐ
Source Han Serif Traditional Chinese  : 思源宋體 sīyuán sòngtǐ    : CTS ; S stand for sòngtǐ

Source Han Sans Japanese              : 源ノ角ゴシック Gen-no-kaku-goshikku : JPG ; G stand for Goshikku
Source Han Serif Japanese             : 源ノ明朝       Gen-no-mincho        : JPM ; M stand for Mincho

Source Han Sans Korean                : 본고딕   Bon GoTik       : KRG ; G stand for Gotik
Source Han Serif Korean               : 본명조   Bon MyeongJo    : KRM ; M stand fo Myeongjo


If you still do not agree the langage-specific name,
langage-agnostic names based on English could be possible
for example CSSS, CSSR, CTSS, CTSR, JPSS, JPSR, KRSS, KRSR as followings:

Source Han Sans Simplified Chinese    :: CSSS  SS stand for Sans-serif
Source Han Serif Simplified Chinese   :: CSSR  SR stand for Serif

Source Han Sans Traditional Chinese   :: CTSS
Source Han Serif Traditional Chinese  :: CTSR

Source Han Sans Japanese              :: JPSS
Source Han Serif Japanese             :: JPSR

Source Han Sans Korean                :: KRSS
Source Han Serif Korean               :: KRSR

Actually, I still recommend the formaer langage-specific ones.
But I accept the latter langage-agnostic ones if you do not agree my opinion.

Best,

(file #55487)

TANAKA Takuji <takuji>
Thu 10 Aug 2023 01:27:44 PM UTC, comment #9: 

Quoting Oliver Corff from the mailing list post cited in comment #7:


I had a look at https://savannah.gnu.org/bugs/?62830 and Tanaka Takuji
mentions the work of UKAI Fumitoshi et al., defining M for Japanese
Mincho and G for Japanese Gothic style.

While Mincho and Gothic loosely compare to serif and sans serif styles
(let's say, Times Roman and Helvetica for the first idea), the problem
with those Japanese fonts which are publicly available or free (as in
free to use) is that they are heavily leaning towards Japanese usage of
the CJK character set. Many of these fonts are derived from earlier work
which covered only Japanese and still do not fully cover the CJK Unified
Character set.

I attached three screenshots of my character map, showing the region
where the characters grouped under the radical 人 (rén, "(hu)man") are
assembled. IPAMincho is visibly defective in its coverage, the bold sans
serif characters are substituted from the Droid Sans font.
WenQuanYi_Zen_Hei is a Chinese-Made sans serif CJK Unified Ideograph
font which covers everything but is a good approximation to "Gothic"
typeface, and ZYSong18030 is the same in Serif-style.

This unintentional mixup of character representations from different
fonts, until recently, was a headache when writing e.g. presentations
for a PR China audience using Microsoft's PowerPoint. At least on
machines until Windows 7 (don't know the current state) an
out-of-the-box slide full of Chinese text would show a horrible mix of
typeface because the "CJK" preset of the OS and of the Office software
suite used to be a Mincho variant, with the result that many frequent
CJK characters which are only used in Mainland China were displayed in a
different style. Setting the document default East Asian font to
something different would solve the problem.

A word on names:   The Japanese loanword "gothic" corresponds to what is
historically known as "blackletter". The Chinese term for this is simply
a translation: "hei" (black), as in "heiti" ("ti", pronounced "tee",
stands for "body"). The "serif" equivalent Mincho name implies the Ming
Dynasty, however the more common usage in China is Song (who were two
centuries before the Ming), universally known for their cultural
refinement, also expressed in calligraphy and typeface.  Ming (or
Japanese "Mincho", "-cho" [long vowel] simply means "dynasty") in Japan
or Song (in China) is the most commonly used typeface for books, serious
newspapers etc.

So I think the font family classification terminology should be
language-agnostic (i.e., avoiding terms like "Mincho" or "heiti" because
fonts explicitely labelled as such might be designed for the needs of
one particular language only (e.g., Chinese (traditional or
simplified?), Japanese (even here: old style vs. new style), Korean),
and by design, might be severely limited for any CJK display outside
that language.

Visible and fundamental differences between different CJK presentations
are shown on the Wikipedia page "List of CJK fonts":
https://en.wikipedia.org/wiki/List_of_CJK_fonts

Figure 1 shows the differences between different typeface styles (NOT
language-specific), while figure 2 shows the graphematical differences
of the same characters in different languages. Notable are columns 1 and
3. NB: This image is not composed of characters taken from a font but
was created as a SVG file.

Please note that CJK Unified Ideographs contain most but not all of the
language-specific differences in stroke order etc. So, for demanding
typographical work one will always use a font which is specifically
designed to match a certain style. For the vast majority of generic text
in modern everyday life (be the genre sciences, humanities, non-fiction,
fiction, newspapers) it is possible to use a single pan CJK font to
cover several CJK languages; think of a Japanese text book for Chinese
learners, of a Chinese-Japanese dictionary, of a Japanese analysis of a
Chinese newspaper article with portions of original text in quotes. Yet,
there are exceptional situations where the usage of a generic CJK font
is such a no-go that it has the potential to create a diplomatic
incident; simply because minutiae in stroke order have a high impact on
the (language/national) identity perception of the reader.

Unfortunately, I cannot say for certain what a typical standard font in
a typical off-the-shelf Japanese Linux version is. I'll have to download
a representative version and run it, at least in a virtual machine or
from a USB live stick, but I'll certainly explore that and share my
findings.


It would be helpful to me if Messrs. Tanaka and Corff could reach a consensus on a schema for naming CJK font families.  :)

G. Branden Robinson <gbranden>
Group administrator
Fri 05 May 2023 12:19:41 AM UTC, comment #8: 

Oliver Corff posted a lengthy commentary about CJK fonts: http://lists.gnu.org/r/groff/2023-05/msg00008.html

Dave <barx>
Group Member
Sat 15 Apr 2023 07:18:20 AM UTC, comment #7: 

I updated my patch.

1. Font description files

There is a precedent of font description file for Japanese support of groff by Japanese developers (Fumitoshi UKAI et al.)

https://answers.launchpad.net/ubuntu/+source/groff/1.18.1.1-12

They defined font description named "M", "G" for Japanese.
M : Japanese Mincho style
G : Japanese Gothic style

"M", "G" are possible candidates.
But I wonder if Chinese/Korean people might feel uncomfortable.
It is the reason that I proposed font description "JPM", "JPG" and CK fonts.


2. src/devices/grohtml/post-html.cpp:

2a & 2e. Encoding US-ASCII or UTF-8, -U option

I tried three step of option setting:
 -U0 : US-ASCII : use named character references or numerical character references
 -U1 : UTF-8 (partial) : use named character references for known characters, UTF-8 literals for unknown characters (default)
 -U2 : UTF-8 (full) : use UTF-8 literals


2b. `to_utf8_string`.

I have moved it to libgroff/font.cpp for trial.

2c. switching text styling properties.

I have removed the function from my patch.

2d. `to_unicode`.

I have renamed it to_unicode() to to_numerical_char_ref().


3. src/devices/grops/ps.cpp

3a. I have renamed is_utf16 to  is_utf16be

3b. I have replaced wchar_t by uint16_t.

3c. postscript name and encoding.

For CJK fonts, encoding is always explicitly shown in PostScript font name
by the structure of (Specific font name)-(style)(-(character set))-(encoding)-(direction).
For example:

/Ryumin-Light-Identity-H
/Ryumin-Light-UniJIS-UTF16-H
/Ryumin-Light-UniJIS-UTF8-H
/Ryumin-Light-EUC-H
/Ryumin-Light-RKSJ-H
/GothicBBB-Medium-Identity-H
/GothicBBB-Medium-UniJIS-UTF16-H
/GothicBBB-Medium-UniJIS-UTF8-H
/GothicBBB-Medium-EUC-H
/GothicBBB-Medium-RKSJ-H

This is a sample PostScript file:
https://github.com/t-tk/PostScript-CJK-samples/blob/master/box-multi.eps

Therefore, I think it is reasonable to get encoding information from PostScript font names.
I guess most of PostScript interpreters do so.


4. src/include/font.h, src/libs/libgroff/font.cpp

I removed "ENABLE_UCSRANGE" macro from my patch.


5. smoke tests.

I replaced UTF-8 literal by octal code expression.


(file #54631)

TANAKA Takuji <takuji>
Fri 31 Mar 2023 11:14:13 AM UTC, comment #6: 

Updating fields to reflect my intention to work on this once groff 1.23.0 is kicked out of the nest.

G. Branden Robinson <gbranden>
Group administrator
Sun 04 Dec 2022 12:23:26 AM UTC, comment #5: 

Thank you for the update.

I don't think I am going to have time to properly consider this patch in the depth it requires before the groff 1.23.0 release, which I am hoping to get at least a release candidate out for before the end of the calendar year.  Only 5 open Savannah tickets remain for that.

I would however like to take a fresh look at this issue (and Russian localization, bug #63076) early in the groff 1.24 development cycle.  How soon that begins will depend on how many urgent bug reports we get against 1.23.0.

On the bright side you can expect relatively few changes to occur between now and that might make your patches difficult to apply or maintain out of the tree.

Here are some thoughts I have for when I can return to this work (or for another groff developer to step up and consider discussing or addressing).

1. I was uncertain about the wisdom of shipping more font description files, but it's not like there isn't a precedent; except for the FreeEuro font, we don't ship any fonts proper--just descriptions of fonts that the user must obtain elsewhere.  So  shipping CSH, CSS, CTH, CTS, JPG, JPM, KOG, and KOM font descriptions for the "dvi", "html", "ps", and "utf8" output devices is not without precedent.

2. src/devices/grohtml/post-html.cpp:

2a. I wonder if defaulting to ASCII for the html output device is   necessary.  Apparently UTF-8 is overwhelmingly the encoding used by most web pages in the world.  https://w3techs.com/technologies/details/en-utf8

2b. The new `to_utf8_string` function might be better housed in libgroff or libdriver, if in fact there is not already a suitable function present in one of those libraries.  Another possibility is that there is some gnulib module we could use here, and not have to carry our own implementation at all.

2c. I am uneasy with switching text styling properties (bold, italic) off based on the groff font name in use.  I think it might be better to have a new font description file directive (see groff_font(5)) that tags a font as being unstyled.  Any font with this property would cause the disablement of bold and italic flags.

2d. Maybe the existing `to_unicode` function should be renamed; from the name along, it's not obvious how it is distinct from `to_utf8_string`.

2e. The `-U` option seems like a good idea, and perhaps is a flag letter we can re-use elsewhere in groff as we improve its Unicode support.

3. src/devices/grops/ps.cpp

3a. `is_utf16` should be renamed to reflect whether it uses UTF-16BE or UTF-16LE.

3b. I'm uneasy with the use of wchar_t.  I think maybe we want to use int32_t, or if that can't be assumed to be available in C++98 (check this), then we should have a type alias ("typedef" [sic]) and use an int, which must be at least 32 bits on any GNU system.

3c. Again we're inferring properties from font names, it looks like:


+  const char *psname = f->get_internal_name();
+
+  if (psname && strstr(psname,"-UTF16-")) {


And again I think I'd prefer a font description file property to communicate this information.

4. src/include/font.h, src/libs/libgroff/font.cpp

I wouldn't have a preprocessor-based feature gate like this "ENABLE_UCSRANGE" macro.  I would enable the feature for all builds.  This will give it exercise and help uncover bugs.

5. Thank you for the 'dvi' and 'ps' device smoke tests!  It might be necessary to rewrite the UTF-8-encoded literals for CJK glyphs as octal escape sequences to the printf(1) command for portability, sadly.  Surprising things go wrong on *BSD and macOS systems.

I emphasize that I don't require any changes to be made at this time to address the above points; they are for consideration and discussion by developers (including the patch author!) before any revision occurs.  I simply wanted to get these points down while they were fresh in my mind.

G. Branden Robinson <gbranden>
Group administrator
Wed 23 Nov 2022 01:28:17 AM UTC, comment #4: 

I updated my patch.

- sync with master of https://git.savannah.gnu.org/cgit/groff.git

I hope the maintainers to consider accepting my request.

Thanks,
Takuji TANAKA

(file #53990)

TANAKA Takuji <takuji>
Sat 03 Sep 2022 08:47:02 AM UTC, comment #3: 

I have updated my patch (cjk-ps-html_20220902.patch).

I followed up the latest repository.
I added CJK support for html, dvi, tty(utf8).
I added CJK font settings for html, dvi, tty(utf8).

In post-grohtml, I added option "-U" to enable output in charset="UTF-8".

I do not need to patch on grodvi for dvi output and patch on grotty for tty(utf8) output.

More test results are shown at https://github.com/t-tk/groff-CJK-font.

TANAKA Takuji <takuji>
Thu 04 Aug 2022 12:26:17 AM UTC, comment #2: 

An information-rich discussion thread is at < https://lists.gnu.org/archive/html/groff/2022-07/msg00003.html >.

G. Branden Robinson <gbranden>
Group administrator
Fri 29 Jul 2022 06:16:56 AM UTC, comment #1: 

(removed a duplicate attachment)

Dave <barx>
Group Member
Thu 28 Jul 2022 12:56:27 PM UTC, original submission:  

I wrote a set of patches in order to support CJK fonts in PostScript outputs.
Attached you will find the patches.

- It enables to support CJK non-embedded fonts in UTF16 encoding by grops.
- It enables to define charset by a range of Unicode code points in font definition.
  It is useful for CJK fonts because the number of glyphs with the same metric are very large.


More tests are shown in GitHub:
https://github.com/t-tk/groff-CJK-font


I hope the maintainers to consider accepting my request.

Thanks,
Takuji TANAKA

TANAKA Takuji <takuji>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #55489:  tanaka.pdf added by deri (33KiB - application/pdf)
file #55487:  cjk-ps-html_20231223.patch added by takuji (87KiB - application/octet-stream)
file #54631:  cjk-ps-html_20230415.patch added by takuji (87KiB - application/octet-stream)
file #53990:  cjk-ps-html_20221118.patch added by takuji (86KiB - application/octet-stream)
file #53654:  cjk-ps-html_20220902.patch added by takuji (86KiB - application/octet-stream - I have updated my patch. I added support CJK for html, dvi and tty (utf8).)
file #53481:  cjk-ps.patch added by takuji (33KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by deri (Updated the item)
  • -email is unavailable- added by barx (has knowledge of the issue)
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by takuji (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 19 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-11-21 gbranden StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2024-11-21 gbranden StatusNeed Info In Progress
    2024-01-01 gbranden Dependencies- bugs #65099 is dependent
    2023-12-29 deri Attached File- Added tanaka.pdf, #55489
    2023-12-29 barx Carbon-Copy- Added -email is unavailable-
    2023-12-29 takuji Attached File- Added cjk-ps-html_20231223.patch, #55487
    2023-08-10 gbranden StatusPostponed Need Info
    2023-04-15 takuji Attached File- Added cjk-ps-html_20230415.patch, #54631
    2023-03-31 gbranden Assigned toNone gbranden
        Planned ReleaseNone 1.24.0
    2023-03-31 gbranden StatusNone Postponed
    2022-11-23 takuji Attached File- Added cjk-ps-html_20221118.patch, #53990
    2022-09-03 barx Summary[grops] support CJK fonts encoded in UTF16 [PATCH] [grops] support CJK fonts encoded in UTF16
    2022-09-03 takuji Attached File- Added cjk-ps-html_20220902.patch, #53654
    2022-07-29 barx Severity3 - Normal 1 - Wish
    2022-07-29 barx Attached File#53482 Removed
    2022-07-28 takuji Attached File- Added cjk-ps.patch, #53482
    2022-07-28 takuji Attached File- Added cjk-ps.patch, #53481

    Back to the top

    Powered by Savane 3.14-9aa3.
    Corresponding source code