bugGNU troff - Bugs: bug #62830, [PATCH] [grops] support CJK fonts...


bug #62830: [PATCH] [grops] support CJK fonts encoded in UTF16

Submitter:  TANAKA Takuji <takuji>
Submitted:  Thu 28 Jul 2022 12:56:27 PM UTC
Category:  Driver grops Severity:  1 - Wish
Item Group:  Feature change Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment (Rich Markup)

Comment Type & Canned Response:
       No canned response available


Sun 04 Dec 2022 12:23:26 AM UTC, comment #5: 

Thank you for the update.

I don't think I am going to have time to properly consider this patch in the depth it requires before the groff 1.23.0 release, which I am hoping to get at least a release candidate out for before the end of the calendar year.  Only 5 open Savannah tickets remain for that.

I would however like to take a fresh look at this issue (and Russian localization, bug #63076) early in the groff 1.24 development cycle.  How soon that begins will depend on how many urgent bug reports we get against 1.23.0.

On the bright side you can expect relatively few changes to occur between now and that might make your patches difficult to apply or maintain out of the tree.

Here are some thoughts I have for when I can return to this work (or for another groff developer to step up and consider discussing or addressing).

1. I was uncertain about the wisdom of shipping more font description files, but it's not like there isn't a precedent; except for the FreeEuro font, we don't ship any fonts proper--just descriptions of fonts that the user must obtain elsewhere.  So  shipping CSH, CSS, CTH, CTS, JPG, JPM, KOG, and KOM font descriptions for the "dvi", "html", "ps", and "utf8" output devices is not without precedent.

2. src/devices/grohtml/post-html.cpp:

2a. I wonder if defaulting to ASCII for the html output device is   necessary.  Apparently UTF-8 is overwhelmingly the encoding used by most web pages in the world.  https://w3techs.com/technologies/details/en-utf8

2b. The new `to_utf8_string` function might be better housed in libgroff or libdriver, if in fact there is not already a suitable function present in one of those libraries.  Another possibility is that there is some gnulib module we could use here, and not have to carry our own implementation at all.

2c. I am uneasy with switching text styling properties (bold, italic) off based on the groff font name in use.  I think it might be better to have a new font description file directive (see groff_font(5)) that tags a font as being unstyled.  Any font with this property would cause the disablement of bold and italic flags.

2d. Maybe the existing `to_unicode` function should be renamed; from the name along, it's not obvious how it is distinct from `to_utf8_string`.

2e. The `-U` option seems like a good idea, and perhaps is a flag letter we can re-use elsewhere in groff as we improve its Unicode support.

3. src/devices/grops/ps.cpp

3a. `is_utf16` should be renamed to reflect whether it uses UTF-16BE or UTF-16LE.

3b. I'm uneasy with the use of wchar_t.  I think maybe we want to use int32_t, or if that can't be assumed to be available in C++98 (check this), then we should have a type alias ("typedef" [sic]) and use an int, which must be at least 32 bits on any GNU system.

3c. Again we're inferring properties from font names, it looks like:

+  const char *psname = f->get_internal_name();
+  if (psname && strstr(psname,"-UTF16-")) {

And again I think I'd prefer a font description file property to communicate this information.

4. src/include/font.h, src/libs/libgroff/font.cpp

I wouldn't have a preprocessor-based feature gate like this "ENABLE_UCSRANGE" macro.  I would enable the feature for all builds.  This will give it exercise and help uncover bugs.

5. Thank you for the 'dvi' and 'ps' device smoke tests!  It might be necessary to rewrite the UTF-8-encoded literals for CJK glyphs as octal escape sequences to the printf(1) command for portability, sadly.  Surprising things go wrong on *BSD and macOS systems.

I emphasize that I don't require any changes to be made at this time to address the above points; they are for consideration and discussion by developers (including the patch author!) before any revision occurs.  I simply wanted to get these points down while they were fresh in my mind.

G. Branden Robinson <gbranden>
Project Administrator
Wed 23 Nov 2022 01:28:17 AM UTC, comment #4: 

I updated my patch.

- sync with master of https://git.savannah.gnu.org/cgit/groff.git

I hope the maintainers to consider accepting my request.


(file #53990)

TANAKA Takuji <takuji>
Sat 03 Sep 2022 08:47:02 AM UTC, comment #3: 

I have updated my patch (cjk-ps-html_20220902.patch).

I followed up the latest repository.
I added CJK support for html, dvi, tty(utf8).
I added CJK font settings for html, dvi, tty(utf8).

In post-grohtml, I added option "-U" to enable output in charset="UTF-8".

I do not need to patch on grodvi for dvi output and patch on grotty for tty(utf8) output.

More test results are shown at https://github.com/t-tk/groff-CJK-font.

TANAKA Takuji <takuji>
Thu 04 Aug 2022 12:26:17 AM UTC, comment #2: 

An information-rich discussion thread is at < https://lists.gnu.org/archive/html/groff/2022-07/msg00003.html >.

G. Branden Robinson <gbranden>
Project Administrator
Fri 29 Jul 2022 06:16:56 AM UTC, comment #1: 

(removed a duplicate attachment)

Dave <barx>
Project Member
Thu 28 Jul 2022 12:56:27 PM UTC, original submission:  

I wrote a set of patches in order to support CJK fonts in PostScript outputs.
Attached you will find the patches.

- It enables to support CJK non-embedded fonts in UTF16 encoding by grops.
- It enables to define charset by a range of Unicode code points in font definition.
  It is useful for CJK fonts because the number of glyphs with the same metric are very large.

More tests are shown in GitHub:

I hope the maintainers to consider accepting my request.


TANAKA Takuji <takuji>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #53990:  cjk-ps-html_20221118.patch added by takuji (86KiB - application/octet-stream)
file #53654:  cjk-ps-html_20220902.patch added by takuji (86KiB - application/octet-stream - I have updated my patch. I added support CJK for html, dvi and tty (utf8).)
file #53481:  cjk-ps.patch added by takuji (33KiB - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by takuji (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.





    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-11-23 takuji Attached File- Added cjk-ps-html_20221118.patch, #53990
    2022-09-03 barx Summary[grops] support CJK fonts encoded in UTF16 [PATCH] [grops] support CJK fonts encoded in UTF16
    2022-09-03 takuji Attached File- Added cjk-ps-html_20220902.patch, #53654
    2022-07-29 barx Severity3 - Normal 1 - Wish
    2022-07-29 barx Attached File#53482 Removed
    2022-07-28 takuji Attached File- Added cjk-ps.patch, #53482
    2022-07-28 takuji Attached File- Added cjk-ps.patch, #53481

    Back to the top

    Powered by Savane 3.9