bugGNU troff - Bugs: bug #63076, [tmac] add Russian language support

 
 

bug #63076: [tmac] add Russian language support

Submitter:  Nikita Ivanov <nikitaivanov>
Submitted:  Sat 17 Sep 2022 05:41:41 PM UTC
 
Category:  Macro - others/general Severity:  1 - Wish
Item Group:  Feature change Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       No canned response available

 

( Jump to the original submission )

Sun 23 Oct 2022 08:27:09 PM UTC, comment #8: 

Forgot to provide a link.

https://en.wikipedia.org/wiki/History_of_sentence_spacing#French_and_English_spacing

And me further clarify; using each of a pair of words to mean precisely opposite things is often a way to achieve lucidity.

What I mean is that what one speaker calls "widows", another calls "orphans", and vice versa; the same is, infuriatingly, true of "French" and "English" spacing.  So in practice these terms communicate nothing except a topic indicator.  ("Uhhh, something to do with lines of a paragraph being broken across pages.")

G. Branden Robinson <gbranden>
Project Administrator
Sun 23 Oct 2022 08:24:22 PM UTC, comment #7: 

comment #5:

> Another question: Why do nearly all language macro packages set additional inter-sentence space to 0?


The short answer is English-language typography diverged from European practices over the 19th and 20th centuries.  With respect to this specific issue, practices within English-speaking communities have fragmented.

It is a deeply confusing issue.  As with "widows" and "orphans", "French spacing" and "English spacing" are each used to mean precisely opposite things.

Most European languages using the Latin alphabet appear to now predominantly use only one word space after sentence-endings.  I believe there is a European Union mandate (applicable to that organization's own official output only, but adhered to voluntarily by many private organizations for want of a different style manual) along these lines.

> Should the same be done in the Russian one?


I would consult one or authoritative style manuals for Russian language professional writing and typography.

G. Branden Robinson <gbranden>
Project Administrator
Wed 28 Sep 2022 07:01:50 PM UTC, comment #6: 

Please review my patch groff-ru-1.patch. It includes translated strings for macro packages and hyphenation rules. The Cyrillic symbols are encoded in KOI8-R (koi8-ru.tmac adds support for the encoding).

(file #53764)

Nikita Ivanov <nikitaivanov>
Tue 20 Sep 2022 10:11:33 PM UTC, comment #5: 

Another question: Why do nearly all language macro packages set additional inter-sentence space to 0? Should the same be done in the Russian one?

Nikita Ivanov <nikitaivanov>
Tue 20 Sep 2022 08:22:25 PM UTC, comment #4: 

Thank you for so much helpful info!

I've noticed that if I enable hyphenation rules for Russian language with .hpf request, hyphenation for text written in English seems to stop working. However, it's not the case for French or German languages. Is it because those languages have similar hyphenation rules to English? Should I include English hyphenation rules in Russian hyphenation rules (or perhaps append English ones to Russian ones with .hpfa request) to make hyphenation work for both languages?

Nikita Ivanov <nikitaivanov>
Sun 18 Sep 2022 04:37:41 PM UTC, comment #3: 

original submission:

> I would love to help the project by adding support for Russian language (in a similar manner to Italian, German and so on, by providing hyphenation rules and strings in Russian for macro packages). The Russian language is spoken by around 258 million people in the world, so I think this addition may potentially help a lot of new and old groff users. What are your thoughts?


I see nothing objectionable here.

comment #1:

> When I wrote it, I forgot that the built-in fonts do not support Cyrillic symbols and you always has to use a 3rd-party font to write in Russian.


Yes, but that's true of Chinese and Japanese as well, yet we have the localization macro files zh.tmac and ja.tmac.

> So now I'm not sure if adding strings and hyphenation rules makes sense... Unless I also add Cyrillic symbols to the built-in fonts.


I don't think that is necessary.  I don't know if you saw my lengthy feedback to a recent request to support UTF-16-encoded fonts in grops, but some similar considerations are present.  People sometimes think they have to bite off more of a task than they really do.

Specifically, for Russian language support, the main things to check are the ones you need to customize: correctly localized strings, and correct hyphenation.

The really good news is that both are straightforward to verify for a fluent Russian speaker even without Cyrillic fonts installed.  All that is required is a partial understanding of GNU troff's device-independent output format ("grout", I like to call it) and either the ability to recognize Unicode code points for Cyrillic letters or a crude filter that will translate them for spot-checking at a terminal.

Consider the "simple example" from this message to the groff mailing list about a month ago.

If you can follow it, you are well on your way to knowing all the "grout" you need to to verify Russian language support.

For example, with a reasonable ru.tmac file in place, I expect to be able to produce output much like this using "groff -kZ -Tuf8".

x T utf8
x res 240 24 40
x init
x F -
p1
x font 1 R
f1
s10
V40
H0
md
DFd
Cu0431
H24
Cu043B
h24
Cu0430
h24
Cu0433
h24
Cu043E
h24
Chy
h24
n40 0
V80
H0
Cu0434
H24
Cu0430
h24
Cu0440
h24
Cu044F
h24
n40 0
x trailer
V2640
x stop

(If you're wondering how I got that, I gave groff input consisting of a Cyrillic word familiar to me, turned off adjustment, set the line length to 8n, and manually stuck in a hyphen where it seemed to make sense to me as an English speaker, knowing very little of the morphology of the Slavic-language specimen.)

(The above output can be given as input to grotty on a UTF-8-capable terminal and a terminal font with Cyrillic coverage, which is probably the practical means of checking Russian language support, but I wanted to illustrate how the task could be achieved even in a plain ASCII environment.)

Validating the localized strings is even easier; they simply need to be input in the source files in a natural way (UTF-8 and all) and proofread for spelling errors.  I cannot imagine a GNU troff failure mode where these input characters will be remapped or resequenced.

The "Manipulating Hyphenation" section/node of the groff Texinfo manual should prove useful for constructing the list of hyphenation codes.  There is an example in German already which illustrates the basic principle; each lowercase Cyrillic letter will need a "mapping" to itself, and then each uppercase counterpart can be mapped to the lowercase one.

The adaptation of TeX hyphenation patterns is the part that seems like it might be hardest to me.  All of our current hyphenation pattern files are in ASCII or an ISO-8859 encoding.  I don't know if iconv(1) is up to transforming them to a form that GNU troff's hyphenation pattern file reader is prepared to accept.

https://git.savannah.gnu.org/cgit/groff.git/tree/src/roff/troff/env.cpp#n3784

Making groff support KOI8-R might be easier than getting it to interpret UTF-8-encoded hyphenation patterns, funny as that may sound.

On the other hand, we can integrate the macro file part of this first because it is independent and easy (saith I).  Without hyphenation pattern files, hyphenation simply won't be done.  That won't get us professional-grade typography, but it will improve significantly on the status quo.

(Mainly as a note to myself, one thing to check is what data type is used for hyphenation codes.  If it's a char, we should fix that.)

G. Branden Robinson <gbranden>
Project Administrator
Sat 17 Sep 2022 06:23:44 PM UTC, comment #2: 

Someone has to do the work, preferably someone who has a basic understanding of three subjects: groff, the Russian language, and contributing to Free Software projects.  Professional skills are likely not required in any of these three areas, but some time and diligence are.

I doubt that any of the groff developers would be opposed to improving support for the Russian language (or any other language, no matter how many people speak it), if somebody interested takes the lead in developing the needed patches and helps with integration and ongoing maintenance.

Ingo Schwarze <schwarze>
Project Member
Sat 17 Sep 2022 05:55:27 PM UTC, comment #1: 

When I wrote it, I forgot that the built-in fonts do not support Cyrillic symbols and you always has to use a 3rd-party font to write in Russian. So now I'm not sure if adding strings and hyphenation rules makes sense... Unless I also add Cyrillic symbols to the built-in fonts.

Nikita Ivanov <nikitaivanov>
Sat 17 Sep 2022 05:41:41 PM UTC, original submission:  

I would love to help the project by adding support for Russian language (in a similar manner to Italian, German and so on, by providing hyphenation rules and strings in Russian for macro packages). The Russian language is spoken by around 258 million people in the world, so I think this addition may potentially help a lot of new and old groff users. What are your thoughts?

Nikita Ivanov <nikitaivanov>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #53764:  groff-ru-1.patch added by nikitaivanov (42KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by schwarze (Posted a comment)
  • -email is unavailable- added by nikitaivanov (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-09-28 nikitaivanov Attached File- Added groff-ru-1.patch, #53764
    2022-09-18 gbranden CategoryGeneral Macro - others/general
        SummaryAdding Russian language to groff [tmac] add Russian language support
    2022-09-17 schwarze Severity3 - Normal 1 - Wish
        Item GroupNone Feature change

    Back to the top

    Powered by Savane 3.9