bugGNU roff - Bugs: bug #66112, [PATCH] [l10n:english] map Latin-1...

 
 

bug #66112: [PATCH] [l10n:english] map Latin-1 Supplement character hyphenation codes to base-character equivalents

Submitter:  Dave <barx>
Submitted:  Mon 19 Aug 2024 08:21:05 PM UTC
   
 
Category:  Macro package - others/general Severity:  1 - Wish
Item Group:  Feature change Status:  Fixed
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  1.24.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 16 Oct 2024 07:04:38 PM UTC, comment #3: 


commit d0933894c66d5c4f759bd4a924bdcf6e2c9ca03f
Author: Dave Kemper <saint.snit@gmail.com>
Date:   Thu Sep 12 03:37:33 2024 +0000

    [tmac]: Fix Savannah #66112.

    * tmac/en.tmac: Map hcodes of Latin-1 characters with diacritical marks
      that are used in English words to their unadorned ASCII counterparts.

    Fixes <https://savannah.gnu.org/bugs/?66112>.


G. Branden Robinson <gbranden>
Group administrator
Thu 12 Sep 2024 03:37:33 AM UTC, comment #2: 

comment #1:

> Dave, would you be willing to prepare an appropriate patch to "tmac/en.tmac"?


I'll do better than that--I'll prepare two.

The first contains only those characters that appear in English words, as defined by the SCOWL word list (version 2020.12.07, the latest as of this writing; http://wordlist.aspell.net/).  This is the most complete, easily greppable word list I know of, but it still may be missing some obscurities.

The second contains all alphabetic Latin-1 characters that have bare ASCII counterparts (omitting, e.g., þ and æ).

The letters included in the first set but excluded from the second are ã, ì, ò, õ, ý, and ÿ, and their uppercase counterparts for those that have them.

Both patches have an introductory comment, both of which point to this bug report's URL for the rationale, as copying the extensive explanation in the original submission into tmac-file comments seemed inelegant.  There may be a better way to handle this, or including a rationale in the comments may not even be necessary.  Those judgment calls I leave to someone who's done more tmac-file hacking than I have.

(file #56424, file #56425)

Dave <barx>
Group Member
Wed 11 Sep 2024 07:41:44 AM UTC, comment #1: 

Dave, would you be willing to prepare an appropriate patch to "tmac/en.tmac"?

G. Branden Robinson <gbranden>
Group administrator
Mon 19 Aug 2024 08:21:05 PM UTC, original submission:  

This is a sequel to the recently fixed bug #59397.

Branden presents a simple test file and its result in that bug report.

$ cat EXPERIMENTS/hw-with-special-characters.groff
.ll 1n
r\['e]sum\['e]
.hcode \['e]e
r\['e]sum\['e]
.pl \n[nl]u
$ ~/groff-1.22.3/bin/groff -ww -W break -T utf8 EXPERIMENTS/hw-with-special-characters.groff
résumé
ré-
sumé

(I'm not sure why he chose groff 1.22.3 to illustrate this; 1.22.4 and 1.23 give the same result.)

Running this test file after #59397's pair of fixes (commit 0629380a9 and commit 56e793e73) has been applied changes the output to:

ré-
sumé
ré-
sumé


The #59397 fixes do what that bug report's summary and original submission asked for: assigned hcodes to the Latin-1 Supplement characters.

What those fixes did not do was this extra step Branden wondered about: "it might be an open question as to whether letters from outside the basic Latin alphabet should necessarily be hyphenated like their basic Latin 'base characters'."

I addressed this question, writing: "When a diacritic changes the syllabication, such as 'expose' vs 'exposé', it will pretty much (I hedge, but can't think of any exceptions) always do so by adding a syllable, and thus a potential break point.  The patterns, presumably, are set up for the unaccented form, meaning groff will never use the additional break point offered by the accented form.  But that's fine: it's better to not break a word in an acceptable spot than to break one in an unacceptable spot.

"And anyway, those are the rarer cases.  More commonly, the break points won't change, such as whether 'coöperate', 'doppelgänger', or 'débâcle' are written with or without the diacritics."

This point was not subsequently addressed, so this might still be an open question.  But let's look at some real words.  The below set is limited to the decorated characters è, ë, ç, and ö.

$ cat 59397.newtest
.hy 4
.
.ll 1n
co\[:o]rdinating
fa\[,c]ade
gar\[,c]onni\[`e]re
pre\[:e]mptively
re\[:e]nacted
unco\[:o]perative
$ nroff -Wbreak 59397.newtest | cat -s
coör-
di-
nat-
ing
façade
garçon-
nière
preëmp-
tively
reë-
n-
acted
un-
coöper-
a-
tive

Post-#59397 groff misses several hyphenation points above (and has at least one bogus one).  Now let's see what happens if we make the "open question" change for the four accented characters used in this sample set.

$ ( echo '.hcode \[,c] c \[:e] e \[`e] e \[:o] o'; cat 59397.newtest ) | nroff -Wbreak | cat -s
co-
ör-
di-
nat-
ing
fa-
çade
gar-
çon-
nière
pre-
ëmp-
tively
reën-
acted
un-
co-
öp-
er-
a-
tive

This is a clear improvement.  While "reënacted" should still have an additional hyphenation point, this is unrelated to the diaeresis; the unadorned word is hyphenated that way too.  And certainly one of its previous break points, "reë- nacted", was subpar.

So this report requests that groff adopt these additional mappings.

I'm not certain whether they're more appropriate for tmac/latin1.tmac (which is where #59397 added the setting of hyphenation codes for these characters) or tmac/en.tmac.

If the latter, they may not need to include every Latin-1 alphabetic character; I'm not aware of any English words that use a thorn or O with stroke, for example.  There's also not any reasonable single ASCII letter to map the thorn to; the O with stroke, for completeness, could be mapped to an ordinary O.

Dave <barx>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #56424:  66112-selected-chars.patch added by barx (969B - text/x-diff - patches to add hcodes to accented Latin-1 characters)
file #56425:  66112-all-chars.patch added by barx (1KiB - text/x-diff - patches to add hcodes to accented Latin-1 characters)

 

Depends on the following items: None found

Items that depend on this one

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 15 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-10-16 gbranden StatusIn Progress Fixed
        Open/ClosedOpen Closed
        Planned ReleaseNone 1.24.0
    2024-10-14 gbranden StatusConfirmed In Progress
        Assigned toNone gbranden
    2024-09-28 barx Summary[l10n:english] map Latin-1 Supplement character hyphenation codes to base-character equivalents [PATCH] [l10n:english] map Latin-1 Supplement character hyphenation codes to base-character equivalents
    2024-09-12 gbranden Dependencies- bugs #65099 is dependent
    2024-09-12 gbranden StatusNone Confirmed
        SummaryMap Latin-1 Supplement character hyphenation codes to their base-character equivalents [l10n:english] map Latin-1 Supplement character hyphenation codes to base-character equivalents
    2024-09-12 barx StatusNeed Info None
        Assigned tobarx None
    2024-09-12 barx Attached File- Added 66112-selected-chars.patch, #56424
        Attached File- Added 66112-all-chars.patch, #56425
    2024-09-11 gbranden StatusNone Need Info
        Assigned toNone barx

    Back to the top

    Powered by Savane 3.14-9aa3.
    Corresponding source code