bugGNU roff - Bugs: bug #66919, [troff] behavior change in some...

 
 

bug #66919: [troff] behavior change in some .hcode calls when a special character is the first argument

Submitter:  Dave <barx>
Submitted:  Sun 16 Mar 2025 02:59:17 PM UTC
   
 
Category:  Core Severity:  3 - Normal
Item Group:  Incorrect behaviour Status:  Need Info
Privacy:  Public Assigned to:  barx
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Post a Comment

Add a New Comment Rich Markup
   

Discussion

Jump to the original submission

Mon 09 Jun 2025 02:28:43 AM UTC, comment #36: 

comment #35:

> I think one of the reasons you perceive a behavior change from
> groff 1.23.0 to the Git trunk has nothing to do with `hcode`'s
> implementation, but bug #59397.


To combat the influence of that change (and effects of other startup files), I've tried to confine my test cases to characters not given .hcode values in any startup file.  (Per the last paragraph of the original submission, "õ" is given no .hcode value in either latin1.tmac or en.tmac.)  If you see any test cases where a startup file affects the behavior, please point it out.

I hope to respond to the meat of comment #34 soon.

Dave <barx>
Group Member
Wed 07 May 2025 05:48:31 PM UTC, comment #35: 

I think one of the reasons you perceive a behavior change from groff 1.23.0 to the Git trunk has nothing to do with `hcode`'s implementation, but bug #59397.

G. Branden Robinson <gbranden>
Group administrator
Sat 03 May 2025 12:56:15 AM UTC, comment #34: 

Hi Dave,

At 2025-05-02T20:48:57-0400, Dave wrote:

> Follow-up Comment #33, bug #66919 (group groff):
>
> More later, but this part is easy:
>
> comment #32:
> > Again I have lost track of your definition of reflexivity here,
>
> I defined it at the end of comment #4.  (If comments are ordered
> earliest-to-latest, it's the first use of the term "reflexive" that
> comes up in this ticket.)


Yes, and it may be my fault that we're using it at all, as it may have
been I who dragged the discrete math syllabus into this mega-thread.

> The term is my way to distinguish between the two distinct things
> .hcode does.
> * ".hcode a a" (for any a) "creates" a previously unused hyphenation
> code for character a.
> * ".hcode a b" (for any a != b) sets the hyphenation code of character
> a to that of character b.


I can readily endorse this definition, but I think doing so only
highlights our clashing understandings elsewhere.

Ordinary characters are not special characters and indexed characters
are neither.  All are distinct kinds of object, both in the lay sense
and in the C++ implementation.

Consequently,


.hcode \[~o] õ


cannot ever be a reflexive hyphenation code assignment, no matter what
hyphenation code values are in play.  A novice, given your definition of
"reflexive", would say that that's obvious from inspection.  I think we
should strive not to overturn that intuition.

> I termed the former syntax "reflexive" by analogy with the
> mathematical reflexive property.  But I have no attachment to this
> term, so I'm happy to change to a better one.  It's just something
> (like "non-copy mode") that needed a name for the sake of discussion,
> so I assigned it one.


Fair.  My present hypothesis is that my immediately previous observation
is where we are clashing.  But I could be mistaken.

G. Branden Robinson <gbranden>
Group administrator
Sat 03 May 2025 12:48:52 AM UTC, comment #33: 

More later, but this part is easy:

comment #32:

> Again I have lost track of your definition of reflexivity here,


I defined it at the end of comment #4.  (If comments are ordered earliest-to-latest, it's the first use of the term "reflexive" that comes up in this ticket.)

The term is my way to distinguish between the two distinct things .hcode does.

  • ".hcode a a" (for any a) "creates" a previously unused hyphenation code for character a.
  • ".hcode a b" (for any a != b) sets the hyphenation code of character a to that of character b.


I termed the former syntax "reflexive" by analogy with the mathematical reflexive property.  But I have no attachment to this term, so I'm happy to change to a better one.  It's just something (like "non-copy mode") that needed a name for the sake of discussion, so I assigned it one.

Dave <barx>
Group Member
Wed 23 Apr 2025 03:33:40 AM UTC, comment #32: 

At 2025-04-16T17:45:11-0400, Dave wrote:

> Follow-up Comment #31, bug #66919 (group groff):
>
> comment #29:
> > comment #27:
> >> Clearly, "can't" overstates this, because it was regarded as one
> >> in past groffs.
> >
> > ...unless you were on an EBCDIC system, or loaded "latin2.tmac".
> >
> >> Additionally, in my example input, at the time that .hcode is run,
> >> the character encoding is known,
> >
> > Not to the formatter.
>
> Both these statements can't be true.  If it was regarded as one in
> past groffs

[snip]

> Determining which one is true is, once again, thwarted by past
> releases lacking a way to query hyphenation codes, and my attempts to
> deduce them through observations of hyphenation behavior have given me
> seemingly contradictory results.


Well, I went to the trouble of preparing a patch to add the erstwhile
`phcode` request to groff 1.23.0 (attached, or at least I'll try, and
hope Savannah's mail gateway gets it into the ticket tracker).

I quickly found out that we didn't set up hyphenation codes for most
"extended ASCII" character codes in 1.23.0, except in the language
localization files!

Go ahead, look for 'em!  (We're gonna see an ocean of Unicode
replacement characters here, because the file of interest are not
encoded in UTF-8, but a variety of other encodings).


$ git describe
1.23.0
$ grep -a hcode tmac/*
tmac/LOCALIZATION:        for the locale are set with the .hcode request.
tmac/X.tmac:.  hcode \\$1\\$4
tmac/X.tmac:.hcode \[S ,]s
tmac/X.tmac:.hcode \[s ,]s
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/cs.tmac:.hcode � �  � �
tmac/de.tmac:.hcode � �  � �  � �  � �  � �  � �  � �
tmac/de.tmac:.hcode � �
tmac/de.tmac:.hcode � �  � �  � �  � �
tmac/de.tmac:.hcode � �  � �  � �  � �
tmac/de.tmac:.hcode � �
tmac/de.tmac:.hcode � �  � �  � �  � �  � �
tmac/de.tmac:.hcode � �  � �  � �
tmac/de.tmac:.hcode � �  � �  � �  � �  � �  � �  � �
tmac/de.tmac:.hcode � �
tmac/de.tmac:.hcode � �  � �  � �  � �
tmac/de.tmac:.hcode � �  � �  � �  � �
tmac/de.tmac:.hcode � �
tmac/de.tmac:.hcode � �  � �  � �  � �  � �
tmac/de.tmac:.hcode � �  � �  � �
tmac/de.tmac:.hcode � �
tmac/dvi.tmac:.  hcode \\$1\\$4
tmac/dvi.tmac:.hcode \[,C]c
tmac/dvi.tmac:.hcode \[,c]c
tmac/dvi.tmac:.hcode \[S ,]s
tmac/dvi.tmac:.hcode \[s ,]s
tmac/ec.tmac:.\" hcode values are not handled.
tmac/fallbacks.tmac:.\"fchar \[u2011] -\" non-breaking hyphen (won't break w/o .hcode or \:)
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/fr.tmac:.hcode � �  � �
tmac/lbp.tmac:.  hcode \\$1\\$4
tmac/lbp.tmac:.hcode \[S ,]s
tmac/lbp.tmac:.hcode \[s ,]s
grep: tmac/mdoc: Is a directory
tmac/ps.tmac:.  hcode \\$1\\$4
tmac/ps.tmac:.hcode \[S ,]s
tmac/ps.tmac:.hcode \[s ,]s
tmac/psold.tmac:.ie '\\$3'\(.i' .hcode \\$1i
tmac/psold.tmac:.el .hcode \\$1\\$3
tmac/sv.tmac:.hcode � �  � �
tmac/sv.tmac:.hcode � �  � �
tmac/sv.tmac:.hcode � �  � �
tmac/sv.tmac:.hcode � �  � �
grep: tmac/tests: Is a directory


> in latin1 but not in latin2, then the formatter did somehow take the
> input encoding into account.  Conversely, if the formatter knew
> nothing about the encoding, then past groffs must have treated the
> request the same regardless of whether latin1 or latin2 was in effect.


I can't easily simulate a CCSID 1047 environment (EBCDIC), but I can
look at English, the default language, which uses Latin-1 (and has a õ
character), and Czech, which uses Latin-2 (and lacks a õ character).

Here are the results.


$ printf '.phcode ab\\[:e]\\[~o]\n' | ./build/test-groff
a       97
b       98
\[:e]   0
\[~o]   0
$ printf '.phcode ab\\[:e]\\[~o]\n' | ./build/test-groff -men
a       97
b       98
\[:e]   0
\[~o]   0
$ printf '.phcode ab\\[:e]\\[~o]\n' | ./build/test-groff -mcs
a       97
b       98
\[:e]   0
\[~o]   0


> I was trying to reconcile these when my attention got diverted to
> other (non-groff) matters, where it may remain a while yet.  So this
> question remains unresolved unless you care to tackle it before I'm
> able to return to it.
>
> But the answer would seem to determine what direction this ticket
> should take.  The byte 0xF5 represents LATIN SMALL LETTER O WITH TILDE
> in Latin-1 and LATIN SMALL LETTER O WITH DOUBLE ACUTE in Latin-2.  So,
> in older groffs with Latin-2 loaded, for the input ".hcode \[~o]"
> followed by the byte 0xF5:
> * If the formatter treated this as reflexive, then this was a bug,
> which goes a long way toward quashing objections to the behavior
> change.


It treated that input as assigning a hyphenation code to a character
that didn't have one before.  Observe.


$ printf '.phcode \\[o~]\n.hcode \\[o~] \365\n.phcode \\[o~]\n' | ./build/test-groff
\[~o]   0
\[~o]   245
$ printf '.phcode \\[~o]\n.hcode \\[~o] \365\n.phcode \\[~o]\n' | ./build/test-groff -men
\[~o]   0
\[~o]   245
$ printf '.phcode \\[~o]\n.hcode \\[~o] \365\n.phcode \\[~o]\n' | ./build/test-groff -mcs
\[~o]   0
\[~o]   245


> * If the formatter didn't treat this as reflexive, then it somehow
> knew the encoding, undermining much of the justification for the
> behavior change.


Again I have lost track of your definition of reflexivity here, so I
remain uncertain as to what you think the above illustrates.

Once reminded where we weren't setting up hyphenation codes as
recently as the 1.23.0 release, I saw nothing in the foregoing that
surprised me.


(file #57163)

G. Branden Robinson <gbranden>
Group administrator
Wed 16 Apr 2025 09:45:07 PM UTC, comment #31: 

comment #29:

> comment #27:
> > Clearly, "can't" overstates this, because it was regarded as one in past groffs.
>
> ...unless you were on an EBCDIC system, or loaded "latin2.tmac".
>
> >  Additionally, in my example input, at the time that .hcode is run, the character encoding is known,
>
> Not to the formatter.


Both these statements can't be true.  If it was regarded as one in past groffs in latin1 but not in latin2, then the formatter did somehow take the input encoding into account.  Conversely, if the formatter knew nothing about the encoding, then past groffs must have treated the request the same regardless of whether latin1 or latin2 was in effect.

Determining which one is true is, once again, thwarted by past releases lacking a way to query hyphenation codes, and my attempts to deduce them through observations of hyphenation behavior have given me seemingly contradictory results.  I was trying to reconcile these when my attention got diverted to other (non-groff) matters, where it may remain a while yet.  So this question remains unresolved unless you care to tackle it before I'm able to return to it.

But the answer would seem to determine what direction this ticket should take.  The byte 0xF5 represents LATIN SMALL LETTER O WITH TILDE in Latin-1 and LATIN SMALL LETTER O WITH DOUBLE ACUTE in Latin-2.  So, in older groffs with Latin-2 loaded, for the input ".hcode \[~o]" followed by the byte 0xF5:

  • If the formatter treated this as reflexive, then this was a bug, which goes a long way toward quashing objections to the behavior change.
  • If the formatter didn't treat this as reflexive, then it somehow knew the encoding, undermining much of the justification for the behavior change.
Dave <barx>
Group Member
Thu 10 Apr 2025 07:00:28 PM UTC, comment #30: 

Hi Dave,

At 2025-04-03T07:08:32-0400, G. Branden Robinson wrote:

> Yes.  I don't disagree with documenting the change in NEWS, but I'd
> like your help in constructing it.  Collaborating on this language
> will also give me the opportunity to correct lingering
> misunderstandings on both our parts.


I haven't heard back from you on this, so would you like me to make my
own stab at it?

Regards,
Branden

G. Branden Robinson <gbranden>
Group administrator
Thu 03 Apr 2025 11:08:24 AM UTC, comment #29: 

comment #27:

> comment #26:
> > Here's another clash of terminology we have.  I assert that:
> >
> > .hcode \[~o] õ
> >
> > ...*is not a reflexive hyphenation code assignment*.
>
> This, I believe, is the clash at the root of this ticket.
>
> > It can't be, because the meaning of "õ" depends on the
> > character encoding in a way that "\[~o]" does not.
>
> Clearly, "can't" overstates this, because it was regarded as one in past groffs.


...unless you were on an EBCDIC system, or loaded "latin2.tmac".  I know the latter group of people exists because they periodically write to the groff list and other forums looking for better font coverage and support for Czech, Polish, and so on.

>  Additionally, in my example input, at the time that .hcode is run, the character encoding is known,


Not to the formatter.  The character set localization files are simply piles of `trin` and `hcode` requests.  Nothing tells the formatter "hey, you're using Latin-9".

Yes, a user-defined register or string could be created to track this information, but as a rule, the formatter, by which I mean the GNU troff executable, does not alter its behavior based on the values of registers or strings that the user (here defined to include startup files) that defines.
 

>  so it could still be interpreted as reflexive.


I think that's a big stretch in this scenario.  We seem to be concerned about different minority groups.

1.  You're concerned about regressing (or altering) the hyphenation behavior of Latin-1/English documents that are sensitive to the automatic hyphen processes applied to words that use letters not occurring in the English language.  I admit I'm pretty skeptical of this usage, and wonder why authors of such documents with this much sensitivity haven't been using `hw` all along.

2.  I'm concerned about unjustifiably extending assumptions that are valid (only) in Latin-1 and English to other character encodings and other languages.  We know these people exist because they've gone to the trouble of submitting macro files preparing GNU troff to correctly handle these encodings and languages.

> These facts suggest two possible ways forward.
> * Use character-encoding knowledge, if available at the time .hcode is run, to interpret any 8-bit arguments; or


I don't think this information is available to the formatter, per the above.

> * Decide this is a back-compatibility-breaking change.  This should then require two other things, and suggest an optional third:
> ** A justification for what larger purpose is being served by the behavior change;


See item 2 above.

> ** Clear notice in NEWS that the behavior has changed;


I invite you to draft this notice, scoped carefully to the impact of the change.

> ** Optionally, a warning when the user invokes an .hcode that previously did one thing and now does another.


Again, the formatter doesn't have this information.  All it sees is, say, character code 245.  It doesn't know what that means.


.hcode \[~o] õ \" Latin-1
.hcode \[~o] 5 \" EBCDIC
.hcode \[~o] ő \" Latin-2


The "source" arguments are all encoded identically on disk and in memory.
 

> I get the resistance to warn about something that's not actually incorrect.  (Though, arguably, that's exactly what a "warning" is.)  But in this instance, the effect of an .hcode behavior change will typically only be visible in a document at some remove from its cause.


Fair point.

> Users who might otherwise spend significant and frustrating time tracking down the source of a change could instead be given a helpful arrow pointing directly to it.


Yes.  I don't disagree with documenting the change in NEWS, but I'd like your help in constructing it.  Collaborating on this language will also give me the opportunity to correct lingering misunderstandings on both our parts.

> For hyphenation in particular, even the smallest changes (in wording, in paper or margin size, in any number of other things) can affect where words are broken throughout a document.  So it's just as plausible that after a groff upgrade, a user will see no change at all to their document, even though its interpretation of some .hcode calls does something different, simply because of where word breaks happen to fall.  The change in behavior may only occur during a later, possibly quite minor, revision of the document.  This is why, for this change in particular, it's especially useful for groff to emit notice that .hcode is doing something new.


This argument makes sense to me.

comment #28:

> Also, this older remark has bearing on my last comment.
>
>
> comment #19:
> > But generally in language development, if you break someone's
> > reliance on undefined behavior, you don't owe them an apology
> > and possibly not even notice.
>
> True, and I argued exactly that in my last reply to Deri in bug #50770.  But we need to be scrupulous in how we define "undefined."
>
> A formal language specification tells you exactly what the defined behavior is, and therefore, by implication, what is undefined.  The roff language has never had such a spec.
>
> Even absent that, documentation might specifically call out certain constructions as having undefined behavior.
>
> But undocumented does not necessarily imply undefined (unless the documentation so states); it could merely mean no one's written it down yet.  And if that's the case, it's perfectly reasonable for users to determine the behavior experimentally.  (In many cases, you've done that in order to write something down.)  And when that experimentally determined behavior persists across multiple releases across multiple decades, it becomes harder to justify changing it with impunity just because no one's gotten around to writing down what it did across all those releases and all those decades.


I think the magnitude of "harder" is a pretty wild variable.

Consider this NEWS item for 1.24:


*  GNU troff no longer accepts a newline as a delimiter for the
   parameterized escape sequences `\A`, `\b`, `\o`, `\w`, `\X`, and
   `\Z`.


No deprecation period, no nothing.  ("No warning, no second chance!")

But nobody has yet seen fit to defend the practice being foreclosed by the above change.

That's why I want your input on a NEWS item for this ticket.  As the voice of caution, I can trust you to paint the change as dramatically as possible, and if I catch you in overstatement, I can file that down.  What remains should suffice to coach users that bother to read the "NEWS" file at all to consider their use of the affected features.

I think the number of people using Latin-1 input and setting up hyphenation codes for accented Latin letters not found in English words is small.  But they could exist; as I noted, people might be writing fantasy novels in English using the occasional funny letter.  I don't mind warning them to double-check their documents against `hcode`'s changed behavior.

So: please propose a "NEWS" item for this change.

G. Branden Robinson <gbranden>
Group administrator
Tue 01 Apr 2025 11:49:54 AM UTC, comment #28: 

Also, this older remark has bearing on my last comment.


comment #19:

> But generally in language development, if you break someone's
> reliance on undefined behavior, you don't owe them an apology
> and possibly not even notice.


True, and I argued exactly that in my last reply to Deri in bug #50770.  But we need to be scrupulous in how we define "undefined."

A formal language specification tells you exactly what the defined behavior is, and therefore, by implication, what is undefined.  The roff language has never had such a spec.

Even absent that, documentation might specifically call out certain constructions as having undefined behavior.

But undocumented does not necessarily imply undefined (unless the documentation so states); it could merely mean no one's written it down yet.  And if that's the case, it's perfectly reasonable for users to determine the behavior experimentally.  (In many cases, you've done that in order to write something down.)  And when that experimentally determined behavior persists across multiple releases across multiple decades, it becomes harder to justify changing it with impunity just because no one's gotten around to writing down what it did across all those releases and all those decades.

Dave <barx>
Group Member
Fri 28 Mar 2025 04:04:41 PM UTC, comment #27: 

This is a lot to digest!  So I'll focus on the point that seems central to the issue.

comment #26:

> Here's another clash of terminology we have.  I assert that:
>
> .hcode \[~o] õ
>
> ...*is not a reflexive hyphenation code assignment*.


This, I believe, is the clash at the root of this ticket.

> It can't be, because the meaning of "õ" depends on the
> character encoding in a way that "\[~o]" does not.


Clearly, "can't" overstates this, because it was regarded as one in past groffs.  Additionally, in my example input, at the time that .hcode is run, the character encoding is known, so it could still be interpreted as reflexive.

These facts suggest two possible ways forward.

  • Use character-encoding knowledge, if available at the time .hcode is run, to interpret any 8-bit arguments; or
  • Decide this is a back-compatibility-breaking change.  This should then require two other things, and suggest an optional third:
    • A justification for what larger purpose is being served by the behavior change;
    • Clear notice in NEWS that the behavior has changed;
    • Optionally, a warning when the user invokes an .hcode that previously did one thing and now does another.


I get the resistance to warn about something that's not actually incorrect.  (Though, arguably, that's exactly what a "warning" is.)  But in this instance, the effect of an .hcode behavior change will typically only be visible in a document at some remove from its cause.  Users who might otherwise spend significant and frustrating time tracking down the source of a change could instead be given a helpful arrow pointing directly to it.

For hyphenation in particular, even the smallest changes (in wording, in paper or margin size, in any number of other things) can affect where words are broken throughout a document.  So it's just as plausible that after a groff upgrade, a user will see no change at all to their document, even though its interpretation of some .hcode calls does something different, simply because of where word breaks happen to fall.  The change in behavior may only occur during a later, possibly quite minor, revision of the document.  This is why, for this change in particular, it's especially useful for groff to emit notice that .hcode is doing something new.

Dave <barx>
Group Member
Wed 26 Mar 2025 10:21:07 AM UTC, comment #26: 

Okay, I told you to hold your horses in comment #21, but like me, you have a loose grip on the reins when galloping through Esoteric Argument Pass.

comment #23:

> Now we get to where my conceptual groundwork of comment #20 starts interacting with concrete examples.
>
> comment #19:
> > So I would say that this:
> >
> > $ printf '.hcode A A\n.pchar aA\n' | groff 2>&1 | grep -E '(char|hyph)'
> > [snip]
> >
> > ...illustrates neither "generation" of a (presumptively new) hyphenation
> > code
>
> No character previously had a hyphenation code of 65, so groff has, conceptually, generated one.  Your input above tells groff, "I want the character 'A' to be considered a potential hyphenation point, and to be considered not equivalent to any existing characters with hyphenation codes."


No, that's not what that means.

If I have this sequence:


.hcode A A
.hcode B A
.hcode A a
.hcode A A


...then my second invocation of `hcode A A` does in fact make its hyphenation code equivalent to that of an existing character: "B".  This possibility is one reason I indulged the seemingly tedious digression into surjective relations.  We're not dealing with pointers, references, or other forms of indirection here, as popular as they are in programming.  What we have is a translation table from character codes (in the case of ordinary characters) to hyphenation codes.

`tr` works similarly.  When we update the map of "from" to "to" in a pairing, the request pays no heed to the existing mapping of "from".

If I have:


.tr aZbZcZ


...no *roff forgets how to distinguish "a", "b", and "c" from each other.  It knows, and we can therefore undo the equivalence relation.


.tr aabbcc


Other operations, like request deletion and special-built-in-register deletion, do work ultimately with references of a kind, and are therefore irreversible.  When the only reference to a thing is discarded, it is gone forever.  (Until a new GNU troff process is constructed afresh.)

>  This matches the typical user's (i.e., one who isn't peering into the implementation) understanding of what it means to generate a new hyphenation code.
>
> Your own words, in fact, betray your view being colored by knowledge of formatter internals:
>
> > It was 97, and we knocked it back to 65.
>
> It was, in no sense meaningful to a user, knocked back to anything, as it never had 65 as a hyphenation code.  You happen to know James Clark's initialization algorithm, but even that is arbitrary: Clark could just as easily have initialized "A" and "a" to the same value without going through his set-it-then-change-it routine, at which point the "back" in your sentence becomes an actual and not merely conceptual misstatement.


That's true, but he didn't.  And I plead guilty to coloration by knowledge of GNU troff internals.  It's been a lot of effort to acquire what I have, which isn't enough to suit me.

> > I say that, for English, you cannot assume that `\[o~]` is
> > going to behave just like `õ`.  The latter is _not defined in the
> > English alphabet_, so you can't rely on its hyphenation code having any
> > particular value.
>
> That's a fair statement.  But even though I'm running groff with its default startup (English) files, the behavior I'm talking about in this ticket is in the formatter, not in any startup files.  What I'm talking about has nothing to do with the input language and everything to do with input encoding.  (You'll notice that I'm not providing any sample input with any English words.  The two words I've used, lanteronial, and lanterõnial--and then only to work around the lack of .pchar in older groffs--aren't part of any language that I'm aware of.  So I'm talking about general formatter behavior, independent of any language setting.)


Now that I think I've root-caused this, I think I can pretty much agree with that.
 

> In all other respects, groff treats \[o~] and the Latin-1 õ as the same character.


That, I think, is too bold a claim.  The only reason groff does that, ever, is because "latin1.tmac" sets it up.

https://git.savan ... tmac?h=1.23.0#n96

This hasn't changed in HEAD.

And of course in other encodings, "char245" is the same as something else.

https://git.savan ... mac?h=1.23.0#n224

> If you want to make .hcode treat them as different, the documentation should clearly highlight this difference.


I won't rule out that we can improve our documentation of hyphenation codes, but we should reach a meeting of the minds on the topic of this ticket first.
 

> But I suspect that, once you start trying to explain to readers why ".hcode \[o~] õ" and ".hcode õ õ" behave differently in Latin-1 input, you might start to think that's not actually such a wise thing for groff to do.


I don't agree, because whether they should be equivalent depends on whether you're writing in English.  Although, if you are, even groff HEAD treats them the same for hyphenation purposes: they both get a big fat zero.

> > But one thing's for sure, in the KOI8-R locale:
> >
> > .hcode \[~o] õ
> >
> > ...is *not* reflexive, because in that locale it really looks like
> > this:
> >
> > .hcode \[~o] Т \" CYRILLIC CAPITAL LETTER TE
>
> Right, the meaning of any character with the 8th bit set depends on the input encoding.  That's true whether we're talking about .hcode or any other part of groff input.  So this point is not really relevant to .hcode itself.


But it is!  From groff 1.23.0 back to the Dawn of Man, employing `hcode` using an ordinary character with its 8th bit set as the source could create a "new" hyphenation code!
 

> My examples are all Latin-1 input, which I've tried to be clear about (in the comment #0 example, by running "file" on the input; in the comment #4 one, by stating the encoding before presenting the input file).  If groff offered a special character for CYRILLIC CAPITAL LETTER TE, I'm certain that using that with the above .hcode example in KOI8-R encoding would reveal the same behavior.


You've been crystal clear about which encoding you're using.  I confess I'm being a stickler that Latin-1 doesn't necessarily mean English, and, per our discussion in bug #66112, there are accented Latin letters that do not play a role in English orthography and therefore should not be assigned default hyphenation codes by groff.

Moreover, if the English localization file doesn't assign a hyphenation code one to a character, `hcode` should not be able to magic one up out of thin air--unless the user explicitly asks for reflexive code assignment.  The user can then knock themselves out setting up special character synonyms.  By hand, by loading "rfc1345.tmac", or what-have-you.
 

> > I claim that, in groff, special characters cannot, and have never
> > been able to, participate in reflexive hyphenation code assignments.
>
> I crafted the example in comment #4 precisely to show the change between older groffs and the current one when using a special character in a reflexive hcode.


Here's another clash of terminology we have.  I assert that:


.hcode \[~o] õ


...*is not a reflexive hyphenation code assignment*.

It can't be, because the meaning of "õ" depends on the character encoding in a way that "\[~o]" does not.

I agree with you that


.hcode õ õ


...where õ is any ordinary character, is a reflexive hyphenation code assignment.  But I claim that the source and destination have to be input-character-identical for it to be the case.

G. Branden Robinson <gbranden>
Group administrator
Wed 26 Mar 2025 09:37:05 AM UTC, comment #25: 

This might show up twice.  The Savannah bug tracker just glitched for me HTTP-wise in a kind of weird way.

Since I know that was a boatload of stuff to read, here's a simpler illustrator of one of my claims.

> we might retitle this ticket "[troff] behavior change in some
> .hcode calls when an ordinary character is the second argument",
> and you can guess what my proposed resolution is.


Here's my evidence.  Let's throw out the confounding factor of a special character altogether.


$ iconv -f iso-8859-1 EXPERIMENTS/lanteronial-Z.groff
.ll 1n
lanteronial
lanterZnial
.hcode Z õ
lanterZnial
$ ~/groff-HEAD/bin/groff -a -Wbreak EXPERIMENTS/lanteronial-Z.groff
<beginning of page>
lantero<hy>
nial
lanterZ<hy>
nial
lanterZnial
$ ~/groff-1.23.0/bin/groff -a -Wbreak EXPERIMENTS/lanteronial-Z.groff
<beginning of page>
lantero<hy>
nial
lanterZ<hy>
nial
lanterZ<hy>
nial
$ ~/groff-1.22.4/bin/groff -a -Wbreak EXPERIMENTS/lanteronial-Z.groff
<beginning of page>
lantero<hy>
nial
lanterZ<hy>
nial
lanterZ<hy>
nial
$ ~/groff-1.22.3/bin/groff -a -Wbreak EXPERIMENTS/lanteronial-Z.groff
<beginning of page>
lantero<hy>
nial
lanterZ<hy>
nial
lanterZ<hy>
nial


A behavior change, yes, one that is completely symmetric with yours except for my use of an ordinary character in place of your special one.

I claim the formatter cannot assume which hyphenation codes are valid (apart from, arguably, a-z and A-Z, because inertia and Western-centrism).

Selection of any other valid hyphenation codes is up to the encoding macro files and, I submit eventually, solely the localization macro files.

G. Branden Robinson <gbranden>
Group administrator
Wed 26 Mar 2025 09:22:12 AM UTC, comment #24: 

comment #23:

> Now we get to where my conceptual groundwork of comment #20 starts interacting with concrete examples.


Good, 'cause I have some too!  :D

First let me start with this illustrator.


$ printf '.ll 3n\ndomain\n' | groff -a -Wbreak
<beginning of page>
do<hy>
main


GNU troff will break in the same place any word with a letter equivalent to "o" in the same place.


$ printf '.ll 3n\nd\[`o]main\n' | groff -a -Wbreak
<beginning of page>
d<`o><hy>
main


Recalling from our discussion in bug #66112, and my selection of your first suggestion over your second, o-with-tilde-accent is not equivalent to "o" in English, so it shouldn't break...


$ printf '.ll 3n\nd\[~o]main\n' | groff -a -Wbreak
<beginning of page>
d<~o>main


...and indeed it doesn't.

That established...

> That's a fair statement.  But even though I'm running groff with its default startup (English) files, the behavior I'm talking about in this ticket is in the formatter, not in any startup files.  What I'm talking about has nothing to do with the input language and everything to do with input encoding.


I agree!

>  (You'll notice that I'm not providing any sample input with any English words.  The two words I've used, lanteronial, and lanterõnial--and then only to work around the lack of .pchar in older groffs--aren't part of any language that I'm aware of.  So I'm talking about general formatter behavior, independent of any language setting.)


But you're not talking about general formatter behavior, you're talking about formatter behavior *after the "latin1.tmac" file is loaded*.

Observe.


$ printf '.ll 3n\nd\\[~o]main\n' | ~/groff-HEAD/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main
$ printf '.ll 3n\nd\\[~o]main\n' | ~/groff-1.23.0/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main
$ printf '.ll 3n\nd\\[~o]main\n' | ~/groff-1.22.4/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main
$ printf '.ll 3n\nd\\[~o]main\n' | ~/groff-1.22.3/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main


Let's try it with the "raw" o with tilde accent character, Latin-1 245 decimal (365 octal).


$ printf '.ll 3n\nd\365main\n' | ~/groff-HEAD/bin/troff -Ra -Wbreak
<beginning of page>
/home/branden/groff-HEAD/bin/troff:<standard input>:2: warning: character with input code 245 not defined
dmain
$ printf '.ll 3n\nd\365main\n' | ~/groff-1.23.0/bin/troff -Ra -Wbreak
<beginning of page>
/home/branden/groff-1.23.0/bin/troff:<standard input>:2: warning: character with input code 245 not defined
dmain
$ printf '.ll 3n\nd\365main\n' | ~/groff-1.22.4/bin/troff -Ra -Wbreak
<beginning of page>
/home/branden/groff-1.22.4/bin/troff: <standard input>:2: warning: can't find character with input code 245
dmain
$ printf '.ll 3n\nd\365main\n' | ~/groff-1.22.3/bin/troff -Ra -Wbreak
<beginning of page>
<standard input>:2: warning: can't find character with input code 245
dmain


This makes sense, because in all released versions of groff, the formatter doesn't yet know, before loading startup files, whether it's going to be operating in a Latin-1 or EBCDIC (code page 1047) environment.  (Well, technically it can know just by checking the character code of, say, "a", but it stays as agnostic as it can and lets macro files do most of the lifting.)

Let's macro-load "latin1.tmac" in our examples and see if that changes anything.


$ printf '.mso latin1.tmac\n.ll 3n\nd\365main\n' | ~/groff-HEAD/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main
$ printf '.mso latin1.tmac\n.ll 3n\nd\365main\n' | ~/groff-1.23.0/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main
$ printf '.mso latin1.tmac\n.ll 3n\nd\365main\n' | ~/groff-1.22.4/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main
$ printf '.mso latin1.tmac\n.ll 3n\nd\365main\n' | ~/groff-1.22.3/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main


The character code is now recognized, and translated on input (`trin`) to the special character `~o`.  But it still doesn't hyphenate.

For completeness, let's see if explicitly specifying the special character changes behavior.


$ printf '.mso latin1.tmac\n.ll 3n\nd\\[~o]main\n' | ~/groff-HEAD/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main
$ printf '.mso latin1.tmac\n.ll 3n\nd\\[~o]main\n' | ~/groff-1.23.0/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main
$ printf '.mso latin1.tmac\n.ll 3n\nd\\[~o]main\n' | ~/groff-1.22.4/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main
$ printf '.mso latin1.tmac\n.ll 3n\nd\\[~o]main\n' | ~/groff-1.22.3/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main


Still no.  Finally let's load "en.tmac", which didn't exist prior to 1.23.0.


$ printf '.mso en.tmac\n.ll 3n\nd\\[~o]main\n' | ~/groff-HEAD/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main
$ printf '.mso en.tmac\n.ll 3n\nd\\[~o]main\n' | ~/groff-1.23.0/bin/troff -Ra -Wbreak
<beginning of page>
d<~o>main
$ printf '.mso en.tmac\n.ll 3n\nd\\[~o]main\n' | ~/groff-1.22.4/bin/troff -Ra -Wbreak
/home/branden/groff-1.22.4/bin/troff: <standard input>:1: warning: can't find macro file 'en.tmac'
<beginning of page>
d<~o>main
$ printf '.mso en.tmac\n.ll 3n\nd\\[~o]main\n' | ~/groff-1.22.3/bin/troff -Ra -Wbreak
<standard input>:1: warning: can't find macro file `en.tmac'
<beginning of page>
d<~o>main


So here are a bunch more cases where formatter behavior doesn't change, all using the same special character you've chosen.


$ printf '.ll 3n\nd\\[~o]main\n' | ~/groff-HEAD/bin/groff -a -Wbreak
<beginning of page>
d<~o>main
$ printf '.ll 3n\nd\\[~o]main\n' | ~/groff-1.23.0/bin/groff -a -Wbreak
<beginning of page>
d<~o>main
$ printf '.ll 3n\nd\\[~o]main\n' | ~/groff-1.22.4/bin/groff -a -Wbreak
<beginning of page>
d<~o>main
$ printf '.ll 3n\nd\\[~o]main\n' | ~/groff-1.22.3/bin/groff -a -Wbreak
<beginning of page>
d<~o>main


Why does "lanteronial" (not an English word) hyphenate differently from "domain" (definitely an English word)?  To answer that requires a source dive, which is coming shortly.  But first, I must ask:

So the hyphenation of a non-English word using a letter that doesn't exist in the English alphabet has changed from groff 1.23.0 to (what will become) 1.24.0.  Is it fair to call that a regression?

I think you've identified a relatively dusty crevice in a corner case, and that it arises solely due a presumption that was being made in `set_hyphenation_code()` for many years.

So why did commit a52141ac46eef95dd1f85e4c2e0a336affa9bcc9 change things?

Let's look at the diff again.


diff --git a/src/roff/troff/input.cpp b/src/roff/troff/input.cpp
index cc7d9dd71..946b93570 100644
--- a/src/roff/troff/input.cpp
+++ b/src/roff/troff/input.cpp
@@ -7309,25 +7309,26 @@ static void set_hyphenation_codes()
       error("cannot use the hyphenation code of a numeral");
       break;
     }
-    unsigned char new_code = 0; // TODO: int
+    unsigned char new_code = 0;
     charinfo *cisrc = tok.get_char();
-    if (csrc != 0)
-      new_code = csrc;
-    else {
+    if (cisrc != 0 /* nullptr */)
+      // Common case: assign destination character the hyphenation code
+      // of the source character.
+      new_code = cisrc->get_hyphenation_code();
+    if (0 == csrc) {
       if (0 /* nullptr */ == cisrc) {
        error("expected ordinary or special character, got %1",
              tok.description());
        break;
       }
-      // source character is special
-      if (0 == cisrc->get_hyphenation_code()) {
-       error("second member of hyphenation code pair must be an"
-             " ordinary character, or a special character already"
-             " assigned a hyphenation code");
-       break;
-      }
       new_code = cisrc->get_hyphenation_code();
     }
+    else {
+      // If assigning a ordinary character's hyphenation code to itself,
+      // use its character code point as the value.
+      if (csrc == cdst)
+       new_code = tok.ch();
+    }
     cidst->set_hyphenation_code(new_code);
     if (cidst->get_translation()
        && cidst->get_translation()->get_translation_input())


...and at your test case (the UTF-8 version for readability in Savannah, *not* bug-reproducibility).


$ cat EXPERIMENTS/lanteronial-utf8.groff
.ll 1n
lanteronial
lanter\[~o]nial
.hcode \[~o] õ
lanter\[~o]nial


You've only got the one `hcode` invocation, so that's good.

What was its path through the old code?

https://git.savan ... 5d9e59b7ca1#n7278

Well, we have a bunch of validity checking/error handling first.

Eventually, if we've got two (mostly) valid arguments, we end up on line 7312.

If `csrc` is not zero, the source character is "ordinary".  (If it is zero, it could be anything, like a horizontal motion escape sequence.  But in valid cases, if it's zero it's a special or indexed character.)  And so that branch should be taken for the "lanteronial" file.  `new_code` becomes its value (7315) and we skip to 7331, where the `charinfo` of the destination character is set to that value.

We then worry about whether the destination character is "translated" (which I *think* refers to `tr` translation but I haven't ruled out `trin` or `trnt` translations instead, because it seems that no good item of terminology should be permitted to apply to only one concept in a program), if it is, that new code is immediately superseded by that of its translation (7334).

Then the function ends.

Okay, what about after the "bad commit"?

https://git.savan ... 36affa9bcc9#n7278

We start off again at line 7312.

We don't make a decision about `csrc` right away.  Instead we gather the source character's hyphenation code immediately, if it has one (7314-7317), then if the source character is special, we proceed as before (7319-7324).  But in this case, the source character is ordinary, so we check to see if the character is being assigned to itself, and if so apply this "reflexive case" (7329-7330).  But we won't take that branch either because the test on line 7329 will fail: `csrc` is 245 decimal, but `cdst` is 0 because it's a special character.  We then hit line 7332 where we assign `new_code` to `cidst`.  But remember line 7317.  `cisrc`'s hyphenation code would be zero, because because that's the value it has when the formatter starts up ("troff -R"), and neither "en.tmac" nor "latin1.tmac" ever assigned it a hyphenation code.

The bottom line is that there is a logic change.  Before "bad commit", `new_code` got populated presumptively with the character code of the source character, *if the character was ordinary*.

https://git.savan ... 5d9e59b7ca1#n7315

In the new logic, it doesn't.  It didn't occur to me that that assumption was warranted.  The character code might not be meaningful as a hyphenation code in the language.  `set_hyphenation_code()` has, for many years, been aggressively assuming that it was, if you had the audacity to use an ordinary character as the source character (second argument) in an `hcode` request.

I'd say the "bad commit" is a bug fix.

So we might retitle this ticket "[troff] behavior change in some .hcode calls when an ordinary character is the second argument", and you can guess what my proposed resolution is.

But I want to hear your take.

G. Branden Robinson <gbranden>
Group administrator
Wed 26 Mar 2025 12:09:42 AM UTC, comment #23: 

Now we get to where my conceptual groundwork of comment #20 starts interacting with concrete examples.

comment #19:

> So I would say that this:
>
> $ printf '.hcode A A\n.pchar aA\n' | groff 2>&1 | grep -E '(char|hyph)'
> [snip]
>
> ...illustrates neither "generation" of a (presumptively new) hyphenation
> code


No character previously had a hyphenation code of 65, so groff has, conceptually, generated one.  Your input above tells groff, "I want the character 'A' to be considered a potential hyphenation point, and to be considered not equivalent to any existing characters with hyphenation codes."  This matches the typical user's (i.e., one who isn't peering into the implementation) understanding of what it means to generate a new hyphenation code.

Your own words, in fact, betray your view being colored by knowledge of formatter internals:

> It was 97, and we knocked it back to 65.


It was, in no sense meaningful to a user, knocked back to anything, as it never had 65 as a hyphenation code.  You happen to know James Clark's initialization algorithm, but even that is arbitrary: Clark could just as easily have initialized "A" and "a" to the same value without going through his set-it-then-change-it routine, at which point the "back" in your sentence becomes an actual and not merely conceptual misstatement.

> I say that, for English, you cannot assume that `\[o~]` is
> going to behave just like `õ`.  The latter is _not defined in the
> English alphabet_, so you can't rely on its hyphenation code having any
> particular value.


That's a fair statement.  But even though I'm running groff with its default startup (English) files, the behavior I'm talking about in this ticket is in the formatter, not in any startup files.  What I'm talking about has nothing to do with the input language and everything to do with input encoding.  (You'll notice that I'm not providing any sample input with any English words.  The two words I've used, lanteronial, and lanterõnial--and then only to work around the lack of .pchar in older groffs--aren't part of any language that I'm aware of.  So I'm talking about general formatter behavior, independent of any language setting.)

In all other respects, groff treats \[o~] and the Latin-1 õ as the same character.  If you want to make .hcode treat them as different, the documentation should clearly highlight this difference.

But I suspect that, once you start trying to explain to readers why ".hcode \[o~] õ" and ".hcode õ õ" behave differently in Latin-1 input, you might start to think that's not actually such a wise thing for groff to do.

> But one thing's for sure, in the KOI8-R locale:
>
> .hcode \[~o] õ
>
> ...is *not* reflexive, because in that locale it really looks like
> this:
>
> .hcode \[~o] Т \" CYRILLIC CAPITAL LETTER TE


Right, the meaning of any character with the 8th bit set depends on the input encoding.  That's true whether we're talking about .hcode or any other part of groff input.  So this point is not really relevant to .hcode itself.

My examples are all Latin-1 input, which I've tried to be clear about (in the comment #0 example, by running "file" on the input; in the comment #4 one, by stating the encoding before presenting the input file).  If groff offered a special character for CYRILLIC CAPITAL LETTER TE, I'm certain that using that with the above .hcode example in KOI8-R encoding would reveal the same behavior.

> I claim that, in groff, special characters cannot, and have never
> been able to, participate in reflexive hyphenation code assignments.


I crafted the example in comment #4 precisely to show the change between older groffs and the current one when using a special character in a reflexive hcode.  Using the made-up word seemed easier to me than backporting .pchar, but if the latter is easy for you to do, it can show the effect of various .hcode requests directly.

Dave <barx>
Group Member
Tue 25 Mar 2025 11:32:45 PM UTC, comment #22: 

comment #21:

> You might be able to hold that thought.


I can't hold it long!

> The startup files are squarely on point.


They're on point for some things, sure.  However, the issue reported by this ticket is agnostic about them.  It describes a behavior change in core groff, run without any startup files at all.  (I've taken care to make my tests work with the English startup files, since that's what groff runs by default, which simplifies pasting self-contained examples here.  But disable all startup files and run my tests again to verify that they're not necessary to illustrate the behavior I'm demonstrating.)

Dave <barx>
Group Member
Tue 25 Mar 2025 07:06:02 PM UTC, comment #21: 

At 2025-03-25T14:59:13-0400, Dave wrote:

> I was speaking in the abstract--or, perhaps more pertinently, within
> what is documented, which, as you note, omits (with good reason)
> implementation details.  But I appreciate the mathematical detail in
> which you quibbled. ;-)


Heh.

> > Here's where I disagree with your model.  I'd have to dig deeper
> > into the startup code to be sure, but conceptually, when GNU troff
> > initializes hyphenation codes, they're a perfect image of the
> > character code values.  It then goes through each of them and "fixes
> > them up".
>
> I disagree with your adverb.  Conceptually, the user is presented
> with a startup state where certain characters have certain hyphenation
> codes.  What those codes are, and how they got initialized, are
> irrelevant to the user's conceptual framework.  Without delving into
> the code, the user knows what's presented in the documentation, which
> is that A-Z have hyphenation codes that map to a-z--but,
> intentionally, not what those codes are.  That's beyond the
> conceptual model; that's an implementation detail, one that's
> undocumented and therefore not guaranteed to remain the same across
> releases.  (Or even, as far as the user knows, across multiple runs of
> the same groff executable.)
>
> Most users then have additional hyphenation codes added by groff
> startup files.  How those get added is more discoverable to the casual
> user, because they're in startup files to which the user has direct
> access, and that are written in roff syntax.  The manual documents the
> fact that startup files will likely set some additional hyphenation
> codes, though for obvious reasons gives no further detail.
>
> I submit that the foregoing is the extent of the user's conceptual
> framework.
>
> This may sound like some hair-splitting semantics, but it's laying
> groundwork for some practical points I'll get to later.


You might be able to hold that thought.  I did some more thinking of my
own and a way of making a more concrete illustration (finally) occurred
to me.  The startup files are squarely on point.

I'll follow-up once I have an opportunity to review groff 1.23.0
behavior.  And earlier versions, if need be.

G. Branden Robinson <gbranden>
Group administrator
Tue 25 Mar 2025 06:59:05 PM UTC, comment #20: 

comment #19:

> At 2025-03-22T14:01:14-0400, Dave wrote:
> > It can either do what it has always done, which is to generate a new,
> > unique hyphenation code;
>
> I quibble with the word "generate" here.


I was speaking in the abstract--or, perhaps more pertinently, within what is documented, which, as you note, omits (with good reason) implementation details.  But I appreciate the mathematical detail in which you quibbled. ;-)

> Here's where I disagree with your model.  I'd have to dig deeper into
> the startup code to be sure, but conceptually, when GNU troff
> initializes hyphenation codes, they're a perfect image of the character
> code values.  It then goes through each of them and "fixes them up".


I disagree with your adverb.  Conceptually, the user is presented with a startup state where certain characters have certain hyphenation codes.  What those codes are, and how they got initialized, are irrelevant to the user's conceptual framework.  Without delving into the code, the user knows what's presented in the documentation, which is that A-Z have hyphenation codes that map to a-z--but, intentionally, not what those codes are.  That's beyond the conceptual model; that's an implementation detail, one that's undocumented and therefore not guaranteed to remain the same across releases.  (Or even, as far as the user knows, across multiple runs of the same groff executable.)

Most users then have additional hyphenation codes added by groff startup files.  How those get added is more discoverable to the casual user, because they're in startup files to which the user has direct access, and that are written in roff syntax.  The manual documents the fact that startup files will likely set some additional hyphenation codes, though for obvious reasons gives no further detail.

I submit that the foregoing is the extent of the user's conceptual framework.

This may sound like some hair-splitting semantics, but it's laying groundwork for some practical points I'll get to later.

Dave <barx>
Group Member
Sun 23 Mar 2025 08:21:20 AM UTC, comment #19: 

At 2025-03-22T14:01:14-0400, Dave wrote:

> comment #16:
> > Follow-up Comment #14, bug #66919 (group groff):
> >> But it seems that this mechanism to clear a code needs to carve
> >> out an exception for what I'm terming "reflexive hcode," right?
> >
> > I think it has one.
>
> Call me old-fashioned, but I prefer to go by how the code behaves
> rather than by what it says.


They're the same thing--given sufficient understanding.  ;-)

(Which I'm sometimes lacking...)

> And the results of running the hcode_test in the [comment #0 original
> submission] tell us that "reflexive hcode" assigns a new, unique
> hyphenation code (what it has always done) when the first parameter is
> Latin-1, but not when the first parameter is a special character--even
> the special character representing the same character that works in
> its Latin-1 form.
>
> >> Because the purpose of [reflexive hcode] is to generate a new
> >> hyphenation code for a character that may never have had one.
> >
> > ...or reset one to its virgin state after meddling.
>
> Sorry, I'm not understanding how ".hcode x x" can ever reset a
> hyphenation code.


I missed out the correct context, which you helpfully clarified with
brackets.

> It can either do what it has always done, which is to generate a new,
> unique hyphenation code;


I quibble with the word "generate" here.  Presently, there are 256
possible hyphenation codes, 0-255, because they are `unsigned char`s.

Hyphenation code assignment, reflexive or not, updates the surjective
relation of character codes to hyphenation codes.

Since "surjective" is heavy math jargon, let me explain.

The character codes and hyphenation codes are both drawn from the exact
same set of possible values, 0-255.

Hyphenation code assignment creates a relation, or map, from one to the
other.  We therefore say that the "domain" (of character codes) and
"codomain" (of hyphenation codes) have the same elements.

However, while all character codes in the domain are in use (GNU troff
deals with them one way or another, and has semantics for nearly all),
in a typical run of the formatter the same is not true of the
codomain.

Most possible values of the hyphenation code are not in use.  Most
character codes map to hyphenation codes of zero.  GNU troff's
convention is to map the hyphenation codes of uppercase letters to their
lowercase counterparts.

Observe:


$ echo '.pchar aA' | groff 2>&1 | grep -E '(char|hyph)'
character 'a'
  hyphenation code: 97
character 'A'
  hyphenation code: 97


The character code of 'A' is 65.  I can prove it!


$ printf '.hcode A A\n.pchar aA\n' | groff 2>&1 | grep -E '(char|hyph)'
character 'a'
  hyphenation code: 97
character 'A'
  hyphenation code: 65


Thus, given a hyphenation code value, we don't necessarily know which
character code its character bears.

Equivalently, the relation between character and hyphenation codes is
not one-to-one.  The relation is surjective.  Some people use the
term "onto" for this kind of relation.  (I think the codomain has to
also be a [not necessarily proper] subset of the domain for this usage
to be correct.)

> or it can "reset" the hyphenation code of character x to... the value
> it already has.


Here's where I disagree with your model.  I'd have to dig deeper into
the startup code to be sure, but conceptually, when GNU troff
initializes hyphenation codes, they're a perfect image of the character
code values.  It then goes through each of them and "fixes them up".
Most hyphenation codes get set to zero, the uppercase letters get set to
that of their lowercase counterparts, and the lowercase letters are left
as-is.

Later, when troffrc is read, hyphenation codes may (typically do) get
updated.

So I would say that this:


$ printf '.hcode A A\n.pchar aA\n' | groff 2>&1 | grep -E '(char|hyph)'
character 'a'
  hyphenation code: 97
character 'A'
  hyphenation code: 65


...illustrates neither "generation" of a (presumptively new) hyphenation
code nor "reset" of "A"'s hyphenation code to "the value it already
has".  It was 97, and we knocked it back to 65.

It would be fair to complain that I might have infected your imagination
with the notion of generated hyphenation codes, because I did so to
myself in bug #66051.

"We have a lot more integers available to us than just 0-255," I
thought.  Wouldn't it be neat if we could generate a new one for
any special character that wanted one, starting at 256, say?"

That idea and plan hit the wall, hard.  See comment #7 of bug #66051.

We are confined to the `unsigned char` straitjacket for now.

> OK, but punting the question implies restoring the .hcode behavior to
> its 1.23 state.


Not if we can argue that the aspect of the behavior you observe is
undefined or otherwise undocumented.

> That would be the more conservative approach,


In a sense.  But generally in language development, if you break
someone's reliance on undefined behavior, you don't owe them an apology
and possibly not even notice.  (In this case, I don't object to "notice"
unless it makes the "NEWS" file "too long"--an assessment I personally
am unlikely to make.)

> but it's overkill for resolving this ticket, which is concerned only
> with the combination of reflexive hcode, and the first character being
> a special character.


I concur.

> I'd change the Summary to specify this ticket is limited only to
> reflexive hcode calls if that were an accepted term rather than one I
> invented in the course of talking about this issue.


I have trouble interpreting


.hcode \[~o] õ


as "reflexive" because in the part of GNU troff we're talking about, the
source and destination characters are distinguishable.  We can observe
that fact by inspecting how `pchar` talks about them, even ignoring the
reported hyphenation code values.

I concede that other aspects of GNU troff's input processing, namely the
canned set of special character definitions that completely covers the
ISO Latin-1 character set, make the issue muddier.

Stepping outside of ISO Latin-1 might clarify.


.hcode \[b=] б


You might not assume, absent knowledge that English speakers find
specialized, that the foregoing is a "reflexive" use of `hcode`.

In fact GNU troff can, for 1.24.0, just about get away with this, thanks
to the new "tmac/koi8-r.tmac" file.

https://git.savan ... /tmac/koi8-r.tmac

But someone unconcerned with Cyrillic, and not using it in their
document, might want to define a special character named `b=` and to
feel free either to assume that its hyphenation code will default to
zero, or that they can assign it at liberty.  Maybe they want this:


.char \[b=] \o'b='\" I want a special B for names in my fantasy novel...
.hcode \[b=] b\" ...but it's still a kind of "b".


...are they wrong?

That's why I say that, for English, you cannot assume that `\[o~]` is
going to behave just like `õ`.  The latter is _not defined in the
English alphabet_, so you can't rely on its hyphenation code having any
particular value.

Running groff in a Latin-1 terminal, I get this:


$ groff
.pchar õ
character code 245 (U+00F5)
  is translated
  does not have a macro
  special translation: 0
  hyphenation code: 0
  flags: 0
  ASCII code: 245
  asciify code: 0
  is found
  is transparently translatable
  is not translatable as input
  mode: normal


Running groff in a KOI8-R terminal, I get this...

(...er, uh, one trip through "dpkg-reconfigure locales" later...)

I can't even input the character, because õ doesn't exist in that
character encoding.  But I can stab at it anyway.


$ printf '.pchar \365\n' | groff
character code 245 (U+00F5)
  is translated
  does not have a macro
  special translation: 0
  hyphenation code: 0
  flags: 0
  ASCII code: 245
  asciify code: 0
  is found
  is transparently translatable
  is not translatable as input
  mode: normal


"They're the same picture." -- Pam from The Office (U.S.)

Yes.  And no.  But one thing's for sure, in the KOI8-R locale:


.hcode \[~o] õ


...is *not* reflexive, because in that locale it really looks like
this:


.hcode \[~o] Т \" CYRILLIC CAPITAL LETTER TE


> > The question I have for you right now is whether groff master is
> > working as you expect and desire specifically for Latin-1 characters
> > whose hyphenation codes we configure in "en.tmac",
>
> Yes.  I have no quarrel with this, and if I did, it would be out of
> scope for this ticket.


Okay.

> > I also want to know whether spelling them as 8-bit ordinary
> > characters or special characters works as you expect.
>
> Ordinary characters, yes.  Special characters, in reflexive context,
> no.


I claim that, in groff, special characters cannot, and have never
been able to, participate in reflexive hyphenation code assignments.

Have I persuaded you?

G. Branden Robinson <gbranden>
Group administrator
Sat 22 Mar 2025 06:01:09 PM UTC, comment #18: 

comment #16:

> Follow-up Comment #14, bug #66919 (group groff):
> > But it seems that this mechanism to clear a code needs to carve
> > out an exception for what I'm terming "reflexive hcode," right?
>
> I think it has one.


Call me old-fashioned, but I prefer to go by how the code behaves rather than by what it says.  And the results of running the hcode_test in the original submission tell us that "reflexive hcode" assigns a new, unique hyphenation code (what it has always done) when the first parameter is Latin-1, but not when the first parameter is a special character--even the special character representing the same character that works in its Latin-1 form.

> > Because the purpose of [reflexive hcode] is to generate a new
> > hyphenation code for a character that may never have had one.
>
> ...or reset one to its virgin state after meddling.


Sorry, I'm not understanding how ".hcode x x" can ever reset a hyphenation code.  It can either do what it has always done, which is to generate a new, unique hyphenation code; or it can "reset" the hyphenation code of character x to... the value it already has.

> I am inclined to punt this question out into the Great
> Post-1.24 Hyphenation Reform Chinwag that Peter proposed.


OK, but punting the question implies restoring the .hcode behavior to its 1.23 state.  That would be the more conservative approach, but it's overkill for resolving this ticket, which is concerned only with the combination of reflexive hcode, and the first character being a special character.

I'd change the Summary to specify this ticket is limited only to reflexive hcode calls if that were an accepted term rather than one I invented in the course of talking about this issue.

> The question I have for you right now is whether groff master
> is working as you expect and desire specifically for Latin-1
> characters whose hyphenation codes we configure in "en.tmac",


Yes.  I have no quarrel with this, and if I did, it would be out of scope for this ticket.

> I also want to know whether spelling them as 8-bit ordinary
> characters or special characters works as you expect.


Ordinary characters, yes.  Special characters, in reflexive context, no.

Dave <barx>
Group Member
Fri 21 Mar 2025 07:11:32 PM UTC, comment #17: 

Comment #16 threw the ball back into Dave's court.

"The question I have for you right now is whether groff master is working as you expect and desire specifically for Latin-1 characters whose hyphenation codes we configure in "en.tmac", like "ó".  By listing them there, we bless them as honorary letters of the English alphabet. I also want to know whether spelling them as 8-bit ordinary characters or special characters works as you expect."

G. Branden Robinson <gbranden>
Group administrator
Fri 21 Mar 2025 01:07:49 AM UTC, comment #16: 

At 2025-03-18T15:49:28-0400, Dave wrote:

> Follow-up Comment #14, bug #66919 (group groff):
>
> comment #13:
> > In this case, the change is deliberate:
> >
> > "Support clearing a character's hyphenation code by copying
> > that of a character that lacks one."
>
> Ah!
>
> But it seems that this mechanism to clear a code needs to carve out an
> exception for what I'm terming "reflexive hcode," right?


I think it has one.

https://git.savan ... 650a8731ff7#n8058

> Because the purpose of that is to generate a new hyphenation code for
> a character that may never have had one.


...or reset one to its virgin state after meddling.  A theme of my
changes to groff is that we should reduce the number of one-way paths in
the formatter's configuration space.  Anything you can create, you
should be able to delete (and create again).  Any parameter you can put
some English on, giving it some wacky value, you can unspin back to
normality.

A couple of accepted exceptions exist: deletion of requests, and
deletion of registers with special semantics (like `.i` or `.ps`).  If
you kill those, there's no going back.  Doug McIlroy had to talk me into
the latter.

> It's also curious that "reflexive hcode" works differently depending
> on how the character is spelled: ".hcode \[~o] õ" now has a different
> effect from ".hcode õ õ".


I don't have many useful thoughts on this yet.  I understand that a set
of normalization processes is internally applied to special character
names.  For instance:

$ printf "hell\\[o aa]\n" | groff -a
<beginning of page>
hell<'o>

(same output in groff 1.22.3 through master HEAD).

Also, you can (try to) remove a special character and recreate it as a
user-defined character.  Its hyphenation code will remain unchanged
throughout these processes.

$ printf ".pchar \\[o aa]\n.rchar \\[o aa]]\n.pchar \\[o aa]\n.char \\[o aa] world\n.pchar \\[o aa]\n" | ~/groff-HEAD/bin/groff -a
special character "'o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 111
  flags: 0
  ASCII code: 0
  asciify code: 243
  is found
  is transparently translatable
  is translatable as input
  mode: normal
special character "'o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 111
  flags: 0
  ASCII code: 0
  asciify code: 243
  is found
  is transparently translatable
  is translatable as input
  mode: normal
special character "'o"
  is not translated
  has a macro: "world"
  special translation: 0
  hyphenation code: 111
  flags: 0
  ASCII code: 0
  asciify code: 243
  is found
  is transparently translatable
  is translatable as input
  mode: normal

Is the foregoing correct/desirable?  I don't know.  Since they've been
separately configurable for either ~23 years (`char`) or 34+ (`hcode`),
I guess so.  Maybe?

I am inclined to punt this question out into the Great Post-1.24
Hyphenation Reform Chinwag that Peter proposed.  There seems to be a
cluster of related nettles to grasp in this area.

The question I have for you right now is whether groff master is
working as you expect and desire specifically for Latin-1 characters
whose hyphenation codes we configure in "en.tmac", like "ó".  By listing
them there, we bless them as honorary letters of the English alphabet.
I also want to know whether spelling them as 8-bit ordinary characters
or special characters works as you expect.

Once those questions are decided, then I think it is time to decide
whether it's NEWS-worthy to announce changed hyphenation support, in
English, of letters that don't properly occur in English words.  I lean
against it, because people can and will install fancy fonts with support
for Thai and Cherokee scripts and their hyphenation codes are all going
to be stone zeroes as well.

G. Branden Robinson <gbranden>
Group administrator
Fri 21 Mar 2025 12:34:07 AM UTC, comment #15: 

Updating status: as far as I can tell, this is no longer waiting on info.

Dave <barx>
Group Member
Tue 18 Mar 2025 07:49:22 PM UTC, comment #14: 

comment #13:

> In this case, the change is deliberate:
>
> "Support clearing a character's hyphenation code by copying
> that of a character that lacks one."


Ah!

But it seems that this mechanism to clear a code needs to carve out an exception for what I'm terming "reflexive hcode," right?  Because the purpose of that is to generate a new hyphenation code for a character that may never have had one.

It's also curious that "reflexive hcode" works differently depending on how the character is spelled: ".hcode \[~o] õ" now has a different effect from ".hcode õ õ".

Dave <barx>
Group Member
Tue 18 Mar 2025 07:28:51 PM UTC, comment #13: 

Bear in mind that "bad" is merely a label.  In this case it refers to the change from groff 1.23.0 and earlier behavior.


a52141ac46eef95dd1f85e4c2e0a336affa9bcc9 is the first bad commit
commit a52141ac46eef95dd1f85e4c2e0a336affa9bcc9
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Thu Aug 8 06:53:23 2024 -0500

    [troff]: Continue fixing Savannah #66054.

    * src/roff/troff/input.cpp (set_hyphenation_codes): Handle the common
      case (copying the existing hyphenation code of one character to
      another) first.  Support clearing a character's hyphenation code by
      copying that of a character that lacks one.  Reorganize for clarity,
      and add comments.

    Continues fixing <https://savannah.gnu.org/bugs/?66054>.

 ChangeLog                | 10 ++++++++++
 src/roff/troff/input.cpp | 23 ++++++++++++-----------
 2 files changed, 22 insertions(+), 11 deletions(-)


In this case, the change is deliberate:

"Support clearing a character's hyphenation code by copying that of a character that lacks one."

Here's the entire diff (of "input.cpp").


diff --git a/src/roff/troff/input.cpp b/src/roff/troff/input.cpp
index cc7d9dd71..946b93570 100644
--- a/src/roff/troff/input.cpp
+++ b/src/roff/troff/input.cpp
@@ -7309,25 +7309,26 @@ static void set_hyphenation_codes()
       error("cannot use the hyphenation code of a numeral");
       break;
     }
-    unsigned char new_code = 0; // TODO: int
+    unsigned char new_code = 0;
     charinfo *cisrc = tok.get_char();
-    if (csrc != 0)
-      new_code = csrc;
-    else {
+    if (cisrc != 0 /* nullptr */)
+      // Common case: assign destination character the hyphenation code
+      // of the source character.
+      new_code = cisrc->get_hyphenation_code();
+    if (0 == csrc) {
       if (0 /* nullptr */ == cisrc) {
        error("expected ordinary or special character, got %1",
              tok.description());
        break;
       }
-      // source character is special
-      if (0 == cisrc->get_hyphenation_code()) {
-       error("second member of hyphenation code pair must be an"
-             " ordinary character, or a special character already"
-             " assigned a hyphenation code");
-       break;
-      }
       new_code = cisrc->get_hyphenation_code();
     }
+    else {
+      // If assigning a ordinary character's hyphenation code to itself,
+      // use its character code point as the value.
+      if (csrc == cdst)
+       new_code = tok.ch();
+    }
     cidst->set_hyphenation_code(new_code);
     if (cidst->get_translation()
        && cidst->get_translation()->get_translation_input())


G. Branden Robinson <gbranden>
Group administrator
Tue 18 Mar 2025 07:14:00 PM UTC, comment #12: 

comment #11:

> When instrumenting `hcode` with the debugging statement
> shown previously, I see "ps.tmac" and "en.tmac" getting busy.


Fair point.  I discounted en.tmac because it's a relatively recent addition, so it can't be affecting all the older releases, and I verified that it doesn't touch õ's hyphenation code in its current incarnation.

Looking now at ps.tmac past and present, its .hcodes also seem to not touch õ.

> My objection is pretty much the haste with which you reached a
> diagnosis, which I think may have led me to undertake invalid
> experiments.


Fair objection.  I've updated the Summary again to hopefully more accurately reflect the situation.

Dave <barx>
Group Member
Tue 18 Mar 2025 06:48:04 PM UTC, comment #11: 

At 2025-03-18T14:35:22-0400, Dave wrote:

> Follow-up Comment #10, bug #66919 (group groff):
>
> Not trying to be difficult.  What I'm trying to get at is that the
> comment #4 test file shows observable output changing from all prior
> groffs I have in my grasp (which span nearly 20 years).


Fair, and I'm not upset with you about that.  (Or, really, upset at
all.  Just frustrated at bugs that hide from the light.)

> You confirmed this output change but ascribed it to a change in
> startup tmac files, which I refuted in comment #7.


Not completely.  You refuted, I think, that a change in "latin1.tmac"
was the culprit.  There are other startup files, and perhaps one of
those is to blame.  When instrumenting `hcode` with the debugging
statement shown previously, I see "ps.tmac" and "en.tmac" getting busy.

Maybe I really did manage to change the way way `hcode` works.  But I'm
confident, given the aforementioned instrumentation, that "not
recognizing a special character" isn't what's going on.

> So if not that, what is the source of the behavior change?  Either
> my refutation needs refuting, or the parser is doing something
> different with the same input.  And if the latter, is this an
> accidental or intentional change?  I suspect accidental, only because
> if intentional I think by now you would have remembered doing it and
> seized up a ticket or commit that explained it.


Yeah, I think it's time to bust out a Git bisection.

> As ever, I don't object to breaking back compatibility per se.  But I
> believe a compatibility-breaking change should be justifiable to serve
> some larger purpose.  If this was an intentional parser change with
> such a purpose, we can log it in NEWS and move on.  But right now it
> seems we don't yet know why the output has changed.  (Or at least
> I don't; if you do, maybe we've gone back to talking past each
> other.)


No, I can't account for it, either, which means we need to RCA out the
truffle.

> Does this clarify the principles I'm working from?


Yup.  My objection is pretty much the haste with which you reached a
diagnosis, which I think may have led me to undertake invalid
experiments.

> Do you disagree with any of the foregoing?


Nope.  I'll get to bisecting.

G. Branden Robinson <gbranden>
Group administrator
Tue 18 Mar 2025 06:35:15 PM UTC, comment #10: 

Not trying to be difficult.  What I'm trying to get at is that the comment #4 test file shows observable output changing from all prior groffs I have in my grasp (which span nearly 20 years).  You confirmed this output change but ascribed it to a change in startup tmac files, which I refuted in comment #7.

So if not that, what is the source of the behavior change?  Either my refutation needs refuting, or the parser is doing something different with the same input.  And if the latter, is this an accidental or intentional change?  I suspect accidental, only because if intentional I think by now you would have remembered doing it and seized up a ticket or commit that explained it.

As ever, I don't object to breaking back compatibility per se.  But I believe a compatibility-breaking change should be justifiable to serve some larger purpose.  If this was an intentional parser change with such a purpose, we can log it in NEWS and move on.  But right now it seems we don't yet know why the output has changed.  (Or at least I don't; if you do, maybe we've gone back to talking past each other.)

Does this clarify the principles I'm working from?  Do you disagree with any of the foregoing?

Dave <barx>
Group Member
Tue 18 Mar 2025 03:34:29 PM UTC, comment #9: 

Okay, then, we need to burn this down to first principles, because you are blinding me with your claims that the `hcode` request is *ever* "failing to accept a special character as a first argument".

As far as I can tell, it is doing so with perfect reliability.  But something is going wrong that you're not happy with and we need to identify what that is.

Apply the following patch.


$ git diff
diff --git a/src/roff/troff/input.cpp b/src/roff/troff/input.cpp
index 0bf7533c5..1ced88ebe 100644
--- a/src/roff/troff/input.cpp
+++ b/src/roff/troff/input.cpp
@@ -8030,6 +8030,7 @@ static void set_hyphenation_codes()
              " got %1", tok.description());
        break;
       }
+      debug("assigning hyphenation code to special or indexed character");
     }
     tok.next();
     if (!has_arg()) {


Running the patched GNU troff with your "66919.groff" file (comment #0) and my subsequent "66919[bcd].groff" files, I get the following.


$ ./build/test-groff -a -Wbreak ATTIC/66919.groff
troff:ps.tmac:84: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:85: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:86: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:87: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:88: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:58: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:58: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:59: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:59: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:60: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:60: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:61: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:61: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:62: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:62: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:63: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:63: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:64: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:64: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:65: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:65: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:66: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:66: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:67: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:67: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:68: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:68: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:69: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:69: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:70: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:70: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:71: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:71: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:72: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:72: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:73: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:73: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:74: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:74: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:75: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:75: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:76: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:76: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:77: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:77: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:78: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:78: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:79: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:79: debug: assigning hyphenation code to special or indexed character
troff:ATTIC/66919.groff:1: debug: assigning hyphenation code to special or indexed character
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 0
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 245
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
$ ./build/test-groff -a -Wbreak ATTIC/66919b.groff
troff:ps.tmac:84: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:85: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:86: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:87: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:88: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:58: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:58: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:59: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:59: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:60: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:60: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:61: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:61: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:62: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:62: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:63: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:63: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:64: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:64: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:65: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:65: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:66: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:66: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:67: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:67: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:68: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:68: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:69: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:69: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:70: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:70: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:71: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:71: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:72: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:72: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:73: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:73: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:74: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:74: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:75: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:75: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:76: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:76: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:77: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:77: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:78: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:78: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:79: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:79: debug: assigning hyphenation code to special or indexed character
<beginning of page>
lantero<hy>
nial
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 0
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
lanter<~o>nial
troff:ATTIC/66919b.groff:5: debug: assigning hyphenation code to special or indexed character
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 0
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
lanter<~o>nial
$ ./build/test-groff -a -Wbreak ATTIC/66919c.groff
troff:ps.tmac:84: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:85: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:86: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:87: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:88: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:58: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:58: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:59: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:59: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:60: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:60: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:61: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:61: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:62: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:62: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:63: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:63: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:64: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:64: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:65: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:65: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:66: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:66: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:67: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:67: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:68: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:68: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:69: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:69: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:70: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:70: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:71: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:71: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:72: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:72: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:73: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:73: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:74: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:74: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:75: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:75: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:76: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:76: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:77: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:77: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:78: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:78: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:79: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:79: debug: assigning hyphenation code to special or indexed character
<beginning of page>
lantero<hy>
nial
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 0
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
lanter<~o>nial
troff:ATTIC/66919c.groff:7: debug: assigning hyphenation code to special or indexed character
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 111
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
lanter<~o><hy>
nial
$ ./build/test-groff -a -Wbreak ATTIC/66919d.groff
troff:ps.tmac:84: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:85: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:86: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:87: debug: assigning hyphenation code to special or indexed character
troff:ps.tmac:88: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:58: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:58: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:59: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:59: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:60: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:60: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:61: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:61: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:62: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:62: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:63: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:63: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:64: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:64: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:65: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:65: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:66: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:66: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:67: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:67: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:68: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:68: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:69: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:69: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:70: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:70: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:71: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:71: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:72: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:72: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:73: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:73: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:74: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:74: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:75: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:75: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:76: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:76: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:77: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:77: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:78: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:78: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:79: debug: assigning hyphenation code to special or indexed character
troff:en.tmac:79: debug: assigning hyphenation code to special or indexed character
<beginning of page>
lantero<hy>
nial
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 0
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
lanter<~o>nial
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 111
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
lanter<~o><hy>
nial


G. Branden Robinson <gbranden>
Group administrator
Tue 18 Mar 2025 12:45:22 PM UTC, comment #8: 

comment #6:

> Or, rather, how I correct the input document to not assume an
> English-language document can spell words using non-English letters and
> make assumptions about how they will undergo hyphenation.


My test file makes no assumptions.  It explicitly sets the hyphenation code it wants to get the breaking it wants.  It was purposely designed to rely on no Latin-1 hyphenation codes set by any past or present startup files.

> we don't even need to invoke `hcode` with a
> special character as the first argument.


True, in this case we don't need to.  However:

  • It used to work, and now it doesn't.
  • The manual currently says the first .hcode argument can be a special character.
  • We should want to minimize the amount of Latin-1 users are required to supply in input.  At present nothing can be done about the second .hcode argument, but the first could previously be specified using only ASCII characters.
Dave <barx>
Group Member
Tue 18 Mar 2025 12:06:56 PM UTC, comment #7: 

comment #5:

> In English, õ used to get a (nonzero) hyphenation code
> in groff (because "latin1.tmac" gave it one), and now
> it no longer does.


This must be another set of alternative facts. ;-)  According to git, latin1.tmac had no .hcode requests in it until August 2024, commit 0629380a9 (its most recent).  Indeed, my test file asks groff to hyphenate my invented word both before and after invoking the .hcode to expressly demonstrate the absence of a hyphenation code for õ in the first instance.  This absence shows up in all tested groffs, including the latest git has to offer.

Dave <barx>
Group Member
Tue 18 Mar 2025 06:06:20 AM UTC, comment #6: 

At 2025-03-18T02:00:43-0400, G. Branden Robinson wrote:

> And that's how I fix the bug you're reporting.


Or, rather, how I correct the input document to not assume an
English-language document can spell words using non-English letters and
make assumptions about how they will undergo hyphenation.

I promptly realized I can go one better.  Because `\[~o]` is equivalent
to õ under groff's hood, we don't even need to invoke `hcode` with a
special character as the first argument.  You get the desired
hyphenation (and groff-tradition) automagically.


$ iconv -f iso-8859-1 ATTIC/66919d.groff
.ll 1n
lanteronial
.pchar \[~o]
lanter\[~o]nial
.\" Now let's pretend we speak Portuguese.
.hcode õ o
.pchar \[~o]
lanter\[~o]nial
$ ./build/test-groff -a -Wbreak ATTIC/66919c.groff
<beginning of page>
lantero<hy>
nial
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 0
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
lanter<~o>nial
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 111
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
lanter<~o><hy>
nial


G. Branden Robinson <gbranden>
Group administrator
Tue 18 Mar 2025 06:00:38 AM UTC, comment #5: 

Hi Dave,

At 2025-03-18T01:25:14-0400, Dave wrote:

> Follow-up Comment #4:
>
> comment #3:
> > We seem to be talking totally past each other.
>
> It's our superpower!


Fortunately, some kryptonite appears to be arriving on the scene.

> Happily, after only 5 minutes of frustration I've managed to craft an
> input file that shows the behavior change from older groffs (tested on
> 1.19.2, 1.22.4, and 1.23.0) to current groff.  (Or current as of when
> this bug was opened a couple days ago.  I haven't built your latest
> push.)


I don't think it's changed.  I've thrashed the hell out of `pline`
behavior but not touched hyphenation logic in a little while.

> The following has to be encoded via Latin-1 to accommodate .hcode's
> limitation on accepting only an 8-bit character as its second
> argument.


Right.

> .ll 1n
> lanteronial
> lanter\[~o]nial
> .hcode \[~o] õ
> lanter\[~o]nial
>
> I recommend running the file with groff's "-a -Wbreak" options, though
> any other output format should also suffice.


Yes.  That pair of options should be near the top of anyone's bag of
tricks when debugging hyphenation problems.

> Here's the -a output I see in all listed older groffs:
>
> <beginning of page>
> lantero<hy>
> nial
> lanter<~o>nial
> lanter<~o><hy>
> nial
>
> And here's current groff:
>
> <beginning of page>
> lantero<hy>
> nial
> lanter<~o>nial
> lanter<~o>nial


Confirmed.  I can reproduce this.

> The .hcode that formerly worked can be seen here to not work, based on
> the different breaking behavior.  And, notably, changing the first
> hcode parameter from the special-character version to the Latin-1
> version of the character does make this match older groff's
> behavior.


That makes sense to me; since it's an "ordinary character", it forces
the character õ to adopt its own code point value as its hyphenation
code.

We can do the same thing with the equals sign, and summon a bespoke
hyphenation code from the vasty deep.


$ printf '.pchar =\n.hcode = =\n.pchar =\n' | ./build/test-groff 2>&1 | grep hyph
  hyphenation code: 0
  hyphenation code: 61


And that's how I fix the bug you're reporting.


$ iconv -f iso-8859-1 ATTIC/66919c.groff
.ll 1n
lanteronial
.pchar \[~o]
lanter\[~o]nial
.\" Now let's pretend we speak Portuguese.
.hcode õ o
.hcode \[~o] õ
.pchar \[~o]
lanter\[~o]nial
$ ./build/test-groff -a -Wbreak ATTIC/66919c.groff
<beginning of page>
lantero<hy>
nial
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 0
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
lanter<~o>nial
special character "~o"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 111
  flags: 0
  ASCII code: 0
  asciify code: 245
  is found
  is transparently translatable
  is translatable as input
  mode: normal
lanter<~o><hy>
nial


> > The assertion in the bug summary is that ".hcode no longer accepts a
> > special character as its first argument".
>
> You're right, I failed to qualify the summary with a "sometimes" in
> light of your modified example.  Done now.


That helps!

> > How are we to distinguish special characters that are created with a
> > default hyphenation code of zero from ones have that a hyphenation
> > code of zero assigned to them by copying from an ordinary character
> > that has a hyphenation code of zero?
>
> But "reflexive hcode," if you will (that is, ".hcode x x," for any
> value of x) is a special case that doesn't assign the hyphenation code
> of x to x, but assigns a new, unique hyphenation code to x.


Agreed, as illustrated above.  Where "new, unique" just means its code
point value in the character encoding, but the groff documentation is
coy about this fact for reasons that I think are mostly sound but carry
a cost in situations like this.

> (In some past bug report I ruminated on the unwisdom of using the same
> syntax for "reflexive hcode" as for other hcode assignments, but this
> is the syntax we've inherited.)


It makes me uneasy as well.

I continue to think that what you're diagnosing here is not any sort of
new problem with the `hcode` request, but the fact that I went with
option 1 in bug #66112.  In English, õ used to get a (nonzero)
hyphenation code in groff (because "latin1.tmac" gave it one), and now
it no longer does.

If we had a Portuguese localization file, I would strongly expect to it
assign a hyphenation code to õ.

Have I persuaded you?  If so, I propose that this ticket is
re-item-groupable to "Documentation", demanding of a "NEWS" item.

G. Branden Robinson <gbranden>
Group administrator
Tue 18 Mar 2025 05:25:03 AM UTC, comment #4: 

comment #3:

> We seem to be talking totally past each other.


It's our superpower!

> Right, I rapidly blundered into the same frustration.


Happily, after only 5 minutes of frustration I've managed to craft an input file that shows the behavior change from older groffs (tested on 1.19.2, 1.22.4, and 1.23.0) to current groff.  (Or current as of when this bug was opened a couple days ago.  I haven't built your latest push.)  The following has to be encoded via Latin-1 to accommodate .hcode's limitation on accepting only an 8-bit character as its second argument.

.ll 1n
lanteronial
lanter\[~o]nial
.hcode \[~o] õ
lanter\[~o]nial

I recommend running the file with groff's "-a -Wbreak" options, though any other output format should also suffice.

Here's the -a output I see in all listed older groffs:

<beginning of page>
lantero<hy>
nial
lanter<~o>nial
lanter<~o><hy>
nial

And here's current groff:

<beginning of page>
lantero<hy>
nial
lanter<~o>nial
lanter<~o>nial

The .hcode that formerly worked can be seen here to not work, based on the different breaking behavior.  And, notably, changing the first hcode parameter from the special-character version to the Latin-1 version of the character does make this match older groff's behavior.

> The assertion in the bug summary is that ".hcode no longer accepts a
> special character as its first argument".


You're right, I failed to qualify the summary with a "sometimes" in light of your modified example.  Done now.

> How are we to distinguish special characters that are created with a
> default hyphenation code of zero from ones have that a hyphenation code
> of zero assigned to them by copying from an ordinary character that has
> a hyphenation code of zero?


But "reflexive hcode," if you will (that is, ".hcode x x," for any value of x) is a special case that doesn't assign the hyphenation code of x to x, but assigns a new, unique hyphenation code to x.  (In some past bug report I ruminated on the unwisdom of using the same syntax for "reflexive hcode" as for other hcode assignments, but this is the syntax we've inherited.)

Dave <barx>
Group Member
Tue 18 Mar 2025 02:44:50 AM UTC, comment #3: 

Hi Dave,

We seem to be talking totally past each other.

At 2025-03-17T22:12:04-0400, Dave wrote:

> Follow-up Comment #2, bug #66919 (group groff):
>
> comment #1:
> > I don't think you diagnosed this problem correctly.
>
> Admittedly, it's hard to show the behavior change between git HEAD and
> any previous version of groff, because no version before about a month
> ago had a .pchar request.  So I need to either backport .pchar into an
> older groff, or craft a test that shows an observable output change.


Right, I rapidly blundered into the same frustration.

> > I have alternative facts.  Er, I mean, an alternative input file.
>
> OK, but this at best shows that the bug only occurs sometimes, not all
> the time.


The assertion in the bug summary is that ".hcode no longer accepts a
special character as its first argument".

I believe that acceptance of a special charcter as the first `hcode`
request argument precisely what my input file illustrates.  The
hyphenation code of the "source" (second argument, a lowercase letter
'a' with acute accent) is extracted and assigned to the special
character `\[~o]`.

> It doesn't change the erroneous behavior of my test.


How is your input file behaving erroneously?

> > ...and therefore its hyphenation code is 0, the default.
>
> Yes, which is my intent.


How are we to distinguish special characters that are created with a
default hyphenation code of zero from ones have that a hyphenation code
of zero assigned to them by copying from an ordinary character that has
a hyphenation code of zero?

If we can't, then short of a GDB session, I don't see how you can
sustain that claim.  For what it's worth, I did actually run GDB on the
"troff" in Git HEAD and the control flow is correct.  `\[~o]` is
recognized as a special character.

I could assign it a hyphenation code from a basic Latin letter instead.
That's no trouble.


$ printf '.pchar \\[~o]\n.hcode \\[~o] z\n.pchar \\[~o]\n' | ./build/test-groff 2>&1 | grep hyph
  hyphenation code: 0
  hyphenation code: 122


> I suspect your alternative input using a character that is given a
> hyphenation code in latin1.tmac accounts for the difference in
> behavior.


My claim is that the fact that the assignment "took" and got applied to
the special character `\[~o]` is evidence that ".hcode no longer accepts
a special character as a first argument" is not a factual statement.

> > There are multiple opinions about ...
>
> I'm happy to kick that can down the road; for now, having a character
> that has no hyphenation code defined in any startup file is useful for
> demonstrating the bug.


We're unlikely to run out of those.


$ echo '.pchar =' | ./build/test-groff
character '='
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 0
  flags: 0
  ASCII code: 61
  asciify code: 0
  is found
  is transparently translatable
  is not translatable as input
  mode: normal


> > I propose that there is no bug here, or at a minimum, that GNU
> > troff is in fact accepting a special character as the first
> > argument to the `hcode` request.  What do you think?
>
> I'm not clear on how "I can craft different input where .hcode does
> work" invalidates the bug demonstrated by my input where it doesn't.


I guess I don't understand where your siting the claim of the buggy
behavior.  (Deri had a great expression for this, but I can't remember
it right now.)

I reiterate:

How are we to distinguish special characters that are created with a
default hyphenation code of zero from ones have that a hyphenation code
of zero assigned to them by copying from an ordinary character that has
a hyphenation code of zero?

Here's another demonstrator.  Let's take a special character that starts
life (after startup files are read, importantly) with a nonzero
hyphenation code and see if we can get the `hcode` request to fail to
assign it a new one, illustrating your claim that ".hcode no longer
accepts a special character as a first argument".


$ printf '.pchar \\[`a]\n.hcode \\[`a] z\n.pchar \\[`a]\n' | ./build/test-groff 2>&1 | grep hyph
  hyphenation code: 97
  hyphenation code: 122


In what sense is the `hcode` request not working?

My working copy does tend to race ahead of Savannah, so I checked out
master and confirmed the foregoing exhibits using this commit.


commit 6469e004b219f35786e02d886d56808bba846cee (HEAD, origin/master, origin/HEAD)
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Sun Mar 16 09:35:30 2025 -0500

    [troff]: Fix logic error.


Here's another illustrator.


$ printf '.ll 1n\n.pchar \\[`a]\n.hcode \\[`a] z\n.pchar \\[`a]\nfizzle fi\\[`a]zle\n' | ./build/test-groff -Wbreak -a 2>&1
special character "`a"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 97
  flags: 0
  ASCII code: 0
  asciify code: 224
  is found
  is transparently translatable
  is translatable as input
  mode: normal
special character "`a"
  is not translated
  does not have a macro
  special translation: 0
  hyphenation code: 122
  flags: 0
  ASCII code: 0
  asciify code: 224
  is found
  is transparently translatable
  is translatable as input
  mode: normal
<beginning of page>
fiz<hy>
zle
fi<`a><hy>
zle


As I predicted, with this input, the \[`a] special character is
participating in hyphenation as the basic Latin letter `z` does.

G. Branden Robinson <gbranden>
Group administrator
Tue 18 Mar 2025 02:12:00 AM UTC, comment #2: 

comment #1:

> I don't think you diagnosed this problem correctly.


Admittedly, it's hard to show the behavior change between git HEAD and any previous version of groff, because no version before about a month ago had a .pchar request.  So I need to either backport .pchar into an older groff, or craft a test that shows an observable output change.

> I have alternative facts.  Er, I mean, an alternative input file.


OK, but this at best shows that the bug only occurs sometimes, not all the time.  It doesn't change the erroneous behavior of my test.

> ...and therefore its hyphenation code is 0, the default.


Yes, which is my intent.  I suspect your alternative input using a character that is given a hyphenation code in latin1.tmac accounts for the difference in behavior.

> There are multiple opinions about ...


I'm happy to kick that can down the road; for now, having a character that has no hyphenation code defined in any startup file is useful for demonstrating the bug.

> I propose that there is no bug here, or at a minimum, that GNU
> troff is in fact accepting a special character as the first
> argument to the `hcode` request.  What do you think?


I'm not clear on how "I can craft different input where .hcode does work" invalidates the bug demonstrated by my input where it doesn't.

Dave <barx>
Group Member
Mon 17 Mar 2025 08:25:08 AM UTC, comment #1: 

Hi Dave,

original submission:

> The .hcode entry in the manual says:
>
>  -- Request: .hcode dst1 src1 [dst2 src2] ...
>      DST1 must be an ordinary character (other than a numeral) or a special character
>
> And a special character did previously work here.  But it no longer does.


I don't think you diagnosed this problem correctly.


> $ file hcode_test
> hcode_test: ISO-8859 text
> $ iconv -f iso-8859-1 hcode_test
> .hcode \[~o] õ
> .pchar \[~o]
> .hcode õ õ
> .pchar \[~o]
> $ groff-latest hcode_test 2>&1 | fgrep 'hyphenation code:'
>   hyphenation code: 0
>   hyphenation code: 245


I have alternative facts.  Er, I mean, an alternative input file.


$ file ATTIC/66919alt.groff
ATTIC/66919alt.groff: ISO-8859 text
$ iconv -f iso-8859-1 ATTIC/66919alt.groff
.hcode \[~o] á
.pchar \[~o]
.hcode õ á
.pchar \[~o]
$ ./build/test-groff ATTIC/66919alt.groff 2>&1 | grep -F 'hyphenation code'
  hyphenation code: 225
  hyphenation code: 225


> ("õ" is given no .hcode value in either latin1.tmac or en.tmac.


...and therefore its hyphenation code is 0, the default.

> Its absence from latin1.tmac seems like a bug, but at present
> it's a useful one, so I hope it sticks around for a bit.)


There are multiple opinions about whether hyphenation codes should bind to the character encoding or to the hyphenation language, and right now my stance is the latter.

The comment in "en.tmac" explains.


.\" Map hcodes of Latin-1 characters with diacritical marks that are
.\" used in English words to their unadorned ASCII counterparts.
.\" See http://savannah.gnu.org/bugs/?66112 for rationale.


In part I am influenced by the exclusively UTF-8 future I can foresee, where every language uses the same character encoding for input to GNU troff, and we must rely upon localization files to set up all of the hyphenation codes anyway, since the mappings will sometimes differ (hello, Turkish).

I went with your first suggestion in bug #66112, rather than your second.

I propose that there is no bug here, or at a minimum, that GNU troff is in fact accepting a special character as the first argument to the `hcode` request.  What do you think?

G. Branden Robinson <gbranden>
Group administrator
Sun 16 Mar 2025 02:59:17 PM UTC, original submission:  

The .hcode entry in the manual says:

 -- Request: .hcode dst1 src1 [dst2 src2] ...
     DST1 must be an ordinary character (other than a numeral) or a special character

And a special character did previously work here.  But it no longer does.

$ file hcode_test
hcode_test: ISO-8859 text
$ iconv -f iso-8859-1 hcode_test
.hcode \[~o] õ
.pchar \[~o]
.hcode õ õ
.pchar \[~o]
$ groff-latest hcode_test 2>&1 | fgrep 'hyphenation code:'
  hyphenation code: 0
  hyphenation code: 245

("õ" is given no .hcode value in either latin1.tmac or en.tmac.  Its absence from latin1.tmac seems like a bug, but at present it's a useful one, so I hope it sticks around for a bit.)

Dave <barx>
Group Member

 

Attached Files

Attached Files

(Note: upload size limit is set to 4.0MiB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Dependencies

This item does not depend on any other items.

No items depend on this one.

 

Mail Notification Carbon-Copy List

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by barx (Submitted the item)
  •  

    Votes

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

    History

    Follow 14 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2025-04-23 gbranden Attached File- Added phcode-for-groff-1.23.0.diff, #57163
    2025-03-26 gbranden Assigned toNone barx
    2025-03-26 gbranden StatusNone Need Info
    2025-03-22 barx Assigned tobarx None
    2025-03-22 barx StatusNeed Info None
    2025-03-21 gbranden Assigned toNone barx
    2025-03-21 gbranden StatusNone Need Info
    2025-03-21 barx StatusNeed Info None
        Assigned tobarx None
    2025-03-18 barx Summary[troff] .hcode sometimes fails to accept a special character as a first argument [troff] behavior change in some .hcode calls when a special character is the first argument
    2025-03-18 barx Summary[troff] .hcode no longer accepts a special character as a first argument [troff] .hcode sometimes fails to accept a special character as a first argument
    2025-03-17 gbranden StatusNone Need Info
        Assigned toNone barx
        Summary.hcode no longer accepts a special character as a first argument [troff] .hcode no longer accepts a special character as a first argument

    Back to the top

    Powered by Savane 3.16-a7ba.
    Corresponding source code