bugGNU roff - Bugs: bug #62264, [troff] string iteration handles...

 
 

bug #62264: [troff] string iteration handles escape sequences inconsistently (want `for` request)

Submitter:  G. Branden Robinson <gbranden>
Submitted:  Thu 07 Apr 2022 04:49:45 PM UTC
   
 
Category:  Core Severity:  3 - Normal
Item Group:  Incorrect behaviour Status:  Postponed
Privacy:  Public Assigned to:  None
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 18 Aug 2024 12:09:46 PM UTC, comment #6: 

Kicking this out past groff 1.24.  I may still need some machinery to solve the problem Deri mentions in comment #5.  But that may not require a string iterator: resolving bug #63074 might suffice.

I hope to find out soon.

 G. Branden Robinson <gbranden>
Group administrator
Mon 26 Jun 2023 04:00:37 PM UTC, comment #5: 

As you may be aware one of the goals for the new version of gropdf I have been working on is to make it "friendlier" for unicode support. To do this I had to drop using asciify and pass the original string to gropdf since the special chars such as \[uXXXX] produced by preconv cannot be asciified and just disappear, whereas gropdf (which has access to the font information) can convert this to utf16. This is now all working.

One problem remains. In Keith's pdfmark macros (upon which pdf.tmac is based) he introduced the convention that when placing a link in the document to an internal destination within the document using:-

.pdfhref L [-D <dest-name>] [-P <prefix-text>] [-A <affixed-text>] \
[-X] [--] [descriptive text ...]

If "Descriptive text" is missing it will be replaced with text from when the target destination is created. In pdf-mom.pdf this concept is explained as "expandos" (+ and *). What this means is that the descriptive text used when the destination is created is stored in a troff string register called pdf:look(<dest-name>). The problem is that if the dest-name is not ascii, for example, if the document is written in unicode cyrillic, it is natural that the dest-name will be cyrillic as well. The problem is that if the name includes unicode (\[uXXXX]) then you receive an error:-

error: special character is not allowed in an identifier

So, at the moment, all destinations must be given in ascii to avoid the error occurring. I don't know whether the proposed change in this bug will help.

One simple solution would be a new request, .stringhex, a similar code to that used in .stringup/down. This would effectively hide the special characters within the string but still produce a unique identifier to be used as a dest-name in the pdf:look() register.

Or is there a better way of solving this problem.

Deri James <deri>
Group Member
Fri 23 Jun 2023 06:42:32 PM UTC, comment #4: 


> I had wondered idly if we needed a reverse iterator.


Straw-man interface:


.for str mac anything
.rfor str mac anything

Iterate through `mac` (rfor: backwards), automatically assigning each element in turn to `str`, interpreting `anything` at each iteration.


Open questions:

1.  What to put in `str` if the iterator hits a newline?
2.  Support something like Python's "StopIteration"?  Since we have no exception mechanism, maybe just set a new read-only register \n[.stop] to indicate that traversal was complete.  Might not be necessary; since I plan to support conditional blocks and `break` (and `continue`) inside `for` and `rfor` anyway, any logic that would prematurely exit the loop can set its own register to indicate this.

 G. Branden Robinson <gbranden>
Group administrator
Sun 30 Apr 2023 12:08:18 PM UTC, comment #3: 

A corollary to this idea is that we could move `asciify` and `unformat` out of the formatter, making them macros that employ `for`.

 G. Branden Robinson <gbranden>
Group administrator
Thu 27 Apr 2023 01:46:24 PM UTC, comment #2: 

I had wondered idly if we needed a reverse iterator.

I thought, "surely not".

I'm thinking again.

See bug #64114, where I want to traverse backward along a string and truncate it (toward the front) upon first hitting a slash.

Another possibility would be to have a node-list reversing request.

But that seems like a dangerous gun to hand the user.

On the other hand you can already suicide as hard as you like by walking strings/macros/diversions with .substring and mangling them.  So...maybe.

 G. Branden Robinson <gbranden>
Group administrator
Wed 22 Mar 2023 08:54:03 AM UTC, comment #1: 

Here are some notes from a private email I sent to Alex Colomar and Deri James last December.


That bigger fish arises from the observation that seemingly everywhere
we prepare groff strings/macros/diversions for handoff to a device
control escape sequence, we end up with some new variation on bespoke
logic to tediously walk a string (more or less).  These sequences of
code are long and surely not easy for the novice to understand.

So one of the things I want to look into for groff 1.24 is giving the
language an actual string iterator--a `for` request.  And a couple of
new conditional expression operators to perform tests on the items
returned by that iterator.

Some background for this is in <https://savannah.gnu.org/bugs/?62264>
("string iteration handles escape sequences inconsistently").

Here's the idea.

.di div
Here's my \f[CI]crazy\f[] diversion!
.di
.
.ds div*scrubbed \" empty
.
.for ch \*[div] \{\
.  if !N \*[ch]  .as div*scrubbed \n*[ch]
.  if '\*[ch]'@' .break
.  if e          .continue
.  \" some crazy stuff you do only on odd pages
.\}

The above is really contrived, but the idea is to communicate as much of
the semantics as I think we could want.

1. No messing with `length` or `substring` operations.
2. Address #62264.  Document that string iteration can hand you back any
   of (A) a Basic Latin character; (B) a special character; or (C) a
   "node" (like a type face or size changing operation, but the details
   aren't important as its "formatty" stuff, not "plain text" stuff).
3. A new 'N' conditional expression operator tests string contents for
   node identity.  I don't know whether this should test just the first
   element of the string or scan the whole thing.  In the example above,
   it doesn't matter--`for` guarantees that the `ch` string is a
   singleton.

   Giving the *roff programmer a way to cope with this is the correct
   way to solve this old chestnut.

     can't transparently output node at top level

4. Need to decide whether the string `ch` is left defined after the for
   loop exits.
5. You should be able to `break` or `continue` a `for` loop just as you
   can a `while` loop.
6. A lingering issue is our other old friend.

     can't translate character code 233 to special character ''e' in
     transparent throughput

   This is <https://savannah.gnu.org/bugs/?63074>.

Here's another use case, way less hypothetical.  It's some Deri magic.

.\" Remove '\%' from string used as bookmark destination
.de an*cln
.  ds \\$1
.  als an*cln:res \\$1
.  shift
.  ds an*cln:res \\$*\"
.  ds an*cln:char \\*[an*cln:res]
.  substring an*cln:char 0 0
.  if '\\*[an*cln:char]'\%' .substring an*cln:res 1
.  rm an*cln:char
..

Here's how you'd do it with `for`.

.ds output \" empty
.
.for ch \*[input] \{\
.  if !'\*[ch]'\%' .as output \*[ch]
.\}

As syntactic sugar goes, I'd say that enables considerable slimming.

This would probably also compel us to clear up our documentation (and
our thinking) a lot with respect to what's really a "character" in
groff.  \- is.  \% is.  \f isn't (if the remainder is well-formed, it
becomes a node).  What about the "leader character" (Ctrl+A)?  Or the
uninterpreted leader character \a?  Many of these things have the word
"character" in their names but, for example, you can't test them with
".if c".

Consider:

.ds string \&\%
.ds char \*[string]
.substring char 0 0
.if c \*[char] .tm it is a character
troff:<standard input>:6: error: expected ordinary or special character, got an escaped '&'
.ds char \*[string]
.substring char 1 1
.if c \*[char] .tm it is a character
troff:<standard input>:9: error: expected ordinary or special character, got an escaped '%'

Keith Marshall wrote an entire macro file to deal with this sort of
thing.[1]

Maybe consideration of these issues is affecting my priorities and
making me want to get over the release hump so I can work on them.


 G. Branden Robinson <gbranden>
Group administrator
Thu 07 Apr 2022 04:49:45 PM UTC, original submission:  

So I'm trying to fix Savannah bug #62257 and I ran into good news and bad news.

The bad news, I already knew, which is why the bug got filed--you can't just treat a groff string as a character array like you would a character string that used a fixed-width character encoding (like ASCII, ISO 8859, UTF-16LE).

No, sometimes an index will land you in the middle of an escape sequence, which means if you edit the string at that point you risk creating syntactical nonsense.

The good news is that the strings I'm trying to edit (to abbreviate them to keep them overrunning parts of man page headers and footers) are arguments to the man(7) `TH` macro, which tend to be really well behaved: we abbreviate the page title, which will almost always be (nearly) pure ASCII due to keyboard input practicalities, and the "extra2" argument, which by preponderant tradition is the name and version of the project responsible for the man page.  This also tends to be a well-behaved string because it needs to be easily googled and similar.

So these strings confine themselves to ASCII except for damnable old friend, the hyphen-minus.  And that's not too rare, which seems bad (although rarer than it should be thanks to the efforts of the ASCII Puritan Reactionary Underground)...

BUT

The even better news is that GNU troff's string iterator recognizes the \- escape sequence and hands it back to you as an atomic unit!  It also does this for \`, \', and \_, none of which we really need, but which may aid future research.

I started writing a macro to scan a string for escape sequences so I could bail out of the abbreviation process instead of corrupting an escape-happy string.

Here are my findings.


$ cat EXPERIMENTS/string-contains-escape.man
.an*string-contains-escape foo\-bar
.an*string-contains-escape foo\`bar
.an*string-contains-escape foo\'bar
.an*string-contains-escape foo\_bar
.an*string-contains-escape foo\(hybar
.an*string-contains-escape foo\[hy]bar
.an*string-contains-escape caf\['e] bar
.an*string-contains-escape caf\[e aa] bar
$ ./build/test-groff -b -ww -Tutf8 -man EXPERIMENTS/string-contains-escape.man
GBR: an*string-contains-escape: 'foo\-bar'
GBR: string[0]='f'
GBR: string[1]='o'
GBR: string[2]='o'
GBR: string[3]='\-'
GBR: string[4]='b'
GBR: string[5]='a'
GBR: string[6]='r'
GBR: an*string-contains-escape: 'foo\`bar'
GBR: string[0]='f'
GBR: string[1]='o'
GBR: string[2]='o'
GBR: string[3]='\`'
GBR: string[4]='b'
GBR: string[5]='a'
GBR: string[6]='r'
GBR: an*string-contains-escape: 'foo\'bar'
GBR: string[0]='f'
GBR: string[1]='o'
GBR: string[2]='o'
GBR: string[3]='\''
GBR: string[4]='b'
GBR: string[5]='a'
GBR: string[6]='r'
GBR: an*string-contains-escape: 'foo\_bar'
GBR: string[0]='f'
GBR: string[1]='o'
GBR: string[2]='o'
GBR: string[3]='\_'
GBR: string[4]='b'
GBR: string[5]='a'
GBR: string[6]='r'
GBR: an*string-contains-escape: 'foo\(hybar'
GBR: string[0]='f'
GBR: string[1]='o'
GBR: string[2]='o'
GBR: string[3]='\'
GBR: string[4]='('
GBR: string[5]='h'
GBR: string[6]='y'
GBR: string[7]='b'
GBR: string[8]='a'
GBR: string[9]='r'
GBR: an*string-contains-escape: 'foo\[hy]bar'
GBR: string[0]='f'
GBR: string[1]='o'
GBR: string[2]='o'
GBR: string[3]='\'
GBR: string[4]='['
GBR: string[5]='h'
GBR: string[6]='y'
GBR: string[7]=']'
GBR: string[8]='b'
GBR: string[9]='a'
GBR: string[10]='r'
GBR: an*string-contains-escape: 'caf\['e] bar'
GBR: string[0]='c'
GBR: string[1]='a'
GBR: string[2]='f'
GBR: string[3]='\'
GBR: string[4]='['
GBR: string[5]='''
GBR: string[6]='e'
GBR: string[7]=']'
GBR: string[8]=' '
GBR: string[9]='b'
GBR: string[10]='a'
GBR: string[11]='r'
GBR: an*string-contains-escape: 'caf\[e aa] bar'
GBR: string[0]='c'
GBR: string[1]='a'
GBR: string[2]='f'
GBR: string[3]='\'
GBR: string[4]='['
GBR: string[5]='e'
GBR: string[6]=' '
GBR: string[7]='a'
GBR: string[8]='a'
GBR: string[9]=']'
GBR: string[10]=' '
GBR: string[11]='b'
GBR: string[12]='a'
GBR: string[13]='r'


This will make completion of the fix for bug #62257 straightforward as long as I can use the output comparison operator.

I'm filing this bug because the behavior was so surprising and as far as I know it's not documented anywhere.  It feels squicky that some escape sequences get extracted atomically, though I can imagine why that's true (the simple cases currently handed don't require recursive interpolation).

The inconsistency should either be fixed or documented.  Even to be documented, it needs to be better understood.

I don't remotely want to tackle this before groff 1.23 is released.

 G. Branden Robinson <gbranden>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one
bugs[pdfpic] when falling back to EPS, should create file in temporary directory, but doesn't (bug #64114, None)

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by deri (Posted a comment)
  • -email is unavailable- added by gbranden (Submitted the item)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-08-18 gbranden DependenciesRemoved dependency from bugs #65099 -
    2024-08-18 gbranden StatusNone Postponed
    2024-01-01 gbranden Dependencies- bugs #65099 is dependent
    2023-08-10 gbranden StatusPostponed None
    2023-08-10 gbranden Dependencies- bugs #64114 is dependent
    2023-06-23 gbranden Summarystring iteration handles escape sequences inconsistently (want `for` request) [troff] string iteration handles escape sequences inconsistently (want `for` request)
    2023-03-22 gbranden Summarystring iteration handles escape sequences inconsistently string iteration handles escape sequences inconsistently (want `for` request)

    Back to the top

    Copyright © 2025 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.14-8e77.
    Corresponding source code