bugGNU roff - Bugs: bug #62300, [preconv] does not handle U+00A0...

 
 

bug #62300: [preconv] does not handle U+00A0 (NBSP) as it should

Submitter:  G. Branden Robinson <gbranden>
Submitted:  Tue 12 Apr 2022 03:14:52 AM UTC
   
 
Category:  Preprocessor preconv Severity:  3 - Normal
Item Group:  Incorrect behaviour Status:  Fixed
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  1.23.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 13 Apr 2022 10:53:56 PM UTC, comment #4: 


> Convert input U+00A0 to \~ as troff would, not to \[u00A0].


...or as troff should, were it not for bug #58962.

Dave <barx>
Group Member
Wed 13 Apr 2022 02:17:38 AM UTC, comment #3: 


commit a22ceaea8df79608cf31b8080565053c35a0c7cc
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Tue Apr 12 13:19:19 2022 +1000

    [preconv]: Fix Savannah #62300.

    * src/preproc/preconv/preconv.cpp (unicode_entity): Convert input U+00A0
      to \~ as troff would, not to \[u00A0].

    Fixes <https://savannah.gnu.org/bugs/?62300>.


G. Branden Robinson <gbranden>
Group administrator
Tue 12 Apr 2022 10:53:36 AM UTC, comment #2: 

Hi Bjarni,

comment #1:

> commit f47b7dd139525bf3b8b4fbe767c3a45816c8445a
> Author: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
> Date:   Sat Nov 17 15:59:09 2018 +0000
>
>     The character \[u00A0] is not recognized
>    
>       The input character "no-break space" (' ', 0xA0) is mapped by "groff"
>     to '\~' (groff_char(7)), but only the character name '\[char160]' is
>     translated in the file "tmac/troffrc".


Yes.
    

>       The "preconv" translates the no-break space to the name '\[u00A0]'.


That was an error and is the subject of this ticket.
    

> diff --git a/tmac/troffrc b/tmac/troffrc
> index 1bd4aa8c9..8895a9a01 100644
> --- a/tmac/troffrc
> +++ b/tmac/troffrc
> @@ -33,10 +33,14 @@ troffrc!X100 troffrc!X100-12 troffrc!lj4 troff!lbp troffrc!html troffrc!pdf
>  .
>  .\" Test whether we work under EBCDIC and map the no-breakable space
>  .\" character accordingly.
> -.do ie '\[char97]'a' \
> +.do ie '\[char97]'a' \{\
>  . do tr \[char160]\~
> -.el \
> +. do tr \[u00A0]\~
> +.\}
> +.el \{\
>  . do tr \[char65]\~
> +. do tr \[u0041]\~
> +.\}
>  .
>  .\" Set the hyphenation language to 'us'.
>  .do hla us
>


I'm not sure I agree with this patch.  It's preconv's job to produce valid (GNU) troff input.  It was not doing so.

The input sequence '\[u00A0]' is syntactically valid...but like '\[uFFFF]' and '\[u0000]', it's not meaningful, and should be warned about.

Here is the patch I have pending.


diff --git a/src/preproc/preconv/preconv.cpp b/src/preproc/preconv/preconv.cpp
index 83feef8f7..b1027af17 100644
--- a/src/preproc/preconv/preconv.cpp
+++ b/src/preproc/preconv/preconv.cpp
@@ -404,9 +404,13 @@ unicode_entity(int u)
   if (u < 0x80)
     putchar(u);
   else {
-    // Handle soft hyphen specially -- it is an input character only,
-    // not a glyph.
-    if (u == 0xAD) {
+    // Handle no-break space and soft hyphen specially--they are input
+    // characters only, not glyphs.  See groff_char(7).
+    if (u == 0xA0) {
+      putchar('\\');
+      putchar('~');
+    }
+    else if (u == 0xAD) {
       putchar('\\');
       putchar('%');
     }


G. Branden Robinson <gbranden>
Group administrator
Tue 12 Apr 2022 03:37:18 AM UTC, comment #1: 

commit f47b7dd139525bf3b8b4fbe767c3a45816c8445a
Author: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
Date:   Sat Nov 17 15:59:09 2018 +0000

    The character \[u00A0] is not recognized
   
      The input character "no-break space" (' ', 0xA0) is mapped by "groff"
    to '\~' (groff_char(7)), but only the character name '\[char160]' is
    translated in the file "tmac/troffrc".
   
      The "preconv" translates the no-break space to the name '\[u00A0]'.
   
      Example:
   
    .pl 3v
    A\[char161]B\[char160]C
    .br
    .\".tr \[u00A0]\~
    A\[u00A1]B\[u00A0]C
   
      or
   
    echo ' ' | preconv
   
    Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>

diff --git a/tmac/troffrc b/tmac/troffrc
index 1bd4aa8c9..8895a9a01 100644
--- a/tmac/troffrc
+++ b/tmac/troffrc
@@ -33,10 +33,14 @@ troffrc!X100 troffrc!X100-12 troffrc!lj4 troff!lbp troffrc!html troffrc!pdf
 .
 .\" Test whether we work under EBCDIC and map the no-breakable space
 .\" character accordingly.
-.do ie '\[char97]'a' \
+.do ie '\[char97]'a' \{\
 . do tr \[char160]\~
-.el \
+. do tr \[u00A0]\~
+.\}
+.el \{\
 . do tr \[char65]\~
+. do tr \[u0041]\~
+.\}
 .
 .\" Set the hyphenation language to 'us'.
 .do hla us

Bjarni Ingi Gislason <bjarniig>
Tue 12 Apr 2022 03:14:52 AM UTC, original submission:  

preconv handles the soft hyphen by translating it into an appropriate escape sequence (\%), but does not do the same for the no-break space.  groff_char(7) has long defined the semantics in these as input code points (for ISO character encodings).


$ cat whaaa.man
.TH ISO_8859-2 7 2014-10-02 "Linux" "Linux Programmer's Manual"
.TS
l l l c lp-1.
240     160     A0              NO-BREAK SPACE
255     173     AD      ­       SOFT HYPHEN
.TE
$ xxd whaaa.man
00000000: 2e54 4820 4953 4f5f 3838 3539 2d32 2037  .TH ISO_8859-2 7
00000010: 2032 3031 342d 3130 2d30 3220 224c 696e   2014-10-02 "Lin
00000020: 7578 2220 224c 696e 7578 2050 726f 6772  ux" "Linux Progr
00000030: 616d 6d65 7227 7320 4d61 6e75 616c 220a  ammer's Manual".
00000040: 2e54 530a 6c20 6c20 6c20 6320 6c70 2d31  .TS.l l l c lp-1
00000050: 2e0a 3234 3009 3136 3009 4130 09c2 a009  ..240.160.A0....
00000060: 4e4f 2d42 5245 414b 2053 5041 4345 0a32  NO-BREAK SPACE.2
00000070: 3535 0931 3733 0941 4409 c2ad 0953 4f46  55.173.AD....SOF
00000080: 5420 4859 5048 454e 0a2e 5445 0a         T HYPHEN..TE.
$ groff -t -kz -man whaaa.man # groff 1.22.4
troff: whaaa.man:4: warning: can't find special character 'u00A0'
$ ./build/test-groff -ww -t -kz -man whaaa.man $ groff Git HEAD
troff:whaaa.man:4: warning: can't find special character 'u00A0'
$ preconv whaaa.man # groff 1.22.4 and Git HEAD
.lf 1 whaaa.man
.TH ISO_8859-2 7 2014-10-02 "Linux" "Linux Programmer's Manual"
.TS
l l l c lp-1.
240     160     A0      \[u00A0]        NO-BREAK SPACE
255     173     AD      \%      SOFT HYPHEN
.TE


preconv should put \~ on the output as documented in groff_char(7) even in groff 1.22.4.


       160    the ISO latin1 no‐break space is mapped to ‘\~’, the
              stretchable space character.

       173    the soft hyphen control character.  groff never uses
              this character for output (thus it is omitted in the
              table below); the input character 173 is mapped onto
              ‘\%’.


This remapping should occur because the diagnostic itself is not the problem; there are many Unicode code points that are not valid groff input; expressing them as special character escape sequences does not change that fact.

Working on this.

G. Branden Robinson <gbranden>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by bjarniig (Posted a comment)
  • -email is unavailable- added by gbranden (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-04-13 gbranden StatusIn Progress Fixed
        Open/ClosedOpen Closed
        Planned ReleaseNone 1.23.0

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code