bugGNU roff - Bugs: bug #45034, mdoc(7): add support for the...

 
 

bug #45034: mdoc(7): add support for the mdocmx(7) reference extension

Submitter:  None
Submitted:  Wed 06 May 2015 11:53:12 AM UTC
   
 
Category:  Macro mdoc Severity:  1 - Wish
Item Group:  Feature change Status:  Rejected
Privacy:  Public Assigned to:  schwarze
Open/Closed:  Closed Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 30 Mar 2023 10:59:19 PM UTC, comment #17: 

See bug #63042 for an attempted resurrection of this issue.

G. Branden Robinson <gbranden>
Group administrator
Tue 25 May 2021 04:36:08 PM UTC, comment #16: 

In the meantime less got support for OSC 8 "hyperlinks in terminal", and i changed the three lines in mdocmx to create \X'tty: osc8 ...' for the TTY output device instead.

Please find attached a patch for grotty that implements \X'tty: osc8' sequences, which any converter can easily use to create manual pages with references. (For grotty git as of 2021-05-2?.)

As well as a patch that implements OSC 8 id= searches in less(1) (v586) for active references and even external manual lookups.

I will not update the mdocmx macros etc. patch here.  But it exists, shall interest arise.  (It is based upon groff 1.22.3.)

(file #51476, file #51477)

Anonymous
Tue 13 Oct 2020 09:04:17 PM UTC, comment #15: 
Ingo Schwarze <schwarze>
Group Member
Fri 23 Dec 2016 03:15:08 PM UTC, comment #14: 

Hello again,

yes, this is Steffen, and it is still not the second iteration.
But i have found a bug in the "free" links and references implementation, ".Mx -sx" and ".Mx -ix".  It is almost a one-line fix.
The commit message apologizes to Ingo Schwarze.

Christians a merry Christmas, and anyone a good 2017 (i have started praying a while back already).

Ciao!  Looking forward to 2017!

Anonymous
Sat 03 Oct 2015 01:26:02 PM UTC, comment #13: 

Hello again (yes it's Steffen),

this is not about the second iteration but two real bugs have been found in the  meanwhile and since i really enjoy mdocmx(7) i'm eager for posting the fixes.  Sorry for the noise :)

Two bugs in the preprocessor:

. for one processing an already preprocessed document didn't work, a bug that has been introduced when output device specifications for ".Mx -enable" had been added.

. second, creating the temporary file has at some time be changed to the colon : shell special built-in command, but shells have to hard exit once such a command fails, therefore it was an error to keep {;} and not encapsulate in a subshell.

. remove some redundancy in shell traps.

And then the groTTY output now prints references in blue.  This seems to have become common on GNU/Linux, and not being very hip causes hops.

More hopefully for christmas.  Ciao.

(file #35046)

Anonymous
Sat 27 Jun 2015 11:18:58 AM UTC, comment #12: 

And sorry for the for weird resulting format, this tracker is too fancy for me.

Anonymous
Sat 27 Jun 2015 11:17:11 AM UTC, comment #11: 

Ingo's #9:

> Bad idea. Don't use macros in header lines.


Hm, tja, na ja...

>> For now i think it is absolutely sufficient to state --
>> for mdocmx -- that people should use a headline like
>> "Device daemon events" as the headline, especially since
>> the next output line contains the same reference again.


> Exactly, and not just for mdocmx, but for mdoc(7) in general,
> and not just for now, but for good. I should probably add
> a warning message to mandoc(1) to be printed when people
> make the mistake of putting macros into header lines.


..it seems some people want to do so and we have to deal with that!

> In the context of mdocmx, i don't think you need to worry
> about such craziness at all.


You know, i like it and i don't consider this dead!
For the current release of the MUA i maintain i have spend the time necessary to fix at least the worst misgenerations of grohtml(1) and placed a mdocmx(7)ified version of its manual on the web [1], and there you can see in action mdoc(7) effectively doing great when .Mx anchors are used!
And i personally like it even more on the terminal and in less(1), since then the .Xr references also work.

 [1] http://sdaoden.users.sourceforge.net/code-nail.html

But in the meanwhile i looked around a bit and the worst cases of .S[hs] i have found look like the following:

-----------
Source:

.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'

.Sh DESCRIPTION
.Ss Dq Blabla
.Ss "\s-1OPTIONS\s0"
.Ss "Options Controlling \*(C+ Dialect"
.Ss "Using \fBgcov\fP with \s-1GCC\s0 Optimization"
.Ss "Extension based on \fI.h\fP and \fI.c\fP files"
.Ss "Fixed parsing of $$<digit>, &$<digit>, etc."
.Ss "\-s, \-\-shuffle"
.Ss "new ( [\s-1OPTIONS\s0] )
.Ss "report Bundle|Distribution|Module"
.Ss "upgrade [Module|/Regex/]..."
.Ss "_search_module_tree( type => \s-1TYPE\s0, allow => \e@regexex, [data => \e@previous_results ] )"
.
Provide a bus- and machine-independent
.Dq DMA mapping interface .

-----------
mandoc(1):

DESCRIPTION
   “Blabla”

   OPTIONS

   Options Controlling C++ Dialect

   Using gcov with GCC Optimization

   Extension based on .h and .c files

   Fixed parsing of $$<digit>, &$<digit>, etc.

   -s, --shuffle

   new ( [OPTIONS] )

   report Bundle|Distribution|Module

   upgrade [Module|/Regex/]...

   _search_module_tree( type => TYPE, allow => \@regexex, [data =>
   \@previous_results ] )
     Provide a bus- and machine-independent “DMA mapping interface”.

-----------
groff(1) with mdocmx(7) (debug mode):

(TOC):
     DESCRIPTION[4]
           Dq Blabla[11]
           OPTIONS[12]
           Options Controlling C++ Dialect[13]
           Using gcov with GCC Optimization[14]
           Extension based on .h and .c files[15]
           Fixed parsing of $$<digit>, &$<digit>, etc.[16]
           -s, --shuffle[17]
           new ( [OPTIONS] )[18]
           report Bundle|Distribution|Module[19]
           upgrade [Module|/Regex/]...[20]
           _search_module_tree( type => TYPE, allow => \@regexex, [data =>
                             \@previous_results ] )[21]

DESCRIPTION@4@[4]
   @11@[11]“Blabla”
   OPTIONS@12@[12]
   Options Controlling C++ Dialect@13@[13]
   Using gcov with GCC Optimization@14@[14]
   Extension based on .h and .c files@15@[15]
   Fixed parsing of $$<digit>, &$<digit>, etc.@16@[16]
   -s, --shuffle@17@[17]
   new ( [OPTIONS] )@18@[18]
   report Bundle|Distribution|Module@19@[19]
   upgrade [Module|/Regex/]...@20@[20]
   _search_module_tree( type => TYPE, allow => \@regexex, [data =>
     \@previous_results ] )@21@[21]
     Provide a bus- and machine-independent “DMA mapping interface”.

-----------
So mdocmx would be capable to deal with all of that, except that mdoc(7) resolves "Dq BlaBla" by recursing, and we cannot do nothing about that unless mdoc(7) is rewritten the way i imagined.
Until then i will implement (debug) warnings shall we detect macros (we know about).
I think that is all that can be done.
I'll post the updated macros / preprocessor / manuals then.  (After S-roff is building.)

Anonymous
Fri 26 Jun 2015 05:05:25 PM UTC, comment #10: 

Regarding comment #9:

> .Ss Xr devd 8 Events


Bad idea.  Don't use macros in header lines.

> For now i think it is absolutely sufficient to state --
> for mdocmx -- that people should use a headline like
> "Device daemon events" as the headline, especially since
> the next output line contains the same reference again.


Exactly, and not just for mdocmx, but for mdoc(7) in general,
and not just for now, but for good.  I should probably add
a warning message to mandoc(1) to be printed when people
make the mistake of putting macros into header lines.

In the context of mdocmx, i don't think you need to worry
about such craziness at all.

Ingo Schwarze <schwarze>
Group Member
Sat 20 Jun 2015 11:24:06 AM UTC, comment #9: 

For your interest: today i have seen a case were mdocmx fails.

FreeBSD commit r284611 (Xin LI):
share/man/man4/acpi_ibm.4:

-.Ss Xr devd 8 Ss Events
+.Ss Xr devd 8 Events
 Hotkey events received by
 .Xr devd 8
 provide the following information:

We resolve this as

  .Mx -anchor-spass Ss "Xr devd 8 Events" 4

which (in debug mode) ends up as

  @11@[11]devd(8) Events[35]

(instead of "devd(8)[35] Events@11@[11]").
[35] is the (correct) external manual reference devd(8)!

In fact i don't understand why we are able to resolve this at all, i'll inspect that once S-roff is usable.
My current plan would be to provide debug logs for the preprocessor and i don't know yet what i can do in mdocmx(7) in order to warn the user.
Can this ever work correctly in non-multipass troff(1) and without rewriting mdoc(7) as imagined?  I don't know yet.
Like i've said, i'm wondering why we are able to provide correct anchor and links as such.
For now i think it is absolutely sufficient to state -- for mdocmx -- that people should use a headline like "Device daemon events" as the headline, especially since the next output line contains the same reference again.

Final note, since some manuals, including GNU roff ones, sometimes use .Xr without a reference: if the manual would have been changed to

 .Ss Xr devd Events

we would have got

 @11@[11]devd(Events)[35]

(or "devd(Events)" in mandoc(1)), so that i truly see the use case as above as an error.

Anonymous
Tue 12 May 2015 06:59:24 PM UTC, comment #8: 

So ok, i've added a "mx-toc-disable" string as well as enabling PDF annotations.

I've not yet added macro-hooks to back .Xr with functionality for formats other than TTY.  It would maybe make sense to add a complete set of hooks so that indices etc. could be generated in separate files etc.  Otherwise anyone who can define a macro can as well just "hook" .Xr itself and ensure the original is called by itself, which is what mdocmx(7) will see then.
(I've used that technique to add some .pdfinfo's, too, for example.)

In the end however it must be said that PDF support of the mdoc(7) macros seems completely broken!

I have even shortly thought about disabling PDF support again because if i compare the raw textual output of the examples i've attached before (file #33965 and file #33966) then the PDF output even misses text!  But this is a mdoc(7) problem.

Anyway, i'll attach the updated mdocmx.7 documentation and the new mdocmx.tmac macros.

Thanks and Ciao.

(file #33988, file #33989)

Anonymous
Fri 08 May 2015 11:17:00 AM UTC, comment #7: 

ah - and i wonder what could be done to support those .Xr external references in an output format like, especially, PDF: i think i'll also add a ".if d MACRO" to optionally build an URL?  I see your program does support .Xr, being able to specify a target.
Thanks again!

Anonymous
Fri 08 May 2015 11:07:57 AM UTC, comment #6: 

Hello,

:The nail.pdf looks quite similar to this:-
:
:http://chuzzlewit.co.uk:8080/cgi-bin/WebManPDF.pl/man:/nail
:
:Which is a program which converts man pages, of all types - not :just mdoc, to pdf format. I do prefer having the "Contents" :section in the links panel, rather than the top of the page.

Yes, adding support for the PDF format was done in a simple minded way right before i posted the last RC to the mailing-list.
(I wrote "as a goody" back then: I initially didn't want to add support for other output formats for the current mdoc(7) state because of the "TEXT[START/END]" / "[START]TEXT[/END]" problem: the former simply doesn't take advantage of formats like PDF etc., i.e., the way references stand out should be a matter of a stylesheet or the PDF viewer, but definetely should not include any textual information like "[112]"!  I think adding support for PDF annotations is only a matter of the way PDFMARK is used.  Which needs to be proven.)

Maybe there should be some kind of additional mx-toc-disable switch, so that it is possible to suppress creation of the TOC (that is shown in the document body)?!  For now only the opposite is supported: I'll do so!

Thanks for your suggestions!

Anonymous
Fri 08 May 2015 10:43:08 AM UTC, comment #5: 

The nail.pdf looks quite similar to this:-

http://chuzzlewit.co.uk:8080/cgi-bin/WebManPDF.pl/man:/nail

Which is a program which converts man pages, of all types - not just mdoc, to pdf format. I do prefer having the "Contents" section in the links panel, rather than the top of the page.

Deri James <deri>
Group Member
Fri 08 May 2015 10:13:08 AM UTC, comment #4: 

Ingo, sorry for being so loud.

I also forget to mention in this bug report that you were the initial designer of the interface -- you are of course mentioned in the manual page!


I now also think it is best to give some easy-accessible examples, so please let me attach the rendered output of a manual page which already makes use of the extension in question.

- The first is the TTY output: each number in [] brackets denotes a reference that can be followed.
  Shall you apply the small patch from this bugreport onto less(1) , this example file is interactive, too: via ^A (control-a) these references can be followed, as in
    $ MDOCMX_ENABLE=1 less FILE
  Again: document-internal as well as -- after confirmation -- external references will do.

- The second is PDF output: Interactive, but only for document-internal links.
  Since i haven't mentioned it in this bug report yet: because of the way mdoc(7) is implemented references (and anchors) can only be produced in a "TEXT[START/END]" way, "[START]TEXT[/END]" is not possible, unfortunately.  It would be possible with further changes on the mdoc(7) macros.

Sorry, Ingo.

Thanks for your consideration.

(file #33965, file #33966)

Anonymous
Thu 07 May 2015 10:18:50 AM UTC, comment #3: 

"Significant simplification"?
"Completely unstable"?
"Purely experimental"?

NONE of these are true!
Where did you get ANY evidence that back your words?
I have extended the following manuals with mdocmx(7):

- mdocmx.7, mdocmx.1.  Small and simple.

- nail.1: this is a VERY large and complicated manual page with multiple hundreds of anchors and references.  (As is it is shipped with ArchLinux (and CRUX Linux and Void Linux) in a preprocessed state and can thus be viewed in the wild if you only patch mdoc(7), grotty(1) and less(1).  With or without TOC, just the way you desire.  It can also be viewed if you don't patch them, but then without references and possible TOC.  Hallelujah.)

- mdoc(7) of S-roff.  Also a VERY large and complicated manual with hundreds of anchors and references.

- NetBSDs bus_dma.9 (C interface).  ~27000 bytes.

There is no evidence of just any problem!
Should anyone find any problem i'll fix them, of course.
Just see HOW simple the patches for grotty(1), less(1), mdoc(7) actually are!
Sorry for rising the voice somewhat.

Maybe -- maybe! -- there are manuals in the wild which can exceed the capabilities of the preprocessor (which has nothing to do with mdocmx(7) itself, but only circumvents missing multipass capabilities of troff(1)).
But note that nail.1 and tmac-mdoc.7.in are really, really complicated and i don't see any problems.
And if there is still a manual page which is even more complicated, then possibly it should be adjusted a bit, then.

Btw. the manual can now also be browsed online [1].
Of course that HTML version doesn't support references to external manual pages, as could be easily followed on the TTY via grotty(1) and in less(1).

  http://sdaoden.users.sourceforge.net/code-mdocmx.html

Thanks for your consideration.

Anonymous
Wed 06 May 2015 06:33:35 PM UTC, comment #2: 

It's nice to have this code in the bugtracker, but please don't include it into groff yet.  I very strongly urge people to not use these features for real-world manuals yet for lack of portability.

Before this can become usable in practice, it will likely need very significant simplification, so we should consider the interface completely unstable and purely experimental and not encourage the use for any real work yet.

Ingo Schwarze <schwarze>
Group Member
Wed 06 May 2015 12:12:39 PM UTC, comment #1: 

Oops, groff-1_22_3.diff includes addition of a .hash troff(1) request!  This is of course no(t) (longer) needed!  Find attached the diff without .hash (otherwise identical).
Sorry!!

(file #33939)

Anonymous
Wed 06 May 2015 11:53:12 AM UTC, original submission:  

The mdoc(7) manual semantic markup language doesn't support any kind of anchoring: whereas you can exactly state what "x" is whenever you refer to it -- variable, function etc. --, you have no option to define the exact place where "x" is itself defined, or wether it is at all defined in a given manual page.

Also, whereas mdoc(7) does support differentiation in between anchors and references for headlines (.Sh/.Ss and .Sx, respectively), referencing a headline is only of notational interest, the reference is in no way "active", never.

mdocmx(7) extends mdoc(7) by adding all the missing functionality and (much) more with a single new multiplexer command: .Mx.
The attached manual page mdocmx.7 documents in detail; the manual is itself mdocmx(7)-enabled.

And because non-multipass troff(1) implementations are not capable to generate forward references to anchors not yet defined there is a preprocessor necessary to circumvent this problem: mdocmx(1), implemented in portable sh(1) and awk(1).
(A nice property of mdocmx(7) is that it "knows" wether a document has been preprocessed or not, therefore making it possible to distribute preprocessed manuals and still being backward- and forward-compatible.)

For security reasons all parts of the usual manual pipeline require the environment variable MDOCMX_ENABLE=1 to be set to a non-empty value (it must be non-empty because of troff(1) deficiencies):

  mdoc() {
        ( export MDOCMX_ENABLE=1
          : ${MDOCMXFLAGS:=-dmx-toc-force=tree}
          mdocmx.sh "${1}" |
          groff -Tutf8 -mdoc ${MDOCMXFLAGS} |
          LESS= less --RAW-CONTROL-CHARS --ignore-case --no-init )
  }
  mdocdbg() {
        ( export MDOCMX_ENABLE=1
          : ${MDOCMXFLAGS:=-dmx-toc-force=tree}
          mdocmx.sh -vv "${1}" |
          groff -Tutf8 -mdoc -dmx-debug=1 ${MDOCMXFLAGS} |
          LESS= less --RAW-CONTROL-CHARS --ignore-case --no-init )
  }

(The output devices -Tpdf and -Thtml may also be used, though i reiterate that the same output as for TTY is necessarily created due to mdoc(7) deficiencies: it is impossible to generate "<START>text</END>" but only "text[START-END]".  A solution to gain the former is possible, but requires mdoc(7) itself to be changed.)

mdocmx(7) as it is posted here is the result of a seven month evolution following an idea on the GNU roff mailing-list [1,2,3].

  [1] http://lists.gnu.org/archive/html/groff/2014-09/msg00145.html
  [2] http://lists.gnu.org/archive/html/groff/2014-12/msg00045.html
  [3] http://lists.gnu.org/archive/html/groff/2015-04/msg00043.html

Changes compared to [3]:
- a fix in mdoc(7) patch (re-add lost .nop's; was installed like that for months, but not committed).
- the mandoc(1) originator asked for freely definable anchors and references to them, which he needs for his texi2mdoc(1) etc. converters, so .Mx got new -ix and -sx modes.

P.S.: since this form only allows four attachments i had to create a ball:
- groff-1_22_3.diff: add mdocmx(7) support to grotty(1) and mdoc(7).
- less-471.diff: add mdocmx(7) (in fact: "anchor") support to less(1) v471.
- mdocmx.tar.xz: a ball with mdocmx.tmac (the macros), mdocmx.7 (the .Mx manual), mdocmx.sh (the preprocessor) and mdocmx.1 (preprocessor manual).
- mdocmx.7: the manual once again, but this time ball-less ;)

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #51476:  grotty-osc8.patch added by None (4KiB - application/octet-stream)
file #51477:  less-osc8-search.patch added by None (12KiB - application/octet-stream)
file #35046:  groff-mdocmx-fix1.diff added by None (3KiB - application/octet-stream)
file #33988:  mdocmx.7 added by None (14KiB - application/octet-stream)
file #33989:  mdocmx.tmac added by None (25KiB - application/octet-stream)
file #33966:  s-nail.pdf added by None (407KiB - application/pdf)
file #33965:  s-nail.txt added by None (267KiB - text/plain)
file #33939:  groff-1_22_3.diff added by None (23KiB - text/plain)
file #33935:  groff-1_22_3.diff added by None (26KiB - text/plain)
file #33936:  mdocmx.tar.xz added by None (18KiB - application/x-xz)
file #33937:  less-471.diff added by None (10KiB - text/plain)
file #33938:  mdocmx.7 added by None (14KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Updated the item)
  • -email is unavailable- added by deri (Posted a comment)
  • -email is unavailable- added by schwarze (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 21 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-06-10 gbranden Summary[PATCH] mdoc(7): add support for the mdocmx(7) reference extension mdoc(7): add support for the mdocmx(7) reference extension
    2021-05-25 None Attached File- Added grotty-osc8.patch, #51476
        Attached File- Added less-osc8-search.patch, #51477
    2020-10-21 gbranden Assigned toNone schwarze
    2020-10-21 gbranden CategoryNone Macro mdoc
        Item GroupNone Feature change
    2020-10-13 schwarze Severity3 - Normal 1 - Wish
        StatusNone Rejected
        Open/ClosedOpen Closed
    2020-10-13 barx Carbon-CopyRemoved 93119 -
    2020-10-13 barx Summarymdoc(7): add support for the mdocmx(7) reference extension [PATCH] mdoc(7): add support for the mdocmx(7) reference extension
    2015-10-03 None Attached File- Added groff-mdocmx-fix1.diff, #35046
    2015-05-12 None Attached File- Added mdocmx.7, #33988
        Attached File- Added mdocmx.tmac, #33989
    2015-05-08 None Attached File- Added s-nail.txt, #33965
        Attached File- Added s-nail.pdf, #33966
    2015-05-06 None Attached File- Added groff-1_22_3.diff, #33939
    2015-05-06 None Attached File- Added groff-1_22_3.diff, #33935
        Attached File- Added mdocmx.tar.xz, #33936
        Attached File- Added less-471.diff, #33937
        Attached File- Added mdocmx.7, #33938

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code