bugGNU roff - Bugs: bug #59664, grog: Increase the number of files...

 
 

bug #59664: grog: Increase the number of files that are identified correctly

Submitter:  Bjarni Ingi Gislason <bjarniig>
Submitted:  Sat 12 Dec 2020 01:48:51 AM UTC
   
 
Category:  Utilities Severity:  3 - Normal
Item Group:  Incorrect behaviour Status:  Fixed
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  1.23.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 28 Jun 2021 04:47:45 AM UTC, comment #3: 


commit 2ba599980081fb700e09c10abcc136657ae93070
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Jun 28 14:28:32 2021 +1000

    [grog]: Heavily refactor.

    * src/utils/grog/grog.pl:
      - Drop import of unused module `Data::Dumper`.
      - Drop unused scalars `Sp` and `correct_tmac`.
      - Simplify determination of version number.  Drop hash `at_at` which
        only stored one key, `GROFF_VERSION`.  Initialize scalar
        `groff_version` to "DEVELOPMENT".  Rename scalar `before_make` to
        `in_source_tree` and initialize to zero.  Update `groff_version`
        with Automake-determined version variable if it is defined (i.e.,
        grog is not running in an unbuilt source tree).
      - Drop unused `Mparams` list.  Replace it with new list
        `requested_package`, which stores the arguments to any grog `-m`
        options specified by the user.
      - Rename many objects so that I, and others, can better comprehend
        their purpose, and for consistent letter casing.
        . @Command -> @command
        . @devices -> @device
        . $Prog -> $program_name
        . %macros -> %user_macro
        . $have_any_valid_args -> $have_any_valid_arguments
        . &handle_args -> &process_arguments
        . &handle_whole_files -> &process_input
        . @preprograms -> @preprocessor
        . &make_groff_device -> &infer_device
        . &make_groff_preproc -> &infer_preprocessors
        . &make_groff_tmac_man_ms -> &infer_man_or_ms_package
        . &make_groff_line_rest -> &construct_command
      - Drop many unused keys in `Groff` hash.
      - Add new lists, `macro_ms`, `macro_man`, and `macro_man_or_ms` to
        support new scoring technique to disambiguate input documents
        between these two packages.
      - Append the foregoing 3 lists to new list `standard_macro`, and add
        these as keys to the `Groff` hash.
      - Add new list `main_package` to keep track of full-service package
        names.
      - Add new scalars `man_score`, `ms_score`, and `inside_tbl_table` to
        aid disambiguation of .TH macro calls and the many macro names
        shared between man(7) and ms(7).

      (process_arguments): Strip '-m' off of argument before storing the
      remainder in `@requested_package`.

      (do_line): Detect .TH macro call even if white space occurs between
      the control character and the macro name.

      (do_line): Inflate `$man_score` by 100 if .TH is the first macro call
      seen in a document.

      (do_line): Fix bug; clear `$before_first_command` in correct
      scope--after any macro call, not just if we saw a .TH as the first
      macro call.

      (do_line): Set `$inside_tbl_table` when we see a .TS call.

      (do_line): Clear `$inside_tbl_table` when we see a .TE call.  Also
      increment `$Groff{'tbl')' again, increasing the "score" of tbl(1)
      usage evidence.

      (do_line): Drop a lot of code that manually increments %Groff keys
      corresponding to man and ms macros.  This is now done differently and
      elsewhere.

      (do_line): Drop "P" from list of characteristic mm(7) macros.

      (do_line): Simplify matching of mom(7) macros (match $command, not
      $line).  Extend list of characteristic mom(7) macros.

      (do_line): Increment $Groff{$key} if $key is in @standard_macro.

      (infer_man_or_ms_package): Rewrite.  Compute a score for each package
      by counting occurrences of their characteristic macros.  If both have
      a score of zero, assume that the input is a raw roff document.  If the
      scores are equal (doc/webpage.ms, startlingly, comes within 1 point of
      a tied score), infer ms(7) if 'TH' was never called, and if it was,
      issue a diagnostic advising user to supply a disambiguating `-m`
      option.  Otherwise, the scores are unequal, and infer the package of
      the winner.  Set scalar `inferred_main_package` instead of pushing
      `-m` options onto `@m`.

      (infer_macro_packages): Set scalar `inferred_main_package` instead of
      pushing `-m` options onto `@m`.  Explicitly return 0 if we fall off
      the end of the function.

      (construct_command): Rewrite handling of -m options.  Add new list
      `msupp` to store supplementary (non-main) macro package arguments.  If
      a full-service package was explicitly requested, it had better not
      clash with what we inferred.  If it does, explicitly unset
      $inferred_main_package so that the -m arguments are placed in the same
      order that the user gave them; caveat dictator.  If `--run` option was
      given, just print the command; don't preface it with __FILE__ and
      __LINE__ noise.

      - Remove comments documenting shared variables used by subroutines.
        These are far from useless but too tedious to keep up to date while
        the code is in flux.
      - Note several places for further code review or refactoring with
        "XXX" comments.
      - Add Vim modeline.

    grog now passes all its tests and correctly infers arguments for all
    in-tree groff documents (except for a known, and already documented in
    grog(1), false positive detection of soelim in soelim(1)).  This
    refactor also obviates or resolves several outstanding Savannah tickets.

    Fixes <https://savannah.gnu.org/bugs/?44707> by obviating it; grog no
    longer cares about file name extensions on man pages (or any other
    input).

    Fixes <https://savannah.gnu.org/bugs/?55302>; same.  The quality of
    diagnostic messages has been improved as well.

    Fixes <https://savannah.gnu.org/bugs/?59753>; same.

    Fixes <https://savannah.gnu.org/bugs/?59664>.  The attached patch was a
    less aggressive refactor of &do_line and %Groff.  Its author made the
    following claim for it: "With this patch, all 'man', 'me', 'mom, and
    'ms' files in the repository are correctly identified.  The only example
    of a 'mm'-file is "letter.mm", which is not recognized correctly."  As
    noted above, the present refactor achieves correct recognition of all of
    the files including letter.mm.


G. Branden Robinson <gbranden>
Group administrator
Fri 11 Jun 2021 10:05:37 AM UTC, comment #2: 

Discussion in (now-closed) bug #59676 indicates that Bjarni feels this patch needs modifications.  Branden requested in that bug that Bjarni update this patch.  So for now, removing "[PATCH]" from the Summary, since no one seems to think the existing patch here is valid.  Will restore it when a corrected patch is posted.

Dave <barx>
Group Member
Mon 07 Jun 2021 09:49:41 PM UTC, comment #1: 

Categorizing grog bugs as Utilities, per discussion.

Dave <barx>
Group Member
Sat 12 Dec 2020 01:48:51 AM UTC, original submission:  

Subject: [PATCH] grog/subs.pl: Increase the number of files that are
 identified correctly

src/roff/grog/subs.pl: Increase the number of files that are identified
correctly.

1) Don't remove commands whoes lowercase names are used as a detection
for 'me'-files.

2) The macro ".TH" can also be a part of a table.
Don't interpret that as an indicator for a 'man' or a 'ms' file.

3)  Don't use the macros ".TL", ".PP", and ".SS" as indicators,
as they can lead to a false result.

4) Add the arguments to the regular expression for macros in "mom"
files.

  With this patch, all 'man', 'me', 'mom, and 'ms' files in the
repository are correctly identified.

  The only example of a 'mm'-file is "letter.mm",
which is not recognized correctly.

Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
---
 src/roff/grog/subs.pl | 48 ++++++++++++++++++++++++-------------------
 1 file changed, 27 insertions(+), 21 deletions(-)

diff --git a/src/roff/grog/subs.pl b/src/roff/grog/subs.pl
index 1c40dd95..54d195b5 100644
--- a/src/roff/grog/subs.pl
+++ b/src/roff/grog/subs.pl
@@ -114,11 +114,11 @@ my %Groff =
    'IP' => 0,                # man and ms
    'LP' => 0,                # man and ms
    'P' => 0,                # man and ms
-   'PP' => 0,                # man and ms
+   'PP' => 0,                # man, mom,  and ms
    'SH' => 0,                # man and ms

    'OP' => 0,                # man
-   'SS' => 0,                # man
+   'SS' => 0,                # man and mom
    'SY' => 0,                # man
    'TH_first' => 0,        # TH as 1st command is man
    'TP' => 0,                # man
@@ -569,16 +569,22 @@ sub do_line {
   return if ( $line =~ /^\.\.$/ );        # ignore ..

   # Ignore "Standard preamble" from perl module "Pod::Man"
-  return if ( $line =~ /^\.[[:lower:]]/ );
-  return if ( $line =~ /^\.\s*\\}/ );
-  return if ( $line =~ /^\.IX / );
+  # Keep commands, that are in 'me' files
+  if ( $line !~ /^\.(
+                    [ilnp]p|
+                    sh
+                  )$/x ) {
+    return if ( $line =~ /^\.[[:lower:]]/ );
+    return if ( $line =~ /^\.\s*\\}/ );
+    return if ( $line =~ /^\.IX / );
+  }

   if ( $before_first_command ) { # so far without 1st command
-    if ( $line =~ /^\.TH/ ) {
+    if ( ($line =~ /^\.TH/ ) && ($Groff{'tbl'} == 0 ) ) {
       # check if .TH is 1st command for man
       $Groff{'TH_first'} = 1 if ( $line =~ /^\.\s*TH/ );
     }
-    if ( $line =~ /^\./ ) {
+    if ( ( $line =~ /^\./ ) && ($Groff{'tbl'} == 0) ) {
       $before_first_command = 0;
     }
   }
@@ -691,7 +697,7 @@ sub do_line {
     return;
   }
   if ( $command =~ /^\.TH$/ ) {
-    unless ( $Groff{'TH_first'} ) {
+    unless ( $Groff{'TH_first'} || ($ext eq 'mom') ) {
       $Groff{'TH_later'}++;                # for tbl
     }
     return;
@@ -760,10 +766,10 @@ sub do_line {
     $Groff{'NH'}++;                # for ms
     return;
   }
-  if ( $command =~ /^\.TL$/ ) {
-    $Groff{'TL'}++;                # for ms
-    return;
-  }
+#  if ( $command =~ /^\.TL$/ ) {
+#    $Groff{'TL'}++;                # for ms
+#    return;
+#  }
   if ( $command =~ /^\.XP$/ ) {
     $Groff{'XP'}++;                # for ms
     return;
@@ -786,10 +792,10 @@ sub do_line {
     $Groff{'P'}++;                # for man and ms
     return;
   }
-  if ( $command =~ /^\.PP$/ ) {
-    $Groff{'PP'}++;                # for man and ms
-    return;
-  }
+#  if ( $command =~ /^\.PP$/ ) {
+#    $Groff{'PP'}++;                # for man, mom, and ms
+#    return;
+#  }
   if ( $command =~ /^\.SH$/ ) {
     $Groff{'SH'}++;                # for man and ms
     return;
@@ -807,10 +813,10 @@ sub do_line {
     $Groff{'OP'}++;
     return;
   }
-  if ( $command =~ /^\.SS$/ ) {        # for man
-    $Groff{'SS'}++;
-    return;
-  }
+#  if ( $command =~ /^\.SS$/ ) {        # for man and mom
+#    $Groff{'SS'}++;
+#    return;
+#  }
   if ( $command =~ /^\.SY$/ ) {        # for man
     $Groff{'SY'}++;
     return;
@@ -886,7 +892,7 @@ sub do_line {
                    PRINTSTYLE|
                    PT_SIZE|
                    T_MARGIN
-                 )$/x ) {
+                 ).*$/x ) {
     $Groff{'mom'}++;                # for mom
     return;
   }
--
2.29.2


Bjarni Ingi Gislason <bjarniig>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Updated the item)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by bjarniig (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-06-28 gbranden StatusIn Progress Fixed
        Open/ClosedOpen Closed
        Planned ReleaseNone 1.23.0
        Summarygrog/subs.pl: Increase the number of files that are identified correctly grog: Increase the number of files that are identified correctly
    2021-06-28 gbranden StatusNeed Info In Progress
        Assigned toNone gbranden
    2021-06-11 barx StatusNone Need Info
    2021-06-11 barx Summary[PATCH] grog/subs.pl: Increase the number of files that are identified correctly grog/subs.pl: Increase the number of files that are identified correctly
    2021-06-07 barx Carbon-CopyRemoved 93119 -
    2021-06-07 barx CategoryCore Utilities

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code