bugGNU troff - Bugs: bug #61439, [tmac] test...

 
 

bug #61439: [tmac] test "an_inner-footer-abbreviation-works.sh" fails if package defaults changed

Submitted by:  Bjarni Ingi Gislason <bjarniig>
Submitted on:  Sun 07 Nov 2021 05:11:16 PM UTC
 
Category:  Macro man Severity:  3 - Normal
Item Group:  Test Status:  Need Info
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       No canned response available

 

Sun 06 Feb 2022 05:32:07 PM UTC, comment #4: 

It's been almost 3 months; I propose to close this as invalid in
a day or two.

G. Branden Robinson <gbranden>
Project Administrator
Mon 08 Nov 2021 09:24:21 AM UTC, comment #3: 

Hi, Bjarni,

comment #2:

>
> comment #1:
> [...]
>
> > I need more to go on.  What is in the test log file?  It should be in your build tree:
> >
> > tmac/tests/an_inner-footer-abbreviation-works.sh.log
> >
>
> Yes, you need more to go on.
>   You need information (and thinking) but primarily from
> yourself.


Another groff contributor has recently pointed out how you shouldn't hold your breath waiting on me to think correctly.  Explain issues so that many people can understand.

> > I have a suspicion that your personal groff branch is setting the CHECKSTYLE register to a nonzero value, perhaps in tmac/man.local or tman/an.tmac.
> >
>
>   The change is in "git/groff/tmac/an.tmac" among other additions of
> mine.


Okay.  All is well.  We've been around this block before--maybe Dave can remember in which ticket(s).  The groff test suite is not a generic test harness for *roff implementations in general.

That would be a wonderful thing to have but it's out of scope for the groff project itself.  As I understand it, Bertrand put the infrastructure and first few tests in because groff had no automated regression testing at all and it was sorely needed, given the project's size and complexity.

Some of the tests are "white box"[1] tests, coded with knowledge of implementation internals.  A general *roff test suite would probably take a "black box" approach, caring about nothing but generated, user-facing output.

One of the consequences is that the test suite is coupled to the branch that it's in.  That's why I say, if you're going to make changes to the implementation, you may have to modify the tests as well.

> > If you've made such a change, the responsibility is similarly on you to change the tests to update their expectations.
> >
>
>   No, it is not.  The defect is in the test code, not in the
> environment of the user.  It is your expectation that is wrong, not
> that of any other user.


If someone later changes the default CHECKSTYLE to be "truthy", it likely will break at least one test.  That's a consequence of another test design goal: I want freedom to tweak the language of diagnostic messages, and I don't want to have to complicate the tests by pattern-matching diagnostic output, especially when it's not the point of the test.

The test at issue is written to deliberately expect zero output to the standard error stream.  If you hack up your tmac/an.tmac to instrument it, it and other tests will fail.  This happens to me when I troubleshoot our man(7) macros.  Once I figure out the macro programming problem that has bedeviled me, I back out the instrumentation and write a test for root-cause problem I discovered.  And if I can't figure it out, as with hyperlink macros in paragraph tags recently, I certainly don't leave the instrumentation in and rewrite all the failing tests to expect it.

I make no claim to be the world's greatest software tester but I do think the approach I'm using helps keep the tests short, focussed, and relatively easy to understand (notwithstanding the big readability penalty that the POSIX shell language imposes).

>   If a test fails by returning a value different from zero, it
> may be expected that the observer investigates the cause, if he knows
> how, in my case by issuing


Yes, "sh -x".  Often I find myself editing the test to just dump the output of the offending test ('echo "$output"').  Frequently I can tell quickly by visual inspection what has gone wrong, and there is little other output to distract me.  "sh -x" can be chatty.

>   Your issue now is similar to the bug report
>
>  #58164] First html-test fails when a non-utf-8 charmap is used
>
> where you deny a general solution, but then later did change the code
> to fulfil your own constraints.


Yes.  I had access to a Mac OS X build environment and, curious, I built groff on it to see what would happen.

You should note that the test was altered to skip itself if the environment didn't prove adequate.

> ---cut---
> (commit
> f51cf27208e70ccb60afc4a245f293a0146f77ed
> Author: G. Branden Robinson <g.branden.robinson@gmail.com>
> Date:   Thu Oct 28 00:00:33 2021 +1100
>
>     [tests]: Fix portability problems.
>
> [...]
>
>     * src/roff/groff/tests/smoke-test_html_device.sh: Stop trying to set
>       $LC_CTYPE to "C.UTF-8"; some systems don't support this
> expediency.
>       Skip the test if the tester hasn't configured the environment
>       adequately.
> ---cut---
>
>
>   An update of my proposal was put forward in
>
> https://lists.gnu.org/archive/html/groff/2020-11/msg00138.html
> (Subject: Testing the test)
>
>   I later reduced the cases to be tested in my repository.
>
>   Fact is: that test does not need any locale, "groff" handles that.
>
>   My current example (removing irrelevant lines):
>
> echo "testing -k -Thtml" >&2
> printf '\303\241' | "$groff" -KUTF-8 -Thtml | grep -qx '<p>&aacute;</p>'
>
> # We test compatibility-mode HTML output somewhat differently since
> # preconv only emits groffish \[uXXXX] escapes for non-ASCII codepoints.
> echo "testing -C -k -Thtml" >&2
> printf "\('a" | "$groff" -C -k -Thtml | grep -qx '<p>&aacute;</p>'
>
> ###
>
>   "The problems of the real world are primarily those you are
> left with when you refuse to apply their effective solutions."


The lesson here is that real world experience with a Mac OS X environment proved more persuasive to me than your occasionally context-light and sometimes hectoring reports, particularly when you sprinkle them with quotations as evangelists do with their favorite texts.

There is much philosophical literature than we can read for wisdom, certainly including EWD, but quoting it at length in an attempt at persuasive discourse suggests that your are not molding your arguments to your context and interlocutor, but expecting your counterparty to reshape themselves to fit your dogma.  Ultimately, that is an authoritarian approach, not a collegial one.

In simpler language, you need to work harder to convince me using fresh language appropriate to the scenario.  Dropping scripture in front of people and inviting them to proof-text it in order to find their way into agreement with you is not likely to succeed.

(Okay, I brought up proof-texting.  Sometimes I fail to achieve simpler language.)

Normally I'd close the ticket as invalid based on the customization revelation, but since I got all philosophical I'll leave it open and invite you (and others) to try the approach to persuasion that I claim is more effective.

Admittedly, I may remain a stick in the mud nevertheless, and thereby similarly fail to be an effective advocate for my principles.

Regards,
Branden

[1] "Clear box" would probably be a better metaphor but it's not one I've seen in use.

G. Branden Robinson <gbranden>
Project Administrator
Mon 08 Nov 2021 12:51:19 AM UTC, comment #2: 

comment #1:
[...]

> I need more to go on.  What is in the test log file?  It should be in your build tree:
>
> tmac/tests/an_inner-footer-abbreviation-works.sh.log
>


Yes, you need more to go on.
  You need information (and thinking) but primarily from
yourself.

 This is there
---cut---
testing long inner footer with sufficient space to set it
testing long inner footer with insufficient space to set it
testing for graceful behavior when TH has no 3rd argument
...FAILED
FAIL tmac/tests/an_inner-footer-abbreviation-works.sh (exit status: 1)
---cut ---

> I have a suspicion that your personal groff branch is setting the CHECKSTYLE register to a nonzero value, perhaps in tmac/man.local or tman/an.tmac.
>


  The change is in "git/groff/tmac/an.tmac" among other additions of
mine.

> If you've made such a change, the responsibility is similarly on you to change the tests to update their expectations.
>


  No, it is not.  The defect is in the test code, not in the
environment of the user.  It is your expectation that is wrong, not
that of any other user.

  If a test fails by returning a value different from zero, it
may be expected that the observer investigates the cause, if he knows
how, in my case by issuing

<definition of the path to test-groff> sh -x an_inner-footer-abbreviation-works.sh

> If I'm correct, this bug should be closed as invalid.  If I'm not, I definitely need more information.


###

  Your issue now is similar to the bug report

 #58164] First html-test fails when a non-utf-8 charmap is used

where you deny a general solution, but then later did change the code
to fulfil your own constraints.

---cut---
(commit
f51cf27208e70ccb60afc4a245f293a0146f77ed
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Thu Oct 28 00:00:33 2021 +1100

    [tests]: Fix portability problems.

[...]

    * src/roff/groff/tests/smoke-test_html_device.sh: Stop trying to set
      $LC_CTYPE to "C.UTF-8"; some systems don't support this
expediency.
      Skip the test if the tester hasn't configured the environment
      adequately.
---cut---

  An update of my proposal was put forward in

https://lists.gnu.org/archive/html/groff/2020-11/msg00138.html
(Subject: Testing the test)

  I later reduced the cases to be tested in my repository.

  Fact is: that test does not need any locale, "groff" handles that.

  My current example (removing irrelevant lines):

echo "testing -k -Thtml" >&2
printf '\303\241' | "$groff" -KUTF-8 -Thtml | grep -qx '<p>&aacute;</p>'

# We test compatibility-mode HTML output somewhat differently since
# preconv only emits groffish \[uXXXX] escapes for non-ASCII codepoints.
echo "testing -C -k -Thtml" >&2
printf "\('a" | "$groff" -C -k -Thtml | grep -qx '<p>&aacute;</p>'

###

  "The problems of the real world are primarily those you are
left with when you refuse to apply their effective solutions."

Page xxxviii in:

On the Cruelty of Really Teaching Computing Science

Edsger W. Dykstra (Dijkstra)

SIGCSE Bulletin 1989, 21(1), pages xxv-xxxix.
Also "www.cs.utexas.edu/users/EWD/"
#

        It is necessary to consider all consequences of
        the proposed action in all states of nature.

There are lots of consequences of any given action.  Many of the
problems of society today are at least partly due to the failure
to realize that the "obvious" action taken had so many bad side
effects.

Herman Rubin on the Usenet forum "misc.education.science".
#

Bjarni Ingi Gislason <bjarniig>
Sun 07 Nov 2021 05:59:14 PM UTC, comment #1: 

original submission:

>  The option "-rCHECKSTYLE=0" is needed to exclude external
> influence.
>


I need more to go on.  What is in the test log file?  It should be in your build tree:

tmac/tests/an_inner-footer-abbreviation-works.sh.log

I have a suspicion that your personal groff branch is setting the CHECKSTYLE register to a nonzero value, perhaps in tmac/man.local or tman/an.tmac.

If you've made such a change, the responsibility is similarly on you to change the tests to update their expectations.

If I'm correct, this bug should be closed as invalid.  If I'm not, I definitely need more information.

G. Branden Robinson <gbranden>
Project Administrator
Sun 07 Nov 2021 05:11:16 PM UTC, original submission:  

 The option "-rCHECKSTYLE=0" is needed to exclude external
influence.

Bjarni Ingi Gislason <bjarniig>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by bjarniig (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-11-08 gbranden Summary[tmac]: test "an_inner-footer-abbreviation-works.sh" fails => [tmac] test "an_inner-footer-abbreviation-works.sh" fails if package defaults changed
    2021-11-07 gbranden StatusNone => Need Info
        Assigned toNone => gbranden

    Back to the top


    Powered by Savane 3.9