mainAutoconf - Support: sr #110403, autoconf-2.70 trips bug in...

 
 

sr #110403: autoconf-2.70 trips bug in here-doc handling in OmniOS /bin/sh

Submitter:  Bob Friesenhahn <bfriesen>
Submitted:  Wed 16 Dec 2020 08:40:22 PM UTC
   
 
Priority:  * 5 - Unprioritized Severity:  3 - Normal
Status:  Not Autoconf Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Operating System:  Solaris
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 07 Mar 2021 08:09:23 PM UTC, comment #12: 

The bug exists also in the /bin/sh of Solaris OpenIndiana 20171031.

The symptom is this failure during configure:

configure: creating ./config.status
config.status: creating Makefile
gawk: ./conf0h1pqt/subs.awk:1691: cat >>"$ac_tmp/subs1.awk" <<_ACAWK &&
gawk: ./conf0h1pqt/subs.awk:1691:     ^ syntax error
config.status: error: could not create Makefile


Find attached a tarball that contains
- the configure script,
- the broken config.status,
- the correct config.status (generated with CONFIG_SHELL=/usr/bin/bash).

(file #51026)

Bruno Haible <haible>
Tue 05 Jan 2021 06:33:23 PM UTC, comment #11: 

I see that https://www.illumos.org/issues/13405 addresses updating Illumos ksh93 to a recent version.  The description says that the task is 'hard' but the task is showing progress made.

Regardless, a note about the bug is certainly useful since a fix would not be deployed and available to all for some time.

Bob Friesenhahn <bfriesen>
Tue 05 Jan 2021 06:01:47 PM UTC, comment #10: 

I have filed https://www.illumos.org/issues/13434 for the shell bug.  Do you think it's worth mentioning this issue in the Autoconf release notes?

Zack Weinberg <zackw>
Group administrator
Thu 24 Dec 2020 09:54:26 PM UTC, comment #9: 

Since the shell delivered with OmniOSce is provided by Illumos, the appropriate bug tracker would be at https://www.illumos.org/projects/illumos-gate/issues

Bob Friesenhahn <bfriesen>
Thu 24 Dec 2020 08:44:24 PM UTC, comment #8: 

Given that

  • OmniOS has an active development [1],
  • The OpenSolaris community is approximately equally divided among OmniOS, SmartOS, and OpenIndiana [2],
  • You haven't found this bug in other Solaris derivatives,
  • The workaround is to set the environment variable CONFIG_SHELL=/path/to/bash,


I would recommend to just document the problem and workaround in two places:
1) in the OmniOS community,
2) at https://gitlab.com/ghwiki/gnow-how/-/wikis/Platforms/Configuration for GNU people.

It's not worth the complexity in Autoconf for a short-lived bug in a small-community system, when a workaround is so easy to put in place.

References:
[1] https://en.wikipedia.org/wiki/Comparison_of_OpenSolaris_distributions
[2] trends.google.com

Bruno Haible <haible>
Thu 24 Dec 2020 08:04:14 PM UTC, comment #7: 

OK, I've verified that this is a bug in OmniOS /bin/sh.

OmniOS /bin/sh reads scripts in 8192-byte chunks.  If the last line of a here-document straddles a chunk boundary, and the last character of that line is not a backslash, but the last character of the first chunk is a backslash, and the first character of the second chunk is a dollar sign, then the here-document will be written out without a final newline.  (I'm not sure whether the boundary has to fall within the last line of the here-document, or whether the first character of the second chunk has to be a dollar sign.  I know \# doesn't work, though.)

The attached file t-001fa2.sh.gz is a self-contained reproducer for the bug.  If you uncompress it and run it with a shell that has this bug, it will print something like this:


cmp: EOF on /tmp/tmp.LCyimJxG8v after byte 1, in line 1
+ xxd /tmp/tmp.q5iPccsLOn
00000000: 240a                                     $.
+ xxd /tmp/tmp.LCyimJxG8v
00000000: 24                                       $
+ exit 1
+ rm -f /tmp/tmp.q5iPccsLOn /tmp/tmp.LCyimJxG8v


(Note that the script does not have a `#!` line.  Also, you need to have `xxd` for it to work as intended, but both invocations of this utility are after the critical part of the code, so you can safely change them to any other hex-dump program you may happen to have around without breaking the test.  I have also attached the program I used to generate the test script, which may be easier to tinker with.)

(The name t-001fa2 refers to the number of padding characters in the script, before the actual code.)

> I wonder why autoconf-2.69b does not exhibit the issue


I believe this was pure luck.  In your testing, autoconf-2.69b happened not to generate a script that had \$, inside a here document, at exactly the right offset.  I don't see any reason why this couldn't have happened with any version of Autoconf.

It's also possible that the bug was recently introduced -- did you patch IP65-66-246-71.simplesystems.org recently?  I wasn't able to reproduce the bug with any of /bin/sh, /bin/ksh, or /usr/xpg4/bin/sh on heritage Sun Solaris 10 or 11.




Now we know what's wrong, the question is what to do about it.  Of course OmniOS should be notified.  I looked at https://omniosce.org/ a bit and was not able to figure out how to report the bug.  (It says that development happens on Github, and I found a link to what seems to be the repo where development happens, namely https://github.com/omniosorg/illumos-omnios -- but that repo has no open issues at all, so I'm guessing that is not the place to file bug reports.)

It is not feasible to make Autoconf avoid ever writing out backslashes inside here-documents.  Generation of config.status, in particular, involves using here-documents to emit shell code, in which backslashes are often necessary.

What we could do is add a test for this sh bug to _AS_DETECT_BETTER_SHELL.  It would be a little fiddly but not difficult.  I hesitate, though, because of how sensitive this test would be to unrelated modifications to the shell -- for instance, if someone happens to decide tomorrow that OmniOS sh should read files in 16kB chunks instead of 8kB chunks, a test that only probed the 8kB boundary would be invalidated.  How many different offsets do we need to try before we have confidence that the bug is not present?

cc:ing several other autoconf maintainers for their thoughts.

(file #50563, file #50564)

Zack Weinberg <zackw>
Group administrator
Wed 23 Dec 2020 10:02:46 PM UTC, comment #6: 

I wonder why autoconf-2.69b does not exhibit the issue, and in fact I have not seen any other issues with configure scripts on this system?  This system has been one of my core development systems for quite some time now.

Note that this system has a config.site script at the default location, but that is normal for my systems.

Bob Friesenhahn <bfriesen>
Wed 23 Dec 2020 09:15:27 PM UTC, comment #5: 

I just ran the autoconf testsuite myself on this system, using the tip of branch-2.70.  I do not see AC_TYPE_INTMAX_T fail, but I do see two other failures which appear to have the same root cause: config.status files with a missing line break at the end of a here-document in the parent configure script.


SHELL=${CONFIG_SHELL-/bin/sh}
export SHELL## -------------------- ##
## M4sh Initialization. ##


and


test -n "$AWK" || AWK=awk# The default lists apply if the user does not specify any file.


In Bourne shell syntax, `#` only begins a comment if it is the beginning of a shell word, so both of these turn into syntax errors.

To my surprise, this is not an intermittent fault.  It happens every single time I run the configure script for one of the tests that fails for me.  However, it doesn't happen if I run `bash ./configure` instead of `./configure`, and it goes away if I add a single blank line near the top of the offending configure script. 

I think what we're looking at here is a bug in OmniOS /bin/sh, in which here-documents lose their final \n character if they are at particular offsets within a shell script.  I'll see if I can come up with a precise test for that hypothesis.

Zack Weinberg <zackw>
Group administrator
Wed 23 Dec 2020 07:44:34 PM UTC, comment #4: 

I just verified that this test passes for autoconf-2.69b.  This is the test result using autoconf-2.69b, which is what I had tested on this system previously:

## ------------- ##
## Test results. ##
## ------------- ##

474 tests behaved as expected.
49 tests were skipped.

Bob Friesenhahn <bfriesen>
Wed 23 Dec 2020 07:41:39 PM UTC, comment #3: 

The external host name appears to be "IP65-66-246-71.simplesystems.org".  Sorry for not making it more memorable in DNS.

Previously we determined that the ksh93-based shell behaves poorly if /usr/xpg4/bin is in the path.  So I removed it from my path.

My path setting is /usr/local/bin:/opt/DTT/bin:/opt/local/bin:/opt/local/sbin:/usr/gnu/bin:/usr/bin:/usr/sbin:/usr/ucb

The compiler used comes from /opt/local/bin/gcc

Bob Friesenhahn <bfriesen>
Wed 23 Dec 2020 07:31:22 PM UTC, comment #2: 

Thanks for the testsuite directory attachment.  The problem is clearer now:


# Set up the scripts for CONFIG_HEADERS section.
# No need to generate them if there are no CONFIG_HEADERS.
# This happens for instance with `./config.status Makefile'.
if test -n "$CONFIG_HEADERS"; then
cat >"$ac_tmp/defines.awk" <<\_ACAWK ||
BEGIN {
D["PACKAGE_NAME"]=" \"\""
...
}
{ print }
_ACAWK  as_fn_error $? "could not setup config headers machinery" "$LINENO" 5
fi # test -n "$CONFIG_HEADERS"


In a correctly generated config.status, the `as_fn_error` would not be on the same line as `_ACAWK`.  However, the code that generates this part of config.status hasn't changed in years either, and the corresponding piece of the configure script in this test is exactly as it ought to be:


{ print }
_ACAWK
_ACEOF
cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
  as_fn_error $? "could not setup config headers machinery" "$LINENO" 5
fi # test -n "$CONFIG_HEADERS"


So I'm still suspicious of some kind of intermittent fault in the shell executing this configure script.

> As a reminder, you have login access to this system.


I seem to have lost the message in which you told me the name of this system, could you tell me what it is again?

Zack Weinberg <zackw>
Group administrator
Wed 23 Dec 2020 06:58:34 PM UTC, comment #1: 

This is bizarre; nothing has changed in a very long time that would plausibly cause this failure.  AC_TYPE_INTMAX_T hasn't been changed at all since it was added, the core logic to check for missing types hasn't been changed since 2008, and the logic to check for `long long' hasn't been changed since 2012.  And AC_TYPE_INTMAX_T is almost exactly the same as AC_TYPE_UINTMAX_T, but the test for that didn't fail.

Could you please try running `make check TESTSUITEFLAGS=537` over and over again, at least ten times, and report whether it fails every single time?  Could you please also tar up the contents of tests/testsuite.dir/537, from a run where it failed, and attach that to this bug report?

Zack Weinberg <zackw>
Group administrator
Wed 16 Dec 2020 08:40:22 PM UTC, original submission:  

While running the autoconf-2.70 test suite under OmniOS I encountered a new failure that I have not seen before.  This failure did not occur with at least some of the release candidates.

A testsuite.log file is attached.

Bob Friesenhahn <bfriesen>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #51026:  sh-bug.tar.gz added by haible (363KiB - application/gzip)
file #50563:  t-001fa2.sh.gz added by zackw (177B - application/gzip)
file #50564:  test-gen.py added by zackw (1KiB - text/x-python)
file #50557:  autoconf-2.70-test537.tar.gz added by bfriesen (906KiB - application/gzip - I have attached the test 537 content. As a reminder, you have login access to this system.)
file #50496:  testsuite.log.gz added by bfriesen (19KiB - application/gzip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by zackw
  • -email is unavailable- added by zackw
  • -email is unavailable- added by zackw
  • -email is unavailable- added by zackw (Posted a comment)
  • -email is unavailable- added by bfriesen (Submitted the item)
  • -email is unavailable- added by bfriesen
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-03-07 haible Attached File- Added sh-bug.tar.gz, #51026
    2021-01-05 zackw StatusNeed Info Not Autoconf
        Open/ClosedOpen Closed
    2020-12-24 zackw Summaryautoconf-2.70 AC_TYPE_INTMAX_T test failure under OmniOS autoconf-2.70 trips bug in here-doc handling in OmniOS /bin/sh
    2020-12-24 zackw Attached File- Added t-001fa2.sh.gz, #50563
        Attached File- Added test-gen.py, #50564
        Carbon-Copy- Added eggert
        Carbon-Copy- Added ericb
        Carbon-Copy- Added haible
    2020-12-23 bfriesen Attached File- Added autoconf-2.70-test537.tar.gz, #50557
    2020-12-23 zackw StatusNone Need Info
    2020-12-16 bfriesen Attached File- Added testsuite.log.gz, #50496
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code