mainThe GNU Bourne-Again SHell - Support: sr #108491, Is `set -o histexpand` meant to...

 
 

sr #108491: Is `set -o histexpand` meant to trump `set -o posix`? [eg, echo "#!/"] (Because it does.)

Submitter:  None
Submitted:  Sat 01 Feb 2014 01:32:34 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  2 - Minor Status:  Done
Privacy:  Public Assigned to:  None
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  *BSD
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 17 Jan 2023 09:50:16 PM UTC, comment #4: 

This was changed in bash-4.4: posix mode allows double quotes to quote the history expansion character.

Chet Ramey <chet>
Group administrator
Fri 13 Nov 2015 09:04:31 PM UTC, comment #3: 

The thing is, there are plenty of things that Posix doesn't mention to which bash attaches meaning.  For instance, Posix has nothing to say about brace expansion or some of the word expansions bash implements.  All shells do this: ksh93, for instance, allows variable names containing a `.' when Posix clearly states that valid shell identifiers cannot contain a dot.

Maybe it's enough to make one specific change: double quotes can quote the history expansion character when the shell is in Posix mode, since Posix specifies the effect of double-quoting a string.

Since the intent of posix mode is that scripts written to Posix run unmodified in bash, not that bash turn off every feature that Posix doesn't specify, maybe this will work.

Bash-5.0 will disable history expansion by default in any case.

Chet Ramey <chet>
Group administrator
Wed 25 Mar 2015 11:06:50 AM UTC, comment #2: 

Ok, this is not a bug.

The work around is to disable history expansion (set +H), also, this doesn't affect scripts at all (history expansion is only enabled for interactive shells).

Eduardo Bustamante <dualbus>
Mon 03 Feb 2014 07:46:49 AM UTC, comment #1: 

Got it. Couple of things:

  • To be super clear, I wasn't the one to ask that first question; the obvious answer

   to that is 'its improperly quoted'.

  • I'm not taking any issue with the xpg_echo business. POSIX definitely munged

   echo up, and that isn't bash's fault. Moreover, in other shells, printf
   builtins are very frequently botched and insecure too, so myself I usually just:
 

   printf(){ command printf '%s' "$@"; }; print(){ printf '%s' "$@\n";}


   and then use print "..." . Slow, but at least its predictable.

  • I asked the second question (and sent in this bug) because history expansion

   in POSIX mode immediately smacked of a bug to me, even prior to revisiting
   the docs/specs for this issue. And it seems from what you've told me, I'm not
   the first to raise this issue.


Because we're not talking about an error, but a choice in implementation, its only a 'bug' if you say it is. You say its not, so its not, and thats fully
within the prerogative of any developer I think. But if I might note:

> It's a compromise between one of the oldest bash features -- history expansion

  predates posix mode by a lot of years -- and the desire to provide users a way
  to get a strictly-conforming Posix environment if they really want one.

Thats slightly inaccurate. It predates POSIX by a number of years, not
just posix mode ! Neither is it really one of the oldest bash features. It's
one of the oldest csh features. It predates bash by some ten years I think,
probably straight out of Bill Joy's dorm room circa '78.

The error message is a dead giveaway here that that not really 'bash':

From (en.wikipedia.org/wiki/C_shell#Criticism):

  'The implementation is also criticized for its notoriously poor error
  messages, e.g., "0 event not found", which yields no useful information
  about the problem.'

And so I think its fair to say we're basically now revisiting the csh v. ksh
debate that in large part was the very reason IEEE 1003.2 was necessary in the
first place. POSIX went with ksh, and on this issue in particular
it is in no way ambiguous.

The "Shell Command Language" spec says that ! is a reserved word, and spells out
precisely when and where it is to be so.

Moreover, it explicitly and repeatedly calls out bang history expansion:

Page 2997, 'nl':

  "The logical page delimiter is "!+". Note that the ’!’ has to
  be escaped when using csh as a command interpreter because of its history
  substitution syntax. For ksh and sh the escape is not necessary, but does
  not do any harm."

Page 3316, 'uucp':

  "Note that the ’!’ character in addresses has to be escaped
  when using csh as a command interpreter because of its history substitution
  syntax. For ksh and sh the escape is not necessary, but may be used."

So given that, I would venture to say this:

if RC2 were released today as GM, it might not be shipping with a bug in
bash, but it would have several serious inaccuracies in the documentation.

Particularly, the phraseology of, for example:

  --posix

      Change the behavior of Bash where the default operation differs from
      the POSIX standard to match the standard. This is intended to make Bash
      behave as a strict superset of that standard.'

Should have 'behavior' become 'some behaviors', and have 'intended' and
'strict' removed, since this is not your intention (you've just told me
as much, no?) and it's therefore an improper superset, and a close subset, but not a strict superset.




If that seems unfair, it's not meant to be. I understand couching compliance
terminology in verbiage like that in the context of 'to our knowledge'. But I'm
not really sure I get the concept of a 'compromise' in this case. If there is
an intentional deviation, as I understand it, that means in the most literal
sense, 'non-compliant'. I think its kind of black and white...

> Bash-4.3 has been in testing for almost a year (4.3-alpha came out in March,

2013), and public test versions have been available for six months.

That was my point about the git repo! I definitely would have brought this up
earlier, but since I wasn't subscribed to the mailing list, I never knew that
until now, despite pulling that repo daily.


> that's going to have to wait for a future version


Well... that's a bummer. Version, not patch, correct? Given that bash seems to
get releases approximately every two years, does this means to look for
this maybe sometime in 2016?


> I would rather just disable it entirely by default.


I would rather you did too!


I suppose until then, in cases where (for example) one drops to an interactive
shell from another from another language and can't know which shell will be
invoked for this, a zsh-style "compatibility mode command" like:

  [ x"$BASH_VERSION != x"" ] && set +o histexpand

...might work?


Only the highest regard and best of intentions,
 
  Geoff Nixon
  -email is unavailable-



On 2/1/14, 7:35 PM, Geoff Nixon wrote:

> So having done some more playing around with this, I've found that when
> bash is configured and compiled with '--enable-strict-posix-default', this
> doesn't occur.  I see this is documented in the change log for 4.3alpha, as:
>
>   k.  When compiled for strict Posix conformance, history expansion is
> disabled by default.
>
> But why is this a compile-time-only behavior?


It's a compromise between one of the oldest bash features -- history
expansion predates posix mode by a lot of years -- and the desire to
provide users a way to get a strictly-conforming Posix environment if they
really want one.

I got several requests to disable it entirely by default in bash-4.3, but
that's going to have to wait for a future version.

> There's nothing else in the documentation about any differences between
> bash compiled with -DSTRICT_POSIX v.s. enabling this mode with `bash
> --posix`. It just says that POSIX mode will become a default option.


It doesn't say that exactly.  What the configure help string says is that
it will force bash to be conformant by default.  Even when you enable
Posix mode, the behavior controlled by xpg_echo and histexpand has to be
tweaked to get a really strictly conforming environment.  Since bash enters
Posix mode when it's called as `sh', I judged it would break too many
scripts to do away with `echo -n' and too many interactive user
expectations to disable history expansion.  So, posix mode and history
expansion (and echo behavior) are decoupled.

Geoff Nixon <gsnixon>
Sat 01 Feb 2014 01:32:34 PM UTC, original submission:  

 From: -email is unavailable-
To:  chet@po.cwru.edu, chet.ramey@case.edu, chet@cwru.edu, bash-testers@case.edu, -email is unavailable-
Subject: Is `set -o histexpand` meant to trump `set -o posix`? [eg, echo "#!/"] (It does.)

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: darwin13.1.0
Compiler: /usr/bin/clang
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='darwin13.1.0' -DCONF_MACHTYPE='x86_64-apple-darwin13.1.0' -DCONF_VENDOR='apple' -DLOCALEDIR='/Users/Shared/usr/local/opt/bash/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -DMACOSX -DDEBUG -DMALLOC_DEBUG -I.  -I. -I./include -I./lib -g -O2
uname output: Darwin air.local 13.1.0 Darwin Kernel Version 13.1.0: Thu Jan 16 19:40:37 PST 2014; root:xnu-2422.90.20~2/RELEASE_X86_64 x86_64
Machine Type: x86_64-apple-darwin13.1.0

Bash Version: 4.3
Patch Level: 0
Release Status: rc2

Description:

Please see these:

 http://stackoverflow.com/questions/21494213/why-cant-i-echo-the-characters-in-a-string-in-bash
http://stackoverflow.com/questions/21496049/what-is-the-intended-effective-ordering-of-set-o-options-in-bash-does-histe

Its still not at all clear to me if I'm correct or not; and I kind of got a serious beatdown for calling it a bug on Stack Overflow. So I'm not sure if this is a 'blocker' per se or even something you'll consider a bug. But it certainly blocks me from wanting to using bash to execute a POSIX-portable script!

(Note that I now fully realize my statement there to the effect of 'bash never sees any updates so use a different shell' is erroneous. I made this claim because a keep a git checkout on my hard drive which gets pulled daily - but your master branch is configured to track 4.2 rather than devel, so that's what I thought was indeed the case. I was completely aghast to find that 4.3 is in rc2 having just made that statement, and I intend to redact it a soon as there's some resolution to this "bug", be it a bug or not.)

I apologize if this appears in your inbox two dozen times. I don't know if sendmail even works at all on my system; and there are three slightly different emails in the bashbug script, in git, and the top hit in a web search for your name. I'm trying to make sure this gets seen before the 'release' button gets pushed for 4.3.

Repeat-By:
set -o histexpand && set -o posix && echo "#!/"
  set -o histexpand && set -o posix && echo "#!/bin/bash"
  echo $?

Fix:
Disable `set -o histexpand` while parsing and expanding double-quoted strings, at least for lines which begin with a hash symbol, at least when `set -o posix` is set, and possibly always?

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by chet (Posted a comment)
  • -email is unavailable- added by dualbus (Posted a comment)
  • -email is unavailable- added by gsnixon (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-01-17 chet StatusIn Progress Done
    2015-11-13 chet Severity5 - Blocker 2 - Minor
        StatusNone In Progress

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code