helpThe GNU Bourne-Again SHell - Support: sr #108491, Is `set -o histexpand` meant to...

 
 

sr #108491: Is `set -o histexpand` meant to trump `set -o posix`? [eg, echo "#!/"] (Because it does.)

Submitted by:  None
Submitted on:  Sat 01 Feb 2014 01:32:34 PM UTC  
 
Category: NonePriority: 5 - Normal
Severity: 5 - BlockerStatus: None
Privacy: PublicAssigned to: None
Originator Email: -unavailable-Open/Closed: Open
Operating System: *BSD

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 03 Feb 2014 07:46:49 AM UTC, comment #1:

Got it. Couple of things:

  • To be super clear, I wasn't the one to ask that first question; the obvious answer

to that is 'its improperly quoted'.

  • I'm not taking any issue with the xpg_echo business. POSIX definitely munged

echo up, and that isn't bash's fault. Moreover, in other shells, printf
builtins are very frequently botched and insecure too, so myself I usually just:

and then use print "..." . Slow, but at least its predictable.

  • I asked the second question (and sent in this bug) because history expansion

in POSIX mode immediately smacked of a bug to me, even prior to revisiting
the docs/specs for this issue. And it seems from what you've told me, I'm not
the first to raise this issue.

Because we're not talking about an error, but a choice in implementation, its only a 'bug' if you say it is. You say its not, so its not, and thats fully
within the prerogative of any developer I think. But if I might note:

> It's a compromise between one of the oldest bash features -- history expansion

predates posix mode by a lot of years -- and the desire to provide users a way
to get a strictly-conforming Posix environment if they really want one.

Thats slightly inaccurate. It predates POSIX by a number of years, not
just posix mode ! Neither is it really one of the oldest bash features. It's
one of the oldest csh features. It predates bash by some ten years I think,
probably straight out of Bill Joy's dorm room circa '78.

The error message is a dead giveaway here that that not really 'bash':

From (en.wikipedia.org/wiki/C_shell#Criticism):

'The implementation is also criticized for its notoriously poor error
messages, e.g., "0 event not found", which yields no useful information
about the problem.'

And so I think its fair to say we're basically now revisiting the csh v. ksh
debate that in large part was the very reason IEEE 1003.2 was necessary in the
first place. POSIX went with ksh, and on this issue in particular
it is in no way ambiguous.

The "Shell Command Language" spec says that ! is a reserved word, and spells out
precisely when and where it is to be so.

Moreover, it explicitly and repeatedly calls out bang history expansion:

Page 2997, 'nl':

"The logical page delimiter is "!+". Note that the ’!’ has to
be escaped when using csh as a command interpreter because of its history
substitution syntax. For ksh and sh the escape is not necessary, but does
not do any harm."

Page 3316, 'uucp':

"Note that the ’!’ character in addresses has to be escaped
when using csh as a command interpreter because of its history substitution
syntax. For ksh and sh the escape is not necessary, but may be used."

So given that, I would venture to say this:

if RC2 were released today as GM, it might not be shipping with a bug in
bash, but it would have several serious inaccuracies in the documentation.

Particularly, the phraseology of, for example:

--posix

Change the behavior of Bash where the default operation differs from
the POSIX standard to match the standard. This is intended to make Bash
behave as a strict superset of that standard.'

Should have 'behavior' become 'some behaviors', and have 'intended' and
'strict' removed, since this is not your intention (you've just told me
as much, no?) and it's therefore an improper superset, and a close subset, but not a strict superset.


If that seems unfair, it's not meant to be. I understand couching compliance
terminology in verbiage like that in the context of 'to our knowledge'. But I'm
not really sure I get the concept of a 'compromise' in this case. If there is
an intentional deviation, as I understand it, that means in the most literal
sense, 'non-compliant'. I think its kind of black and white...

> Bash-4.3 has been in testing for almost a year (4.3-alpha came out in March,

2013), and public test versions have been available for six months.

That was my point about the git repo! I definitely would have brought this up
earlier, but since I wasn't subscribed to the mailing list, I never knew that
until now, despite pulling that repo daily.

> that's going to have to wait for a future version


Well... that's a bummer. Version, not patch, correct? Given that bash seems to
get releases approximately every two years, does this means to look for
this maybe sometime in 2016?

> I would rather just disable it entirely by default.


I would rather you did too!

I suppose until then, in cases where (for example) one drops to an interactive
shell from another from another language and can't know which shell will be
invoked for this, a zsh-style "compatibility mode command" like:

[ x"$BASH_VERSION != x"" ] && set +o histexpand

...might work?

Only the highest regard and best of intentions,

Geoff Nixon
-unavailable-


On 2/1/14, 7:35 PM, Geoff Nixon wrote:

> So having done some more playing around with this, I've found that when
> bash is configured and compiled with '--enable-strict-posix-default', this
> doesn't occur. I see this is documented in the change log for 4.3alpha, as:
>
> k. When compiled for strict Posix conformance, history expansion is
> disabled by default.
>
> But why is this a compile-time-only behavior?


It's a compromise between one of the oldest bash features -- history
expansion predates posix mode by a lot of years -- and the desire to
provide users a way to get a strictly-conforming Posix environment if they
really want one.

I got several requests to disable it entirely by default in bash-4.3, but
that's going to have to wait for a future version.

> There's nothing else in the documentation about any differences between
> bash compiled with -DSTRICT_POSIX v.s. enabling this mode with `bash
> --posix`. It just says that POSIX mode will become a default option.


It doesn't say that exactly. What the configure help string says is that
it will force bash to be conformant by default. Even when you enable
Posix mode, the behavior controlled by xpg_echo and histexpand has to be
tweaked to get a really strictly conforming environment. Since bash enters
Posix mode when it's called as `sh', I judged it would break too many
scripts to do away with `echo -n' and too many interactive user
expectations to disable history expansion. So, posix mode and history
expansion (and echo behavior) are decoupled.

Geoff Nixon <gsnixon>
Sat 01 Feb 2014 01:32:34 PM UTC, original submission:

From: -unavailable-
To: chet@po.cwru.edu, chet.ramey@case.edu, chet@cwru.edu, bash-testers@case.edu, -unavailable-
Subject: Is `set -o histexpand` meant to trump `set -o posix`? [eg, echo "#!/"] (It does.)

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: darwin13.1.0
Compiler: /usr/bin/clang
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='darwin13.1.0' -DCONF_MACHTYPE='x86_64-apple-darwin13.1.0' -DCONF_VENDOR='apple' -DLOCALEDIR='/Users/Shared/usr/local/opt/bash/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -DMACOSX -DDEBUG -DMALLOC_DEBUG -I. -I. -I./include -I./lib -g -O2
uname output: Darwin air.local 13.1.0 Darwin Kernel Version 13.1.0: Thu Jan 16 19:40:37 PST 2014; root:xnu-2422.90.20~2/RELEASE_X86_64 x86_64
Machine Type: x86_64-apple-darwin13.1.0

Bash Version: 4.3
Patch Level: 0
Release Status: rc2

Description:

Please see these:

http://stackoverflow.com/questions/21494213/why-cant-i-echo-the-characters-in-a-string-in-bash
http://stackoverflow.com/questions/21496049/what-is-the-intended-effective-ordering-of-set-o-options-in-bash-does-histe

Its still not at all clear to me if I'm correct or not; and I kind of got a serious beatdown for calling it a bug on Stack Overflow. So I'm not sure if this is a 'blocker' per se or even something you'll consider a bug. But it certainly blocks me from wanting to using bash to execute a POSIX-portable script!

(Note that I now fully realize my statement there to the effect of 'bash never sees any updates so use a different shell' is erroneous. I made this claim because a keep a git checkout on my hard drive which gets pulled daily - but your master branch is configured to track 4.2 rather than devel, so that's what I thought was indeed the case. I was completely aghast to find that 4.3 is in rc2 having just made that statement, and I intend to redact it a soon as there's some resolution to this "bug", be it a bug or not.)

I apologize if this appears in your inbox two dozen times. I don't know if sendmail even works at all on my system; and there are three slightly different emails in the bashbug script, in git, and the top hit in a web search for your name. I'm trying to make sure this gets seen before the 'release' button gets pushed for 4.3.

Repeat-By:
set -o histexpand && set -o posix && echo "#!/"
set -o histexpand && set -o posix && echo "#!/bin/bash"
echo $?

Fix:
Disable `set -o histexpand` while parsing and expanding double-quoted strings, at least for lines which begin with a hash symbol, at least when `set -o posix` is set, and possibly always?

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gsnixon (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    No Changes Have Been Made to This Item

    Back to the top


    Powered by Savane 3.1-cleanup