mainThe GNU Bourne-Again SHell - Support: sr #109629, quoting scrambled within functions...

 
 

sr #109629: quoting scrambled within functions and variables

Submitter:  None
Submitted:  Fri 18 Jan 2019 07:01:52 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  4 - Important Status:  Invalid
Privacy:  Public Assigned to:  None
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 20 Jun 2019 06:29:50 AM UTC, comment #10: 

hmmm... not what i get with my current (older) version 4.4.19(1)-release (x86_64-slackware-linux-gnu) :
$ bash /tmp/t.sh
+ a='$foo'
+ b='$bar'
++ echo '$foo'
++ sed 's/\$/\\$/g'
+ a='\$foo'
+ echo 'whatever\$foo-$bar/'
whatever\$foo-$bar/
+ echo 'whatever\$foo-$bar'
whatever\$foo-$bar

which is the expected behaviour you get with your dash...

Zui
201906204

Anonymous
Wed 19 Jun 2019 08:04:27 PM UTC, comment #9: 

Not exactly, that's an issue with backslash quoting in glob patterns. Coincidentally, there's a discussion about this exact thing taking place on the Posix mailing list. This one I can fix.

Chet Ramey <chet>
Group administrator
Tue 18 Jun 2019 02:57:05 PM UTC, comment #8: 

Similar case i think:

$ cat > /tmp/t.sh <<'EOF'
#!/bin/bash

set -x

a='$foo'
b='$bar'
a=$(echo "$a" | sed 's/\$/\\$/g')
echo 'whatever'$a'-'$b'/'
echo 'whatever'$a'-'$b
EOF

$ bash /tmp/t.sh
+ a='$foo'
+ b='$bar'
++ echo '$foo'
++ sed 's/\$/\\$/g'
+ a='\$foo'
+ echo '\w\h\a\t\e\v\e\r\$foo\-$bar/'
\w\h\a\t\e\v\e\r\$foo\-$bar/
+ echo 'whatever\$foo-$bar'
whatever\$foo-$bar

$ dash /tmp/t.sh
+ a=$foo
+ b=$bar
+ echo $foo
+ sed s/\$/\\$/g
+ a=\$foo
+ echo whatever\$foo-$bar/
whatever\$foo-$bar/
+ echo whatever\$foo-$bar
whatever\$foo-$bar

$ bash --version | sed 1q
GNU bash, version 5.0.7(1)-release (x86_64-pc-linux-gnu)

$ uname -a
Linux saturn 5.1.4-arch1-1-ARCH #1 SMP PREEMPT Wed May 22 08:06:56 UTC 2019 x86_64 GNU/Linux

Anonymous
Tue 12 Feb 2019 11:54:00 AM UTC, comment #7: 

Wooow, finally over.
Here's the result that took me so long.


    #!/bin/sh
    set -x

    # <content of $1>, as in my example, could be sthg as awfull as : blah/blah ( bl$ah [bl^a.h) \][; bl:ah.0/0)
    # and, yes, things look/are already messy, unpaired, weird chars, etc... who cares, let them be, keep them safe and warm.
    # in fact each use case is users' case and leave it to them
    OptionValue=$1
    OptionName='--myOption'
    OptionExpr="${OptionName}=$(echo ${OptionValue} |
                sed 's/[ ()\/\.:;|^$\\&]/\\&/g' |
                sed 's/.*/\\\"&\\\"/' )"
    # go ahead and add more chars to be protected within the 1st sed, the more the merrier,
    # you can even escape them all with \ for extra(futil?) safety
    # now the long awaited call (fireworks here!!!)
    eval echo myCmd "${OptionExpr}"

    # yes, i did put echo to display what was about to be executed...
    # the rest is for you to build upon you imagination/situation (what an experience)


and this will hopefully run as expected. Notice the (crucial) presence/absence of ' within the lines :


    + OptionValue='blah/blah ( bl$ah [bl^a.h) \][; bl:ah.0/0)'
    + UsrAgtOpt=--myOption
    ++ sed 's/.*/\\\"&\\\"/'
    ++ sed 's/[ ()\/\.:;|^$\\&]/\\&/g'
    ++ echo blah/blah '(' 'bl$ah' '[bl^a.h)' '\][;' 'bl:ah.0/0)'
    + UsrAgtExpr='--myOption=\"blah\/blah\ \(\ bl\$ah\ [bl\^a\.h\)\ \\][\;\ bl\:ah\.0\/0\)\"'
    + eval echo myCmd '--myOption=\"blah\/blah\ \(\ bl\$ah\ [bl\^a\.h\)\ \\][\;\ bl\:ah\.0\/0\)\"'
    ++ echo myCmd '--myOption="blah/blah ( bl$ah [bl^a.h) \][; bl:ah.0/0)"'
    myCmd --myOption="blah/blah ( bl$ah [bl^a.h) \][; bl:ah.0/0)"


Obviously "too much work" for one value... but what about two or more?
And sure, this can be improved a lot. I guess one could start by
hiding away the assignement of $OptionExpr in a function for reusability.
What are your suggestions?

Zui
201902122

Anonymous
Mon 04 Feb 2019 12:46:46 AM UTC, comment #6: 

this is not a 1st time i meet the beast, just the 1st time to post about it.
The case is very common for mr: composing strings of options to feed a command with.
Of course in the case of a variable(non predetermined) set of options. Let me give a concrete case :
you wanna process multimedia files with ffmpeg(or compress with 7zip, or transfert with scp, or... you get the gist of it).
what i want is prepare the pieces apart and concatenate them all together at the end. in the case of ffmpeg :
if you have subs
aa=<subs string> else aa=""
audio
bb=<audio string> else bb=""
and so on
then you do eval, indeed, for :
ffmpeg $aa $bb ...
and it should just stick all together without messing inside the strings... well, so i hoped and searched.

bah, i always find ways but each time looks like a different one since it behaves almost unpredictably. ex, with () or [] in the strings.

anyway, thank for the care.
Zui
201902041

Anonymous
Sun 03 Feb 2019 10:04:09 PM UTC, comment #5: 


>> So your desired goal is to have the second word be
>> "--options=arg1 arg2 arg3"
>>
>> (this is pretty bad utility design, but here we are.)
> well, dunno about design but no, i don't want
> "--options=arg1 arg2 arg3"
> .   because it would gather the option selector and the args (could be a
> filename with spaces).


Yes, into a single word. There's no way `util' is going to be able to parse its options if they're not single words. What do
you expect `util' to be able to do with that?

> recap of it all:
> .  assignment :
> .    instruct='util --options="arg1 arg2 arg3"'
> .  call :
> .    ${instruct} <more args>
> .  desired (notice the position of the double quotes?):
> .    util --options="arg1 arg2 arg3" <more args>


You want literal double quotes in the second word, but it's still treated as a single word? And `util' is supposed to deal with them?

You can use eval, just add an escaped pair of double quotes to the string.

Chet Ramey <chet>
Group administrator
Sun 03 Feb 2019 08:31:08 PM UTC, comment #4: 


> So your desired goal is to have the second word be
> "--options=arg1 arg2 arg3"
>
> (this is pretty bad utility design, but here we are.)

well, dunno about design but no, i don't want
"--options=arg1 arg2 arg3"
.   because it would gather the option selector and the args (could be a filename with spaces).
recap of it all:
.  assignment :
.    instruct='util --options="arg1 arg2 arg3"'
.  call :
.    ${instruct} <more args>
.  desired (notice the position of the double quotes?):
.    util --options="arg1 arg2 arg3" <more args>
with "set -x" debuging to see what comes out of eval in :
eval $instruct <more args>
  still not  it...
if there is no solution within bash, that OK, i'll suck it up and play with functions...
why so much $รน% in this world :-?

however thank you for your help (just playing with eval taught a thing or two)
Zui
201902037

Anonymous
Sat 02 Feb 2019 07:20:33 PM UTC, comment #3: 


> Thank you for your reply,
> but, hmmmm, how shall i put it: i feel we are not talking about the same
> thing.
> I'm not sure that what i'm trying to do is, in your words, "processing quotes
> in variables".


It is. You want quotes embedded in expanded variable values to have the same special meaning as they would if you had entered them in the command directly. Quotes are processed while parsing, not after word expansion.

> So, allow me to setup an illustrative example.
> I want to be able to generate instructions dynamicaly, according to context
> and all, so
> to make it simple and concrete, here comes the code :
> ----
>
> #my script sample
> #i want to see what is going on
> set -x
>
> # i build part of my instruction earlier in the script
> instruct='util --options="arg1 arg2 arg3"'
>
> # and call it later down the lines
> ${instruct} "more args"
>
> ----
>
> i was expecting the whole command to become :
>
> util --options='arg1 arg2 arg3' "more args"


You have mixed the quotes and misunderstood how quote processing works (and when quote removal takes place).

Logically, this becomes, after expansion (bounding double quotes to show individual expanded words), as you discovered:

"util" "--options="arg1" "arg2" "arg3"" "more args"

It would be clearer if you did something like

instruct='util --options="arg1 arg2 arg3"'
printf "<%s> " ${instruct} "more args"
echo

to show that word splitting still takes place but the double quotes resulting from the expansion don't have any special meaning:

<util> <--options="arg1> <arg2> <arg3"> <more args>

but you are correct in how it was expanded. This is expected behavior.


> so i tried swapping simple and double quotes, then escaping inner quoting in
> both
> cases but not getting any closer to my desired goal.

So your desired goal is to have the second word be
"--options=arg1 arg2 arg3"

(this is pretty bad utility design, but here we are.)

> Would 'eval' play a onforting role in here?


If you want quotes in the expanded value of a variable to act as they would if they were entered in the command directly, yes.

> If so, how should i fill my variable and use it?


If you want "more args" to be word split, just use eval in front of the command:

eval ${instruct} "more args"

If you don't want word splitting on "more args", you need to add a second set of quotes for it:

eval ${instruct} "\"more args\""

Chet Ramey <chet>
Group administrator
Sat 02 Feb 2019 04:15:56 PM UTC, comment #2: 

Thank you for your reply,
but, hmmmm, how shall i put it: i feel we are not talking about the same thing.
I'm not sure that what i'm trying to do is, in your words, "processing quotes in variables".
So, allow me to setup an illustrative example.
I want to be able to generate instructions dynamicaly, according to context and all, so
to make it simple and concrete, here comes the code :




#my script sample
#i want to see what is going on
set -x

# i build part of my instruction earlier in the script
instruct='util --options="arg1 arg2 arg3"'

# and call it later down the lines
${instruct} "more args"




i was expecting the whole command to become :

util --options='arg1 arg2 arg3' "more args"


but instead it became :

util '--options="arg1' 'arg2' 'arg3" "more args"


so i tried swapping simple and double quotes, then escaping inner quoting in both
cases but not getting any closer to my desired goal.

Would 'eval' play a onforting role in here?
If so, how should i fill my variable and use it?
Otherwise, any idea on how to make this quoting work to get the expected result?

thank you so much for your kind help.

Zui
201902026

Anonymous
Fri 01 Feb 2019 03:21:40 PM UTC, comment #1: 

If you want quotes in variables to be processed, you have to use `eval'.

Chet Ramey <chet>
Group administrator
Fri 18 Jan 2019 07:01:52 PM UTC, original submission:  

since xidel doesn't provide a conf file (rather i did not find mention of it in docs) to gather
 some default options such as UA, silent mode...
I buried it all in either a function or a variable for reusability.
I did not put it in a distinct file for speed reasons;

Also, it could have been any other utility, as you can see with the debug mode set to 'on', below.

Here is the scenario with the error msgs (indentation added for readability).
Though the processed "page" file does not matter, here a sample :
    <html>
        <head>
            <title>201901175 : xidel+bash</title>
        </head>
        <body>
            <div class="c1">
                <a href="link1.1">lien 1/1</a><br>
            </div>
            <div class="c2">
                <a href="link2.1">lien 1/2</a><br>
                <a href="link2.2">lien 2/2</a><br>
            </div>
            <div class="c3">
                <a href="link3.1">lien 1/3</a><br>
            </div>
        </body>
    </html>

And the script to reproduce the resulting errors on quoted strings:
    #!/bin/bash
        echo linux version : $(uname -a)
        echo bash version  : $(bash --version)
        echo xidel version : $(xidel --version)

        # setting debug ON
        set -x

    echo calling xidel directly
        xidel --user-agent 'my UA' page -e 'css(".c2 a")/@href'

    echo calling xidel in a variable with external double-quotes
        xidv="xidel --user-agent=\"myUA\""
        $xidv page -e 'css(".c2 a")/@href'

    echo calling xidel in a variable with external simple-quotes
        xidv='xidel --user-agent="myUA"'
        $xidv page -e 'css(".c2 a")/@href'

    echo calling xidel in a function
        xidf () {
          xidel --user-agent "myUA" $*
        }
        xidf page -e 'css(".c2 a")/@href'

running it will display the following :
=====================================================================================
    + . ./x.sh.1
    +++ uname -a
    ++ echo linux version : Linux porteus 4.9.0-porteus '#1' SMP PREEMPT Sat Dec 17 19:22:10 BRST 2016 x86_64...
    linux version : Linux porteus 4.9.0-porteus #1 SMP PREEMPT Sat Dec 17 19:22:10 BRST 2016 x86_64...
    +++ bash --version
    ++ echo bash version : GNU bash, version '4.3.46(1)-release' ...
    bash version : GNU bash, version 4.3.46(1)-release (x86_64-slackware-linux-gnu) Copyright (C) 2013 Free Software Foundation, Inc.... +++ xidel --version
    ++ echo xidel version : Xidel 0.9.8 '(20180421.6162.1f357eaaf5f3)' http://www.videlibri.de/xidel.html by '<benito@benibela.de>'...
    xidel version : Xidel 0.9.8 (20180421.6162.1f357eaaf5f3) http://www.videlibri.de/xidel.html by <benito@benibela.de> ...
    ++ set -x
    ++ echo calling xidel directly
    calling xidel directly
    ++ xidel --user-agent 'my UA' page -e 'css(".c2 a")/@href'
    ** Retrieving: page **
    ** Processing: page **
    link2.1
    link2.2
    link2.3
    ++ echo calling xidel in a variable with external double-quotes
    calling xidel in a variable with external double-quotes
    ++ xidv='xidel --user-agent="myUA"'
    ++ xidel '--user-agent="myUA"' page -e 'css(".c2 a")/@href'
    ** Retrieving: page **
    ** Processing: page **
    link2.1
    link2.2
    link2.3
    ++ echo calling xidel in a variable with external simple-quotes
    calling xidel in a variable with external simple-quotes
    ++ xidv='xidel --user-agent="myUA"'
    ++ xidel '--user-agent="myUA"' page -e 'css(".c2 a")/@href'
    ** Retrieving: page **
    ** Processing: page **
    link2.1
    link2.2
    link2.3
    ++ echo calling xidel in a function
    calling xidel in a function
    ++ xidf page -e 'css(".c2 a")/@href'
    ++ xidel --user-agent myUA page -e 'css(".c2' 'a")/@href'
    ** Retrieving: page **
    ** Processing: page **
    Error:
    err:XPST0003: Unclosed string
    in: css(".c [<- error occurs before here] 2
=====================================================================================

I tried many quoting "strategies" with no success : any hint on how to get Bash to NOT over-quote and change the "semantic"-grouping?

thanks in advance for your help.
and, of course, if i can provide some testing to help (testing an stuff)...

Zui
201901175

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by chet (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-02-01 chet StatusNone Invalid

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code