bugmake - Bugs: bug #35063, Suggest conventional $(%XX) syntax...


bug #35063: Suggest conventional $(%XX) syntax for special characters

Submitter:  David Boyce <boyski>
Submitted:  Wed 14 Dec 2011 02:27:53 AM UTC
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.0 Operating System:  Any
Fixed Release:  None Triage Status:  Medium Effort
* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Sun 29 Jan 2012 10:07:13 AM UTC, comment #7: 

I just suggested a more general solution which covers your issues too.

Eldar Abusalimov <abusalimov>
Sun 29 Jan 2012 03:15:37 AM UTC, comment #6: 

I think the root of our disagreement is in the goals. This is not at all about adding url encode/decode capability to make; it's about a consistent, safe way of getting special characters past the make parser and I suggested url encoding (aka percent-encoding) as a well-known mechanism which is easy to implement. Another encoding would serve just as well.

In particular, there's no call for url encoding support as I see it. If you can make a case that there's a reason to support both then maybe your proposal would be better. But that's not the stated goal here.

Your question about 'Hello%2C%20world%21' is answered in the original proposal; if the variable reference begins with '$(%' then the string is subjected to standard url decoding, which is well defined to produce "Hello, World!".

WRT special functions I think the choices are pretty clear. Given that %3A is a colon:

$(call %3A) returns :
$(value %3A) returns :
$(origin %3A) returns default
$(flavor %3A) returns simple

In essence $(%2C) is defined to behave exactly as if the line

%2C := ,

had been seen (actually not quite, it would be in the list of predefined variables such that --no-builtin-variables would suppress the feature).

BTW see <http://www.cmcrossroads.com/ask-mr-make/8442-gnu-make-escaping-a-walk-on-the-wild-side> for some background on the problem.

David Boyce <boyski>
Sat 28 Jan 2012 08:06:17 PM UTC, comment #5: 

Well, when saying "altering the basic syntax" I mean changing the semantics of '$(...)' notation, which currently stands for expanding a variable reference or invoking a built-in function.

Personally for me it is unclear what '$(%20)' or '$(.%20)' means. If we perceive it just as a variable reference, then what would '$(value %20)' or '$(call %20)' return? I guess, it still should be a space. Also, what's about computed references, like '$(%$(twenty))', or the one I told about in my previous comment? In such case the problem can't be solved only by the tokenizer and requires '%XX' special variables to be defined.

Finally, encoding/decoding multiple characters at once becomes a pain. For example, how do I properly unescape 'Hello%2C%20world%21' string (assuming it is computed somewhere, that is not literal)?

However, I agree, that Paul is to make the final decision.

Eldar Abusalimov <abusalimov>
Sat 28 Jan 2012 06:40:54 PM UTC, comment #4: 

I don't agree with this, and I don't know what you mean about altering the basic syntax. The original proposal makes no change to syntax (although I see that I used the word incorrectly in the subject line), it adds a new range of built-in variables. But Paul will make the final call.

David Boyce <boyski>
Sat 28 Jan 2012 05:39:36 PM UTC, comment #3: 

I would vote up for just adding two new functions, i.e. `url-encode` and `url-decode`, without altering the basic syntax. Both functions are simple enough to implement.

Thus, the suggested usage is the following:

_space := $(url-decode %20)

The argument can be, of course, computed as usual:

_space := $(url-decode %$(words foo bar)0)

Eldar Abusalimov <abusalimov>
Tue 17 Jan 2012 05:05:38 AM UTC, comment #2: 

According to RFC 1738 (http://www.ietf.org/rfc/rfc1738.txt):

... octets may be encoded by a character triplet consisting of the character "%" followed by the two hexadecimal digits (from "0123456789ABCDEF") which forming the hexadecimal value of the octet. (The characters "abcdef" may also be used in hexadecimal encodings.)

Therefore I believe URL encoding aka percent encoding requires all three characters, and thus omitting a leading 0 would be disallowed by definition.

Though the unreserved set seems pretty inert wrt make, I still think this feature should not be limited to the restricted set. Its main value IMO is that while currently there's one trick for getting a space past the makefile parser and another for backslashes and another for colons etc, the percent-encoding scheme is regular and predictable. A user could easily write a script to convert a string to its make-percent-encoded equivalent. Having to consider each character separately to determine whether its a "may", "must", or "must not" would bring back the pain of special casing.

Although makefiles are of course not URLs, the practice of allowing any octet to be encoded has always been tolerated. From RFC 1738 again:

"... characters that are not required to be encoded (including alphanumerics) may be encoded ..."

WRT when the expansion is done, I'm not well enough versed in the intricacies to have an opinion.

David Boyce <boyski>
Mon 16 Jan 2012 08:01:47 PM UTC, comment #1: 

A useful idea.  FMI:


Note we already have a $% variable (for archive members) and their associated $(%D) and $(%F) (which are both legal hex values BTW).

So we will have to decide whether we'd prefer the "uncleanliness of $(.%<token>), or the potential backward-compatibility issue with $(%xx)... as long as we require two digits and don't allow omitting of the leading 0.

Also we should consider whether this encoding should be limited to the strict URL restricted character set, or whether it will extend to any legal UTF-8 character with code <128.

We'd need to decide when the expansion is done.  Is it done using normal expansion rules?  Or is it done very early by the makefile parser?

Paul D. Smith <psmith>
Group administrator
Wed 14 Dec 2011 02:27:53 AM UTC, original submission:  

This idea seems too obvious to not have been brought up before but I haven't seen it and it seems both helpful and easy to implement so here goes. There's a lot of clever trickery around for inserting special characters in make syntax, e.g. the famous

_empty :=
_space := $(_empty) $(_empty)

So why not support a special syntax for "predefined" variables which uses URL encoding? With this feature, a space would be $(%20) and a backslash $(%5C), etc. The "." namespace is already reserved for internal variables, e.g. $(.DEFAULT_GOAL), so the same could be done for the "%" namespace with a low likelihood of breaking anything. Or, to be perfectly safe, it could be placed in the "." namespace as in $(.%20).

Just to be clear, what I'm suggesting here would be a simple token replacement which requires no actual use of the symbol table. Whenever the sequence "$(%XX)" is encountered while doing variable evaluation it's replaced by its character equivalent.

The primary use would be for single characters but I see no reason url-encoded strings couldn't be allowed. As long as the variable string starts with % (or .%), the whole thing is replaced by its url-decoded equivalent.

David Boyce <boyski>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by abusalimov (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by boyski (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.


    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-01-16 psmith Triage StatusNone Medium Effort

    Back to the top

    Powered by Savane 3.13-04b1.
    Corresponding source code