bugGNU Wget - Bugs: bug #20863, Encode URL to UTF-8 before escaping


bug #20863: Encode URL to UTF-8 before escaping

Submitter:  None
Submitted:  Wed 22 Aug 2007 01:49:12 AM UTC
Category:  Feature Request Severity:  3 - Normal
Priority:  5 - Normal Status:  Duplicate
Privacy:  Public Assigned to:  None
Originator Name:  KJK::Hyperion Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  1.8.2
Operating System:  Microsoft Windows Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  No Work Required:  None
Patch Included:  No
* Mandatory Fields

Add a New Comment (Rich Markup)

Comment Type & Canned Response:


Thu 04 Oct 2007 09:58:36 PM UTC, comment #5: 

(Forgot to mention the target this bug is marked duplicate of: bug 20422.)

Micah Cowan <micahcowan>
Fri 24 Aug 2007 08:00:20 PM UTC, comment #4: 

I interpret --restrict-file-names as a hint to wget about how to "do the right thing" with naming files on my file system. Could we define rules for non-ASCII characters for Unix, Windows, etc. and apply them based on the setting of --restrict-file-names?

I'm willing to bet that will work for 99% of wget users. The other 1% can work on an enhancement to give them more fine-grained control over file names. :-)

Tony Lewis <tdlewis77>
Thu 23 Aug 2007 11:28:51 PM UTC, comment #3: 

> Wouldn't --restrict-file-names work for that purpose?

That lets you specify what characters (or, probably more accurately, what bytes) are allowed in a file name; it doesn't let you specify how those characters should be encoded.

Suppose that a file fetched through recursive mode has some non-ASCII characters in it (such as "résumé.html"). Say that web file's name is encoded using Unicode, but the user's filesystem or terminal environment uses ISO-8859-1. The user isn't going to want wget to do what it currently does, which is just to convert the percent encoding right back to literal byte values (producing "résumé.pdf" in latin1); he's going to want it to be encoded to the user's preferred encoding (or the one supported by his platform, etc).

Micah Cowan <micahcowan>
Thu 23 Aug 2007 10:53:47 PM UTC, comment #2: 

> As to translating it back for local filenames, the right way to do that isn't obvious to me, as there's no real way to find out what encoding your filesystem uses to store files.

Wouldn't --restrict-file-names work for that purpose?

Tony Lewis <tdlewis77>
Wed 22 Aug 2007 04:49:07 AM UTC, comment #1: 

Actually, what you say is not generally true. How wget interprets a URI with non-ASCII characters depends entirely on the bytes it actually sees on the commandline. If you are using a UTF-8 terminal, wget will see UTF-8-encoded text. If you are using a terminal set to use ISO-8859-1, that's what the encoding will be that you send to wget. If you paste UTF-8-encoded text into your ISO-8859-1 terminal, you'll see DNS² (DNS%C2%B2 in ISO-8859-1 encoding), but wget will still just see the bytes for DNA%C2%B2 and ask for that.

For instance, when I copy-paste your DNA² link into my gnome-terminal, which is using UTF-8, wget attempts to get http://anime.plan9.de/anime/manga/DNA%C2%B2/, contrary to your description. This is because those are the bytes it sees.

This is in keeping with historical practice. Note that it's actually never been legal for URIs to contain non-ASCII characters; anything outside that range is supposed to be percent-encoded.

There is now a proposed standard for something called IRIs, or Internationalized Resource Identifiers, in which Unicode characters are allowed. They are basically URIs that have been expanded to allow non-ASCII, Unicode characters.

Under that specification, it might be considered appropriate for wget to translate from whatever the current locale is, as specified by the appropriate environment variables, to UTF-8. As to translating it back for local filenames, the right way to do that isn't obvious to me, as there's no real way to find out what encoding your filesystem uses to store files. I suppose using the locale settings is probably a good guess; but providing some option to specify the output encoding might be a good idea.

There's already a feature request filed for wget to respect IRIs (I generalized it from a request having to do specifically with internationalized domain names), so I'll mark this one as a duplicate of that, bug 20422.

In the meantime, if you want wget to interpret its argument as UTF-8 data, you should give the argument as UTF-8 data, or else as a valid URL, complete with percent-encoding.

Micah Cowan <micahcowan>
Wed 22 Aug 2007 01:49:12 AM UTC, original submission:  

Some servers require the URL to be encoded to UTF-8 before escaping: in other words, the unescaped form must be the binary representation of the UTF-8 encoding of the URL.

For example: try the URL "http://anime.plan9.de/anime/manga/DNA²/". wget will currently encode it as "http://anime.plan9.de/anime/manga/DNA%B2/" (and get a 404 error back), while browsers encode it to UTF-8 first, sending a request for "http://anime.plan9.de/anime/manga/DNA%C2%B2/"

wget needs a new option (--utf8-url, boolean?), maybe even on by default (like it is for browsers), that forces all URLs to be encoded to UTF-8 before being sent to the server, and decoded from UTF-8 in the opposite direction (for example, to generate filenames)



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by tdlewis77 (Posted a comment)
  • -email is unavailable- added by micahcowan (Posted a comment)




    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2007-08-22 micahcowan StatusNone Duplicate
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.9