Wed 22 Aug 2007 04:49:07 AM UTC, comment #1:
Actually, what you say is not generally true. How wget interprets a URI with non-ASCII characters depends entirely on the bytes it actually sees on the commandline. If you are using a UTF-8 terminal, wget will see UTF-8-encoded text. If you are using a terminal set to use ISO-8859-1, that's what the encoding will be that you send to wget. If you paste UTF-8-encoded text into your ISO-8859-1 terminal, you'll see DNS² (DNS%C2%B2 in ISO-8859-1 encoding), but wget will still just see the bytes for DNA%C2%B2 and ask for that.
For instance, when I copy-paste your DNA² link into my gnome-terminal, which is using UTF-8, wget attempts to get http://anime.plan9.de/anime/manga/DNA%C2%B2/, contrary to your description. This is because those are the bytes it sees.
This is in keeping with historical practice. Note that it's actually never been legal for URIs to contain non-ASCII characters; anything outside that range is supposed to be percent-encoded.
There is now a proposed standard for something called IRIs, or Internationalized Resource Identifiers, in which Unicode characters are allowed. They are basically URIs that have been expanded to allow non-ASCII, Unicode characters.
Under that specification, it might be considered appropriate for wget to translate from whatever the current locale is, as specified by the appropriate environment variables, to UTF-8. As to translating it back for local filenames, the right way to do that isn't obvious to me, as there's no real way to find out what encoding your filesystem uses to store files. I suppose using the locale settings is probably a good guess; but providing some option to specify the output encoding might be a good idea.
There's already a feature request filed for wget to respect IRIs (I generalized it from a request having to do specifically with internationalized domain names), so I'll mark this one as a duplicate of that, bug 20422.
In the meantime, if you want wget to interpret its argument as UTF-8 data, you should give the argument as UTF-8 data, or else as a valid URL, complete with percent-encoding.
|