bugGNU Wget - Bugs: bug #37564, Problems converting UTF-8 urls...


bug #37564: Problems converting UTF-8 urls into filenames

Submitted by:  Platonides <platonides>
Submitted on:  Sat Oct 13 15:49:40 2012  
Category: NoneSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: NoneOperating System: None
Reproducibility: Every TimeFixed Release: None
Planned Release: NoneRegression: None
Work Required: NonePatch Included: None

Add a New Comment (Rich MarkupRich Markup):

You are not logged in

Please log in, so followups can be emailed to you.


Tue Oct 29 20:14:49 2013, comment #4:

As also stated in the original message “that would allow problematic control characters [such as] %01 or %09”.

> I can't help Windows users, but Wget is a power-user tool.
> And a Windows power-user should be able to start a virtual
> machine with Linux running to use tools like Wget.

Stunning comment.

Platonides <platonides>
Sun Oct 27 20:12:06 2013, comment #3:

You could make --restrict-file-names=nocontrol the default by putting it into /etc/wgetrc or ~/wgetrc:

restrictfilenames = nocontrol

I can't help Windows users, but Wget is a power-user tool. And a Windows power-user should be able to start a virtual machine with Linux running to use tools like Wget.

Tim Ruehsen <rockdaboot>
Project Administrator
Sun Oct 27 19:29:16 2013, comment #2:

The expected behavior is that the filename ends up being "Памятник затопленным кораблям в Севастополе.JPG", without needing --restrict-file-names=nocontrol (which was already mentioned as a workaround in the original report).
Plus Windows users are hopeless.

So it's still broken.

Platonides <platonides>
Fri Sep 13 08:28:23 2013, comment #1:

It works as expected with wget 1.13.4 (Debian Wheezy) and wget 1.14 (Debian unstable). You must have a proper UTF-8 environment and must use --restrict-file-names=nocontrol.

On a Linux you can check your environment with
set|egrep 'LANG|LC_'
and your wget version with
wget --version

If in doubt, please post the output of the two commands above.

Tim Ruehsen <rockdaboot>
Project Administrator
Sat Oct 13 15:49:40 2012, original submission:

wget does not deal correctly with utf-8 urls when choosing the final filenames.

For instance if you download the cyrillic filename
it writes
instead of "Памятник затопленным кораблям в Севастополе.JPG"

If the url was ISO-8859-1 then the urlencoded characters would be control characters, but in utf-8 they are proper characters (by leaving them urlencoded, then the other bytes forming the same codepoint are "broken" and show up as gobbledegook).

If you are using a unix-like OS where the filesystem interface uses utf-8, there is a workaround of using --restrict-file-names=nocontrol (which is still too big, as that would allow problematic control characters %01 or %09).

If you are using Windows, --restrict-file-names=nocontrol still gives garbage (the utf-8 characters are treated as if they were in latin1).

Downstream bug: https://bugzilla.wikimedia.org/show_bug.cgi?id=40844

Platonides <platonides>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by rockdaboot (Posted a comment)
  • -unavailable- added by platonides (Submitted the item)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    No Changes Have Been Made to This Item

    Back to the top

    Powered by Savane 3.1-cleanup