bugGNU Wget - Bugs: bug #37564, Problems converting UTF-8 urls...

 
 

bug #37564: Problems converting UTF-8 urls into filenames

Submitter:  Platonides <platonides>
Submitted:  Sat 13 Oct 2012 03:49:40 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 29 Oct 2013 08:14:49 PM UTC, comment #4: 

As also stated in the original message “that would allow problematic control characters [such as] %01 or %09”.



> I can't help Windows users, but Wget is a power-user tool.
> And a Windows power-user should be able to start a virtual
> machine with Linux running to use tools like Wget.


Stunning comment.

Platonides <platonides>
Sun 27 Oct 2013 08:12:06 PM UTC, comment #3: 

You could make --restrict-file-names=nocontrol the default by putting it into /etc/wgetrc or ~/wgetrc:

restrictfilenames = nocontrol

I can't help Windows users, but Wget is a power-user tool. And a Windows power-user should be able to start a virtual machine with Linux running to use tools like Wget.

Tim Ruehsen <rockdaboot>
Group administrator
Sun 27 Oct 2013 07:29:16 PM UTC, comment #2: 

The expected behavior is that the filename ends up being "Памятник затопленным кораблям в Севастополе.JPG", without needing --restrict-file-names=nocontrol (which was already mentioned as a workaround in the original report).
Plus Windows users are hopeless.

So it's still broken.

Platonides <platonides>
Fri 13 Sep 2013 08:28:23 AM UTC, comment #1: 

It works as expected with wget 1.13.4 (Debian Wheezy) and wget 1.14 (Debian unstable). You must have a proper UTF-8 environment and must use --restrict-file-names=nocontrol.

On a Linux you can check your environment with
  set|egrep 'LANG|LC_'
and your wget version with
  wget --version

If in doubt, please post the output of the two commands above.

Tim Ruehsen <rockdaboot>
Group administrator
Sat 13 Oct 2012 03:49:40 PM UTC, original submission:  

wget does not deal correctly with utf-8 urls when choosing the final filenames.

For instance if you download the cyrillic filename
http://upload.wikimedia.org/wikipedia/commons/a/ad/%D0%9F%D0%B0%D0%BC%D1%8F%D1%82%D0%BD%D0%B8%D0%BA_%D0%B7%D0%B0%D1%82%D0%BE%D0%BF%D0%BB%D0%B5%D0%BD%D0%BD%D1%8B%D0%BC_%D0%BA%D0%BE%D1%80%D0%B0%D0%B1%D0%BB%D1%8F%D0%BC_%D0%B2_%D0%A1%D0%B5%D0%B2%D0%B0%D1%81%D1%82%D0%BE%D0%BF%D0%BE%D0%BB%D0%B5.JPG
it writes
"Р%9FамС%8FС%82РЅРёРє_Р·Р°С%82опленнС%8BРј_РєРѕС%80аблС%8FРј_РІ_СеваС%81С%82ополе"
instead of "Памятник затопленным кораблям в Севастополе.JPG"

If the url was ISO-8859-1 then the urlencoded characters would be control characters, but in utf-8 they are proper characters (by leaving them urlencoded, then the other bytes forming the same codepoint are "broken" and show up as gobbledegook).


If you are using a unix-like OS where the filesystem interface uses utf-8, there is a workaround of using --restrict-file-names=nocontrol (which is still too big, as that would allow problematic control characters %01 or %09).

If you are using Windows, --restrict-file-names=nocontrol still gives garbage (the utf-8 characters are treated as if they were in latin1).

Downstream bug: https://bugzilla.wikimedia.org/show_bug.cgi?id=40844


Platonides <platonides>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by platonides (Submitted the item)
  •  

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.14-49f2.
    Corresponding source code