bugGNU Wget - Bugs: bug #47701, wget 1.17.1 fails to convert from...

 
 

bug #47701: wget 1.17.1 fails to convert from percent encoding to unicode correctly (mingw32)

Submitter:  None
Submitted:  Fri 15 Apr 2016 04:31:08 AM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Anonymous Coward Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  1.17.1
Operating System:  Microsoft Windows Reproducibility:  None
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 22 Apr 2016 07:57:05 AM UTC, comment #6: 

There are several charset patches (also yours, Eli) since 1.17.1.

So please try to build from the 'master' branch (git clone git://git.savannah.gnu.org/wget.git).

Tim Ruehsen <rockdaboot>
Group administrator
Fri 22 Apr 2016 07:48:03 AM UTC, comment #5: 

In order for you to see the files with non-ASCII names correctly named on your Windows disk, all the non-ASCII characters in the file names must be supported by the current system codepage. In addition, your wget must be built with libiconv.  If any of these two conditions is not true, you will see mojibake in the file names, because Windows doesn't support UTF-8 encoded file names.

A way to lift one of these limitations -- that the file names be expressible in the system codepage -- was discussed, but no one has submitted a clean patchset to fix it. (Doing so on Windows requires to replace/wrap C library functions that deal with file names with versions that can accept UTF-8 encoded name, convert it to UTF-16, and then call the appropriate library function, like call _wopen instead of open etc.)

One other thing: a few months back I submitted changes to make non-ASCII file name support more correct, and I'm not sure that patch is in wget 1.17.1.  Perhaps Tim or Giuseppe could tell.  If the patch is not in 1.17.1, I suggest to build wget from the Git repository and see if some of the problems are gone.

Eli Zaretskii <eliz>
Thu 21 Apr 2016 10:21:00 PM UTC, comment #4: 


> Don't get me wrong, it does actually download the files, they're just encoded wrong on my hard drive now.


Oh Now it's stopped. It gave up after track two, after having downloaded seven files losedows gives me a popup box saying 'wget.exe has stopped working.'

Anonymous Coward <anonyninnymouse>
Thu 21 Apr 2016 10:17:16 PM UTC, comment #3: 

I tried adding the --local-encoding=UTF-8 command-line option as suggested by Eli, (Thanks, Eli!) but that also didn't quite work 100% for me.

Instead of simply failing to download anything, as before, now it makes a good attempt at downloading the requested directory, but (presumably because f'ing losedows doesn't use UTF-8 internally) it not only still encodes it wrong, it encodes it wrong differently. In this case it encodes as a capital A with tilde followed by the pilcrow, which would be C3 B6 in Latin-1 or Windows Codepage 1252.

Don't get me wrong, it does actually download the files, they're just encoded wrong on my hard drive now.

When I run wget with debug on, it throws a lot of stuff at me I don't understand, but it might be useful to you so I'll include it here.

It displays it in 'cmd' in a-whole-nother encoding, as two box-drawing characters in DOS Codepage 437! (AAARGH! DIE, MICROSOFT, DIE!)

Due to one or more of these encoding issues, wget seems to decide not to get some of the stuff I asked it to:


Deciding whether to enqueue "https://leoandpeto.com/icons/blank.gif".
Going to "icons" would escape "Music/Peto/Non/o-z/R├╢yksopp - The Understanding" with no_parent on.
Decided NOT to load it.
Deciding whether to enqueue "https://leoandpeto.com/Music/Peto/Non/o-z/R%C3%B6yksopp%20-%20The%20Understanding/?C=N;O=A".
Already on the black list.
Decided NOT to load it.
Deciding whether to enqueue "https://leoandpeto.com/Music/Peto/Non/o-z/R%C3%B6yksopp%20-%20The%20Understanding/?C=M;O=A".
Already on the black list.
Decided NOT to load it.


and something seems to make windows-sockets/winsock output lots of stuff, too. This might be normal for debug mode though.


200 OK
Length: 5409000 (5.2M) [audio/mpeg]
Saving to: 'R\303\266yksopp - The Understanding/01 Triumphant.mp3'

R├╢yksopp - The Understanding/01 Triumphant.   0%[                                                                                              ]       0  --.-KB/s               seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
R├╢yksopp - The Understanding/01 Triumphant.   0%[                                                                                              ]  24.00K   113KB/s               seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
R├╢yksopp - The Understanding/01 Triumphant.   1%[                                                                                              ]  56.00K   119KB/s               seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
R├╢yksopp - The Understanding/01 Triumphant.   1%[>                                                                                             ] 104.00K   152KB/s               seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
seconds 900.00, Winsock error: 0
[.......]


> thanks anonymous for the detailed bug report.


Oh you're quite welcome, I know a detail-less bug report is useless.

-- Anonymous Coward.

Anonymous Coward <anonyninnymouse>
Fri 15 Apr 2016 08:39:49 AM UTC, comment #2: 

Thanks Eli.

And thanks anonymous for the detailed bug report.

Tim Ruehsen <rockdaboot>
Group administrator
Fri 15 Apr 2016 07:18:14 AM UTC, comment #1: 

You need to give Wget the --local-encoding=UTF-8 command-line option, because the URL you are trying to fetch is actually in UTF-8 encoding (and then each byte of the UTF-8 sequence is encoded with percent escapes on top of that).

When I use that switch, the command works for me (with Wget 1.16.1 compiled with MinGW on MS-Windows).

Eli Zaretskii <eliz>
Fri 15 Apr 2016 04:31:08 AM UTC, original submission:  

The version of GNU Wget you were using

1.17.1 (mingw32)

How you invoked wget


wget -d -r -np -nH --cut-dirs=4 "https://leoandpeto.com/Music/Peto/Non/o-z/R%C3%B6yksopp%20-%20The%20Understanding/"


(in case this is some sort of mingw32-compiler bug)
I obtained my binary copy of mingw32 wget from
https://eternallybored.org/misc/wget/ as recommended by
http://wget.addictivecode.org/FrequentlyAskedQuestions#download

What you expected wget to do

Recursively download the requested open directory.

What wget did (include output messages).

didn't download it.

I tried running with debug on, from looking at the
output it seems wget converted from the percent-
encoded URL to w32's native unicode encoding wrong.

(full copy (input and output) of my run of wget is
included as an attachment named "wget-output.txt".)

specifically, the output includes this line (exactly.)


converted 'https://leoandpeto.com/Music/Peto/Non/o-z/R%C3%B6yksopp%20-%20The%20Understanding/' (ASCII) -> 'https://leoandpeto.com/Music/Peto/Non/o-z/RC6yksopp - The Understanding/' (UTF-8)


wget changed the word from "R%C3%B6yksopp" to "RC6yksopp"
with no percent signs.

It seems to be stripping the leading first bit off of
each byte of %C3 (11000011) and %B6 (10110110), and so
converting them into their 7-bit ASCII equivalents:


,-----.----------.-------.
| HEX | BIN      | ASCII |
+-----+----------+-------+
| C3  | 11000011 | --    |
| 43  |  1000011 | C     |
+-----+----------+-------+
| B6  | 10110110 | --    |
| 36  |  0110110 | 6     |
`-----'----------'-------'


Also, I'm not 100% sure that this isn't a duplicate of
http://savannah.gnu.org/bugs/index.php?47689
but I figured it's best to let you developers decide
rather than failing to file a bug-report.

Thank you for making/working on wget, and
please CONTINUE BEING AWESOME! :-D

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #36931:  wget-output.txt added by None (5KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by anonyninnymouse (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by eliz (Posted a comment)
  •  

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-04-15 rockdaboot Open/ClosedOpen Closed
    2016-04-15 None Attached File- Added wget-output.txt, #36931

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code