bugGNU Wget - Bugs: bug #50383, --local-encoding isn't used when...


bug #50383: --local-encoding isn't used when converting a relative link in a recursive download

Submitter:  None
Submitted:  Wed 22 Feb 2017 11:10:42 PM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  William Prescott Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.19
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Thu 23 Feb 2017 10:47:12 AM UTC, comment #1: 

Two problems here:

1. the command-line URL is converted by 'remote_to_utf8()' ind url_parse(). This is wrong, locale_to_utf8() must be taken.
On many locales, this wouldn't make a difference with tilde, but I just recognized it when tracing wget.

2. After dequeing (before download), wget converts the complete URL with remote_to_utf8(). This is wrong - only the part coming from remote should be converted (~foo came from local input).

Suggested fix:
The charset conversion to utf-8 should take place whenever input is taken (from command line or from remote). Internally, wget should work with utf-8 only. That is what Wget2 already does.

I add my Python test script to reproduce this issue, if someone wants to work on it. Copy it to testenv/ and manually start it or add it to Makefile.am.

(file #39814)

Tim Ruehsen <rockdaboot>
Group administrator
Wed 22 Feb 2017 11:10:42 PM UTC, original submission:  

When expanding a relative URL found on a page, Wget doesn't appear to take into
account the local encoding of the URL Wget was called with.

This is apparent when trying to recursively download pages encoded with Shift_JIS
whose URL contains a tilde (Shift_JIS lacks ~ and has ‾ at the same code point).
While the documents themselves cannot have a tilde, they are able to use relative links
to move around within this path.

Wget is currently expanding relative links as if the user-provided URL was in the
document's character encoding. In the case of my example here, this changes the
URL's tilde to ‾.

My expectation is that Wget would use the specified local encoding for the user-provided
part of the base and the remote encoding for the rest of the URL.

Additionally, links on a page retrieved using "IRI fallbacking" will not be
followed (noticeable on bar.html in the example). This may constitute another bug?

EXAMPLE CASE (test files attached as tar archive)
On server:
~foo/baz.html (empty)

~foo/index.html is Shift_JIS encoded and contains
<meta http-equiv="Content-Type" content="text/html;charset=Shift_JIS">
<a href="bar.html">Bar</a>

~foo/bar.html is Shift_JIS encoded and contains
<meta http-equiv="Content-Type" content="text/html;charset=Shift_JIS">
<a href="baz.html">Baz</a>

Results for wget -np -r --local-encoding=utf-8 -d ''
(using Wget 1.19):
~foo/index.html works fine and is saved to ""
~foo/bar.html gets tried as "%E2%80%BEfoo/bar.html" before IRI fallbacking and
is then incorrectly saved to "‾foo/bar.html"
~foo/baz.html is never visited.

Mailing list discussion at http://lists.gnu.org/archive/html/bug-wget/2017-02/msg00111.html



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #39814:  Test-link-shiftjis.py added by rockdaboot (1KiB - text/x-python)
file #39812:  wget_output.txt added by None (6KiB - text/plain)
file #39813:  example.tar.gz added by None (297B - application/gzip)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Updated the item)
  • -email is unavailable- added by None (Submitted the item)

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-02-23 rockdaboot Attached File- Added Test-link-shiftjis.py, #39814
        StatusNone Confirmed
    2017-02-22 None Attached File- Added wget_output.txt, #39812
        Attached File- Added example.tar.gz, #39813

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code