bugGNU Wget - Bugs: bug #50383, --local-encoding isn't used when...


bug #50383: --local-encoding isn't used when converting a relative link in a recursive download

Submitted by:  None
Submitted on:  Wed 22 Feb 2017 11:10:42 PM UTC  
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: Confirmed
Privacy: PublicAssigned to: None
Originator Name: William PrescottOriginator Email: -unavailable-
Open/Closed: OpenRelease: 1.19
Operating System: GNU/LinuxReproducibility: Every Time
Fixed Release: NonePlanned Release: None
Regression: NoneWork Required: None
Patch Included: None

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Thu 23 Feb 2017 10:47:12 AM UTC, comment #1:

Two problems here:

1. the command-line URL is converted by 'remote_to_utf8()' ind url_parse(). This is wrong, locale_to_utf8() must be taken.
On many locales, this wouldn't make a difference with tilde, but I just recognized it when tracing wget.

2. After dequeing (before download), wget converts the complete URL with remote_to_utf8(). This is wrong - only the part coming from remote should be converted (~foo came from local input).

Suggested fix:
The charset conversion to utf-8 should take place whenever input is taken (from command line or from remote). Internally, wget should work with utf-8 only. That is what Wget2 already does.

I add my Python test script to reproduce this issue, if someone wants to work on it. Copy it to testenv/ and manually start it or add it to Makefile.am.

(file #39814)

Tim Ruehsen <rockdaboot>
Project Administrator
Wed 22 Feb 2017 11:10:42 PM UTC, original submission:

When expanding a relative URL found on a page, Wget doesn't appear to take into
account the local encoding of the URL Wget was called with.

This is apparent when trying to recursively download pages encoded with Shift_JIS
whose URL contains a tilde (Shift_JIS lacks ~ and has ‾ at the same code point).
While the documents themselves cannot have a tilde, they are able to use relative links
to move around within this path.

Wget is currently expanding relative links as if the user-provided URL was in the
document's character encoding. In the case of my example here, this changes the
URL's tilde to ‾.

My expectation is that Wget would use the specified local encoding for the user-provided
part of the base and the remote encoding for the rest of the URL.

Additionally, links on a page retrieved using "IRI fallbacking" will not be
followed (noticeable on bar.html in the example). This may constitute another bug?

EXAMPLE CASE (test files attached as tar archive)
On server:
~foo/baz.html (empty)

~foo/index.html is Shift_JIS encoded and contains
<meta http-equiv="Content-Type" content="text/html;charset=Shift_JIS">
<a href="bar.html">Bar</a>

~foo/bar.html is Shift_JIS encoded and contains
<meta http-equiv="Content-Type" content="text/html;charset=Shift_JIS">
<a href="baz.html">Baz</a>

Results for wget -np -r --local-encoding=utf-8 -d ''
(using Wget 1.19):
~foo/index.html works fine and is saved to ""
~foo/bar.html gets tried as "%E2%80%BEfoo/bar.html" before IRI fallbacking and
is then incorrectly saved to "‾foo/bar.html"
~foo/baz.html is never visited.

Mailing list discussion at http://lists.gnu.org/archive/html/bug-wget/2017-02/msg00111.html



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

Attached Files
file #39814:  Test-link-shiftjis.py added by rockdaboot (1KiB - text/x-python)
file #39812:  wget_output.txt added by None (6KiB - text/plain)
file #39813:  example.tar.gz added by None (297B - application/gzip)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by rockdaboot (Updated the item)
  • -unavailable- added by None (Submitted the item)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 23 Feb 2017 10:47:12 AM UTCrockdabootAttached File-=>Added Test-link-shiftjis.py, #39814
    Wed 22 Feb 2017 11:10:42 PM UTCNoneAttached File-=>Added wget_output.txt, #39812
      Attached File-=>Added example.tar.gz, #39813

    Back to the top

    Powered by Savane 3.1-cleanup1