bugGNU Wget - Bugs: bug #20422, Wget should handle IRIs.


bug #20422: Wget should handle IRIs.

Submitted by:  Noël Köthe <nok>
Submitted on:  Mon 09 Jul 2007 09:52:21 PM UTC  
Category:  Protocol Issue Severity:  3 - Normal
Priority:  9 - Immediate Status:  Fixed
Privacy:  Public Assigned to:  Micah Cowan <micahcowan>
Originator Name:  Open/Closed:  Closed
Release:  1.10.2 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  1.12
Planned Release:  1.12 Regression:  None
Work Required:  2 - Weeks Patch Included:  No

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Thu 02 Jul 2009 05:46:03 AM UTC, comment #5: 

Merged in 139a47efb8c5

Micah Cowan <micahcowan>In charge of this item.
Tue 22 Jan 2008 11:21:39 PM UTC, comment #4: 

Bumping down the planned release from 2.0 to 1.12 (tentatively). Some international ccTLDs, including one in Russian intended for domain names in all-Cyrillic, are expected to arrive this year. This will quickly become an in-demand feature.

Micah Cowan <micahcowan>In charge of this item.
Mon 05 Nov 2007 08:05:16 PM UTC, comment #3: 

See RFC 3987, and also http://www.w3.org/TR/REC-html40/appendix/notes.html#non-ascii-chars

Probably, we should carry sufficient information around with the URL that we can try each of the following in turn, until we find one that works:

  1. Transcode from the document's encoding to UTF-8, and percent-encode
  2. Directly percent-encode the actual bytes that were used

There might also be cases where we'd want to try transcoding into ISO-8859-1.

Host-names, of course, should only support punycoding. Perhaps if an internationalized host name is detected, we should forgo the "actual bytes" version, as there really can't be any expected meaning for a URI that has non-ASCII characters in the host name, and non-UTF-8 characters in the remainder.

Micah Cowan <micahcowan>In charge of this item.
Wed 22 Aug 2007 04:49:50 AM UTC, comment #2: 

cf bug 20863, "Encode URL to UTF-8 before escaping".

Micah Cowan <micahcowan>In charge of this item.
Wed 22 Aug 2007 04:25:49 AM UTC, comment #1: 

This is really part of a larger issue, in that wget doesn't handle IRIs in general. In order to do this properly, it'll need to do things like, remember what encoding the HTML file was served as (including when wget is invoked with -c at some point after the HTML file was parsed).

Micah Cowan <micahcowan>In charge of this item.
Mon 09 Jul 2007 09:52:21 PM UTC, original submission:  


wget doesn't support IDN domains:

$ LC_ALL=C wget 'http://www.köln.de/'
--23:49:06--  http://www.k%C3%B6ln.de/
           => `index.html'
Resolving http://www.k\303\266ln.de... failed: Name or service not known.

Translating the URL to the xn-- it works.

Maybe its possible to use GNU libidn http://www.gnu.org/software/libidn/


-email is unavailable-

Noël Köthe <nok>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by sxav (Updated the item)
  • -email is unavailable- added by micahcowan (Updated the item)
  • -email is unavailable- added by nok (Submitted the item)



    Follow 14 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-07-02 micahcowan StatusReady For Test => Fixed
        Open/ClosedOpen => Closed
        Fixed ReleaseNone => 1.12
    2008-08-22 micahcowan Priority6 => 9 - Immediate
        Assigned tosxav => micahcowan
    2008-08-15 sxav StatusConfirmed => Ready For Test
    2008-04-14 micahcowan Assigned toNone => sxav
    2008-01-22 micahcowan Priority5 - Normal => 6
    2008-01-22 micahcowan Planned Release2.0 => 1.12
    2007-08-22 micahcowan Summarywget doesn\'t handle IDN domains => Wget should handle IRIs.
    2007-07-20 micahcowan CategoryFeature Request => Protocol Issue
        Planned ReleaseNone => 2.0
    2007-07-09 micahcowan StatusNone => Confirmed
        Work RequiredNone => 2 - Weeks

    Back to the top

    Powered by Savane 3.7