bugGNU Wget - Bugs: bug #58160, wrong filename encoding used in...


bug #58160: wrong filename encoding used in mirror mode

Submitter:  Tomasz Pala <gotar>
Submitted:  Sat 11 Apr 2020 01:45:22 PM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.19.3 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup

Sat 11 Apr 2020 01:45:22 PM UTC, original submission:  

First, let's see the proper behaviour.  I'm using locale but the same can be seen with --local-encoding=iso8859-1 switch. Provided URL is UTF-8 encoded, there is some automatic fallback that fetches the file somehow:

LC_CTYPE=en_DK.iso88591 wget 'https://www.artcom.pl/wp-content/uploads/2015/05/Informacja_o_udogodnieniach_dla_os%c3%b3b_niepe%c5%82nosprawnych_20150508_DUZA_CZCIONKA.pdf'
--2020-04-11 15:13:33--  https://www.artcom.pl/wp-content/uploads/2015/05/Informacja_o_udogodnieniach_dla_os%C3%83%C2%B3b_niepe%C3%85%C2%82nosprawnych_20150508_DUZA_CZCIONKA.pdf
HTTP request sent, awaiting response... 404 Not Found
2020-04-11 15:13:34 ERROR 404: Not Found.

--2020-04-11 15:13:34--  https://www.artcom.pl/wp-content/uploads/2015/05/Informacja_o_udogodnieniach_dla_os%c3%b3b_niepe%c5%82nosprawnych_20150508_DUZA_CZCIONKA.pdf
Reusing existing connection to www.artcom.pl:443.
HTTP request sent, awaiting response... 200 OK
Length: 72915 (71K) [application/pdf]
Saving to: 'Informacja_o_udogodnieniach_dla_osób_niepełnosprawnych_20150508_DUZA_CZCIONKA.pdf'

Informacja_o_udogodnieniach_dla_osób_niepełnosprawnych_201 100%[=========================================================================================================================================>]  71,21K  --.-KB/s    in 0,002s 

2020-04-11 15:13:34 (33,0 MB/s) - 'Informacja_o_udogodnieniach_dla_osĂłb_niepeĹ\202nosprawnych_20150508_DUZA_CZCIONKA.pdf' saved [72915/72915]

This seems fine. However such command uses the wrong encoding:

LC_CTYPE=en_DK.iso88591 wget --mirror --no-parent -e robots=off --backup-converted --limit-rate=1m -c https://www.artcom.pl/wp-content/uploads/2015/05/

--2020-04-11 15:24:50--  https://www.artcom.pl/wp-content/uploads/2015/05/Informacja_o_udogodnieniach_dla_os%C3%83%C2%B3b_niepe%C3%85%C2%82nosprawnych_20150508_DUZA_CZCIONKA.pdf
Reusing existing connection to www.artcom.pl:443.
HTTP request sent, awaiting response... 404 Not Found
2020-04-11 15:24:51 ERROR 404: Not Found.

The problem is: when the URL source is in not directly provided in commmand line, but automatically scraped from some remote site (like in mirror mode above), it's encoding should be independent on current system locale and --local-encoding value, but on remote party. Only the encoding provided by server should be used to fetch consecutive files, local encoding might be used to save them.

In the case above, I do have LC_CTYPE=pl_PL.iso8859-2 and I'm unable to get the file saved with proper encoding, as using --local-encoding=utf8 is required for wget to actually see the file. This should be definitely possible, as providing ISO-encoded URL:

LC_CTYPE=pl_PL.iso8859-2 wget 'https://www.artcom.pl/wp-content/uploads/2015/05/Informacja_o_udogodnieniach_dla_osób_niepełnosprawnych_20150508_DUZA_CZCIONKA.pdf'

works fine.

Tomasz Pala <gotar>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by gotar (Submitted the item)
  • -email is unavailable- added by gotar

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-04-11 gotar Carbon-Copy- Added gotar

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code