bugGNU Wget - Bugs: bug #42794, Wget doesn't parse HTMLs well if...


bug #42794: Wget doesn't parse HTMLs well if non-ascii characters are present

Submitted by:  None
Submitted on:  Thu 17 Jul 2014 04:11:15 PM UTC  
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: István KissOriginator Email: -unavailable-
Open/Closed: OpenRelease: trunk
Operating System: GNU/LinuxReproducibility: Every Time
Fixed Release: NonePlanned Release: None
Regression: NoneWork Required: None
Patch Included: No

Add a New Comment (Rich MarkupRich Markup):

You are not logged in

Please log in, so followups can be emailed to you.


Sat 20 Dec 2014 08:34:41 PM UTC, comment #2:

If you issue the command, the resulting "http://www.idokapu.com/page/1.html" file, opened in a browser (I use Firefox) shows only the "Half life" and "Trabant" images (lines 321, 383), and for the others a broken alt text is shown, e.g. for the Spongebob one "spongebob ĂŠs patric title=" is shown (which is right if we consider that the closing quote of the alt attribute is missing). (line 196). The image is not shown because it's not downloaded by wget. And this happens only if non-ascii (áÁéÉíÍóÓőŐúÚűÚ) characters are present in the title (and thus in the alt) attribute.

I can't see any Javascript in the img tag which is responsible for showing the images.

One more thing I find remarkable is that opening the original (http://www.idokapu.com/page/1) file, these Unicode characters are OK in the source, but in the one downloaded by wget they aren't. I bet it's a content encoding problem, and this may cause the bug.

Mon 01 Dec 2014 03:42:11 PM UTC, comment #1:

There is so much stuff in page/1... could you please give me two or three names of images not being loaded ?

Also, the 'alt=' values with non-ASCII characters seem to be in Javascript code. Wget will not parse or execute Javascript.

Tim Ruehsen <rockdaboot>
Project Administrator
Thu 17 Jul 2014 04:11:15 PM UTC, original submission:

OS: Debian Jessie (testing)
Wget version: 1.15-1+b1

I tried to archive a website and I realized that Wget didn't download all images.

The original command was longer and mirrored the whole site; for testing I issued:

wget -E -H -Didokapu.com -k -K -p -d -o log http://www.idokapu.com/page/1

as recommended in the manual, to make sure all page-requisites will be downloaded. The bug is present in both cases.

10 main images are on the page (the posts), only 2 were downloaded. The structure of the posts is the same. However, I looked into the HTML source and found syntax errors ("alt" attributes in IMG tags not closed). But it seems not this is what causes the problem (if I tried page nr. 10, there 8 images were downloaded, 2 not, same structure, syntax error in every post).

Finally, I found out that the common thing in the not-downloaded images is this: in the "alt" (and "title") attributes there is at least one non-ascii character. (Try for example /page/10 also, or any page.)

So, wget gets confused by non-ascii characters in these attributes and skips the corresponding images.

(To make sure, I tried playing with --local-encoding, --remote-encoding, --restrict-file-names, but of course they didn't change anything, as they are for filenames.)

Should the result be different on different computers, I attach in a tar.gz what wget downloaded (besides the debug output, of course).



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

Attached Files
file #31735:  log added by None (288kB - application/octet-stream)
file #31736:  www.idokapu.com.tar.gz added by None (1MB - application/gzip)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by rockdaboot (Posted a comment)
  • -unavailable- added by None (Submitted the item)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 17 Jul 2014 04:11:15 PM UTCNoneAttached File-=>Added log, #31735
      Attached File-=>Added http://www.idokapu.com.tar.gz, #31736

    Back to the top

    Powered by Savane 3.1-cleanup