bugGNU Wget - Bugs: bug #42794, Wget doesn't parse HTMLs well if...


bug #42794: Wget doesn't parse HTMLs well if non-ascii characters are present

Submitter:  None
Submitted:  Thu 17 Jul 2014 04:11:15 PM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  István Kiss Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  trunk
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  No
* Mandatory Fields

Add a New Comment (Rich Markup)

Comment Type & Canned Response:


Tue 03 Mar 2015 08:57:58 AM UTC, comment #5: 

However, the Wget command line also has -H included (Host-spanning). And there are external references in the html page/1.

Something is wrong with the parsing...

Using mget as a comparison (mget -E -H -Didokapu.com -k -K -p -d -o log http://www.idokapu.com/page/1), it downloades tons of stuff, mostly because -H. But at least the pic directory holds 10 pictures.
$ l www.idokapu.com/pic/
total 576
drwxr-xr-x 2 oms users   240 03-03-15 09:49:14 .
drwxr-xr-x 7 oms users   180 03-03-15 09:49:13 ..
-rw-r--r-- 1 oms users 69374 16-09-12 22:58:57 1347829016.2209-ltetsrgen-f.jpg
-rw-r--r-- 1 oms users 51571 22-09-12 19:02:48 1348333247.2954-index-f.jpg
-rw-r--r-- 1 oms users 39273 22-09-12 20:37:21 1348338875.5892-Miniclip-f.jpg
-rw-r--r-- 1 oms users 81031 26-09-12 17:33:38 1348673554.5829-jbitkp-f.jpg
-rw-r--r-- 1 oms users 67777 26-09-12 22:01:06 1348689569.2198-spongebobandpatrickstar4857-f.jpg
-rw-r--r-- 1 oms users 67397 27-09-12 20:12:03 1348769330.0948-telefon-f.jpg
-rw-r--r-- 1 oms users 48761 28-09-12 18:37:24 1348850113.9416-halflife1-f.jpg
-rw-r--r-- 1 oms users 46009 02-10-12 22:29:45 1349209599.7523-2-f.jpg
-rw-r--r-- 1 oms users 49748 03-10-12 21:47:22 1349293333.6796-viagra11-f.jpg
-rw-r--r-- 1 oms users 50874 05-10-12 13:56:11 1349438080.7457-trabi60036-f.jpg

We have to reproduce the problem with a test case and fix it.

Tim Ruehsen <rockdaboot>
Project Administrator
Mon 02 Mar 2015 08:30:37 PM UTC, comment #4: 

The only interesting fact is that it does work with wget when there are only ASCII characters in the value of the attribute, regardless of that syntax error.

However, I understand the point of view that it's not a necessity to be able to parse badly written HTML code.

Mon 02 Mar 2015 05:11:17 PM UTC, comment #3: 

In fact when I look at the actual source in Firefox, I see that the images you mentioned, actually have a syntax error. The closing quote mark does not exist. Firefox seems to be smartly understanding the error and working around it. Wget does not, and has no reason to honour a badly written html page.

Darshit Shah <darnir>
Project Administrator
Sat 20 Dec 2014 08:34:41 PM UTC, comment #2: 

If you issue the command, the resulting "www.idokapu.com/page/1.html" file, opened in a browser (I use Firefox) shows only the "Half life" and "Trabant" images (lines 321, 383), and for the others a broken alt text is shown, e.g. for the Spongebob one "spongebob ĂŠs patric title=" is shown (which is right if we consider that the closing quote of the alt attribute is missing). (line 196). The image is not shown because it's not downloaded by wget. And this happens only if non-ascii (áÁéÉíÍóÓőŐúÚűÚ) characters are present in the title (and thus in the alt) attribute.

I can't see any Javascript in the img tag which is responsible for showing the images.

One more thing I find remarkable is that opening the original (http://www.idokapu.com/page/1) file, these Unicode characters are OK in the source, but in the one downloaded by wget they aren't. I bet it's a content encoding problem, and this may cause the bug.

Mon 01 Dec 2014 03:42:11 PM UTC, comment #1: 

There is so much stuff in page/1... could you please give me two or three names of images not being loaded ?

Also, the 'alt=' values with non-ASCII characters seem to be in Javascript code. Wget will not parse or execute Javascript.

Tim Ruehsen <rockdaboot>
Project Administrator
Thu 17 Jul 2014 04:11:15 PM UTC, original submission:  

OS: Debian Jessie (testing)
Wget version: 1.15-1+b1

I tried to archive a website and I realized that Wget didn't download all images.

The original command was longer and mirrored the whole site; for testing I issued:

wget -E -H -Didokapu.com -k -K -p -d -o log www.idokapu.com/page/1

as recommended in the manual, to make sure all page-requisites will be downloaded. The bug is present in both cases.

10 main images are on the page (the posts), only 2 were downloaded. The structure of the posts is the same. However, I looked into the HTML source and found syntax errors ("alt" attributes in IMG tags not closed). But it seems not this is what causes the problem (if I tried page nr. 10, there 8 images were downloaded, 2 not, same structure, syntax error in every post).

Finally, I found out that the common thing in the not-downloaded images is this: in the "alt" (and "title") attributes there is at least one non-ascii character. (Try for example /page/10 also, or any page.)

So, wget gets confused by non-ascii characters in these attributes and skips the corresponding images.

(To make sure, I tried playing with --local-encoding, --remote-encoding, --restrict-file-names, but of course they didn't change anything, as they are for filenames.)

Should the result be different on different computers, I attach in a tar.gz what wget downloaded (besides the debug output, of course).



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #31736:  www.idokapu.com.tar.gz added by None (1MiB - application/gzip)
file #31735:  log added by None (288KiB - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by darnir (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)




    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-03-03 rockdaboot StatusCannot Reproduce Confirmed
    2015-03-02 darnir StatusNone Cannot Reproduce
    2014-07-17 None Attached File- Added log, #31735
        Attached File- Added www.idokapu.com.tar.gz, #31736

    Back to the top

    Powered by Savane 3.9