bugGNU Wget - Bugs: bug #42794, Wget doesn't parse HTMLs well if...

 
 

bug #42794: Wget doesn't parse HTMLs well if non-ascii characters are present

Submitted by:  None
Submitted on:  Thu 17 Jul 2014 04:11:15 PM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: Confirmed
Privacy: PublicAssigned to: None
Originator Name: István KissOriginator Email: -unavailable-
Open/Closed: OpenRelease: trunk
Operating System: GNU/LinuxReproducibility: Every Time
Fixed Release: NonePlanned Release: None
Regression: NoneWork Required: None
Patch Included: No

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 03 Mar 2015 08:57:58 AM UTC, comment #5:

However, the Wget command line also has -H included (Host-spanning). And there are external references in the html page/1.

Something is wrong with the parsing...

Using mget as a comparison (mget -E -H -Didokapu.com -k -K -p -d -o log http://www.idokapu.com/page/1), it downloades tons of stuff, mostly because -H. But at least the pic directory holds 10 pictures.
$ l http://www.idokapu.com/pic/
total 576
drwxr-xr-x 2 oms users 240 03-03-15 09:49:14 .
drwxr-xr-x 7 oms users 180 03-03-15 09:49:13 ..
-rw-r--r-- 1 oms users 69374 16-09-12 22:58:57 1347829016.2209-ltetsrgen-f.jpg
-rw-r--r-- 1 oms users 51571 22-09-12 19:02:48 1348333247.2954-index-f.jpg
-rw-r--r-- 1 oms users 39273 22-09-12 20:37:21 1348338875.5892-Miniclip-f.jpg
-rw-r--r-- 1 oms users 81031 26-09-12 17:33:38 1348673554.5829-jbitkp-f.jpg
-rw-r--r-- 1 oms users 67777 26-09-12 22:01:06 1348689569.2198-spongebobandpatrickstar4857-f.jpg
-rw-r--r-- 1 oms users 67397 27-09-12 20:12:03 1348769330.0948-telefon-f.jpg
-rw-r--r-- 1 oms users 48761 28-09-12 18:37:24 1348850113.9416-halflife1-f.jpg
-rw-r--r-- 1 oms users 46009 02-10-12 22:29:45 1349209599.7523-2-f.jpg
-rw-r--r-- 1 oms users 49748 03-10-12 21:47:22 1349293333.6796-viagra11-f.jpg
-rw-r--r-- 1 oms users 50874 05-10-12 13:56:11 1349438080.7457-trabi60036-f.jpg

We have to reproduce the problem with a test case and fix it.

Tim Ruehsen <rockdaboot>
Project Administrator
Mon 02 Mar 2015 08:30:37 PM UTC, comment #4:

The only interesting fact is that it does work with wget when there are only ASCII characters in the value of the attribute, regardless of that syntax error.

However, I understand the point of view that it's not a necessity to be able to parse badly written HTML code.

Anonymous
Mon 02 Mar 2015 05:11:17 PM UTC, comment #3:

In fact when I look at the actual source in Firefox, I see that the images you mentioned, actually have a syntax error. The closing quote mark does not exist. Firefox seems to be smartly understanding the error and working around it. Wget does not, and has no reason to honour a badly written html page.

Darshit Shah <darnir>
Project Administrator
Sat 20 Dec 2014 08:34:41 PM UTC, comment #2:

If you issue the command, the resulting "http://www.idokapu.com/page/1.html" file, opened in a browser (I use Firefox) shows only the "Half life" and "Trabant" images (lines 321, 383), and for the others a broken alt text is shown, e.g. for the Spongebob one "spongebob ĂŠs patric title=" is shown (which is right if we consider that the closing quote of the alt attribute is missing). (line 196). The image is not shown because it's not downloaded by wget. And this happens only if non-ascii (áÁéÉíÍóÓőŐúÚűÚ) characters are present in the title (and thus in the alt) attribute.

I can't see any Javascript in the img tag which is responsible for showing the images.

One more thing I find remarkable is that opening the original (http://www.idokapu.com/page/1) file, these Unicode characters are OK in the source, but in the one downloaded by wget they aren't. I bet it's a content encoding problem, and this may cause the bug.

Anonymous
Mon 01 Dec 2014 03:42:11 PM UTC, comment #1:

There is so much stuff in page/1... could you please give me two or three names of images not being loaded ?

Also, the 'alt=' values with non-ASCII characters seem to be in Javascript code. Wget will not parse or execute Javascript.

Tim Ruehsen <rockdaboot>
Project Administrator
Thu 17 Jul 2014 04:11:15 PM UTC, original submission:

OS: Debian Jessie (testing)
Wget version: 1.15-1+b1

I tried to archive a website and I realized that Wget didn't download all images.

The original command was longer and mirrored the whole site; for testing I issued:

wget -E -H -Didokapu.com -k -K -p -d -o log http://www.idokapu.com/page/1

as recommended in the manual, to make sure all page-requisites will be downloaded. The bug is present in both cases.

10 main images are on the page (the posts), only 2 were downloaded. The structure of the posts is the same. However, I looked into the HTML source and found syntax errors ("alt" attributes in IMG tags not closed). But it seems not this is what causes the problem (if I tried page nr. 10, there 8 images were downloaded, 2 not, same structure, syntax error in every post).

Finally, I found out that the common thing in the not-downloaded images is this: in the "alt" (and "title") attributes there is at least one non-ascii character. (Try for example /page/10 also, or any page.)

So, wget gets confused by non-ascii characters in these attributes and skips the corresponding images.

(To make sure, I tried playing with --local-encoding, --remote-encoding, --restrict-file-names, but of course they didn't change anything, as they are for filenames.)

Should the result be different on different computers, I attach in a tar.gz what wget downloaded (besides the debug output, of course).

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #31735:  log added by None (288kB - application/octet-stream)
file #31736:  www.idokapu.com.tar.gz added by None (1MB - application/gzip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by darnir (Posted a comment)
  • -unavailable- added by rockdaboot (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 03 Mar 2015 08:57:58 AM UTCrockdabootStatusCannot Reproduce=>Confirmed
    Mon 02 Mar 2015 05:11:17 PM UTCdarnirStatusNone=>Cannot Reproduce
    Thu 17 Jul 2014 04:11:15 PM UTCNoneAttached File-=>Added log, #31735
      Attached File-=>Added http://www.idokapu.com.tar.gz, #31736

    Back to the top


    Powered by Savane 3.1-cleanup