bugGNU Wget - Bugs: bug #34371, recursive HTTP download doesn't...

 
 

bug #34371: recursive HTTP download doesn't obey -A in css urls

Submitter:  None
Submitted:  Fri 23 Sep 2011 09:25:14 PM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Kim Scarborough Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.13.4
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 23 Sep 2011 09:25:14 PM UTC, original submission:  

When I specify -r to a URL download, it downloads all the files, including files specified in the CSS stylesheet. If I limit it with -A, however, it ignores the CSS-specified files, even though they are of the type specified.

Here's my example session:

turkey: /home/kjs/tmp/wget> wget --version
GNU Wget 1.13.4 built on linux-gnu.

+digest +https -ipv6 -iri +large-file -nls +ntlm +opie +ssl/openssl

Wgetrc:
    /etc/wgetrc (system)
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
    -DLOCALEDIR="/share/locale" -I. -I../lib -I../lib -I/include -O2
    -Wall
Link: gcc -O2 -Wall /lib/libssl.so /lib/libcrypto.so /lib/libz.so -Wl,-rpath
    -Wl,/lib -ldl -lz -lrt ftp-opie.o openssl.o http-ntlm.o
    ../lib/libgnu.a

Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.
turkey: /home/kjs/tmp/wget> wget -r -np -nd -nH http://www.unknown.nu/misc/wget-testcase/test.html
--2011-09-23 16:15:22--  http://www.unknown.nu/misc/wget-testcase/test.html
Resolving www.unknown.nu... 66.90.79.99
Connecting to www.unknown.nu|66.90.79.99|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 224 [text/html]
Saving to: `test.html'

100%[======================================>] 224         --.-K/s   in 0s

2011-09-23 16:15:22 (12.4 MB/s) - `test.html' saved [224/224]

Loading robots.txt; please ignore errors.
--2011-09-23 16:15:22--  http://www.unknown.nu/robots.txt
Reusing existing connection to www.unknown.nu:80.
HTTP request sent, awaiting response... 200 OK
Length: 95 [text/plain]
Saving to: `robots.txt'

100%[======================================>] 95          --.-K/s   in 0s

2011-09-23 16:15:22 (8.67 MB/s) - `robots.txt' saved [95/95]

--2011-09-23 16:15:22--  http://www.unknown.nu/misc/wget-testcase/test.css
Reusing existing connection to www.unknown.nu:80.
HTTP request sent, awaiting response... 200 OK
Length: 41 [text/css]
Saving to: `test.css'

100%[======================================>] 41          --.-K/s   in 0s

2011-09-23 16:15:22 (3.82 MB/s) - `test.css' saved [41/41]

--2011-09-23 16:15:22--  http://www.unknown.nu/misc/wget-testcase/html.gif
Reusing existing connection to www.unknown.nu:80.
HTTP request sent, awaiting response... 200 OK
Length: 1326 (1.3K) [image/gif]
Saving to: `html.gif'

100%[======================================>] 1,326       --.-K/s   in 0s

2011-09-23 16:15:22 (70.2 MB/s) - `html.gif' saved [1326/1326]

--2011-09-23 16:15:22--  http://www.unknown.nu/misc/wget-testcase/css.gif
Reusing existing connection to www.unknown.nu:80.
HTTP request sent, awaiting response... 200 OK
Length: 1326 (1.3K) [image/gif]
Saving to: `css.gif'

100%[======================================>] 1,326       --.-K/s   in 0s

2011-09-23 16:15:22 (70.5 MB/s) - `css.gif' saved [1326/1326]

FINISHED --2011-09-23 16:15:22--
Total wall clock time: 0.02s
Downloaded: 5 files, 2.9K in 0s (38.9 MB/s)
turkey: /home/kjs/tmp/wget> ls
css.gif  html.gif  robots.txt  test.css  test.html
turkey: /home/kjs/tmp/wget> rm *
turkey: /home/kjs/tmp/wget> wget -r -np -nd -nH -A gif http://www.unknown.nu/misc/wget-testcase/test.html
--2011-09-23 16:15:35--  http://www.unknown.nu/misc/wget-testcase/test.html
Resolving www.unknown.nu... 66.90.79.99
Connecting to www.unknown.nu|66.90.79.99|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 224 [text/html]
Saving to: `test.html'

100%[======================================>] 224         --.-K/s   in 0s

2011-09-23 16:15:35 (18.6 MB/s) - `test.html' saved [224/224]

Loading robots.txt; please ignore errors.
--2011-09-23 16:15:35--  http://www.unknown.nu/robots.txt
Reusing existing connection to www.unknown.nu:80.
HTTP request sent, awaiting response... 200 OK
Length: 95 [text/plain]
Saving to: `robots.txt'

100%[======================================>] 95          --.-K/s   in 0s

2011-09-23 16:15:35 (8.64 MB/s) - `robots.txt' saved [95/95]

Removing test.html since it should be rejected.

--2011-09-23 16:15:35--  http://www.unknown.nu/misc/wget-testcase/html.gif
Reusing existing connection to www.unknown.nu:80.
HTTP request sent, awaiting response... 200 OK
Length: 1326 (1.3K) [image/gif]
Saving to: `html.gif'

100%[======================================>] 1,326       --.-K/s   in 0s

2011-09-23 16:15:35 (104 MB/s) - `html.gif' saved [1326/1326]

FINISHED --2011-09-23 16:15:35--
Total wall clock time: 0.01s
Downloaded: 3 files, 1.6K in 0s (45.9 MB/s)
turkey: /home/kjs/tmp/wget> ls
html.gif  robots.txt

Note that the second time it only downloaded html.gif, which is specified in the html, but not css.gif (specified in css.gif).

I'll keep that URL live but I'm attaching the html & css files just in case.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #24019:  test.css added by None (41B - text/css)
file #24020:  test.html added by None (224B - text/html)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by None (Submitted the item)
  •  

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-09-23 None Attached File- Added test.css, #24019
        Attached File- Added test.html, #24020

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code