bugGNU Wget - Bugs: bug #37660, Downloading page-requisites breaks...

 
 

bug #37660: Downloading page-requisites breaks when using -O with a file not supporting reads

Submitter:  Jason Hennessey <jayhenn>
Submitted:  Fri 02 Nov 2012 09:07:14 PM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Wont Fix
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  None Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  No
Work Required:  None Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 02 Mar 2015 05:39:54 PM UTC, comment #2: 

While it may be counter-intuitive, it is a necessary step. Keeping the entire page in memory would cause the memory requirement of Wget to shoot up exponentially. Especially so when used alongwith the -r option.

Darshit Shah <darnir>
Group administrator
Thu 15 May 2014 08:31:31 PM UTC, comment #1: 

This is much needed now wget supports warc files. Right now, have to save to two places to create a warc file. One is the warc file itself, and the second copy of the downloaded html to parse.

Anonymous
Fri 02 Nov 2012 09:07:14 PM UTC, original submission:  

Combining --page-requisites and an output file that does not meaningfully support reading (such as /dev/null or stdout) fails.

It fails because the logic for --page-requisites goes back to the output file and reads it in order to build the list of URLs to download. This is not an intuitive behavior for most users since many would assume that an output file would not need to support reading.

To reproduce:
[output appended]

$ wget -O /dev/null --page-requisites www.gnu.org # Stops after initial file
$ wget -O - --page-requisites www.gnu.org # Hangs

Would it be possible to address this either by documenting the behavior within the man page or by parsing the document in memory?

WGET VERSION: 1.12 and 1.14
BEHAVIOR EXPECTED: to fetch an html file and all its prerequisites, throwing the actual data away
BEHAVIOR SEEN: wget either hangs (in the case of using -O -) or does not fetch the requisites (in the case of -O /dev/null)
WHY WOULD ANYONE DO THIS: In order to prime a cache or ensure all of a page's requisites are accessible.

Thank you,
Jason

Output:

$ wget -O /dev/null --page-requisites www.gnu.org
WARNING: combining -O with -r or -p will mean that all downloaded content
will be placed in the single file you specified.

--2012-11-02 13:42:28--  http://www.gnu.org/
Resolving www.gnu.org... 208.118.235.148, 2001:4830:134:3::a
Connecting to www.gnu.org|208.118.235.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `/dev/null'

    [ <=>                                                   ] 19,514      --.-K/s   in 0.1s

2012-11-02 13:42:28 (139 KB/s) - `/dev/null' saved [19514]

FINISHED --2012-11-02 13:42:28--

$ wget -O - --page-requisites www.gnu.org
WARNING: combining -O with -r or -p will mean that all downloaded content
will be placed in the single file you specified.

--2012-11-02 13:43:01--  http://www.gnu.org/
Resolving www.gnu.org... 208.118.235.148, 2001:4830:134:3::a
Connecting to www.gnu.org|208.118.235.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `STDOUT'

    [<=>                                                    ] 0           --.-K/s              <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
[...]
2012-11-02 13:43:28 (139 KB/s) - written to stdout [19514]
[hangs]

Jason Hennessey <jayhenn>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by darnir (Posted a comment)
  • -email is unavailable- added by jayhenn (Submitted the item)
  •  

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-03-02 darnir StatusNone Wont Fix
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code