bugGNU Wget - Bugs: bug #37660, Downloading page-requisites breaks...

 
 

bug #37660: Downloading page-requisites breaks when using -O with a file not supporting reads

Submitted by:  Jason Hennessey <jayhenn>
Submitted on:  Fri 02 Nov 2012 09:07:14 PM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: NoneOperating System: GNU/Linux
Reproducibility: Every TimeFixed Release: None
Planned Release: NoneRegression: No
Work Required: NonePatch Included: No

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 02 Nov 2012 09:07:14 PM UTC, original submission:

Combining --page-requisites and an output file that does not meaningfully support reading (such as /dev/null or stdout) fails.

It fails because the logic for --page-requisites goes back to the output file and reads it in order to build the list of URLs to download. This is not an intuitive behavior for most users since many would assume that an output file would not need to support reading.

To reproduce:
[output appended]

$ wget -O /dev/null --page-requisites http://www.gnu.org # Stops after initial file
$ wget -O - --page-requisites http://www.gnu.org # Hangs

Would it be possible to address this either by documenting the behavior within the man page or by parsing the document in memory?

WGET VERSION: 1.12 and 1.14
BEHAVIOR EXPECTED: to fetch an html file and all its prerequisites, throwing the actual data away
BEHAVIOR SEEN: wget either hangs (in the case of using -O -) or does not fetch the requisites (in the case of -O /dev/null)
WHY WOULD ANYONE DO THIS: In order to prime a cache or ensure all of a page's requisites are accessible.

Thank you,
Jason

Output:

$ wget -O /dev/null --page-requisites http://www.gnu.org
WARNING: combining -O with -r or -p will mean that all downloaded content
will be placed in the single file you specified.

--2012-11-02 13:42:28-- http://www.gnu.org/
Resolving http://www.gnu.org... 208.118.235.148, 2001:4830:134:3::a
Connecting to http://www.gnu.org|208.118.235.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `/dev/null'

[ <=> ] 19,514 --.-K/s in 0.1s

2012-11-02 13:42:28 (139 KB/s) - `/dev/null' saved [19514]

FINISHED --2012-11-02 13:42:28--

$ wget -O - --page-requisites http://www.gnu.org
WARNING: combining -O with -r or -p will mean that all downloaded content
will be placed in the single file you specified.

--2012-11-02 13:43:01-- http://www.gnu.org/
Resolving http://www.gnu.org... 208.118.235.148, 2001:4830:134:3::a
Connecting to http://www.gnu.org|208.118.235.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `STDOUT'

[<=> ] 0 --.-K/s <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
[...]
2012-11-02 13:43:28 (139 KB/s) - written to stdout [19514]
[hangs]

Jason Hennessey <jayhenn>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jayhenn (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    No Changes Have Been Made to This Item

    Back to the top


    Powered by Savane 3.1-cleanup