bugGNU Wget - Bugs: bug #20417, 503 respose is not retryable but...

 
 

bug #20417: 503 respose is not retryable but should be

Submitted by:  Noël Köthe <nok>
Submitted on:  Mon 09 Jul 2007 08:57:35 PM UTC  
 
Category: Feature RequestSeverity: 3 - Normal
Priority: 5 - NormalStatus: Fixed
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Closed
Release: 1.10.2Operating System: GNU/Linux
Reproducibility: NoneFixed Release: 1.19.1
Planned Release: 1.15Regression: None
Work Required: NonePatch Included: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 12 Feb 2017 04:37:57 PM UTC, comment #5:

Starting with version 1.19.1, Wget supports the option --retry-on-http-error which you can use to specify a list of HTTP error codes that should be retried. So, provided that you are running a current version of wget, just add this to the arguments: --retry-on-http-error=503

Anonymous
Sun 24 Nov 2013 08:23:12 AM UTC, comment #4:

In my case an nginx server (from which I download several pages gathered in links.txt file) uses 503 error to slow down the download. If the requests for html files come in periods closer than 3 seconds to one another, then 503 is returned. So "wget -w 3" helps. However I wanted to show this case as the one that would also be better handled by automatic retry of 503.

If I use "-w 3" it always waits 3 seconds, even for helper files like gif, js, which could be loaded immediately instead.

Jarek Czekalski <jarekczek>
Fri 30 Jul 2010 10:36:24 PM UTC, comment #3:

I came here doing research as I was using wget to do simple headless monitoring of our tomcat nodes.

The problem is that the tomcat application becomes corrupted over time (memory leak, thread dead lock, something). At that time the server starts responding with a 503. So I used wget to fetch a search from the server once every 5 minutes. The result worked great except for one thing.

Rebooting the node (automatically once per day) results in the node being in capable of serving traffic for a minute or two. After which it is up and running fine. The error the server gives back while booting is a 503 error.

Thus I am getting down time notifications once a day when the server reboots.

I would like to have wget retry 4 times waiting 20 seconds per failure.

However wget has no option to retry 500 errors. Having that option would be of use for me. Hence I add another comment to this CR has a feature request.

In the mean time I'll see if I can switch out wget with curl or something else.

Mike Power <dodtsair>
Sun 11 Apr 2010 05:24:30 PM UTC, comment #2:

From IETF RFC 2616:

------------------------------ snip -----------------------------

10.5.4 503 Service Unavailable

The server is currently unable to handle the request due to a
temporary overloading or maintenance of the server. The implication
is that this is a temporary condition which will be alleviated after
some delay. If known, the length of the delay MAY be indicated in a
Retry-After header. If no Retry-After is given, the client SHOULD
handle the response as it would for a 500 response.

Note: The existence of the 503 status code does not imply that a
server must use it when becoming overloaded. Some servers may wish
to simply refuse the connection.

------------------------------ snap -----------------------------

There are a lot of services using code 503 for download-queues announcing the availability of the next download slot with the "Retry-After" header. So WGET should honor the "Retry-After" header to handle such download queues correctly. Like CURL, WGET can implement a "--retry-max-time" flag. That way the user can define to give up downloading after e.g. 25200 seconds (7 hours).

Rene Bartsch <renne>
Tue 24 Feb 2009 06:07:46 AM UTC, comment #1:

I see nothing in RFC 2616 that designates it as a "non-fatal" error; and clients are explicitly allowed to treat it as a 500.

In any case, I don't see what Wget could be expected to do with it: while this code does indicate a temporary condition, it's not necessarily so temporary as just a few seconds, and Wget can't be expected to just sit around forever waiting for a change. Not to mention that repeated retries on what is often an overload condition is not a particularly great idea.

Micah Cowan <micahcowan>
Mon 09 Jul 2007 08:57:35 PM UTC, original submission:

Hello,

a forwarded wishlist request from http://bugs.debian.org/347573:

"There is no means to tell wget to retry when it gets a 503 response code. Since many proxies use 503 for proxy errors and 503 is officially a non-fatal error, it should be treated as such.

Currently, all downloads are aborted by wget immediately after 503s and there is no option availble to change that."

thx.

-unavailable-

Noël Köthe <nok>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by rockdaboot (Updated the item)
  • -unavailable- added by jarekczek (Posted a comment)
  • -unavailable- added by nok
  • -unavailable- added by dodtsair (Posted a comment)
  • -unavailable- added by renne (Posted a comment)
  • -unavailable- added by micahcowan (Updated the item)
  • -unavailable- added by nok (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 12 Feb 2017 08:04:13 PM UTCrockdabootStatusInvalid=>Fixed
      Fixed ReleaseNone=>1.19.1
    Mon 09 Jul 2012 05:34:17 PM UTCnokCarbon-Copy-=>Added -unavailable-
    Tue 24 Feb 2009 06:07:46 AM UTCmicahcowanStatusNeeds Discussion=>Invalid
      Open/ClosedOpen=>Closed
    Fri 20 Jul 2007 05:36:59 AM UTCmicahcowanStatusNone=>Needs Discussion
      Planned ReleaseNone=>1.15

    Back to the top


    Powered by Savane 3.1-cleanup1