bugGNU Wget - Bugs: bug #20417, 503 respose is not retryable but...

 
 

bug #20417: 503 respose is not retryable but should be

Submitter:  Noël Köthe <nok>
Submitted:  Mon 09 Jul 2007 08:57:35 PM UTC
   
 
Category:  Feature Request Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  1.10.2 Operating System:  GNU/Linux
Reproducibility:  None Fixed Release:  1.19.1
Planned Release:  1.15 Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 11 Mar 2020 11:07:19 AM UTC, comment #8: 

Please open a new issue with a detailed description - and if possible with a log file (options: --debug -olog.txt) and/or reproducer.

Tim Ruehsen <rockdaboot>
Group administrator
Wed 11 Mar 2020 10:29:04 AM UTC, comment #7: 

Also, this bug is decidedly NOT fixed, as Retry-After is simply ignored and wget immediately retries, hammering the server.

Anonymous
Wed 11 Mar 2020 10:27:13 AM UTC, comment #6: 

While wget now supports --retry-on-http-error, using it breaks both the HTTP spec and the intent, as wget still ignores Retry-After completely and pretty much retries instantly in a loop, which is very bad behaviour and not approrpiate for an automatic download tool.

Also, during the decade that this bug is old, many more sites have switched to using 503 and Retry-After, so lacking support for this HTTP feature makes wget very unsuitable as a well-behaved download tool for a lot of download sites.

Anonymous
Sun 12 Feb 2017 04:37:57 PM UTC, comment #5: 

Starting with version 1.19.1, Wget supports the option --retry-on-http-error which you can use to specify a list of HTTP error codes that should be retried. So, provided that you are running a current version of wget, just add this to the arguments: --retry-on-http-error=503

Anonymous
Sun 24 Nov 2013 08:23:12 AM UTC, comment #4: 

In my case an nginx server (from which I download several pages gathered in links.txt file) uses 503 error to slow down the download. If the requests for html files come in periods closer than 3 seconds to one another, then 503 is returned. So "wget -w 3" helps. However I wanted to show this case as the one that would also be better handled by automatic retry of 503.

If I use "-w 3" it always waits 3 seconds, even for helper files like gif, js, which could be loaded immediately instead.

Jarek Czekalski <jarekczek>
Fri 30 Jul 2010 10:36:24 PM UTC, comment #3: 

I came here doing research as I was using wget to do simple headless monitoring of our tomcat nodes.

The problem is that the tomcat application becomes corrupted over time (memory leak, thread dead lock, something).  At that time the server starts responding with a 503.  So I used wget to fetch a search from the server once every 5 minutes.  The result worked great except for one thing.

Rebooting the node (automatically once per day) results in the node being in capable of serving traffic for a minute or two.  After which it is up and running fine.  The error the server gives back while booting is a 503 error.

Thus I am getting down time notifications once a day when the server reboots.

I would like to have wget retry 4 times waiting 20 seconds per failure.

However wget has no option to retry 500 errors.  Having that option would be of use for me.  Hence I add another comment to this CR has a feature request.

In the mean time I'll see if I can switch out wget with curl or something else.

Mike Power <dodtsair>
Sun 11 Apr 2010 05:24:30 PM UTC, comment #2: 

From IETF RFC 2616:

------------------------------ snip -----------------------------

10.5.4 503 Service Unavailable

   The server is currently unable to handle the request due to a
   temporary overloading or maintenance of the server. The implication
   is that this is a temporary condition which will be alleviated after
   some delay. If known, the length of the delay MAY be indicated in a
   Retry-After header. If no Retry-After is given, the client SHOULD
   handle the response as it would for a 500 response.

      Note: The existence of the 503 status code does not imply that a
      server must use it when becoming overloaded. Some servers may wish
      to simply refuse the connection.

------------------------------ snap -----------------------------


There are a lot of services using code 503 for download-queues announcing the availability of the next download slot with the "Retry-After" header. So WGET should honor the "Retry-After" header to handle such download queues correctly. Like CURL, WGET can implement a "--retry-max-time" flag. That way the user can define to give up downloading after e.g. 25200 seconds (7 hours).

Rene Bartsch <renne>
Tue 24 Feb 2009 06:07:46 AM UTC, comment #1: 

I see nothing in RFC 2616 that designates it as a "non-fatal" error; and clients are explicitly allowed to treat it as a 500.

In any case, I don't see what Wget could be expected to do with it: while this code does indicate a temporary condition, it's not necessarily so temporary as just a few seconds, and Wget can't be expected to just sit around forever waiting for a change. Not to mention that repeated retries on what is often an overload condition is not a particularly great idea.

Micah Cowan <micahcowan>
Mon 09 Jul 2007 08:57:35 PM UTC, original submission:  

Hello,

a forwarded wishlist request from http://bugs.debian.org/347573:

"There is no means to tell wget to retry when it gets a 503 response code. Since many proxies use 503 for proxy errors and 503 is officially a non-fatal error, it should be treated as such.

Currently, all downloads are aborted by wget immediately after 503s and there is no option availble to change that."

thx.

-email is unavailable-

Noël Köthe <nok>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Updated the item)
  • -email is unavailable- added by jarekczek (Posted a comment)
  • -email is unavailable- added by nok
  • -email is unavailable- added by dodtsair (Posted a comment)
  • -email is unavailable- added by renne (Posted a comment)
  • -email is unavailable- added by micahcowan (Updated the item)
  • -email is unavailable- added by nok (Submitted the item)
  •  

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-02-12 rockdaboot StatusInvalid Fixed
        Fixed ReleaseNone 1.19.1
    2012-07-09 nok Carbon-Copy- Added -email is unavailable-
    2009-02-24 micahcowan StatusNeeds Discussion Invalid
        Open/ClosedOpen Closed
    2007-07-20 micahcowan StatusNone Needs Discussion
        Planned ReleaseNone 1.15

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code