bugGNU Wget - Bugs: bug #43876, inifinite loop on retrieval of...

 
 

bug #43876: inifinite loop on retrieval of zero length robots.txt

Submitted by:  Sean Jensen-Grey <seanj>
Submitted on:  Tue 23 Dec 2014 08:38:32 PM UTC  
 
Category: Crash/Freeze/InfloopSeverity: 3 - Normal
Priority: 5 - NormalStatus: Fixed
Privacy: PublicAssigned to: Tim Ruehsen <rockdaboot>
Originator Name: Open/Closed: Closed
Release: 1.16.1Operating System: Mac OS
Reproducibility: Every TimeFixed Release: None
Planned Release: 1.16.2Regression: Yes
Work Required: NonePatch Included: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 27 Dec 2014 10:34:36 PM UTC, comment #4:

Thanks Sean,

I reverted commit d81a8d5f56fca2746c4ad83d764bd24adcfbe927, which introduced the problem.

Tim Ruehsen <rockdaboot>
Project AdministratorIn charge of this item.
Sat 27 Dec 2014 09:23:54 PM UTC, comment #3:

The exact command was

wget --mirror --no-parent http://dcg.ethz.ch/lectures/podc_allstars/
... infinite loop on robots.txt ...

I am using homebrew on OSX,

```
wget --version
GNU Wget 1.16.1 built on darwin13.4.0.

+digest +https +ipv6 -iri +large-file -nls +ntlm +opie -psl +ssl/openssl

Wgetrc:
/usr/local/etc/wgetrc (system)
Compile:
clang -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
-DLOCALEDIR="/usr/local/Cellar/wget/1.16.1/share/locale" -I.
-I../lib -I../lib -I/usr/local/opt/openssl/include -DNDEBUG
Link:
clang -DNDEBUG -L/usr/local/opt/openssl/lib -lssl -lcrypto -ldl -lz
-luuid ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a
```

```
wget --mirror --no-parent http://dcg.ethz.ch/lectures/podc_allstars/
--2014-12-27 13:19:39-- http://dcg.ethz.ch/lectures/podc_allstars/
Resolving dcg.ethz.ch... 129.132.192.162
Connecting to dcg.ethz.ch|129.132.192.162|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Last-modified header missing -- time-stamps turned off.
--2014-12-27 13:19:40-- http://dcg.ethz.ch/lectures/podc_allstars/
Connecting to dcg.ethz.ch|129.132.192.162|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'dcg.ethz.ch/lectures/podc_allstars/index.html'

dcg.ethz.ch/lectures/podc_allstars/in [ <=> ] 21.50K 117KB/s in 0.2s

2014-12-27 13:19:40 (117 KB/s) - 'dcg.ethz.ch/lectures/podc_allstars/index.html' saved [22021]

Loading robots.txt; please ignore errors.
--2014-12-27 13:19:40-- http://dcg.ethz.ch/robots.txt
Connecting to dcg.ethz.ch|129.132.192.162|:80... connected.
HTTP request sent, awaiting response... 200 OK

The file is already fully retrieved; nothing to do.

Cannot open dcg.ethz.ch/robots.txt: No such file or directoryLoading robots.txt; please ignore errors.
--2014-12-27 13:19:41-- http://dcg.ethz.ch/robots.txt
Connecting to dcg.ethz.ch|129.132.192.162|:80... connected.
HTTP request sent, awaiting response... 200 OK

The file is already fully retrieved; nothing to do.

```

Sean Jensen-Grey <seanj>
Sat 27 Dec 2014 07:58:58 PM UTC, comment #2:

Sorry, I can't reproduce it here.
wget -d -r dcg.ethz.ch

The robots.txt with 0 bytes is saved and wget works normally.

So, what exactly did you do (command line) ?

Tim Ruehsen <rockdaboot>
Project AdministratorIn charge of this item.
Tue 23 Dec 2014 08:39:48 PM UTC, comment #1:

This issue is similar but not identical to https://savannah.gnu.org/bugs/?func=detailitem&item_id=23613

Sean Jensen-Grey <seanj>
Tue 23 Dec 2014 08:38:32 PM UTC, original submission:

inifinite loop on retrieval of zero length robots.txt

The url below returns a zero length robots.txt file. The file is not written to disk, causing wget to attempt to download the file again in an infinite loop.

Touching the file

touch dcg.ethz.ch/robots.txt

and restarting wget allows it to continue on.

curl -I dcg.ethz.ch/robots.txt
HTTP/1.1 200 OK
Date: Tue, 23 Dec 2014 20:36:30 GMT
Server: Apache/2.2.15 (Red Hat)
Last-Modified: Fri, 26 Sep 2014 12:36:54 GMT
ETag: "75b0dc-0-503f72c0c7b98"
Accept-Ranges: bytes
Connection: close
Content-Type: text/plain; charset=UTF-8

Sean Jensen-Grey <seanj>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by rockdaboot (Posted a comment)
  • -unavailable- added by seanj (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 27 Dec 2014 10:34:36 PM UTCrockdabootStatusNone=>Fixed
      Assigned toNone=>rockdaboot
      Open/ClosedOpen=>Closed
      Planned ReleaseNone=>1.16.2

    Back to the top


    Powered by Savane 3.1-cleanup1