Sat 27 Dec 2014 09:23:54 PM UTC, comment #3:
The exact command was
wget --mirror --no-parent http://dcg.ethz.ch/lectures/podc_allstars/
... infinite loop on robots.txt ...
I am using homebrew on OSX,
```
wget --version
GNU Wget 1.16.1 built on darwin13.4.0.
+digest +https +ipv6 -iri +large-file -nls +ntlm +opie -psl +ssl/openssl
Wgetrc:
/usr/local/etc/wgetrc (system)
Compile:
clang -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
-DLOCALEDIR="/usr/local/Cellar/wget/1.16.1/share/locale" -I.
-I../lib -I../lib -I/usr/local/opt/openssl/include -DNDEBUG
Link:
clang -DNDEBUG -L/usr/local/opt/openssl/lib -lssl -lcrypto -ldl -lz
-luuid ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a
```
```
wget --mirror --no-parent http://dcg.ethz.ch/lectures/podc_allstars/
--2014-12-27 13:19:39-- http://dcg.ethz.ch/lectures/podc_allstars/
Resolving dcg.ethz.ch... 129.132.192.162
Connecting to dcg.ethz.ch|129.132.192.162|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Last-modified header missing -- time-stamps turned off.
--2014-12-27 13:19:40-- http://dcg.ethz.ch/lectures/podc_allstars/
Connecting to dcg.ethz.ch|129.132.192.162|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'dcg.ethz.ch/lectures/podc_allstars/index.html'
dcg.ethz.ch/lectures/podc_allstars/in [ <=> ] 21.50K 117KB/s in 0.2s
2014-12-27 13:19:40 (117 KB/s) - 'dcg.ethz.ch/lectures/podc_allstars/index.html' saved [22021]
Loading robots.txt; please ignore errors.
--2014-12-27 13:19:40-- http://dcg.ethz.ch/robots.txt
Connecting to dcg.ethz.ch|129.132.192.162|:80... connected.
HTTP request sent, awaiting response... 200 OK
The file is already fully retrieved; nothing to do.
Cannot open dcg.ethz.ch/robots.txt: No such file or directoryLoading robots.txt; please ignore errors.
--2014-12-27 13:19:41-- http://dcg.ethz.ch/robots.txt
Connecting to dcg.ethz.ch|129.132.192.162|:80... connected.
HTTP request sent, awaiting response... 200 OK
The file is already fully retrieved; nothing to do.
```
|