Crawl-delay is host/domain specific. Thus a wget -r 'domain1 domain2 domain3' can't simply wait 'crawl-delay' seconds after a download. We need some specific logic when dequeing the next file. Also how comes --wait into play ? The user might be able to override crawl-delay for domain1 but not for domain2 and domain3.

Today, web servers often allow for 50+ parallel connections from one client - I really don't see the point in implementing crawl-delay.

I could change my mind if someone has a real good reason for it and comes up with a good algorithm / patch to handle all corner cases.

It has the same effect when compliantly implemented.

Crawl-delay is a robots.txt directive that, when applied, instructs any bots with access to throttle their download frequency.

So I guess you could say it's a "default" --wait.

Have wget read and respect the crawl-delay directive in robots.txt

wget --mirror http://localhost

http://robots.txt:

User-agent: *
Crawl-delay: 10

expected:

Wget would wait 10 seconds between retrievals

actual:

wget downloaded like mad.

This bug has been CC'ed to gentoo at http://bugs.gentoo.org/show_bug.cgi?id=336488