Hello,

this is a forward of a Debian bug report to the wget Bugtracker with an answer from Micah (reproduceable with wget 1.12).

From: Micah Cowan <micah@cowan.name>
To: Nicolas Duillier <nicolasduillier@googlemail.com>,
 512138@bugs.debian.org
Cc: Nico Golde <nion@debian.org>,
 Dmitry Baryshev <ksquirrel.iv@gmail.com>
Subject: Re: Bug #512138: wget cannot download some FTP urls
Date: Thu, 14 May 2009 12:39:50 -0700

Nicolas Duillier wrote:

> Nico Golde <nion@debian.org> [Sat, 17 Jan 2009 19:28:15 +0100]:
>> * Dmitry Baryshev <ksquirrel.iv@gmail.com> [2009-01-17 18:41]:
>>> $ LC_ALL=C wget [32]ftp://ftp.apnic.net/apnic/whois-data/APNIC/apnic.RPSL.db.gz
>>> --2009-01-17 19:34:45--
>>> [33]ftp://ftp.apnic.net/apnic/whois-data/APNIC/apnic.RPSL.db.gz
>>>            => `apnic.RPSL.db.gz'
>>> Resolving ftp.apnic.net... 202.12.29.20, 2001:dc0:2001:0:4608:20::
>>> Connecting to ftp.apnic.net|202.12.29.20|:21... connected.
>>> Logging in as anonymous ... Logged in!
>>> ==> SYST ... done.    ==> PWD ... done.
>>> ==> TYPE I ... done.  ==> CWD /apnic/whois-data/APNIC ...
>>> No such directory `apnic/whois-data/APNIC'.
>>>
>>> *
>>>
>>> Opera 9.53 and Firefox 3.0.5 download this file Ok.
>> Looks like wget doesn't follow symlinks.
>
> No.  It's an oddball FTP server that accepts the command
> "RETR /apnic/whois-data/APNIC/apnic.RPSL.db.gz" but rejects
> "CWD /apnic/whois-data/APNIC".
>
> Upstream, why does wget do CWD and RETR when just a
> RETR with the full pathname would suffice?

It's not terribly clear to me why Wget handles FTP in some of the ways it does. The answer is often that it was done to avoid problems with certain servers, whose workarounds may not always have been documented.

However, the relevant RFCs on FTP URLs actually require that it be interpreted as a series of CWDs followed by a RETR, so if we were following RFC 1738, wget should be doing:

CWD apnic
CWD whois-data
CWD APNIC
RETR apnic.RPSL.db.gz

(which would presumably run into the same problem). My preference would be to follow the RFC, and then if need be fall back to other approaches.

The right way to form a URL for a circumstance like this is probably to encode the slashes as %2f, so that a single RETR may result. The following worked for me:

wget ftp://ftp.apnic.net/apnic%2fwhois-data%2fAPNIC%2fapnic.RPSL.db.gz

Oddly, apparently even doing a CWD and then a CWD back into / prevents the RETR command from working... it really is an oddball server.

--
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
Maintainer of GNU Wget and GNU Teseq
http://micah.cowan.name/


http://bugs.debian.org/512138