bugGNU Wget - Bugs: bug #29503, cannot download some FTP URLs


bug #29503: cannot download some FTP URLs

Submitter:  Noël Köthe <nok>
Submitted:  Fri 09 Apr 2010 09:11:10 PM UTC
Category:  Protocol Issue Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.12 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup

Mon 09 Jul 2012 03:48:29 PM UTC, comment #1: 

with 1.13.4. I cannot reproduce the problem with the given URL anymore:

wget ftp://ftp.apnic.net/apnic%2fwhois-data%2fAPNIC%2fapnic.RPSL.db.gz

Works as expected.



Noël Köthe <nok>
Fri 09 Apr 2010 09:11:10 PM UTC, original submission:  


this is a forward of a Debian bug report to the wget Bugtracker with an answer from Micah (reproduceable with wget 1.12).

From: Micah Cowan <micah@cowan.name>
To: Nicolas Duillier <nicolasduillier@googlemail.com>,
Cc: Nico Golde <nion@debian.org>,
 Dmitry Baryshev <ksquirrel.iv@gmail.com>
Subject: Re: Bug #512138: wget cannot download some FTP urls
Date: Thu, 14 May 2009 12:39:50 -0700

Nicolas Duillier wrote:

> Nico Golde <nion@debian.org> [Sat, 17 Jan 2009 19:28:15 +0100]:
>> * Dmitry Baryshev <ksquirrel.iv@gmail.com> [2009-01-17 18:41]:
>>> $ LC_ALL=C wget [32]ftp://ftp.apnic.net/apnic/whois-data/APNIC/apnic.RPSL.db.gz
>>> --2009-01-17 19:34:45--
>>> [33]ftp://ftp.apnic.net/apnic/whois-data/APNIC/apnic.RPSL.db.gz
>>>            => `apnic.RPSL.db.gz'
>>> Resolving ftp.apnic.net..., 2001:dc0:2001:0:4608:20::
>>> Connecting to ftp.apnic.net||:21... connected.
>>> Logging in as anonymous ... Logged in!
>>> ==> SYST ... done.    ==> PWD ... done.
>>> ==> TYPE I ... done.  ==> CWD /apnic/whois-data/APNIC ...
>>> No such directory `apnic/whois-data/APNIC'.
>>> *
>>> Opera 9.53 and Firefox 3.0.5 download this file Ok.
>> Looks like wget doesn't follow symlinks.
> No.  It's an oddball FTP server that accepts the command
> "RETR /apnic/whois-data/APNIC/apnic.RPSL.db.gz" but rejects
> "CWD /apnic/whois-data/APNIC".
> Upstream, why does wget do CWD and RETR when just a
> RETR with the full pathname would suffice?

It's not terribly clear to me why Wget handles FTP in some of the ways it does. The answer is often that it was done to avoid problems with certain servers, whose workarounds may not always have been documented.

However, the relevant RFCs on FTP URLs actually require that it be interpreted as a series of CWDs followed by a RETR, so if we were following RFC 1738, wget should be doing:

CWD apnic
CWD whois-data
RETR apnic.RPSL.db.gz

(which would presumably run into the same problem). My preference would be to follow the RFC, and then if need be fall back to other approaches.

The right way to form a URL for a circumstance like this is probably to encode the slashes as %2f, so that a single RETR may result. The following worked for me:

wget ftp://ftp.apnic.net/apnic%2fwhois-data%2fAPNIC%2fapnic.RPSL.db.gz

Oddly, apparently even doing a CWD and then a CWD back into / prevents the RETR command from working... it really is an oddball server.

Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
Maintainer of GNU Wget and GNU Teseq


Noël Köthe <nok>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #24103:  0001-Add-skip-cwd-option-for-fetching-ftp-urls.patch added by joeythesaint (4KiB - text/x-patch - This seems to work for the servers where I was seeing this problem.)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by joeythesaint (Updated the item)
  • -email is unavailable- added by nok (Submitted the item)

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-10-05 joeythesaint Attached File- Added 0001-Add-skip-cwd-option-for-fetching-ftp-urls.patch, #24103

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code