bugGNU Wget - Bugs: bug #29503, cannot download some FTP URLs

 
 

bug #29503: cannot download some FTP URLs

Submitted by:  Noël Köthe <nok>
Submitted on:  Fri 09 Apr 2010 09:11:10 PM UTC  
 
Category: Protocol IssueSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: 1.12Operating System: GNU/Linux
Reproducibility: Every TimeFixed Release: None
Planned Release: NoneRegression: None
Work Required: NonePatch Included: No

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 09 Jul 2012 03:48:29 PM UTC, comment #1:

with 1.13.4. I cannot reproduce the problem with the given URL anymore:

wget ftp://ftp.apnic.net/apnic%2fwhois-data%2fAPNIC%2fapnic.RPSL.db.gz

Works as expected.

Regards

Noel

Noël Köthe <nok>
Fri 09 Apr 2010 09:11:10 PM UTC, original submission:

Hello,

this is a forward of a Debian bug report to the wget Bugtracker with an answer from Micah (reproduceable with wget 1.12).

From: Micah Cowan <micah@cowan.name>
To: Nicolas Duillier <nicolasduillier@googlemail.com>,
-unavailable-
Cc: Nico Golde <nion@debian.org>,
Dmitry Baryshev <ksquirrel.iv@gmail.com>
Subject: Re: Bug #512138: wget cannot download some FTP urls
Date: Thu, 14 May 2009 12:39:50 -0700

Nicolas Duillier wrote:

> Nico Golde <nion@debian.org> [Sat, 17 Jan 2009 19:28:15 +0100]:
>> * Dmitry Baryshev <ksquirrel.iv@gmail.com> [2009-01-17 18:41]:
>>> $ LC_ALL=C wget [32]ftp://ftp.apnic.net/apnic/whois-data/APNIC/apnic.RPSL.db.gz
>>> --2009-01-17 19:34:45--
>>> [33]ftp://ftp.apnic.net/apnic/whois-data/APNIC/apnic.RPSL.db.gz
>>> => `apnic.RPSL.db.gz'
>>> Resolving ftp.apnic.net... 202.12.29.20, 2001:dc0:2001:0:4608:20::
>>> Connecting to ftp.apnic.net|202.12.29.20|:21... connected.
>>> Logging in as anonymous ... Logged in!
>>> ==> SYST ... done. ==> PWD ... done.
>>> ==> TYPE I ... done. ==> CWD /apnic/whois-data/APNIC ...
>>> No such directory `apnic/whois-data/APNIC'.
>>>
>>> ***
>>>
>>> Opera 9.53 and Firefox 3.0.5 download this file Ok.
>> Looks like wget doesn't follow symlinks.
>
> No. It's an oddball FTP server that accepts the command
> "RETR /apnic/whois-data/APNIC/apnic.RPSL.db.gz" but rejects
> "CWD /apnic/whois-data/APNIC".
>
> Upstream, why does wget do CWD and RETR when just a
> RETR with the full pathname would suffice?


It's not terribly clear to me why Wget handles FTP in some of the ways it does. The answer is often that it was done to avoid problems with certain servers, whose workarounds may not always have been documented.

However, the relevant RFCs on FTP URLs actually require that it be interpreted as a series of CWDs followed by a RETR, so if we were following RFC 1738, wget should be doing:

CWD apnic
CWD whois-data
CWD APNIC
RETR apnic.RPSL.db.gz

(which would presumably run into the same problem). My preference would be to follow the RFC, and then if need be fall back to other approaches.

The right way to form a URL for a circumstance like this is probably to encode the slashes as %2f, so that a single RETR may result. The following worked for me:

wget ftp://ftp.apnic.net/apnic%2fwhois-data%2fAPNIC%2fapnic.RPSL.db.gz

Oddly, apparently even doing a CWD and then a CWD back into / prevents the RETR command from working... it really is an oddball server.

--
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
Maintainer of GNU Wget and GNU Teseq
http://micah.cowan.name/

http://bugs.debian.org/512138

Noël Köthe <nok>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #24103:  0001-Add-skip-cwd-option-for-fetching-ftp-urls.patch added by joeythesaint (4kB - text/x-patch - This seems to work for the servers where I was seeing this problem.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by joeythesaint (Updated the item)
  • -unavailable- added by nok (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 05 Oct 2011 03:52:58 PM UTCjoeythesaintAttached File-=>Added 0001-Add-skip-cwd-option-for-fetching-ftp-urls.patch, #24103

    Back to the top


    Powered by Savane 3.1-cleanup