bugGNU Wget - Bugs: bug #59954, wget assumes ftp server of type...


bug #59954: wget assumes ftp server of type UNIX knows LIST -a command while this is not always true

Submitter:  Arkadiusz Miskiewicz <arekm>
Submitted:  Wed 27 Jan 2021 10:03:24 AM UTC
Category:  Protocol Issue Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  rockdaboot
Originator Name:  Open/Closed:  Open
Release:  1.20 Operating System:  None
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Sun 31 Jan 2021 06:50:50 PM UTC, comment #3: 

So most servers either support -a or ignore it and still return a directory listing (tested your examples).

> Maybe wget could fallback to LIST (without -a) when there is no parsed result in the output, too?

I agree that this seems to be the most sensible way.

Tim Ruehsen <rockdaboot>
Group administrator
Sun 31 Jan 2021 06:28:21 PM UTC, comment #2: 

That ftp server interprets "-a" as file/directory argument and tries to list that.

Doesn't seem to be uncommon for ftp servers to do not return error on LIST on not existing files/directories.

But it is uncommon to not support "-a", that's why this problem doesn't occur so often with wget.

ftp.ubuntu.com - returns success (226) but sends empty list when trying to list not-existing directory
ftp.redhat.com - the same
ftp.pureftpd.org - the same
ftp.wayne.edu - the same
ftp.netbsd.org - the same

ftp.proftpd.com - returns error 450

so only one from above actually returns error. Fortunately all above support "-a" which makes this problem "hidden".

Maybe wget could fallback to LIST (without -a) when there is no parsed result in the output, too?

Don't have strong opinion about this but clearly almost no server returns error on non existing file/directory (which is "-a" file/directory in reported case as server doesn't support "-a" option).

Arkadiusz Miskiewicz <arekm>
Sun 31 Jan 2021 05:45:07 PM UTC, comment #1: 

Wget always tries 'LIST -a' first. If the server returns an error, wget tries 'LIST' as a fallback.

In your case, the server indicates an OK (226 Transfer complete) and puts a custom message as payload. This is good to be parsed by a human, but not by a program that relies on proper success/error codes.

IMO, this is clearly a server issue and should be reported there.

Tim Ruehsen <rockdaboot>
Group administrator
Wed 27 Jan 2021 10:03:24 AM UTC, original submission:  

wget -d ftp://ftp.sudo.ws/pub/sudo/

and there is no listing returned

that's because server doesn't support LIST -a while wget insists on using LIST -a

Server returns "ftpd: -a: No such file or directory" for LIST -a which wget doesn't notice as error.

$ ./wget -d ftp://ftp.sudo.ws/pub/sudo/
DEBUG output created by Wget 1.21.1 on linux-gnu.

Reading HSTS entries from /home/users/arekm/.wget-hsts
URI encoding = ‘UTF-8’
Converted file name 'index.html' (UTF-8) -> 'index.html' (UTF-8)
Using ‘.listing’ as listing tmp file.
--2021-01-27 10:46:45--  ftp://ftp.sudo.ws/pub/sudo/
           => ‘.listing’
Resolving ftp.sudo.ws (ftp.sudo.ws)...
Caching ftp.sudo.ws =>
Connecting to ftp.sudo.ws (ftp.sudo.ws)||:21... connected.
Created socket 3.
Releasing 0x000000000224f430 (new refcount 1).
220 xerxes.sudo.ws FTP server ready.
Logging in as anonymous ...
--> USER anonymous

331 Guest login ok, send your email address as password.

--> PASS -wget@

230-   Welcome to ftp.sudo.ws in sunny Boulder, CO.
230-   This is the master ftp site for the `sudo' sysadmin utility.
230-   For sudo, go to /pub/sudo
230-   For the sudo web page, visit https://www.sudo.ws/
230-   All transfers are logged with your host and email address.
230-   If you don't like this policy, disconnect now!
230-   Illegitimus Non Carborundum Est
230 Guest login ok, access restrictions apply.
Logged in!
==> SYST ...
--> SYST

215 UNIX Type: L8
UNIX TYPE L8: I know it and I will use "LIST -a" as standard list command
==> PWD ...
--> PWD

257 "/" is current directory.
==> TYPE I ...
--> TYPE I

200 Type set to I.
done.  changing working directory
Prepended initial PWD to relative path:
   pwd: '/'
   old: 'pub/sudo'
  new: '/pub/sudo'
==> CWD (1) /pub/sudo ...
--> CWD /pub/sudo

250 CWD command successful.
conaddr is:
==> PASV ...
--> PASV

227 Entering Passive Mode (65,102,237,118,231,91)
trying to connect to port 59227
Created socket 4.
done.    ==> LIST ...
--> LIST -a

150 Opening BINARY mode data connection for '-a'.
(skipping "LIST")done.

.listing                                                       [ <=>                                                                                                                                  ]      36  --.-KB/s    in 0s

Closed fd 4
226 Transfer complete.
2021-01-27 10:46:52 (4.17 MB/s) - ‘.listing’ saved [36]

LINIA: ftpd: -a: No such file or directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - it's my added printf("%s", line) to see what is being returned

UNKNOWN; perms 0;
Removed ‘.listing’.
Converted file name 'index.html' (UTF-8) -> 'index.html' (UTF-8)
Wrote HTML-ized index to ‘index.html’ [212].
Closed fd 3

Arkadiusz Miskiewicz <arekm>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by arekm (Submitted the item)

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-01-31 rockdaboot StatusInspected Confirmed
    2021-01-31 rockdaboot StatusNone Inspected
        Assigned toNone rockdaboot

    Back to the top

    Powered by Savane 3.13-c54d.
    Corresponding source code