bugGNU Wget - Bugs: bug #30479, Failed to resume when downloading...

 
 

bug #30479: Failed to resume when downloading symlinked files via ftp at NcFTPd

Submitted by:  None
Submitted on:  Sun 18 Jul 2010 09:23:00 PM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: Fixed
Privacy: PublicAssigned to: Giuseppe Scrivano <gscrivano>
Originator Name: Leonid PetrovOriginator Email: -unavailable-
Open/Closed: ClosedRelease: 1.12
Operating System: GNU/LinuxReproducibility: Every Time
Fixed Release: NonePlanned Release: None
Regression: NoWork Required: None
Patch Included: Yes

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 20 Jul 2010 10:48:57 AM UTC, comment #1:

I have slightly changed your patch.

I am going to commit this one:

=== modified file 'src/ftp.c'
--- src/ftp.c 2010-07-14 21:06:20 +0000
+++ src/ftp.c 2010-07-20 10:39:12 +0000
@@ -767,6 +767,16 @@
number_to_static_string (expected_bytes));
}

+ if (cmd & DO_RETR && restval > 0 && restval == expected_bytes)
+ {
+ /* Server confirms that file has length restval. We should stop now.
+ Some servers (f.e. NcFTPd) return error when receive REST 0 */
+ logputs (LOG_VERBOSE, "File has already been retrieved \n");
+ fd_close (csock);
+ con->csock = -1;
+ return RETRFINISHED;
+ }
+
/* If anything is to be retrieved, PORT (or PASV) must be sent. */
if (cmd & (DO_LIST | DO_RETR))
{
@@ -1459,7 +1469,7 @@
xfree (hurl);
}
/* Send getftp the proper length, if fileinfo was provided. */
- if (f)
+ if (f && f->type != FT_SYMLINK)
len = f->size;
else
len = 0;

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Sun 18 Jul 2010 09:23:00 PM UTC, original submission:

Dear wgetters,

Thank you for supporting one of the most useful Internet applications.
I came across two problem when restarting the recursive download of symlinked files from NcFTPd Server:

1) wget 1.12 tries to restart even in the case if the file has been completely
downloaded. It issues command restart 0 . Some ftp servers. f.e.
pureftp accepts such a command, NcFTPd raises exception and spits out
"450 Error occurred after 0 of 0 bytes were sent: Unexpected EOF".

As a results, wget falls into the infinite loop.

How to reproduce:

run
wget -c -nH --tries=2 --ftp-user=ftp --password=wget@ --cut-dirs=5 ftp://ladsweb.nascom.nasa.gov/pub/../allData/5/MODATML2/2006/219/MODATML2.A2006219.1120.005.2006221022157.hdf

two times

at the second attempt I see

--2010-07-16 14:42:07-- ftp://ladsweb.nascom.nasa.gov/allData/5/MODATML2/2006/219/MODATML2.A2006219.1120.005.2006221022157.hdf
=> `MODATML2.A2006219.1120.005.2006221022157.hdf'
Resolving ladsweb.nascom.nasa.gov... 198.118.194.40
Connecting to ladsweb.nascom.nasa.gov|198.118.194.40|:21... connected.
Logging in as ftp ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/allData/5/MODATML2/2006/219 ... done.
==> SIZE MODATML2.A2006219.1120.005.2006221022157.hdf ... 2179064
==> PASV ... done. ==> REST 2179064 ... done.
==> RETR MODATML2.A2006219.1120.005.2006221022157.hdf ... done.
Length: 2179064 (2.1M), 0 (0) remaining

0K ,,,,,,,, ,,,,,,,, ,,,,,,,, ,,,,,,,, , 100% 0.00 =0s

2010-07-16 14:42:07 (0.00 B/s) - Data transfer aborted.
Retrying.

--2010-07-16 14:42:08-- ftp://ladsweb.nascom.nasa.gov/allData/5/MODATML2/2006/219/MODATML2.A2006219.1120.005.2006221022157.hdf
(try: 2) => `MODATML2.A2006219.1120.005.2006221022157.hdf'
==> CWD not required.
==> SIZE MODATML2.A2006219.1120.005.2006221022157.hdf ... 2179064
==> PASV ... done. ==> REST 2179064 ... done.
==> RETR MODATML2.A2006219.1120.005.2006221022157.hdf ... done.
Length: 2179064 (2.1M), 0 (0) remaining

0K ,,,,,,,, ,,,,,,,, ,,,,,,,, ,,,,,,,, , 100% 0.00 =0s

2010-07-16 14:42:08 (0.00 B/s) - Data transfer aborted.
Giving up.

2) During recursive download of symlinked files, if connection was lost
or aborted, wget fails to restart. The reason is it takes the file
size from the listing. But the listing contains the length of the symbolic
link, not the length of the file! As a result, it sets a negative offset
for restart. Ftp server returns "501 Syntax error in parameters", and
wget falls into an infinite loop.

How to reproduce.

Run this command two times

wget -r -c -nH --ftp-user=ftp --password=wget@ --retr-symlinks --cut-dirs=4 ftp://ladsweb.nascom.nasa.gov/pub/../allData/5/MODATML2/2006/219/

At the second time I get this:

--2010-07-16 14:47:18-- ftp://ladsweb.nascom.nasa.gov/allData/5/MODATML2/2006/219/MODATML2.A2006219.0000.005.2006220220702.hdf
(try: 2) => `219/MODATML2.A2006219.0000.005.2006220220702.hdf'
==> CWD not required.
==> PASV ... done. ==> REST 2040199 ... done.
==> RETR MODATML2.A2006219.0000.005.2006220220702.hdf ... done.
Length: 78, -2040121 (-204012) remaining

100%[+++++++++++++++++++++++++++++++++++++++] 2,040,199 --.-K/s in 0s

2010-07-16 14:47:18 (0.00 B/s) - Data transfer aborted.
Retrying.

infinite loop :-(

==========================================================================

Patch:

--- src/ftp.c.orig 2009-09-21 22:59:21.000000000 -0400
+++ src/ftp.c 2010-07-16 12:59:34.542415589 -0400
@@ -765,6 +765,15 @@
number_to_static_string (expected_bytes));
}

+ if ( cmd & DO_RETR && restval > 0 && restval == expected_bytes ){
+ /* Server confirms that file has length restval. We should stop now.
+ Some servers (f.e. NcFTPd) return error when receive REST 0 */
+ logputs (LOG_VERBOSE, "File has already been retrieved \n");
+ fd_close (csock);
+ con->csock = -1;
+ return RETRFINISHED;
+ }
+
/* If anything is to be retrieved, PORT (or PASV) must be sent. */
if (cmd & (DO_LIST | DO_RETR))
{
@@ -1455,6 +1464,9 @@
/* Send getftp the proper length, if fileinfo was provided. */
if (f)
len = f->size;
+ /* NB: if the file is symlink, then f->size keeps the size of the symlink,
+ not the size of the file! */
+ if ( f->type == FT_SYMLINK ) len = 0 ;
else
len = 0;
err = getftp (u, len, &qtyread, restval, con);

Does this patch break anything?

Sincerely,
Leonid

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gscrivano (Posted a comment)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 20 Jul 2010 12:19:46 PM UTCgscrivanoStatusNone=>Fixed
      Open/ClosedOpen=>Closed
    Tue 20 Jul 2010 10:48:57 AM UTCgscrivanoAssigned toNone=>gscrivano

    Back to the top


    Powered by Savane 3.1-cleanup1