bugGNU Wget - Bugs: bug #31653, In certain cases, wget drops into...

 
 

bug #31653: In certain cases, wget drops into infinite loop with error message "<filename> has sprung into existence"

Submitter:  Evgeny Kapun <abacabadabacaba>
Submitted:  Sun 14 Nov 2010 01:52:25 PM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  gscrivano
Originator Name:  Open/Closed:  Closed
Release:  1.12 Operating System:  None
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 22 Nov 2010 05:28:59 PM UTC, comment #14: 

I had to revert my last change, it doesn't do what it supposes to fix.

It seems the problem persists only when the response headers are not received, I am going to take a look at this other problem.

Giuseppe Scrivano <gscrivano>
Group administrator
Mon 22 Nov 2010 01:09:16 PM UTC, comment #13: 

Trying to resume the download if count > 0 is wrong sometimes.

Example:

  • Wget is started with -r or -p, so it should overwrite existing files.
  • Wget tries to download a file which already exists locally.
  • Connection terminates before response headers are received.
  • Wget tries again.
  • This time, count > 0, so wget appends to existing file instead of overwriting it.


This is what revision 2444 does.

Evgeny Kapun <abacabadabacaba>
Sun 21 Nov 2010 09:53:09 PM UTC, comment #12: 

I see your point.

Thanks to have looked into this problem.

I am going to change the behaviour and try to resume the download when count > 0 instead of restart it.

Giuseppe Scrivano <gscrivano>
Group administrator
Sun 21 Nov 2010 09:21:34 AM UTC, comment #11: 

No, I didn't mean that.

-c switch means to continue getting partially-downloaded file which was present before wget was started. However, if a file was partially downloaded in the same wget process, then wget should attempt to resume download even if -c was not given.

Example:

  • Wget begins to download a 2GB file.
  • After 1GB has been downloaded, connection terminates.
  • Wget attempts to resume the download.
  • This time, connection terminates before response headers are received.
  • Now, wget discards 1GB of downloaded data and tries to fetch the whole file again. But it should try to resume the download instead.
Evgeny Kapun <abacabadabacaba>
Sat 20 Nov 2010 12:05:54 AM UTC, comment #10: 

Thanks for your followup.

I haven't changed the -c semantic, if -c is not specified then wget doesn't attempt to resume the download.  In other words, if the download fails, wget will attempt to fetch again the file.

Is it what you meant?

Giuseppe Scrivano <gscrivano>
Group administrator
Fri 19 Nov 2010 07:14:42 PM UTC, comment #9: 

This patch has a problem: if connection terminates before response headers are received, then wget would discard everything downloaded so far and download the file from the beginning. I think it should try to continue downloading it instead.

Evgeny Kapun <abacabadabacaba>
Fri 19 Nov 2010 04:16:26 PM UTC, comment #8: 

thanks for the clarification.

I have committed this patch:

=== modified file 'src/ftp.c'
--- src/ftp.c 2010-10-11 20:49:14 +0000
+++ src/ftp.c 2010-11-19 15:36:32 +0000
@@ -241,7 +241,7 @@
    and closes the control connection in case of error.  */
 static uerr_t
 getftp (struct url *u, wgint passed_expected_bytes, wgint *qtyread,
-        wgint restval, ccon *con)
+        wgint restval, ccon *con, int count)
 {
   int csock, dtsock, local_sock, res;
   uerr_t err = RETROK;          /* appease the compiler */
@@ -1172,7 +1172,7 @@
 #endif /* def __VMS [else] */
         }
       else if (opt.noclobber || opt.always_rest || opt.timestamping || opt.dirstruct
-               || opt.output_document)
+               || opt.output_document || count > 0)
         {  
    if (opt.unlink && file_exists_p (con->target))
      {
@@ -1488,7 +1488,7 @@
         len = f->size;
       else
         len = 0;
-      err = getftp (u, len, &qtyread, restval, con);
+      err = getftp (u, len, &qtyread, restval, con, count);
 
       if (con->csock == -1)
         con->st &= ~DONE_CWD;

=== modified file 'src/http.c'
--- src/http.c 2010-10-16 01:04:25 +0000
+++ src/http.c 2010-11-19 15:41:34 +0000
@@ -1494,7 +1494,7 @@
    server, and u->url will be requested.  */
 static uerr_t
 gethttp (struct url *u, struct http_stat *hs, int *dt, struct url *proxy,
-         struct iri *iri)
+         struct iri *iri, int count)
 {
   struct request *req;
 
@@ -2479,7 +2479,7 @@
           fp = fopen (hs->local_file, "ab");
 #endif /* def __VMS [else] */
         }
-      else if (ALLOW_CLOBBER)
+      else if (ALLOW_CLOBBER || count > 0)
         {
    if (opt.unlink && file_exists_p (hs->local_file))
      {
@@ -2756,7 +2756,7 @@
         *dt &= ~SEND_NOCACHE;
 
       /* Try fetching the document, or at least its head.  */
-      err = gethttp (u, &hstat, dt, proxy, iri);
+      err = gethttp (u, &hstat, dt, proxy, iri, count);
 
       /* Time?  */
       tms = datetime_str (time (NULL));


Giuseppe Scrivano <gscrivano>
Group administrator
Fri 19 Nov 2010 02:21:15 PM UTC, comment #7: 

The scenario is as follows:

  1. Wget is told to download a file http://example.org/example.
  2. Wget begins to download it to local file +verbatim+example-verbatim-.
  3. Before file is fully downloaded, connection terminates.
  4. At this point, local file +verbatim+example-verbatim- exists and contains partially downloaded data.
  5. Wget tries to continue download, but this time connection terminates before reply headers are received.
  6. At this point, wget forgets that local file +verbatim+example-verbatim- exists.
  7. Wget tries again to download the file.
  8. Because local file +verbatim+example-verbatim- exists, wget prints "example has sprung into existence" and tries again.
  9. Now, user has to manually delete file +verbatim+example-verbatim-, otherwise wget will never download it.
Evgeny Kapun <abacabadabacaba>
Fri 19 Nov 2010 02:06:01 PM UTC, comment #6: 

The problem is not that wget tries to download a file infinitely many times. The problem is that wget sometimes prints error message "<filename> has sprung into existence" for files which it has created.

Evgeny Kapun <abacabadabacaba>
Fri 19 Nov 2010 01:02:30 PM UTC, comment #5: 

you can specify --tries if you want to limit the number of attempts wget will do.

By default, wget tries unlimited times until the file is fetched.

Giuseppe Scrivano <gscrivano>
Group administrator
Fri 19 Nov 2010 12:24:51 PM UTC, comment #4: 

I'm using wget 1.12 on Ubuntu (Ubuntu package version: wget 1.12-2.1ubuntu1).

It's strange that in your case wget downloads a file without any errors. As you may see by reading the script, it doesn't send response body for the first time, so wget should complain that connection was closed too early and retry.

Anyway, you may try second version of the test case, which should work even if somebody else is trying to connect to port 8000 of your machine while testing.

(file #22059)

Evgeny Kapun <abacabadabacaba>
Fri 19 Nov 2010 11:27:18 AM UTC, comment #3: 

still I wasn't able to reproduce this problem (I have tried your sequence several times), can you tell me which version of wget are you using?

Thanks.


$ ./wget-test &
$ LANG=en wget http://localhost:8000/
--2010-11-19 12:24:54--  http://localhost:8000/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1
Saving to: `index.html.4'

100%[====================================================================================================>] 1           --.-K/s   in 0s     

2010-11-19 12:24:54 (160 KB/s) - `index.html.4' saved [1/1]


Giuseppe Scrivano <gscrivano>
Group administrator
Mon 15 Nov 2010 01:41:43 PM UTC, comment #2: 

You need to kill and restart the script every time you want to do a test. Here's my output:


$ ./wget-test &
[1] 7494
$ wget http://localhost:8000/
--2010-11-15 13:34:53--  http://localhost:8000/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1
Saving to: `index.html'

 0% [                                                                                                     ] 0           --.-K/s   in 0s

2010-11-15 13:34:53 (0.00 B/s) - Connection closed at byte 0. Retrying.

--2010-11-15 13:34:54--  (try: 2)  http://localhost:8000/
Connecting to localhost|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2010-11-15 13:34:56--  (try: 3)  http://localhost:8000/
Connecting to localhost|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1
index.html has sprung into existence.
Retrying.

--2010-11-15 13:34:59--  (try: 4)  http://localhost:8000/
Connecting to localhost|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1
index.html has sprung into existence.
Retrying.

--2010-11-15 13:35:03--  (try: 5)  http://localhost:8000/
Connecting to localhost|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1
index.html has sprung into existence.
Retrying.

^C
$ wget --version
GNU Wget 1.12 built on linux-gnu.

+digest +ipv6 +nls +ntlm +opie +md5/openssl +https -gnutls +openssl
-iri

Wgetrc:
    /etc/wgetrc (system)
Locale: /usr/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
    -DLOCALEDIR="/usr/share/locale" -I. -I../lib -g -O2
    -D_FILE_OFFSET_BITS=64 -O2 -g -Wall
Link: gcc -g -O2 -D_FILE_OFFSET_BITS=64 -O2 -g -Wall -Wl,-Bsymbolic-functions
    /usr/lib/libssl.so /usr/lib/libcrypto.so -Wl,-rpath -Wl,/usr/lib
    -ldl -lrt ftp-opie.o openssl.o http-ntlm.o gen-md5.o
    ../lib/libgnu.a

Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Currently maintained by Micah Cowan <micah@cowan.name>.
Please send bug reports and questions to <bug-wget@gnu.org>.


Evgeny Kapun <abacabadabacaba>
Mon 15 Nov 2010 01:23:36 PM UTC, comment #1: 

thanks for your test case.  I have tried it but I couldn't reproduce the problem:

$ LANG=EN wget  http://localhost:8000
--2010-11-15 14:22:40--  http://localhost:8000/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8000... failed: Connection refused.
Connecting to localhost|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1
Saving to: `index.html'

100%[===================================================================================>] 1           --.-K/s   in 0s     

2010-11-15 14:22:40 (26.2 KB/s) - `index.html' saved [1/1]



can you please provide more information?  Is what I did the right way to test it?

Giuseppe Scrivano <gscrivano>
Group administrator
Sun 14 Nov 2010 01:52:25 PM UTC, original submission:  

If tries to download a file over unstable connection, it can sometimes create the file but then forget about it. After that, wget can't download the file. Instead, it prints error message "<filename> has sprung into existence" and tries again, and again, and so on.

To reproduce, run the attached script and execute +verbatim+wget http://localhost:8000/-verbatim-. The script emulates a server which sometimes closes connection prematurely, which may happen if network connection is unstable.

Evgeny Kapun <abacabadabacaba>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #22059:  wget-test added by abacabadabacaba (162B - application/octet-stream - A script to reproduce the bug, version 2)
file #22017:  wget-test added by abacabadabacaba (252B - application/octet-stream - A script to reproduce the bug)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by abacabadabacaba (Submitted the item)
  •  

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-11-19 gscrivano StatusInspected Fixed
        Open/ClosedOpen Closed
    2010-11-19 gscrivano StatusNone Inspected
        Assigned toNone gscrivano
    2010-11-19 abacabadabacaba Attached File- Added wget-test, #22059
    2010-11-14 abacabadabacaba Attached File- Added wget-test, #22017

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code