bugGNU Wget - Bugs: bug #31653, In certain cases, wget drops into...

 
 

bug #31653: In certain cases, wget drops into infinite loop with error message "<filename> has sprung into existence"

Submitted by:  Evgeny Kapun <abacabadabacaba>
Submitted on:  Sun 14 Nov 2010 01:52:25 PM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: Fixed
Privacy: PublicAssigned to: Giuseppe Scrivano <gscrivano>
Originator Name: Open/Closed: Closed
Release: 1.12Operating System: None
Reproducibility: Every TimeFixed Release: None
Planned Release: NoneRegression: None
Work Required: NonePatch Included: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Mon 22 Nov 2010 05:28:59 PM UTC, comment #14:

I had to revert my last change, it doesn't do what it supposes to fix.

It seems the problem persists only when the response headers are not received, I am going to take a look at this other problem.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Mon 22 Nov 2010 01:09:16 PM UTC, comment #13:

Trying to resume the download if count > 0 is wrong sometimes.

Example:

  • Wget is started with -r or -p, so it should overwrite existing files.
  • Wget tries to download a file which already exists locally.
  • Connection terminates before response headers are received.
  • Wget tries again.
  • This time, count > 0, so wget appends to existing file instead of overwriting it.

This is what revision 2444 does.

Evgeny Kapun <abacabadabacaba>
Sun 21 Nov 2010 09:53:09 PM UTC, comment #12:

I see your point.

Thanks to have looked into this problem.

I am going to change the behaviour and try to resume the download when count > 0 instead of restart it.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Sun 21 Nov 2010 09:21:34 AM UTC, comment #11:

No, I didn't mean that.

-c switch means to continue getting partially-downloaded file which was present before wget was started. However, if a file was partially downloaded in the same wget process, then wget should attempt to resume download even if -c was not given.

Example:

  • Wget begins to download a 2GB file.
  • After 1GB has been downloaded, connection terminates.
  • Wget attempts to resume the download.
  • This time, connection terminates before response headers are received.
  • Now, wget discards 1GB of downloaded data and tries to fetch the whole file again. But it should try to resume the download instead.
Evgeny Kapun <abacabadabacaba>
Sat 20 Nov 2010 12:05:54 AM UTC, comment #10:

Thanks for your followup.

I haven't changed the -c semantic, if -c is not specified then wget doesn't attempt to resume the download. In other words, if the download fails, wget will attempt to fetch again the file.

Is it what you meant?

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Fri 19 Nov 2010 07:14:42 PM UTC, comment #9:

This patch has a problem: if connection terminates before response headers are received, then wget would discard everything downloaded so far and download the file from the beginning. I think it should try to continue downloading it instead.

Evgeny Kapun <abacabadabacaba>
Fri 19 Nov 2010 04:16:26 PM UTC, comment #8:

thanks for the clarification.

I have committed this patch:

=== modified file 'src/ftp.c'
--- src/ftp.c 2010-10-11 20:49:14 +0000
+++ src/ftp.c 2010-11-19 15:36:32 +0000
@@ -241,7 +241,7 @@
and closes the control connection in case of error. */
static uerr_t
getftp (struct url u, wgint passed_expected_bytes, wgint qtyread,
- wgint restval, ccon *con)
+ wgint restval, ccon *con, int count)
{
int csock, dtsock, local_sock, res;
uerr_t err = RETROK; /* appease the compiler */
@@ -1172,7 +1172,7 @@
#endif /* def __VMS [else] */
}
else if (opt.noclobber || opt.always_rest || opt.timestamping || opt.dirstruct
- || opt.output_document)
+ || opt.output_document || count > 0)
{
if (opt.unlink && file_exists_p (con->target))
{
@@ -1488,7 +1488,7 @@
len = f->size;
else
len = 0;
- err = getftp (u, len, &qtyread, restval, con);
+ err = getftp (u, len, &qtyread, restval, con, count);

if (con->csock == -1)
con->st &= ~DONE_CWD;

=== modified file 'src/http.c'
--- src/http.c 2010-10-16 01:04:25 +0000
+++ src/http.c 2010-11-19 15:41:34 +0000
@@ -1494,7 +1494,7 @@
server, and u->url will be requested. */
static uerr_t
gethttp (struct url u, struct http_stat hs, int dt, struct url proxy,
- struct iri *iri)
+ struct iri *iri, int count)
{
struct request *req;

@@ -2479,7 +2479,7 @@
fp = fopen (hs->local_file, "ab");
#endif /* def __VMS [else] */
}
- else if (ALLOW_CLOBBER)
+ else if (ALLOW_CLOBBER || count > 0)
{
if (opt.unlink && file_exists_p (hs->local_file))
{
@@ -2756,7 +2756,7 @@
*dt &= ~SEND_NOCACHE;

/* Try fetching the document, or at least its head. */
- err = gethttp (u, &hstat, dt, proxy, iri);
+ err = gethttp (u, &hstat, dt, proxy, iri, count);

/* Time? */
tms = datetime_str (time (NULL));

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Fri 19 Nov 2010 02:21:15 PM UTC, comment #7:

The scenario is as follows:

  1. Wget is told to download a file http://example.org/example.
Evgeny Kapun <abacabadabacaba>
Fri 19 Nov 2010 02:06:01 PM UTC, comment #6:

The problem is not that wget tries to download a file infinitely many times. The problem is that wget sometimes prints error message "<filename> has sprung into existence" for files which it has created.

Evgeny Kapun <abacabadabacaba>
Fri 19 Nov 2010 01:02:30 PM UTC, comment #5:

you can specify --tries if you want to limit the number of attempts wget will do.

By default, wget tries unlimited times until the file is fetched.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Fri 19 Nov 2010 12:24:51 PM UTC, comment #4:

I'm using wget 1.12 on Ubuntu (Ubuntu package version: wget 1.12-2.1ubuntu1).

It's strange that in your case wget downloads a file without any errors. As you may see by reading the script, it doesn't send response body for the first time, so wget should complain that connection was closed too early and retry.

Anyway, you may try second version of the test case, which should work even if somebody else is trying to connect to port 8000 of your machine while testing.

(file #22059)

Evgeny Kapun <abacabadabacaba>
Fri 19 Nov 2010 11:27:18 AM UTC, comment #3:

still I wasn't able to reproduce this problem (I have tried your sequence several times), can you tell me which version of wget are you using?

Thanks.

$ ./wget-test &
$ LANG=en wget http://localhost:8000/
--2010-11-19 12:24:54-- http://localhost:8000/
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1
Saving to: `index.html.4'

100%[====================================================================================================>] 1 --.-K/s in 0s

2010-11-19 12:24:54 (160 KB/s) - `index.html.4' saved [1/1]

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Mon 15 Nov 2010 01:41:43 PM UTC, comment #2:

You need to kill and restart the script every time you want to do a test. Here's my output:

Evgeny Kapun <abacabadabacaba>
Mon 15 Nov 2010 01:23:36 PM UTC, comment #1:

thanks for your test case. I have tried it but I couldn't reproduce the problem:

$ LANG=EN wget http://localhost:8000
--2010-11-15 14:22:40-- http://localhost:8000/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8000... failed: Connection refused.
Connecting to localhost|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1
Saving to: `index.html'

100%[===================================================================================>] 1 --.-K/s in 0s

2010-11-15 14:22:40 (26.2 KB/s) - `index.html' saved [1/1]

can you please provide more information? Is what I did the right way to test it?

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Sun 14 Nov 2010 01:52:25 PM UTC, original submission:

If tries to download a file over unstable connection, it can sometimes create the file but then forget about it. After that, wget can't download the file. Instead, it prints error message "<filename> has sprung into existence" and tries again, and again, and so on.

Evgeny Kapun <abacabadabacaba>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #22059:  wget-test added by abacabadabacaba (162B - application/octet-stream - A script to reproduce the bug, version 2)
file #22017:  wget-test added by abacabadabacaba (252B - application/octet-stream - A script to reproduce the bug)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by abacabadabacaba (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 19 Nov 2010 04:16:26 PM UTCgscrivanoStatusInspected=>Fixed
      Open/ClosedOpen=>Closed
    Fri 19 Nov 2010 01:02:30 PM UTCgscrivanoStatusNone=>Inspected
      Assigned toNone=>gscrivano
    Fri 19 Nov 2010 12:24:51 PM UTCabacabadabacabaAttached File-=>Added wget-test, #22059
    Sun 14 Nov 2010 01:52:25 PM UTCabacabadabacabaAttached File-=>Added wget-test, #22017

    Back to the top


    Powered by Savane 3.1-cleanup