bugGNU Wget - Bugs: bug #32054, Concurrent processes of wget on...

 
 

bug #32054: Concurrent processes of wget on ftp directories produces comingled files

Submitter:  None
Submitted:  Wed 05 Jan 2011 06:18:16 AM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Status:  Postponed
Privacy:  Public Assigned to:  gscrivano
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  1.11.1
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 25 Jul 2017 10:27:27 AM UTC, comment #4: 

This is a genuine bug and I just spent a day chasing it, starting from random failures in our tarball monitoring system.

If you have two wget processes started at the same time:

$ wget ftp://example.com/foo -O foo.html
$ wget ftp://example.com/bar -O bar.html

These will both write the FTP listing to .listing before processing it to generate HTML, with the expected races.

Sometimes wget will fail because .listing was deleted before they need it. Sometimes foo.html will contain the listing for /bar.

As this is a temporary file, can't wget just use mktemp?

Ross Burton <rossburton>
Tue 11 Jan 2011 06:02:27 PM UTC, comment #3: 

Note that the comingling of data does not happen with normal files.  It only happens when doing a wgets of a ftp directories - the html of the list of files in the directory.

Anonymous
Tue 11 Jan 2011 05:47:37 PM UTC, comment #2: 

I am positive that each instance of wget is outputting different files.  They are, in fact, in different directories by source.  There are locks in place to prevent multiple runs on the same source.

When I run the master script with a 10 second delay between calls of the slave scripts, everything works as expected.

Anonymous
Tue 11 Jan 2011 01:44:01 PM UTC, comment #1: 

wget is not designed to permit simultaneous concurrent accesses to the data.  Are you sure there are not multiple wget processes accessing the same files at the same time?

Giuseppe Scrivano <gscrivano>
Group administrator
Wed 05 Jan 2011 06:18:16 AM UTC, original submission:  

I apologize for not being able to give real info here.  All the specifics are proprietary info belonging to my company.

I have 2 scripts.  One calls another with different parameters, each is in the background.  The second script calls wget on an ftp location to get the list of files on the site.  It parses them and then downloads as appropriate.

Simplified example:
Each of these is being called by a separate instance of the secondary script running in the background.

wget --no-verbose 'ftp://foo.com/' -O '/downloads/foo/foo_listing.html' >foo.log
wget --no-verbose 'ftp://bar.com/' -O '/downloads/bar/bar_listing.html' >bar.log

When I run the secondary script once at a time, everything behaves as expected. I get an html file with a list of files, links to them, and information about the files the same way I would when viewing an ftp url through a browser.

Continued simplified one at a time (and expected) example results:

foo_listing.html:

...
<a href="ftp://foo.com/foo1.xml">foo1.xml</a> ...
<a href="ftp://foo.com/foo2.xml">foo2.xml</a> ...
...

bar_listing.html:

...
<a href="ftp://bar.com/bar3.xml">bar3.xml</a> ...
<a href="ftp://bar.com/bar4.xml">bar4.xml</a> ...
...

When I run the secondary script many times in the background, some of the resulting files, although they have the base urls correct (the one that was passed in) the files listed are from a different run of wget.

Continued simplified multiprocessing (and actual) example results:

foo_listing.html:

...
<a href="ftp://foo.com/bar3.xml">bar3.xml</a> ...
<a href="ftp://foo.com/bar4.xml">bar4.xml</a> ...
...

bar_listing.html
correct, as above

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rossburton (Posted a comment)
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-01-11 gscrivano StatusNone Postponed
        Assigned toNone gscrivano
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-f14d.
    Corresponding source code