bugGNU Wget - Bugs: bug #33825, 'wget -nc' race condition

 
 

bug #33825: 'wget -nc' race condition

Submitted by:  Noël Köthe <nok>
Submitted on:  Sat 23 Jul 2011 08:42:56 AM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: Wont Fix
Privacy: PublicAssigned to: Giuseppe Scrivano <gscrivano>
Originator Name: Open/Closed: Open
Release: 1.12Operating System: GNU/Linux
Reproducibility: NoneFixed Release: None
Planned Release: NoneRegression: None
Work Required: NonePatch Included: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Wed 03 Aug 2011 09:04:42 PM UTC, comment #1:

wget is not designed in a way that multiple instances can write concurrently to the same set of files. In these cases, the behaviour is undefined.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Sat 23 Jul 2011 08:42:56 AM UTC, original submission:

Hello,

I don't know if you aggree with the described problem but I forward it anyway to point you to this possible problem:

http://bugs.debian.org/500395

--<8--
I downloaded all the aircraft for flightgear from their ftp mirrors. To spread the
load over several mirrors I did:

wget -nc ftp://ftp.kingmont.com/flightsims/flightgear/Aircraft/*
wget -nc ftp://ftp.de.flightgear.org/pub/fgfs/Aircraft/*
wget -nc ftp://some.third.mirror/pub/fgfs/Aircraft/*

It worked basically as files started by one wget were not downloaded by other
wgets. However, after one wget decided to download a locally non existant file it
appearently created it only when the data started coming. In this interval - from
decision to file creation - other wgets sometimes would also decide to download
the same file and so two (or possibly three) wgets would download and write the
same file and waste time and bandwidth.

Maybe the destination file should be created right after wget decides to download
it. Maybe these operations can be combined atomically to avoid the race
completely.

If it later turns out that the file cannot be downloaded, it can be deleted.

That might be a new race as other wegts won't try to download a file which one
wget fails to download.

Bernhard
--8<--

Noël Köthe <nok>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by nok (Submitted the item)
  • -unavailable- added by nok (Debian BTS)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 03 Aug 2011 09:04:42 PM UTCgscrivanoStatusNone=>Wont Fix
      Assigned toNone=>gscrivano
    Sat 23 Jul 2011 08:42:56 AM UTCnokCarbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup