Add a New Comment (Rich Markup)
wget is not designed in a way that multiple instances can write concurrently to the same set of files. In these cases, the behaviour is undefined.
Hello, I don't know if you aggree with the described problem but I forward it anyway to point you to this possible problem: http://bugs.debian.org/500395 --<8-- I downloaded all the aircraft for flightgear from their ftp mirrors. To spread the load over several mirrors I did: wget -nc ftp://ftp.kingmont.com/flightsims/flightgear/Aircraft/* wget -nc ftp://ftp.de.flightgear.org/pub/fgfs/Aircraft/* wget -nc ftp://some.third.mirror/pub/fgfs/Aircraft/* It worked basically as files started by one wget were not downloaded by other wgets. However, after one wget decided to download a locally non existant file it appearently created it only when the data started coming. In this interval - from decision to file creation - other wgets sometimes would also decide to download the same file and so two (or possibly three) wgets would download and write the same file and waste time and bandwidth. Maybe the destination file should be created right after wget decides to download it. Maybe these operations can be combined atomically to avoid the race completely. If it later turns out that the file cannot be downloaded, it can be deleted. That might be a new race as other wegts won't try to download a file which one wget fails to download. Bernhard --8<--
(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)
Attach Files: Comment:
No files currently attached
Depends on the following items: None found
Items that depend on this one: None found
Please enter the title of George Orwell's famous dystopian book (it's a date):
Follow 3 latest changes.
Copyright © 2023 Free Software Foundation, Inc. Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved. The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team. Source Code
Powered by Savane 3.11