bugGNU Wget - Bugs: bug #64714, --no-clobber not working with...


bug #64714: --no-clobber not working with --mirror??

Submitter:  None
Submitted:  Sun 24 Sep 2023 04:17:55 PM UTC
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  trunk
Operating System:  GNU/Linux Reproducibility:  None
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Sun 24 Sep 2023 04:17:55 PM UTC, original submission:  

Hi, I'm not entirely certain that the behavior I'm seeing is a bug and not me using it incorrectly. But it definitely is not intuitive.

I tried to mirror multiple sites into the same folder, as  wanted to be able to have them reference each other but get "deeper" at some of the referenced pages and not as deep on others, so I thought I "just" delete the index.html file of these pages and re-invoke wget with --mirror again to mirror that webpage as well and write it into that place (so that the reference from the one mirrored before would still work)

However, even though I specified --no-clobber, wget sometimes overwrote already downloaded and adjusted webpages with a non adjusted version from the server. It looks like this is some kind of recursion issue.

When pages are cross linked linke this:

a.com/index.html => Links to a.com/page2.html which has an out link to b.com/page3.html the first invokation will download all three pages but no out-links of b.com/page3.html (so the created page3.html file will have the original links in it).

b.com/page3.html now has a backlink to a.com/index.html, the 1st invokation of wget tasked to downlaod a.com/index.html doesn't care about this, and may even correctly adjust the backlink to a.com/index.html. HOWEVER when the local copy of "b.com/page3.html" is deleted and wget is invoked a 2nd time and tasked to now only download "b.com/page3.html" potentially with different arguments (like a specified recursion depth or with an option to not download any out-links, or a domain restriction) it'll sometimes overwrite a.com/index.html with a version that no longer has relative (adjusted) urls, but the original non-adjusted ones, effectively breaking the local copy of the the page. Even though "--no-clobber" was specified.

The full commands I used were:

  • `wget --mirror --recursive=on --level=1 --convert-links --adjust-extension --page-requisites --span-hosts --no-clobber -e robots=off`
  • `wget --mirror --http-user=user --http-password=pass --recursive=on --level=1 --convert-links --adjust-extension --page-requisites --span-hosts --no-clobber -e robots=off`
  • `wget --mirror --recursive=on --level=0 --convert-links --adjust-extension --page-requisites --no-parent --no-clobber -e robots=off`

(All three with different target URLs, but each time I first deleted the local index.html to get a "deeper" copy of that subtree of linked-web-pages)



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


CC list is empty


No changes have been made to this item

Back to the top

Powered by Savane 3.13-c54d.
Corresponding source code