bugGNU Wget - Bugs: bug #25340, --mirror and --convert-links...


bug #25340: --mirror and --convert-links mixing poorly?

Submitted by:  Micah Cowan <micahcowan>
Submitted on:  Fri 16 Jan 2009 12:24:58 AM UTC  
Category: Program LogicSeverity: 3 - Normal
Priority: 4Status: Needs Investigation
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: 1.10.2Operating System: None
Reproducibility: NoneFixed Release: None
Planned Release: 1.14Regression: None
Work Required: 1 - DaysPatch Included: None

Add a New Comment (Rich MarkupRich Markup):

You are not logged in

Please log in, so followups can be emailed to you.


Fri 02 Oct 2009 07:47:54 PM UTC, comment #3:

Well, I suppose that's not entirely accurate... -N should imply that the file we timestamped and decided shouldn't be (re-)downloaded corresponds to that URL, and we should be able to parse the links exactly as we found them on the remote site.

With --convert-links, though, we can run into problems if the setting of --restrict-file-names or some other option caused the local file name to differ from the remote URL; it can be difficult to unmunge the local name to obtain the "real" relative URL that should be used to query the server. So, yeah, still easiest to wait for the SIDB.

Micah Cowan <micahcowan>
Project Administrator
Fri 02 Oct 2009 07:44:08 PM UTC, comment #2:

Well, that's obviously exactly what I'd expect -nc to do...

As to the behavior on -N, there's little we can do about that, since it can be difficult to know if and where a previous run had stored files locally that correspond to URLs. There are heuristics available, but...

We won't be fixing this until we have something like the Session Info Database implemented.

Micah Cowan <micahcowan>
Project Administrator
Sun 12 Apr 2009 02:37:41 PM UTC, comment #1:

I can confirm this bug. It seems to occur when -k is used with either -N or -nc, a checked file is not downloaded again, and some link on that file/page has a href that is not valid on the server anymore, such as the ones created with -E or -H. A simpler test case:

wget http://xkcd.com -kpH -nc
cat xkcd.com/index.html|grep xkcdLogo
wget http://xkcd.com -kpH -nc
cat xkcd.com/index.html|grep xkcdLogo

This test case does not work with -N instead of -nc for me, since wget re-downloads the page every time (can't tell why).

This bug stops me from performing backups of websites with a lot of static content (like XKCD), without downloading the whole site or resorting to hacks, such as performing a diff of updated content or restoring .orig files before running wget.

Seems to be a simple fix too, so I hope this will get fixed soon.

Simon Lindholm <simon93>
Fri 16 Jan 2009 12:24:58 AM UTC, original submission:

From http://lists.gnu.org/archive/html/bug-wget/2008-12/msg00050.html :

Micah Cowan <micahcowan>
Project Administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by simon93 (Posted a comment)
  • -unavailable- added by micahcowan (Submitted the item)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 02 Oct 2009 07:44:08 PM UTCmicahcowanPriority5 - Normal=>4

    Back to the top

    Powered by Savane 3.1-cleanup