bugGNU Wget - Bugs: bug #25340, --mirror and --convert-links...


bug #25340: --mirror and --convert-links mixing poorly?

Submitter:  Micah Cowan <micahcowan>
Submitted:  Fri 16 Jan 2009 12:24:58 AM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  4 Status:  Needs Investigation
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.10.2 Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  1 - Days Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Tue 18 Oct 2016 06:35:28 PM UTC, comment #5: 

Ouch!  I've been in the throes of trying to get wget to work downloading and updating a mirror copy if the IANA assignment pages.  But it looks like that can't be made to work until this problem is fixed.

At least in the case of Un*x file systems, as long as wget doesn't run into collisions, the mapping from URLs to filenames is reversible.

But there's still a problem that when doing an update, one wants wget to examine all of the files in the tree, not just the ones that are linked to by files that it downloaded this time.

Dale Worley <worley>
Tue 18 Oct 2016 03:25:46 AM UTC, comment #4: 

This continues to be an issue in v1.18 using the following options:

 wget \
      --mirror \
      --page-requisites \
      --adjust-extension \
      --convert-links \
      --no-parent \
      --no-host-directories \
      -o wget.log \
      --domains charlotteworks.com \

When running this command twice, the background images work on the first attempt then end up getting appended with .html. This is esp noticeably on the PNG and CSS files in the index.html file.

Simplifying this command and removing the --mirror option does not create the problem:

   wget --page-requisites --adjust-extension --convert-links --no-parent --no-host-directories -o wget.log --domains charlotteworks.com www.charlotteworks.com

FYI, the website above will be replaced by a new site after Feb 2017. I'm not sure if this behavior is dependent on the web hosting environment.

William McKee <knowmad>
Fri 02 Oct 2009 07:47:54 PM UTC, comment #3: 

Well, I suppose that's not entirely accurate... -N should imply that the file we timestamped and decided shouldn't be (re-)downloaded corresponds to that URL, and we should be able to parse the links exactly as we found them on the remote site.

With --convert-links, though, we can run into problems if the setting of --restrict-file-names or some other option caused the local file name to differ from the remote URL; it can be difficult to unmunge the local name to obtain the "real" relative URL that should be used to query the server. So, yeah, still easiest to wait for the SIDB.

Micah Cowan <micahcowan>
Fri 02 Oct 2009 07:44:08 PM UTC, comment #2: 

Well, that's obviously exactly what I'd expect -nc to do...

As to the behavior on -N, there's little we can do about that, since it can be difficult to know if and where a previous run had stored files locally that correspond to URLs. There are heuristics available, but...

We won't be fixing this until we have something like the Session Info Database implemented.

Micah Cowan <micahcowan>
Sun 12 Apr 2009 02:37:41 PM UTC, comment #1: 

I can confirm this bug. It seems to occur when -k is used with either -N or -nc, a checked file is not downloaded again, and some link on that file/page has a href that is not valid on the server anymore, such as the ones created with -E or -H. A simpler test case:

wget http://xkcd.com -kpH -nc
cat xkcd.com/index.html|grep xkcdLogo
wget http://xkcd.com -kpH -nc
cat xkcd.com/index.html|grep xkcdLogo

This test case does not work with -N instead of -nc for me, since wget re-downloads the page every time (can't tell why).

This bug stops me from performing backups of websites with a lot of static content (like XKCD), without downloading the whole site or resorting to hacks, such as performing a diff of updated content or restoring .orig files before running wget.

Seems to be a simple fix too, so I hope this will get fixed soon.

Simon Lindholm <simon93>
Fri 16 Jan 2009 12:24:58 AM UTC, original submission:  

From http://lists.gnu.org/archive/html/bug-wget/2008-12/msg00050.html :

I'm trying to mirror a site (http://www.gamingcommission.fgov.be) and
update the mirror at regular intervals.

This is primarily meant to detect new and changed pages and files.  While
at it, I set it up as a local mirror so I get the added benefit of that
(the site has a tendency of going down a LOT).

I'm using wget 1.10.2 on Ubuntu Server 7.10, apparently they don't have
1.11.x in the repository yet (just scanned for and installed updates today
to make sure).
Should I have asked somewhere else in this case?

The initial local copy works perfectly, it is fully browsable, etc.

Then when I run wget again with the exact same command line arguments, this
is what happens:

1) It checks only the top page (index.html) and direct descendants.  If
there are no changes there, it just stops.  In other words, it only detects
a change in a page if the parent of that page (in the link tree) has also

2) It messes up links in the pages it re-checks.  Any link to a page that
isn't downloaded again (because it hasn't changed) is replaced by an
absolute link to the original server, but using the filename of the local

An example of an URL mutilated by (2):

Mirror, initial copy:
Mirror, after one update:

I am using these command line parameters:
   --w 2
   -e robots=off

Is there anything wrong with my parameters?
Is it something in the site?
Is it a bug?  Fixed in 1.11.4?

BTW, in case someone wants to test it out locally: total size of the mirror
is 116MB.  You can significantly reduce that by skipping the .pdf and .doc

Micah Cowan <micahcowan>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by worley (Posted a comment)
  • -email is unavailable- added by knowmad (Posted a comment)
  • -email is unavailable- added by knowmad
  • -email is unavailable- added by simon93 (Posted a comment)
  • -email is unavailable- added by micahcowan (Submitted the item)

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-10-18 knowmad Carbon-Copy- Added knowmad
    2009-10-02 micahcowan Priority5 - Normal 4

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code