bugGNU Wget - Bugs: bug #31781, Bad -k behaviour when -nc is...


bug #31781: Bad -k behaviour when -nc is enabled and the files already exist

Submitted by:  Romain Vimont <romainvimont>
Submitted on:  Sat 04 Dec 2010 02:10:14 PM UTC  
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  Giuseppe Scrivano <gscrivano>
Originator Name:  Open/Closed:  Closed
Release:  1.12 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  1.12.1
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Sun 03 Dec 2017 12:11:15 PM UTC, comment #4:

Thank you for fixing this bug, but it broke the behaviour for some people: https://superuser.com/questions/80060/make-wget-convert-html-links-to-relative-after-download-if-k-wasnt-specified

I just wanted to let anyone stumbling on this thread later know that I built a docker image with wget 1.12 based on Debian Squeeze: https://hub.docker.com/r/berezovskyi/wget1.12/

Andrew Berezovskyi <berezovskyi>
Sat 07 Jul 2012 08:48:54 AM UTC, comment #3:

these two options are inconsistent, at least in the current implementation, newer links will not get converted. If a file is saved with a different encoding then it won't be found by wget.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Sun 17 Jun 2012 07:07:19 AM UTC, comment #2:

There are problems with this bug and with the fix:
You removed a very useful feature, --no-clobber and --convert-links were working correctly if --span-hosts is not set.

Even with the version 1.13.4 of wget, I can still reproduce the bug:
First, use -k with -H: wget -EHkpD lemonde.fr,lemde.fr http://www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html
Now, do it again, without -k but with -nc (-H is not required): wget -nc -Ep http://www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html
=> the .js and .jpg fail with 404 Not Found.

Also, if --timestamping is set and server page hasn't changed, the bug is still reproduced: wget -N -Ep http://www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html
=> Server file no newer than local file `www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html' -- not retrieving.
=> the .js and .jpg fail with 404 Not Found.

The problem is when parsing local files, wget does not handle correctly converted links between different websites.

Pierre ReniƩ <pir>
Thu 09 Dec 2010 12:28:49 PM UTC, comment #1:

thanks for the report, I am going to apply a patch, if both --no-clobber and --convert-links are used at the same time, then only the latter is used.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Sat 04 Dec 2010 02:10:14 PM UTC, original submission:

-k allows to "convert links" for local reading
-nc avoid to download again an already existing file

When you enable both, then -k doesn't convert links correctly for files which already exist.

A very simple test:
wget -nc -EHkpD 'lemonde.fr,lemde.fr' http://www.lemonde.fr/international/article/2010/12/03/wikileaks-la-loi-hadopi-interesse-au-plus-haut-point-washington_1448827_3210.html

The first time, everything is ok. Don't remove the files, and try again, it uses "http://www.lemonde.fr/s1.lemde.fr/medias/www/1.2.339/js/package/lib_mia_deprecated_1.min.js" instead of "http://s1.lemde.fr/medias/www/1.2.339/js/package/lib_mia_deprecated_1.min.js" (which of course give a 404 Not found error)

Remove the files (or disable -nc), and it works fine again.

Romain Vimont <romainvimont>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by berezovskyi (Posted a comment)
  • -email is unavailable- added by pir (Posted a comment)
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by romainvimont (Submitted the item)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    Thu 09 Dec 2010 12:33:00 PM UTC gscrivano StatusNone => Fixed
        Assigned toNone => gscrivano
        Open/ClosedOpen => Closed
        Fixed ReleaseNone => 1.12.1

    Back to the top

    Powered by Savane 3.3