bugGNU Wget - Bugs: bug #31781, Bad -k behaviour when -nc is...

 
 

bug #31781: Bad -k behaviour when -nc is enabled and the files already exist

Submitted by:  Romain Vimont <romainvimont>
Submitted on:  Sat 04 Dec 2010 02:10:14 PM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: Fixed
Privacy: PublicAssigned to: Giuseppe Scrivano <gscrivano>
Originator Name: Open/Closed: Closed
Release: 1.12Operating System: GNU/Linux
Reproducibility: Every TimeFixed Release: 1.12.1
Planned Release: NoneRegression: None
Work Required: NonePatch Included: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 07 Jul 2012 08:48:54 AM UTC, comment #3:

these two options are inconsistent, at least in the current implementation, newer links will not get converted. If a file is saved with a different encoding then it won't be found by wget.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Sun 17 Jun 2012 07:07:19 AM UTC, comment #2:

There are problems with this bug and with the fix:
You removed a very useful feature, --no-clobber and --convert-links were working correctly if --span-hosts is not set.

Even with the version 1.13.4 of wget, I can still reproduce the bug:
First, use -k with -H: wget -EHkpD lemonde.fr,lemde.fr http://www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html
Now, do it again, without -k but with -nc (-H is not required): wget -nc -Ep http://www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html
=> the .js and .jpg fail with 404 Not Found.

Also, if --timestamping is set and server page hasn't changed, the bug is still reproduced: wget -N -Ep http://www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html
=> Server file no newer than local file `http://www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html' -- not retrieving.
=> the .js and .jpg fail with 404 Not Found.

The problem is when parsing local files, wget does not handle correctly converted links between different websites.

Pierre ReniƩ <pir>
Thu 09 Dec 2010 12:28:49 PM UTC, comment #1:

thanks for the report, I am going to apply a patch, if both --no-clobber and --convert-links are used at the same time, then only the latter is used.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Sat 04 Dec 2010 02:10:14 PM UTC, original submission:

-k allows to "convert links" for local reading
-nc avoid to download again an already existing file

When you enable both, then -k doesn't convert links correctly for files which already exist.

A very simple test:
wget -nc -EHkpD 'lemonde.fr,lemde.fr' http://www.lemonde.fr/international/article/2010/12/03/wikileaks-la-loi-hadopi-interesse-au-plus-haut-point-washington_1448827_3210.html

The first time, everything is ok. Don't remove the files, and try again, it uses "http://www.lemonde.fr/s1.lemde.fr/medias/www/1.2.339/js/package/lib_mia_deprecated_1.min.js" instead of "http://s1.lemde.fr/medias/www/1.2.339/js/package/lib_mia_deprecated_1.min.js" (which of course give a 404 Not found error)

Remove the files (or disable -nc), and it works fine again.

Romain Vimont <romainvimont>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by pir (Posted a comment)
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by romainvimont (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 09 Dec 2010 12:33:00 PM UTCgscrivanoStatusNone=>Fixed
      Assigned toNone=>gscrivano
      Open/ClosedOpen=>Closed
      Fixed ReleaseNone=>1.12.1

    Back to the top


    Powered by Savane 3.1-cleanup