bugGNU Wget - Bugs: bug #31781, Bad -k behaviour when -nc is...

 
 

bug #31781: Bad -k behaviour when -nc is enabled and the files already exist

Submitter:  Romain Vimont <romainvimont>
Submitted:  Sat 04 Dec 2010 02:10:14 PM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  gscrivano
Originator Name:  Open/Closed:  Closed
Release:  1.12 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  1.12.1
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 07 Jun 2020 02:00:23 PM UTC, comment #5: 

Thank you berezovskyi for the docker image.
It's helpful to use -k with -nc
But getting "OpenSSL: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version"

After further search found that docker image built with libssl0.9.8 and works only with sites supporting tlsv1

could you please update docker image with latest openssl

Bala <kvt_bk>
Sun 03 Dec 2017 12:11:15 PM UTC, comment #4: 

Thank you for fixing this bug, but it broke the behaviour for some people: https://superuser.com/questions/80060/make-wget-convert-html-links-to-relative-after-download-if-k-wasnt-specified

I just wanted to let anyone stumbling on this thread later know that I built a docker image with wget 1.12 based on Debian Squeeze: https://hub.docker.com/r/berezovskyi/wget1.12/

Andrew Berezovskyi <berezovskyi>
Sat 07 Jul 2012 08:48:54 AM UTC, comment #3: 

these two options are inconsistent, at least in the current implementation, newer links will not get converted.  If a file is saved with a different encoding then it won't be found by wget.

Giuseppe Scrivano <gscrivano>
Group administrator
Sun 17 Jun 2012 07:07:19 AM UTC, comment #2: 

There are problems with this bug and with the fix:
You removed a very useful feature, --no-clobber and --convert-links were working correctly if --span-hosts is not set.

Even with the version 1.13.4 of wget, I can still reproduce the bug:
First, use -k with -H: wget -EHkpD lemonde.fr,lemde.fr http://www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html
Now, do it again, without -k but with -nc (-H is not required): wget -nc -Ep http://www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html
=> the .js and .jpg fail with 404 Not Found.

Also, if --timestamping is set and server page hasn't changed, the bug is still reproduced: wget -N -Ep http://www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html
=> Server file no newer than local file `www.lemonde.fr/politique/article/2012/06/16/le-second-tour-des-elections-legislatives-en-six-questions_1719633_823448.html' -- not retrieving.
=> the .js and .jpg fail with 404 Not Found.

The problem is when parsing local files, wget does not handle correctly converted links between different websites.

Pierre ReniƩ <pir>
Thu 09 Dec 2010 12:28:49 PM UTC, comment #1: 

thanks for the report, I am going to apply a patch, if both --no-clobber and --convert-links are used at the same time, then only the latter is used.

Giuseppe Scrivano <gscrivano>
Group administrator
Sat 04 Dec 2010 02:10:14 PM UTC, original submission:  

-k allows to "convert links" for local reading
-nc avoid to download again an already existing file

When you enable both, then -k doesn't convert links correctly for files which already exist.

A very simple test:
wget -nc -EHkpD 'lemonde.fr,lemde.fr' http://www.lemonde.fr/international/article/2010/12/03/wikileaks-la-loi-hadopi-interesse-au-plus-haut-point-washington_1448827_3210.html

The first time, everything is ok. Don't remove the files, and try again, it uses "http://www.lemonde.fr/s1.lemde.fr/medias/www/1.2.339/js/package/lib_mia_deprecated_1.min.js" instead of "http://s1.lemde.fr/medias/www/1.2.339/js/package/lib_mia_deprecated_1.min.js" (which of course give a 404 Not found error)

Remove the files (or disable -nc), and it works fine again.

Romain Vimont <romainvimont>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by kvt_bk (Posted a comment)
  • -email is unavailable- added by berezovskyi (Posted a comment)
  • -email is unavailable- added by pir (Posted a comment)
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by romainvimont (Submitted the item)
  •  

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-12-09 gscrivano StatusNone Fixed
        Assigned toNone gscrivano
        Open/ClosedOpen Closed
        Fixed ReleaseNone 1.12.1

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code