bugGNU Wget - Bugs: bug #48424, wget fails to convert some URLs...

 
 

bug #48424: wget fails to convert some URLs when the same file path is retrieved via more than one protocol

Submitter:  None
Submitted:  Wed 06 Jul 2016 07:27:23 PM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Invalid
Privacy:  Public Assigned to:  None
Originator Name:  Paul Merchant Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  1.18
Operating System:  Mac OS Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 07 Jul 2016 06:37:13 PM UTC, comment #2: 

NP ;-)

Tim Ruehsen <rockdaboot>
Group administrator
Thu 07 Jul 2016 05:53:17 PM UTC, comment #1: 

This is not a bug, the --protocol-directories flag provides this feature.

Paul Merchant <paulmer>
Wed 06 Jul 2016 07:27:23 PM UTC, original submission:  

If different protocols retrieve the same file path in a recursive crawl that converts URLs, some of the referring URLs will not be rewritten. For example, if file A, in directory wget-test contains these links:

<a href="http://myhost/wget-test/b.html">b - http</a>
<a href="https://myhost/wget-test/b.html">b - https</a>

Then a.html retrieved by this command:
wget -m --convert-links http://myhost/wget-test/a.html

will contain

<a href="http://myhost/wget-test/b.html">b - http</a>
<a href="b.html">b - https</a>

Since the different protocols actually refer to different servers (or ports on the same server that may not sharedirectory aliases), there is no guarantee that the matching url paths actually represent the same file.  Ideally wget should separate paths by protocol, and offer an option to ignore the protocol when making paths so that if http and https (or ftp, or...) are known to correspond to the same directory this can be reflected in the URL conversion.

As wget works now, the no clobber flag cannot be used as a work-around as it is incompatible with the recursive crawl.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by paulmer (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-07-07 rockdaboot StatusNone Invalid
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code