bugGNU Wget - Bugs: bug #48424, wget fails to convert some URLs...

 
 

bug #48424: wget fails to convert some URLs when the same file path is retrieved via more than one protocol

Submitted by:  None
Submitted on:  Wed 06 Jul 2016 07:27:23 PM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: Invalid
Privacy: PublicAssigned to: None
Originator Name: Paul MerchantOriginator Email: -unavailable-
Open/Closed: ClosedRelease: 1.18
Operating System: Mac OSReproducibility: Every Time
Fixed Release: NonePlanned Release: None
Regression: NoneWork Required: None
Patch Included: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 07 Jul 2016 06:37:13 PM UTC, comment #2:

NP ;-)

Tim Ruehsen <rockdaboot>
Project Administrator
Thu 07 Jul 2016 05:53:17 PM UTC, comment #1:

This is not a bug, the --protocol-directories flag provides this feature.

Paul Merchant <paulmer>
Wed 06 Jul 2016 07:27:23 PM UTC, original submission:

If different protocols retrieve the same file path in a recursive crawl that converts URLs, some of the referring URLs will not be rewritten. For example, if file A, in directory wget-test contains these links:

<a href="http://myhost/wget-test/b.html">b - http</a>
<a href="https://myhost/wget-test/b.html">b - https</a>

Then a.html retrieved by this command:
wget -m --convert-links http://myhost/wget-test/a.html

will contain

<a href="http://myhost/wget-test/b.html">b - http</a>
<a href="b.html">b - https</a>

Since the different protocols actually refer to different servers (or ports on the same server that may not sharedirectory aliases), there is no guarantee that the matching url paths actually represent the same file. Ideally wget should separate paths by protocol, and offer an option to ignore the protocol when making paths so that if http and https (or ftp, or...) are known to correspond to the same directory this can be reflected in the URL conversion.

As wget works now, the no clobber flag cannot be used as a work-around as it is incompatible with the recursive crawl.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by rockdaboot (Posted a comment)
  • -unavailable- added by paulmer (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 07 Jul 2016 06:37:13 PM UTCrockdabootStatusNone=>Invalid
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1