bugGNU Wget - Bugs: bug #58354, Wget doesn't parse URIs starting...

 
 

bug #58354: Wget doesn't parse URIs starting with http:/

Submitted by:  Luca Bernardi <f0ff>
Submitted on:  Tue 12 May 2020 10:45:17 AM UTC  
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.14 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  No

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 12 May 2020 11:41:10 AM UTC, comment #3: 

Note that the browser's The WHATWG URL Spec (TWUS) allows that kind of abomination, while RFC 3986 explicitly requires two slashes to be there. (Listed in my "URL interop" page at https://github.com/bagder/docs/blob/master/URL-interop.md)

In fact, TWUS allows an unlimited amount of slashes (and backslashes) to be present and is still fine with it.

Because browsers allow this, instances of such URLs appear in the wild so it makes sense to be somewhat accommodating.

In the curl project (which primarily has RFC3986 as guidance), we've still decided to accept one, two or three slashes as we've seen both one and three happen in the wild. I don't believe in being more lenient than so.

Daniel Stenberg <bagder>
Tue 12 May 2020 11:26:29 AM UTC, comment #2: 

We could accept this while parsing input and having a BASE.

It doesn't make much sense without a BASE, as the host/domain part is skipped here.

I assume that 'https:<absolute-path>' should also be recognized.

Tim Ruehsen <rockdaboot>
Project Administrator
Tue 12 May 2020 10:57:11 AM UTC, comment #1: 

PS This bug has happened when trying to crawl a website with default Wordpress template.

Luca Bernardi <f0ff>
Tue 12 May 2020 10:45:17 AM UTC, original submission:  

Hi,
Wget refuses to parse URIs that start with http:/ (note single slash), e.g. http:/wp-includes/css/dist/block-library/style.min.css?ver=5.4.1. These are widely accepted by browsers.

Command that I've used: `wget --user-agent=Mozilla --content-disposition --page-requisites --adjust-extension --restrict-file-names=windows -d -e robots=off -m -k -E -r -l 10 -p -N -F -P crawl  -nH $IP`

Luca Bernardi <f0ff>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49060:  out.txt added by f0ff (17KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by bagder (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by f0ff (Submitted the item)
  • -email is unavailable- added by f0ff
  •  

     

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-05-12 rockdaboot StatusNone => Confirmed
    2020-05-12 f0ff Attached File- => Added out.txt, #49060
        Carbon-Copy- => Added gnu@lucysan.net

    Back to the top


    Powered by Savane 3.5