bugGNU Wget - Bugs: bug #58354, Wget doesn't parse URIs starting...

 
 

bug #58354: Wget doesn't parse URIs starting with http:/

Submitter:  Luca Bernardi <f0ff>
Submitted:  Tue 12 May 2020 10:45:17 AM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.14 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 12 May 2020 11:41:10 AM UTC, comment #3: 

Note that the browser's The WHATWG URL Spec (TWUS) allows that kind of abomination, while RFC 3986 explicitly requires two slashes to be there. (Listed in my "URL interop" page at https://github.com/bagder/docs/blob/master/URL-interop.md)

In fact, TWUS allows an unlimited amount of slashes (and backslashes) to be present and is still fine with it.

Because browsers allow this, instances of such URLs appear in the wild so it makes sense to be somewhat accommodating.

In the curl project (which primarily has RFC3986 as guidance), we've still decided to accept one, two or three slashes as we've seen both one and three happen in the wild. I don't believe in being more lenient than so.

 Daniel Stenberg <bagder>
Tue 12 May 2020 11:26:29 AM UTC, comment #2: 

We could accept this while parsing input and having a BASE.

It doesn't make much sense without a BASE, as the host/domain part is skipped here.

I assume that 'https:<absolute-path>' should also be recognized.

Tim Ruehsen <rockdaboot>
Group administrator
Tue 12 May 2020 10:57:11 AM UTC, comment #1: 

PS This bug has happened when trying to crawl a website with default Wordpress template.

 Luca Bernardi <f0ff>
Tue 12 May 2020 10:45:17 AM UTC, original submission:  

Hi,
Wget refuses to parse URIs that start with http:/ (note single slash), e.g. http:/wp-includes/css/dist/block-library/style.min.css?ver=5.4.1. These are widely accepted by browsers.

Command that I've used: `wget --user-agent=Mozilla --content-disposition --page-requisites --adjust-extension --restrict-file-names=windows -d -e robots=off -m -k -E -r -l 10 -p -N -F -P crawl  -nH $IP`

 Luca Bernardi <f0ff>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49060:  out.txt added by f0ff (17KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by bagder (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by f0ff (Submitted the item)
  • -email is unavailable- added by f0ff

    •  

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-05-12 rockdaboot StatusNone Confirmed
    2020-05-12 f0ff Attached File- Added out.txt, #49060
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.14-8eb0.
    Corresponding source code