bugGNU Wget - Bugs: bug #27097, Nice to have...


bug #27097: Nice to have --restrict-file-names=mac (or =darwin, etc)

Submitter:  None
Submitted:  Fri 24 Jul 2009 10:35:48 AM UTC
Category:  Feature Request Severity:  2 - Minor
Priority:  4 Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Bjoern Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.11.4
Operating System:  None Reproducibility:  Every Time
Fixed Release:  None Planned Release:  1.14
Regression:  None Work Required:  None
Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup

Tue 01 Sep 2009 06:01:16 AM UTC, comment #3: 

Well, if we did that, it would escape various characters unnecessarily, which wouldn't be ideal.

Probably, it should behave exactly like "unix", except with the additional requirement that colons are also escaped (and, probably, the ":port" should become "@port", as it does in Windows).

The magic is in the filechr_table array in url.c, and the macro FILE_CHAR_TEST. Each element of filechr_table is a bitmask of values for: control character, escape-on-windows, escape-on-unix, with a few macros representing shorthand combinations of those (UW = escape on unix and windows). As you'll see when you look at it, to keep the structure as it currently is, you'd just add a new bit value for mac os, and add a UM macro (for escape on unix and mac: all "U" entries would become "MU"), and a "MW" macro for the colon entry.

The logic for :port -> @port in FN_QUERY_SEP would also need adjusting.

Micah Cowan <micahcowan>
Tue 01 Sep 2009 04:58:03 AM UTC, comment #2: 

Is there a good reason to not just let:


actually do the same thing in the code?

This will be a new branch of code for me to look at.  I'll take the assignment if you like, Micah.

Joseph Rios <alotau>
Fri 24 Jul 2009 08:38:23 PM UTC, comment #1: 

The ":" -> "/" thing is something Mac OS X is doing. If it's going to make that transformation, then it ought to continue supporting it for finding the files again :\

Perhaps this should take the form of an --restrict-file-names=macos option, since it's specific to MacOS.

The ":" in the converted links, though, ought to be percent-encoded, even if they aren't in the original filename, since otherwise it represents a scheme name.

As a workaround, consider using --restrict-file-names=windows, which will include ":" as a character that shouldn't be emitted, but won't currently use any characters that are unsafe for Unix.

Micah Cowan <micahcowan>
Fri 24 Jul 2009 10:35:48 AM UTC, original submission:  

I tried to download the Twitter apiwiki using

wget -rkp http://apiwiki.twitter.com/

Unfortunately the wiki uses ":" in links, which results in converted links of the form "twitter-search-api-method:-search"

Trying to follow these links results in an "unkown protocol" error by Firefox, because "twitter-search-api-method:" is being interpreted as a protocol part of an URL.

Also, the filenames seem to have ":" converted to "/", so the file is called "Twitter-Search-API-Method/-search". That also seems to cause semi-troubles on the OS X file system (some programs can open the file, others can't).



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by alotau (Posted a comment)
  • -email is unavailable- added by micahcowan (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-08-31 micahcowan CategoryProgram Logic Feature Request
        Severity3 - Normal 2 - Minor
        Summary-k does not escape &quot;:&quot; in URLs, produces unusable links Nice to have --restrict-file-names=mac (or =darwin, etc)
    2009-07-24 micahcowan Priority5 - Normal 4
        StatusNone Confirmed
        Planned ReleaseNone 1.14

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code