bugGNU Wget - Bugs: bug #22670, Wget matches accept/reject lists...


bug #22670: Wget matches accept/reject lists against local filenames to decide when to delete

Submitter:  Micah Cowan <micahcowan>
Submitted:  Wed 19 Mar 2008 11:11:25 PM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  7 - High Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.13 Regression:  None
Work Required:  1 - Days Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Fri 25 Sep 2009 06:15:35 PM UTC, comment #6: 

Not a patch-level fix, whether I want it to be or not.

Micah Cowan <micahcowan>
Tue 22 Sep 2009 07:59:05 PM UTC, comment #5: 

The "delete-after" logic, meanwhile, should not follow the file's new, actual local filename (note that that also fails for content-disposition'd files), but should save the delete-after value based on whether the file should otherwise have been rejected, but matched the "traverse" accept/reject rules.

Micah Cowan <micahcowan>
Tue 03 Jun 2008 06:47:23 PM UTC, comment #4: 

retrieval acc/rej and delete-after acc/rej is the wrong paradigm: at some point, if we support streamed processing, there may be no download of recurse-only URLs, and thus nothing to delete. We'll still want to control whether those links are traversed, regardless of whether they must first be downloaded before they can be traversed; it should be download acc/rej versus recurse (links-only mode) acc/rej.

Micah Cowan <micahcowan>
Fri 30 May 2008 08:03:59 PM UTC, comment #3: 

Note that this problem currently results in different behavior between the following two commands:

wget --content-disposition -A'*cd*' micah.cowan.name/cgi-bin/cd?cd=foo
wget -r --content-disposition -A'*cd*' micah.cowan.name/cgi-bin/cd?cd=foo

Both download to a file named "foo" (Content-Disposition header based on the value of the "cd" query-string param); but the latter will delete the file afterwards (the acc/rej deletion stuff is only done in retrieve_tree, not retrieve_url).

Micah Cowan <micahcowan>
Thu 22 May 2008 06:58:54 PM UTC, comment #2: 

Separating the concepts of fetch-accept and delete-after rules had been proposed, and seems like a good idea, combined with no longer automatically grabbing all .html files no matter what the acc/rej rules are.

Micah Cowan <micahcowan>
Mon 14 Apr 2008 04:08:55 AM UTC, comment #1: 

To be clear: it's matched against the URL's "filename" to determine accept/reject, and against the local filename to delete it afterwards.

Micah Cowan <micahcowan>
Wed 19 Mar 2008 11:11:25 PM UTC, original submission:  

It should match against the original URL. This leads to strange cases like "-E -A php" telling Wget that it should download and parse the PHP file, but afterwards deleting it because it had been renamed to "index.php.html", which doesn't match any longer.

Similarly, there can be issues when the local filename is ".html.1", renamed because of the use of -nd.

Micah Cowan <micahcowan>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by micahcowan (Submitted the item)

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-10-02 micahcowan Planned Release1.14 1.13
    2009-09-25 micahcowan Planned Release1.12.x 1.14
        Priority4 7 - High
    2009-09-22 micahcowan Planned Release1.14 1.12.x
    2008-08-21 micahcowan Planned Release1.12 1.14
    2008-05-22 micahcowan Priority5 - Normal 4
        Work Required0 - Hours 1 - Days

    Back to the top

    Powered by Savane 3.13-3cd0.
    Corresponding source code