bugGNU Wget - Bugs: bug #23756, Should rename "foo" to...


bug #23756: Should rename "foo" to "foo/index.html" when necessary.

Submitted by:  Micah Cowan <micahcowan>
Submitted on:  Wed 02 Jul 2008 06:19:54 PM UTC  
Category:  Feature Request Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  0 - Hours Patch Included:  None

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Sat 16 Oct 2010 11:02:28 PM UTC, comment #3:

> Wget will never replace a file with a directory.
> Not any official release of wget, at any rate.

Sorry, Micah, you're wrong. In wget-1.12 url.c (2009-09-22 05:05), lines 1211ff in mkalldirs() we find:

/* If the dir exists as a file name, remove it first. This
is only for Wget to work with buggy old CERN http
servers. Here is the scenario: When Wget tries to
retrieve a directory without a slash, e.g.
http://foo/bar (bar being a directory), CERN server will
not redirect it too http://foo/bar/ -- it will generate a
directory listing containing links to bar/file1,
bar/file2, etc. Wget will lose because it saves this
HTML listing to a file `bar', so it cannot create the
directory. To work around this, if the file of the same
name exists, we just remove it and create the directory
anyway. */
DEBUGP (("Removing %s because of directory danger!\n", t));
unlink (t);

This code ist still present in wget-1.12-2428.

Thu 22 Oct 2009 09:11:13 PM UTC, comment #2:

Wget will never replace a file with a directory. Not any official release of wget, at any rate. The logic to do so is simply not there. Instead, it will fail to construct the directory for the new file, which won't be saved anywhere.

Micah Cowan <micahcowan>
Thu 22 Oct 2009 05:57:01 PM UTC, comment #1:

I find that wget's behavior is currently buggy in this respect:

wget --random-wait -np -r http://www.healthgrades.com/group-directory/

Will create files like http://www.healthgrades.com/group-directory/alabama-al/birmingham

but then it will find links to http://www.healthgrades.com/group-directory/alabama-al/birmingham/117-medical-group-69f8a003

and will always replace the original birmingham file with a directory, even with no clobber, level restrictions, and other settings. I have found no way to keep the original file.


Wed 02 Jul 2008 06:19:54 PM UTC, original submission:

If Wget is made to process the URLs:


and the server doesn't redirect http://host/foo to http://host/foo/, then Wget will fail to download bar, with a "not a directory" error.

It would be nice if, instead, Wget were to try to rename the file to "foo/index.html" (this would require an intermediate rename to a temporary name, and creation of the appropriate directory), so that this sort of usage doesn't throw Wget off.

Usually, one can get around this problem with the use of -nd; but there's no guarantee.

In implementing this, one would need to be careful also to ensure that, if host/foo/ already exists as a directory, then encountering http://host/foo wouldn't result in an error either, but would automatically result in saving the file to host/foo/index.html.

Micah Cowan <micahcowan>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by micahcowan (Submitted the item)



    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-08-21 micahcowan Planned Release1.15 => 1.14

    Back to the top

    Powered by Savane 3.4