bugGNU Wget - Bugs: bug #23756, Should rename "foo" to...

 
 

bug #23756: Should rename "foo" to "foo/index.html" when necessary.

Submitter:  Micah Cowan <micahcowan>
Submitted:  Wed 02 Jul 2008 06:19:54 PM UTC
   
 
Category:  Feature Request Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  0 - Hours Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 16 Oct 2010 11:02:28 PM UTC, comment #3: 


> Wget will never replace a file with a directory.
> Not any official release of wget, at any rate.


Sorry, Micah, you're wrong. In wget-1.12 url.c (2009-09-22 05:05), lines 1211ff in mkalldirs() we find:

          /* If the dir exists as a file name, remove it first.  This
             is only for Wget to work with buggy old CERN http
             servers.  Here is the scenario: When Wget tries to
             retrieve a directory without a slash, e.g.
             http://foo/bar (bar being a directory), CERN server will
             not redirect it too http://foo/bar/ -- it will generate a
             directory listing containing links to bar/file1,
             bar/file2, etc.  Wget will lose because it saves this
             HTML listing to a file `bar', so it cannot create the
             directory.  To work around this, if the file of the same
             name exists, we just remove it and create the directory
             anyway.  */
          DEBUGP (("Removing %s because of directory danger!\n", t));
          unlink (t);

This code ist still present in wget-1.12-2428.

Anonymous
Thu 22 Oct 2009 09:11:13 PM UTC, comment #2: 

Wget will never replace a file with a directory. Not any official release of wget, at any rate. The logic to do so is simply not there. Instead, it will fail to construct the directory for the new file, which won't be saved anywhere.

Micah Cowan <micahcowan>
Thu 22 Oct 2009 05:57:01 PM UTC, comment #1: 

I find that wget's behavior is currently buggy in this respect:

wget --random-wait -np -r http://www.healthgrades.com/group-directory/

Will create files like www.healthgrades.com/group-directory/alabama-al/birmingham

but then it will find links to www.healthgrades.com/group-directory/alabama-al/birmingham/117-medical-group-69f8a003

and will always replace the original birmingham file with a directory, even with no clobber, level restrictions, and other settings. I have found no way to keep the original file.

-Rob

Anonymous
Wed 02 Jul 2008 06:19:54 PM UTC, original submission:  

If Wget is made to process the URLs:

http://host/foo
http://host/foo/bar

and the server doesn't redirect http://host/foo to http://host/foo/, then Wget will fail to download bar, with a "not a directory" error.

It would be nice if, instead, Wget were to try to rename the file to "foo/index.html" (this would require an intermediate rename to a temporary name, and creation of the appropriate directory), so that this sort of usage doesn't throw Wget off.

Usually, one can get around this problem with the use of -nd; but there's no guarantee.

In implementing this, one would need to be careful also to ensure that, if host/foo/ already exists as a directory, then encountering http://host/foo wouldn't result in an error either, but would automatically result in saving the file to host/foo/index.html.

Micah Cowan <micahcowan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by micahcowan (Submitted the item)
  •  

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-08-21 micahcowan Planned Release1.15 1.14

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code