bugGNU Wget - Bugs: bug #23756, Should rename "foo" to...

 
 

bug #23756: Should rename "foo" to "foo/index.html" when necessary.

Submitted by:  Micah Cowan <micahcowan>
Submitted on:  Wed 02 Jul 2008 06:19:54 PM UTC  
 
Category: Feature RequestSeverity: 3 - Normal
Priority: 5 - NormalStatus: Confirmed
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: NoneOperating System: None
Reproducibility: NoneFixed Release: None
Planned Release: 1.14Regression: None
Work Required: 0 - HoursPatch Included: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 16 Oct 2010 11:02:28 PM UTC, comment #3:

> Wget will never replace a file with a directory.
> Not any official release of wget, at any rate.


Sorry, Micah, you're wrong. In wget-1.12 url.c (2009-09-22 05:05), lines 1211ff in mkalldirs() we find:

/* If the dir exists as a file name, remove it first. This
is only for Wget to work with buggy old CERN http
servers. Here is the scenario: When Wget tries to
retrieve a directory without a slash, e.g.
http://foo/bar (bar being a directory), CERN server will
not redirect it too http://foo/bar/ -- it will generate a
directory listing containing links to bar/file1,
bar/file2, etc. Wget will lose because it saves this
HTML listing to a file `bar', so it cannot create the
directory. To work around this, if the file of the same
name exists, we just remove it and create the directory
anyway. */
DEBUGP (("Removing %s because of directory danger!\n", t));
unlink (t);

This code ist still present in wget-1.12-2428.

Anonymous
Thu 22 Oct 2009 09:11:13 PM UTC, comment #2:

Wget will never replace a file with a directory. Not any official release of wget, at any rate. The logic to do so is simply not there. Instead, it will fail to construct the directory for the new file, which won't be saved anywhere.

Micah Cowan <micahcowan>
Project Administrator
Thu 22 Oct 2009 05:57:01 PM UTC, comment #1:

I find that wget's behavior is currently buggy in this respect:

wget --random-wait -np -r http://www.healthgrades.com/group-directory/

Will create files like http://www.healthgrades.com/group-directory/alabama-al/birmingham

but then it will find links to http://www.healthgrades.com/group-directory/alabama-al/birmingham/117-medical-group-69f8a003

and will always replace the original birmingham file with a directory, even with no clobber, level restrictions, and other settings. I have found no way to keep the original file.

-Rob

Anonymous
Wed 02 Jul 2008 06:19:54 PM UTC, original submission:

If Wget is made to process the URLs:

http://host/foo
http://host/foo/bar

and the server doesn't redirect http://host/foo to http://host/foo/, then Wget will fail to download bar, with a "not a directory" error.

It would be nice if, instead, Wget were to try to rename the file to "foo/index.html" (this would require an intermediate rename to a temporary name, and creation of the appropriate directory), so that this sort of usage doesn't throw Wget off.

Usually, one can get around this problem with the use of -nd; but there's no guarantee.

In implementing this, one would need to be careful also to ensure that, if host/foo/ already exists as a directory, then encountering http://host/foo wouldn't result in an error either, but would automatically result in saving the file to host/foo/index.html.

Micah Cowan <micahcowan>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by micahcowan (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 21 Aug 2008 11:44:28 PM UTCmicahcowanPlanned Release1.15=>1.14

    Back to the top


    Powered by Savane 3.1-cleanup