bugGNU Wget - Bugs: bug #29647, wget does not automatically create...

 
 

bug #29647: wget does not automatically create necessary local directories

Submitted by:  Jarl Friis <jarl>
Submitted on:  Fri 23 Apr 2010 08:42:32 AM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: 1.11.4Operating System: GNU/Linux
Reproducibility: Every TimeFixed Release: None
Planned Release: NoneRegression: None
Work Required: NonePatch Included: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 18 Apr 2012 08:40:16 AM UTC, comment #9:

The first patch leaks memory. Sorry!

This one should be better.

~mk

(file #25677)

Anonymous
Mon 16 Apr 2012 09:24:44 PM UTC, comment #8:

Attached patch works for me.

Can't give any guarantee for VMS, Windows, or any other exotic OS.

~mk

(file #25664)

Anonymous
Tue 10 Apr 2012 08:55:04 PM UTC, comment #7:

Jarl Friis wrote:

> When wget reaches out for '/delivery/c/2/file.png' it fails like this:
> ...
> Cannot write to `delivery/c/2/placemark_icon.png' (Not a directory).


The key phrase here is "Not a directory", because when wget tries to download 'delivery/c/2/file.png', 'delivery' already exists as a regular file, so subdirectory 'delivery/c' cannot be created, neither can 'delivery/c/2', and finally there's no place to store 'file.png' to.

This does not directly depend on '--mirror' or '--page-requisites'.

Fri 23 Apr 2010 09:16:31 AM UTC, Jarl Friis wrote:

> with trailing slash
> ...
> then it works as expected.


Yeah, because with a trailing slash you force 'delivery' to be created as a directory. Then wget loads 'http://localhost:3000/delivery/' into 'delivery/index.html', and 'c' and '2' can be created as subdirectories.

Tue 25 May 2010 06:31:39 PM UTC, Steve Wainstead wrote:

> swain.example.com/store.php/to: Not a directory ...


I bet, you already have store.php as a regular file.

Again '--page-requisites', '--mirror', '--restrict-file-names=unix', and '--no-host-directories' are not related to this bug.

Here is a simple test case:

# -x, --force-directories force creation of directories.
# -nH, --no-host-directories don't create host directories.
# -P, --directory-prefix=PREFIX save files to PREFIX/...

'-x' is essential for these tests.
'-nH' just makes the paths shorter. It has no influence on the test results.
'-P' is used to keep the cases apart.

# ok

# no problem, if any number of directories exist

# fail, if any path component except the last one pre-exists as a regular file

# if the last directory pre-exists as a regular file, it is silently removed
# (c.f. https://savannah.gnu.org/bugs/?23756)

# if the destination file exists as a directory,
# wget downloads to a different filename: bug.orig.png.1

~mk

BTW, comment #5 (l...x...com) is spam!

Anonymous
Fri 24 Sep 2010 08:11:20 PM UTC, comment #6:

strace is here:

http://www.panix.com/~swain/strace.txt

Command was:

strace wget -H --domains=testing.polaroidstudio.com,www.polaroidstudio.com,store.polaroidstudio.com --reject '*.pdf' --no-check-certificate --post-data='loginstring=testing&password=112233&subdomain=testing&WGET_TESTING=true' --mirror https://www.polaroidstudio.com/login.php >strace.out 2>&1

Steve Wainstead <wainstead>
Fri 24 Sep 2010 05:29:22 PM UTC, comment #5:

http://loanshark411.xanga.com

Anonymous
Thu 27 May 2010 08:27:30 AM UTC, comment #4:

I couldn't reproduce this problem.

Can you please provide more information?

Can you attach a `strace' log of wget?

Giuseppe Scrivano <gscrivano>
Project Administrator
Tue 25 May 2010 06:31:39 PM UTC, comment #3:

I've tried to work around this same problem, on both OSX and GNU/Linux, but to no avail:

wget --page-requisites --post-data='uname=someone&gallerypassword=secret' --mirror http://swain.example.com/loginpage.php

(I also tried specifying --restrict-file-names=unix and --no-host-directories)

The man page says '/' will be escaped with its hex code, but this does not seem to work:

Connecting to swain.example.com (swain.example.com)|10.1.1.180|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: /store.php/to/select_photo?product=note_cards_12_pk&fromCat=print_11x14_chocolate_frame [following]
--2010-05-25 11:21:40-- http://swain.example.com/store.php/to/select_photo?product=note_cards_12_pk&fromCat=print_11x14_chocolate_frame
Connecting to swain.example.com (swain.example.com)|10.1.1.180|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
swain.example.com/store.php/to: Not a directoryswain.example.com/store.php/to/select_photo?product=note_cards_12_pk&fromCat=print_11x14_chocolate_frame: Not a directory

Cannot write to `swain.example.com/store.php/to/select_photo?product=note_cards_12_pk&fromCat=print_11x14_chocolate_frame' (Not a directory).

Steve Wainstead <wainstead>
Fri 23 Apr 2010 09:16:31 AM UTC, comment #2:

I figured out that if I do a (with trailing slash)

wget -e robots=off --no-host-directories --mirror --page-requisites http://localhost:3000/delivery/

Instead of
wget -e robots=off --no-host-directories --mirror --page-requisites http://localhost:3000/delivery

then it works as expected.

This may very well be documented behaviour. If so, sorry for the inconvenience.

Jarl

Jarl Friis <jarl>
Fri 23 Apr 2010 08:58:38 AM UTC, comment #1:

I figured out that it had nothing to do with the fact that 'http://localhost:3000/delivery/c/2' responds 404, I changed it to respond 200, but the problem still occurs.

If I mirror the file directly like this:
wget -e robots=off --no-host-directories --mirror --page-requisites http://localhost:3000/delivery/c/2/placemark_icon.png

Then all the necessary directories (delivery/c/2/) are created on the fly.

Jarl Friis <jarl>
Fri 23 Apr 2010 08:42:32 AM UTC, original submission:

I have a site with
http://localhost:3000/delivery/index.html which links to '/delivery/c/2/placemark_icon.png'

I use wget like this:
wget -e robots=off --no-host-directories --mirror --page-requisites http://localhost:3000/delivery

When wget reaches out for '/delivery/c/2/file.png' it fails like this:

--2010-04-23 10:36:25-- http://localhost:3000/delivery/c/2/placemark_icon.png
Connecting to localhost|127.0.0.1|:3000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17460 (17K) [image/png]
delivery/c/2: Not a directorydelivery/c/2/placemark_icon.png: Not a directory

Cannot write to `delivery/c/2/placemark_icon.png' (Not a directory).
FINISHED --2010-04-23 10:36:25--
Downloaded: 27 files, 864K in 0.08s (10.7 MB/s)

The problem is that the local directory 'delivery/c/2' is not created automatically by wget (that is expected). The cause of this is most likely that
the page 'http://localhost:3000/delivery/c/2' responds (intentionally, because it is a web-app) with a '404 Not found'. Maybe that is the reason that wget does not create the local directory 'delivery/c/2'

Jarl

Jarl Friis <jarl>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #25677:  05_makedir.diff added by None (2KiB - text/x-patch - new version, memory leak fixed)
file #25664:  05_makedir.diff added by None (2KiB - text/x-patch - patch 05_makedir.diff)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by wainstead (Posted a comment)
  • -unavailable- added by jarl (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 18 Apr 2012 08:40:16 AM UTCNoneAttached File-=>Added 05_makedir.diff, #25677
    Mon 16 Apr 2012 09:24:44 PM UTCNoneAttached File-=>Added 05_makedir.diff, #25664

    Back to the top


    Powered by Savane 3.1-cleanup1