bugGNU Wget - Bugs: bug #29647, wget does not automatically create...

 
 

bug #29647: wget does not automatically create necessary local directories

Submitter:  Jarl Friis <jarl>
Submitted:  Fri 23 Apr 2010 08:42:32 AM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.11.4 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 12 Oct 2018 02:44:54 AM UTC, comment #10: 

Re-reproduced in wget 1.19.5, however the patch here no longer applies
(it needs file_exists_p(dir) -> file_exists_p(dir, NULL))

I attached a patch updated against this version.

(file #45186)

Anonymous
Wed 18 Apr 2012 08:40:16 AM UTC, comment #9: 

The first patch leaks memory. Sorry!

This one should be better.

~mk

(file #25677)

Anonymous
Mon 16 Apr 2012 09:24:44 PM UTC, comment #8: 

Attached patch works for me.

Can't give any guarantee for VMS, Windows, or any other exotic OS.

~mk

(file #25664)

Anonymous
Tue 10 Apr 2012 08:55:04 PM UTC, comment #7: 

Jarl Friis wrote:

> When wget reaches out for '/delivery/c/2/file.png' it fails like this:
> ...
> Cannot write to `delivery/c/2/placemark_icon.png' (Not a directory).


The key phrase here is "*Not a directory*", because when wget tries to download 'delivery/c/2/file.png', 'delivery' already exists as a regular file, so subdirectory 'delivery/c' cannot be created, neither can 'delivery/c/2', and finally there's no place to store 'file.png' to.

This does not directly depend on '--mirror' or '--page-requisites'.

Fri 23 Apr 2010 09:16:31 AM UTC, Jarl Friis wrote:

> with trailing slash
> ...
> then it works as expected.


Yeah, because with a trailing slash you force 'delivery' to be created as a directory. Then wget loads 'http://localhost:3000/delivery/' into 'delivery/index.html', and 'c' and '2' can be created as subdirectories.

Tue 25 May 2010 06:31:39 PM UTC, Steve Wainstead wrote:

> swain.example.com/store.php/to: Not a directory ...


I bet, you already have store.php as a regular file.

Again '--page-requisites', '--mirror', '--restrict-file-names=unix', and '--no-host-directories' are not related to this bug.

Here is a simple test case:


URL=http://savannah.gnu.org/images/Savannah.theme/contexts/bug.orig.png
rm -r t0[123456789]


#  -x,  --force-directories        force creation of directories.
#  -nH, --no-host-directories      don't create host directories.
#  -P,  --directory-prefix=PREFIX  save files to PREFIX/...

'-x' is essential for these tests.
'-nH' just makes the paths shorter. It has no influence on the test results.
'-P' is used to keep the cases apart.


wget -x -nH -Pt01 $URL
ls -lR t01

# ok

# no problem, if any number of directories exist

mkdir -p t02
wget -x -nH -Pt02 $URL
mkdir -p t03/images
wget -x -nH -Pt03 $URL
mkdir -p t04/images/Savannah.theme
wget -x -nH -Pt04 $URL
mkdir -p t05/images/Savannah.theme/contexts
wget -x -nH -Pt05 $URL


# fail, if any path component except the last one pre-exists as a regular file

mkdir -p t06
touch t06/images
wget -x -nH -Pt06 $URL
mkdir -p t07/images
touch t07/images/Savannah.theme
wget -x -nH -Pt07 $URL


# if the last directory pre-exists as a regular file, it is silently removed
# (c.f. https://savannah.gnu.org/bugs/?23756)

mkdir -p t08/images/Savannah.theme
touch t08/images/Savannah.theme/contexts
wget -x -nH -Pt08 $URL


# if the destination file exists as a directory,
# wget downloads to a different filename: bug.orig.png.1

mkdir -p t09/images/Savannah.theme/contexts/bug.orig.png
wget -x -nH -Pt09 $URL


~mk


BTW, comment #5 (l...x...com) is spam!

Anonymous
Fri 24 Sep 2010 08:11:20 PM UTC, comment #6: 

strace is here:

http://www.panix.com/~swain/strace.txt

Command was:

strace wget -H --domains=testing.polaroidstudio.com,www.polaroidstudio.com,store.polaroidstudio.com --reject '*.pdf' --no-check-certificate --post-data='loginstring=testing&password=112233&subdomain=testing&WGET_TESTING=true' --mirror https://www.polaroidstudio.com/login.php >strace.out 2>&1                                       

Steve Wainstead <wainstead>
Fri 24 Sep 2010 05:29:22 PM UTC, comment #5: 
Anonymous
Thu 27 May 2010 08:27:30 AM UTC, comment #4: 

I couldn't reproduce this problem.

Can you please provide more information?


Can you attach a `strace' log of wget?

Giuseppe Scrivano <gscrivano>
Group administrator
Tue 25 May 2010 06:31:39 PM UTC, comment #3: 

I've tried to work around this same problem, on both OSX and GNU/Linux, but to no avail:

wget --page-requisites --post-data='uname=someone&gallerypassword=secret' --mirror http://swain.example.com/loginpage.php

(I also tried specifying --restrict-file-names=unix and --no-host-directories)

The man page says '/' will be escaped with its hex code, but this does not seem to work:

Connecting to swain.example.com (swain.example.com)|10.1.1.180|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: /store.php/to/select_photo?product=note_cards_12_pk&fromCat=print_11x14_chocolate_frame [following]
--2010-05-25 11:21:40--  http://swain.example.com/store.php/to/select_photo?product=note_cards_12_pk&fromCat=print_11x14_chocolate_frame
Connecting to swain.example.com (swain.example.com)|10.1.1.180|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
swain.example.com/store.php/to: Not a directoryswain.example.com/store.php/to/select_photo?product=note_cards_12_pk&fromCat=print_11x14_chocolate_frame: Not a directory

Cannot write to `swain.example.com/store.php/to/select_photo?product=note_cards_12_pk&fromCat=print_11x14_chocolate_frame' (Not a directory).

Steve Wainstead <wainstead>
Fri 23 Apr 2010 09:16:31 AM UTC, comment #2: 

I figured out that if I do a (with trailing slash)

wget -e robots=off --no-host-directories --mirror --page-requisites http://localhost:3000/delivery/

Instead of
wget -e robots=off --no-host-directories --mirror --page-requisites http://localhost:3000/delivery

then it works as expected.

This may very well be documented behaviour. If so, sorry for the inconvenience.

Jarl

Jarl Friis <jarl>
Fri 23 Apr 2010 08:58:38 AM UTC, comment #1: 

I figured out that it had nothing to do with the fact that 'http://localhost:3000/delivery/c/2' responds 404, I changed it to respond 200, but the problem still occurs.

If I mirror the file directly like this:
wget -e robots=off --no-host-directories --mirror --page-requisites http://localhost:3000/delivery/c/2/placemark_icon.png

Then all the necessary directories (delivery/c/2/) are created on the fly.

Jarl Friis <jarl>
Fri 23 Apr 2010 08:42:32 AM UTC, original submission:  

I have a site with
http://localhost:3000/delivery/index.html which links to '/delivery/c/2/placemark_icon.png'

I use wget like this:
wget -e robots=off --no-host-directories --mirror --page-requisites http://localhost:3000/delivery

When wget reaches out for '/delivery/c/2/file.png' it fails like this:

--2010-04-23 10:36:25--  http://localhost:3000/delivery/c/2/placemark_icon.png
Connecting to localhost|127.0.0.1|:3000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17460 (17K) [image/png]
delivery/c/2: Not a directorydelivery/c/2/placemark_icon.png: Not a directory

Cannot write to `delivery/c/2/placemark_icon.png' (Not a directory).
FINISHED --2010-04-23 10:36:25--
Downloaded: 27 files, 864K in 0.08s (10.7 MB/s)


The problem is that the local directory 'delivery/c/2' is not created automatically by wget (that is expected). The cause of this is most likely that
the page 'http://localhost:3000/delivery/c/2' responds (intentionally, because it is a web-app) with a '404 Not found'. Maybe that is the reason that wget does not create the local directory 'delivery/c/2'

Jarl

Jarl Friis <jarl>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #45186:  06_makedir.diff added by None (2KiB - application/octet-stream)
file #25677:  05_makedir.diff added by None (2KiB - text/x-patch - new version, memory leak fixed)
file #25664:  05_makedir.diff added by None (2KiB - text/x-patch - patch 05_makedir.diff)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by wainstead (Posted a comment)
  • -email is unavailable- added by jarl (Submitted the item)
  •  

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-10-12 None Attached File- Added 06_makedir.diff, #45186
    2012-04-18 None Attached File- Added 05_makedir.diff, #25677
    2012-04-16 None Attached File- Added 05_makedir.diff, #25664

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code