bugGNU Wget - Bugs: bug #31909, -r does not work if...

 
 

bug #31909: -r does not work if htaccess/htpassword

Submitted by:  Romain Vimont <romainvimont>
Submitted on:  Fri 17 Dec 2010 09:09:43 PM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: Fixed
Privacy: PublicAssigned to: Giuseppe Scrivano <gscrivano>
Originator Name: Open/Closed: Closed
Release: NoneOperating System: GNU/Linux
Reproducibility: NoneFixed Release: None
Planned Release: NoneRegression: None
Work Required: NonePatch Included: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 21 Jan 2011 11:34:15 AM UTC, comment #5:

it is a problem with your robots.txt.

http://en.wikipedia.org/wiki/Robots_exclusion_standard

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Thu 20 Jan 2011 07:02:20 PM UTC, comment #4:

It works with -e robots=off.

Here is my robots.txt file :
User-agent: *
Disallow: /

Is it a problem from wget, or from my robots.txt configuration?
Why wget does it read this robots file?

Romain Vimont <romainvimont>
Tue 11 Jan 2011 03:19:36 PM UTC, comment #3:

what does happen using -e robots=off ?

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Mon 27 Dec 2010 08:23:10 PM UTC, comment #2:

I just attached the result of the command you asked :
https://savannah.gnu.org/bugs/download.php?file_id=22299

Romain Vimont <romainvimont>
Sat 25 Dec 2010 07:29:18 PM UTC, comment #1:

can you please provide more information?

Can you attach the output you get using -d?

wget -d -rEHkp --user=user --password=password http://mysite.tld

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Fri 17 Dec 2010 09:09:43 PM UTC, original submission:

If I "wget -r" on one of my blog without htaccess/htpassword, it works fine.

If I "wget -r --user=user --password=password" on another blog protected by htaccess/htpasswd, it just download the first page :


$ wget -rEHkp --user=user --password=password http://mysite.tld
--2010-12-17 22:07:30-- http://mysite.tld/
Resolving mysite.tld... 11.22.33.44
Connecting to mysite.tld|11.22.33.44|:80... connected.
HTTP request sent, awaiting response... 401 Authorization Required
Reusing existing connection to mysite.tld:80.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `mysite.tld/index.html'

[ <=> ] 29,538 --.-K/s in 0.01s

2010-12-17 22:07:30 (2.13 MB/s) - `mysite.tld/index.html' saved [29538]

FINISHED --2010-12-17 22:07:30--
Downloaded: 1 files, 29K in 0.01s (2.13 MB/s)
Converting mysite.tld/index.html... 5-1
Converted 1 files in 0.001 seconds.


It should recursively download the whole blog, not just index.html.

Romain Vimont <romainvimont>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #22299:  log added by None (116KiB - application/octet-stream - Here is the output of 'wget -d -rEHkp --user=user --password=password http://mysite.tld')

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by romainvimont (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 21 Jan 2011 11:34:15 AM UTCgscrivanoStatusNeeds Investigation=>Fixed
      Open/ClosedOpen=>Closed
    Mon 27 Dec 2010 08:20:45 PM UTCNoneAttached File-=>Added log, #22299
    Sat 25 Dec 2010 07:29:18 PM UTCgscrivanoStatusNone=>Needs Investigation
      Assigned toNone=>gscrivano

    Back to the top


    Powered by Savane 3.1-cleanup1