bugGNU Wget - Bugs: bug #60865, wget puts robots.txt and headers...

 
 

bug #60865: wget puts robots.txt and headers into the downloaded file --no-clobber -A without robots.txt

Submitter:  Aleksej Serdyukov <watersky>
Submitted:  Wed 30 Jun 2021 11:46:54 PM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  rockdaboot
Originator Name:  Open/Closed:  Closed
Release:  trunk Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 05 Jul 2021 01:51:44 PM UTC, comment #2: 

Thanks for reporting. The fix should be merged into master (on Gitlab) when the CI pipeline succeeds: https://gitlab.com/gnuwget/wget/-/pipelines/331964209

Tim Ruehsen <rockdaboot>
Group administrator
Wed 30 Jun 2021 11:49:09 PM UTC, comment #1: 

Wget versions are 1.21.1 built from source, and 1.21 from Debian.

Aleksej Serdyukov <watersky>
Wed 30 Jun 2021 11:46:54 PM UTC, original submission:  

--no-directories is for convenience only.


wget --no-clobber --page-requisites  --accept "101547:*" --no-directories https://www.linux.org.ru/people/xaizek/profile


robots.txt ist downloaded into robots.txt.tmp and not deleted:


$ ls -1s
итого 2380
   8 101547:617472904.jpg
   4 robots.txt.tmp


Delete the image (only robots.txt.tmp will be left). Run again. Downloading the image takes a long time, and the .jpg file has robots.txt and HTTP headers included before the image file's data.

However, with "robots.txt" in --accept, robots.txt is saved directly as robots.txt, and the image is OK:


./wget --no-clobber --page-requisites  --accept "101547:*,robots.txt" --no-directories https://www.linux.org.ru/people/xaizek/profile


Aleksej Serdyukov <watersky>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by watersky (Submitted the item)
  • -email is unavailable- added by watersky
  •  

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-07-05 rockdaboot StatusNone Fixed
        Assigned toNone rockdaboot
        Open/ClosedOpen Closed
    2021-06-30 watersky Carbon-Copy- Added watersky

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code