bugGNU Wget - Bugs: bug #59086, --page-requisites not always...


bug #59086: --page-requisites not always working when creating a warc file

Submitter:  Thomas Egense <thomasegense>
Submitted:  Wed 09 Sep 2020 08:52:02 AM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  GNU/Linux
Reproducibility:  None Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Wed 09 Sep 2020 08:59:37 AM UTC, comment #1: 

minor correction:
and links are also NOT followed

Thomas Egense <thomasegense>
Wed 09 Sep 2020 08:52:02 AM UTC, original submission:  

Url example: https://jyllands-posten.dk/

How to reproduce:
echo "https://jyllands-posten.dk/" >> url_list.txt

wget --level=1 --recursive --warc-cdx --page-requisites --warc-file=jp --warc-max-size=1G -i url_list.txt

The source code for the page is downloaded in the warc (last record). But none of the images are downloaded and links are also followed (--recursive parameter).

It is probably due to some HTTPS redirection, but since the
source code is downloaded correct, it should still be possible to follow links and download page requisites.

Thomas Egense <thomasegense>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by thomasegense (Submitted the item)

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-3cd0.
    Corresponding source code