bugGNU Wget - Bugs: bug #53322, Add option to let page-requisites...


bug #53322: Add option to let page-requisites bypass no-parent

Submitter:  David <mcdado>
Submitted:  Sun 11 Mar 2018 11:43:25 AM UTC
Category:  Feature Request Severity:  3 - Normal
Priority:  5 - Normal Status:  Invalid
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  None Operating System:  None
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment (Rich Markup)

( Jump to the original submission )

Sun 15 Apr 2018 09:40:43 AM UTC, comment #9: 

Probably I must have missed something, because using `--recursive --page-requisites --no-parent` does indeed do what I want. Strange, I could swear it was behaving differently before.

David <mcdado>
Sun 08 Apr 2018 10:33:27 AM UTC, comment #8: 

Just remove --recursive and you get what you want.
Keep in mind that wget doesn't run Javascript, so no dynamic created requisites can be downloaded.

$ tree addyosmani.com/
├── cdn-cgi
│   └── scripts
│       └── d07b1474
│           └── cloudflare-static
│               └── email-decode.min.js
└── resources
    └── essentialjsdesignpatterns
        ├── book
        │   ├── images
        │   │   ├── base.png
        │   │   └── ns1.png
        │   ├── index.html
        │   ├── scripts
        │   │   └── vendor.js
        │   └── styles
        │       └── vendor.css
        └── cover
            └── cover.jpg

11 directories, 7 files

Tim Ruehsen <rockdaboot>
Group administrator
Sat 07 Apr 2018 11:41:08 PM UTC, comment #7: 

Okay, I've got an example:

wget --recursive --page-requisites --convert-links --adjust-extension https://addyosmani.com/resources/essentialjsdesignpatterns/book/

I would like to download everything under this path: "/resources/essentialjsdesignpatterns/book/", in this example index.html is the only page, but if there was a second html page linked from index, that would be downloaded too.

Plus anything page requisites from other paths, like "../../../cdn-cgi/" but not other pages, like html documents in "/blog/".

David <mcdado>
Thu 05 Apr 2018 07:58:13 AM UTC, comment #6: 

Little correction: the "...is no reference..." should be "... no page requisite..."

Tim Ruehsen <rockdaboot>
Group administrator
Thu 05 Apr 2018 07:54:44 AM UTC, comment #5: 

Maybe I was a bit unclear, sorry for that.

In your document (http://www.oreilly.com/openbook/osfreesoft/book/) is no reference to any other resource on the same domain (www.oreilly.com).

So to download anything with --page-requisite, you'll need -H.

Without it, you just get your index.html and that's it. You didn't request anything else.

So what exactly do you want. Please make a list of documents/resources you would like to see in the end. With that we also have a potential test case, if it comes to implement a new feature.

Tim Ruehsen <rockdaboot>
Group administrator
Wed 04 Apr 2018 11:01:01 PM UTC, comment #4: 

Thanks for your answer, but it's missing the point of the original question.

I basically want to download everything under the current folder (recursive, no parent) but i also want to download css/js that are on the same domain but higher in the hierarchy (page requisites, recursive).

You understand what I mean? For example, in the previous example I'd like to download css/js and the linked pdf files.

David <mcdado>
Wed 04 Apr 2018 12:20:40 PM UTC, comment #3: 

Then you won't need --recursive.
And looking at that page: the links all go to different domains - so will need -H as well.

wget -H --page-requisites http://www.oreilly.com/openbook/osfreesoft/book/

does the job for me:

FINISHED --2018-04-04 14:17:29--
Total wall clock time: 15s
Downloaded: 41 files, 569K in 0.6s (888 KB/s)

Tim Ruehsen <rockdaboot>
Group administrator
Wed 04 Apr 2018 10:54:35 AM UTC, comment #2: 

Well, I don't want to download a whole site!

wget --recursive --page-requisites http://www.oreilly.com/openbook/osfreesoft/book/

It will start downloading everything that is linked on the page, not just the current directory.

David <mcdado>
Wed 04 Apr 2018 09:47:12 AM UTC, comment #1: 

Why don't you leave away --no-parent then ?

--page-requisites alone should do what you want.

Tim Ruehsen <rockdaboot>
Group administrator
Sun 11 Mar 2018 11:43:25 AM UTC, original submission:  

When using `--no-parent` and `--page-requisites`, if the page requires images or other requisites from that domain but higher in hierarchy, it will not proceed to download those requisites because of the `no-parent` option. While `no-parent` is useful for web pages (without it you download a whole site, especially if you `--mirror`) so it would be good to have an extra option to allow downloading page requisites that are higher in the hierarchy.

David <mcdado>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by darnir (Updated the item)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by mcdado (Submitted the item)




    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-11-13 darnir StatusNone Invalid
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.11