bugGNU Wget - Bugs: bug #59363, wget with --wait (crawling...

 
 

bug #59363: wget with --wait (crawling specific domain's specific url with wait)

Submitter:  None
Submitted:  Tue 27 Oct 2020 02:43:35 AM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Makoto Wada Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.20
Operating System:  None Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 06 Nov 2020 03:06:26 AM UTC, comment #2: 

Dear Tim Ruehsen, you have suspected correctly.

comment #1:

> You say that you expect --wait=N to wait N seconds before a retrieval !?


So the word between is not a mistake in documentation (my optimistic wish is dashed).
I presume one condition that satisfies between is when it is recursively downloading.

> How can we improve the docs ?
> Please feel free to suggest a change or the wording that suits best to you.


Since my command example does not satisfy the between condition, here is a suggestion of wording of the documentation. I think it clarifies when between condition is satisfied.
 -w,  --wait=SECONDS              wait SECONDS between retrievals
                                  (e.g. waits between recursive downloads when -r is used)

 Makoto Wada <makwada>
Sat 31 Oct 2020 05:49:14 PM UTC, comment #1: 

Not sure if I understand you correctly.
You say that you expect --wait=N to wait N seconds before a retrieval !?
The documentation is pretty clear saying between retrievals, not before and not after.

In other words, the first retrieval happens immediately. If you want to pause before or after, explicitly sleep/wait before or starting wget or after wget stops.

How can we improve the docs ?
Please feel free to suggest a change or the wording that suits best to you.

Tim Ruehsen <rockdaboot>
Group administrator
Tue 27 Oct 2020 02:43:35 AM UTC, original submission:  

To whom it may concern,

1. I am using the following
    - GNU Wget 1.20.3 built on linux-gnu.
    - Ubuntu 20.04 LTS (Focal Fossa)
    - WSL (v1, File version: 10.0.18362.1049)
    - Windows 10 Pro (64 bit)
2. I am invoking the command directly through WSL default terminal (bash: )
3. I have executed the following command which I thought it would start wget process and suspend it until 10 seconds has elapsed. In other words, it would start downloading after 10 seconds. Instead, it immediately executes.
    wget https://●●●.com/top/index.html --wait=10 --output-document ./top_index.html
I read its documentation which mentions the following (from wget --help):

> -w, --wait=SECONDS       wait SECONDS between retrievals
>     --waitretry=SECONDS  wait 1..SECONDS between retries of a retrieval
>     --random-wait        wait from 0.5WAIT...1.5WAIT secs between retrievals


Maybe I am reading the documentation wrong but the word "between" sounds sketchy. What does the word between refer to? I think the documentation should be more specific.

I tried finding similar post but I do not think it is mentioned in other posts. I looked at the following one but this one talks about wait between retires:

- wget option "--wait" does not work
  https://savannah.gnu.org/bugs/?func=detailitem&item_id=41906

 Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makwada (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    •  

    No changes have been made to this item

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.14-8eb0.
    Corresponding source code