bugGNU Wget - Bugs: bug #56946, Performance of `-r -A` is poor

 
 

bug #56946: Performance of `-r -A` is poor

Submitter:  None
Submitted:  Tue 24 Sep 2019 06:57:18 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  rockdaboot
Originator Name:  Jo Shields Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  1.19.4
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 25 Sep 2019 12:18:55 PM UTC, comment #2: 

The reason is, due to -r wget has to make HEAD requests to all URLs (>280.000) to find out about the content-type (in case it find text/html or text/css, it will scan those as well and follow the found URLs).

What you want is to filter out the URLs before probing. Wget1.x doesn't have this option. Wget2 has --filter-urls which does exactly that. Together with -l1 the download with Wget2 takes here 3s (250Mbit/s). The list contains

Saving 'ca-certificates-mono_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-2.0-1_6.0.0.334-0xamarin1+ubuntu1604b1_i386.deb'
Saving 'libmono-btls-interface4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_amd64.deb'
Saving 'libmono-cairo4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-codecontracts4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-corlib2.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-cscompmgd0.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-csharp4.0c-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-custommarshalers4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-db2-1.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-http4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-corlib4.5-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-2.0-1_6.0.0.334-0xamarin1+ubuntu1604b1_amd64.deb'
Saving 'libmono-2.0-1_6.0.0.334-0xamarin1+ubuntu1604b1_arm64.deb'
Saving 'libmono-2.0-1_6.0.0.334-0xamarin1+ubuntu1604b1_ppc64el.deb'
Saving 'libmono-2.0-dev_6.0.0.334-0xamarin1+ubuntu1604b1_arm64.deb'
Saving 'libmono-2.0-dev_6.0.0.334-0xamarin1+ubuntu1604b1_armhf.deb'
Saving 'libmono-2.0-1_6.0.0.334-0xamarin1+ubuntu1604b1_armhf.deb'
Saving 'libmono-2.0-dev_6.0.0.334-0xamarin1+ubuntu1604b1_i386.deb'
Saving 'libmono-2.0-dev_6.0.0.334-0xamarin1+ubuntu1604b1_ppc64el.deb'
Saving 'libmono-accessibility4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-cil-dev_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-btls-interface4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_arm64.deb'
Saving 'libmono-2.0-dev_6.0.0.334-0xamarin1+ubuntu1604b1_amd64.deb'
Saving 'libmono-cecil-private-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-btls-interface4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_ppc64el.deb'
Saving 'libmono-btls-interface4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_i386.deb'
Saving 'libmono-btls-interface4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_armhf.deb'
Saving 'libmono-microsoft-build-utilities-v4.0-4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-corlib4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-compilerservices-symbolwriter4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-data-tds4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-debugger-soft4.0a-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n-cjk4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n-mideast4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n-other4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n-rare4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n-west4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n4.0-all_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-messaging4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-management4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-ldap4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-messaging-rabbitmq4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-microsoft-build-engine4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-microsoft-build-tasks-v4.0-4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-microsoft-build4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-microsoft-build-framework4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-microsoft-web-infrastructure1.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-parallel4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-peapi4.0a-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-profiler_6.0.0.334-0xamarin1+ubuntu1604b1_i386.deb'
Saving 'libmono-rabbitmq4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-relaxng4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-profiler_6.0.0.334-0xamarin1+ubuntu1604b1_ppc64el.deb'
Saving 'libmono-security4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-microsoft-visualc10.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-microsoft-csharp4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-oracle4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-profiler_6.0.0.334-0xamarin1+ubuntu1604b1_amd64.deb'
Saving 'libmono-posix4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-profiler_6.0.0.334-0xamarin1+ubuntu1604b1_arm64.deb'
Saving 'libmono-profiler_6.0.0.334-0xamarin1+ubuntu1604b1_armhf.deb'


The solution with wget1.x is indeed to download the html file and manually scan the wanted URLs and download those in an extra step.

Tim Ruehsen <rockdaboot>
Group administrator
Wed 25 Sep 2019 11:21:40 AM UTC, comment #1: 

Yeah, quite weird. Thanks for reporting.

Wget2 doesn't have this 100% CPU behavior and just starts downloading.

BTW, in the given URL are ~280.000 URLs - but even with this amount, regex scanning shouldn't take longer than a few milliseconds.

Tim Ruehsen <rockdaboot>
Group administrator
Tue 24 Sep 2019 06:57:18 PM UTC, original submission:  

This takes 4 hours:


wget --quiet -nH --cut-dirs=6 -e robots=off -r --no-parent -A *_6.0.0.334-0xamarin1+ubuntu1604b1_*.deb https://download.mono-project.com/repo/debian/pool/main/m/mono/


This takes under 20 seconds to do the same thing:


wget --quiet -O- https://download.mono-project.com/repo/debian/pool/main/m/mono/ | sed '/.*_6.0.0.334-0xamarin1+ubuntu1604b1_.*.deb/!d; s#.*href="#https://download.mono-project.com/repo/debian/pool/main/m/mono/#; s/\.deb.*/.deb/' > downloadlist.txt
wget --quiet --input-file=downloadlist.txt


The performance difference in filtering is surprising.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-09-25 rockdaboot Assigned toNone rockdaboot
        Open/ClosedOpen Closed
    2019-09-25 rockdaboot StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code