bugGNU Wget - Bugs: bug #57413, No data detection, recursive mode...

 
 

bug #57413: No data detection, recursive mode won;t work if received HTTP responses are always comming in gzip compression

Submitter:  zahlabut <zahlabut>
Submitted:  Sun 15 Dec 2019 10:11:43 AM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  rockdaboot
Originator Name:  Arkady Shtempler Open/Closed:  Closed
Release:  1.14 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  1.20
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 16 Dec 2019 10:23:45 AM UTC, comment #6: 

Hi Tim!

Ok, it did work for me without adding -i flag

Thanks!

zahlabut <zahlabut>
Mon 16 Dec 2019 09:29:26 AM UTC, comment #5: 

Hi zahlabut,

leave away -i, because it consumes '-r' as a file name and thus wget doesn't work recusively.

I just used your command below without '-i' and it seems to work (downloading plenty files).

Tim Ruehsen <rockdaboot>
Group administrator
Mon 16 Dec 2019 08:31:25 AM UTC, comment #4: 

Tim, have you tried to reproduce on "1.19.5" (Current version I have, this is the version that is actually installed using "yum install wget" and I have no idea how to upgrade to 1.20)

I've tried to use --compression flag as you've mentioned with "gzip" or "auto" values, in both cases I'm getting the same bug.
So my command is:
wget -i -r --no-parent --compression=gzip https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e9a/692822/8/check/neutron-tempest-plugin-bgpvpn-bagpipe/e9ade00/


My Operation System is: Red Hat Enterprise Linux release 8.1 (Ootpa)

wget -i -r --no-parent --compression=gzip https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e9a/692822/8/check/neutron-tempest-plugin-bgpvpn-bagpipe/e9ade00/


### Log file ###
--2019-12-16 08:22:04--  https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e9a/692822/8/check/neutron-tempest-plugin-bgpvpn-bagpipe/e9ade00/
Resolving storage.bhs1.cloud.ovh.net (storage.bhs1.cloud.ovh.net)... 142.44.140.9
Connecting to storage.bhs1.cloud.ovh.net (storage.bhs1.cloud.ovh.net)|142.44.140.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 701 [text/html]
Saving to: ‘index.html.2’

     0K                                                       100% 10.9M=0s

2019-12-16 08:22:09 (10.9 MB/s) - ‘index.html.2’ saved [2393]

-r: No such file or directory
No URLs found in -r.
FINISHED --2019-12-16 08:22:09--
Total wall clock time: 4.4s
Downloaded: 1 files, 701 in 0s (10.9 MB/s)


Tim - can you provide the command that did work for you?

zahlabut <zahlabut>
Sun 15 Dec 2019 07:42:31 PM UTC, comment #3: 

I DID reproduce your issue (with 1.20.3 on Debian and with Wget from git master). As I said, you have to enable compression support and remove --header from your example - then your example worked here.

Tim Ruehsen <rockdaboot>
Group administrator
Sun 15 Dec 2019 01:00:15 PM UTC, comment #2: 

Hi Tim!

My bad, I've mentioned incorrect WGET version in Bug field, my version is:

[stack@undercloud-0 LogTool_Python3]$ wget -V
GNU Wget 1.19.5 built on linux-gnu.

Unfortunately I cannot change this version inside the Bug description: https://savannah.gnu.org/bugs/?57413
Anyway, my suggestion to you would be trying to reproduce my problem (you have all needed scenario details to do that) before changing its status to fixed.

Thanks!

zahlabut <zahlabut>
Sun 15 Dec 2019 11:22:49 AM UTC, comment #1: 

Release 1.14 is very old.
Recent versions of Wget allow for --compression=auto or --compression=gzip to solve your issue. You then should remove the --header option.

Tim Ruehsen <rockdaboot>
Group administrator
Sun 15 Dec 2019 10:11:43 AM UTC, original submission:  

I'm trying to use WGET to recursively download all content from some Web page, but in fact only index.html is downloaded, without even being decompressed (It’s saved as “gzip”)
In my case, the site I’m working with, is simply ignores “Accept-Encoding” HTTP request header and always responds with “gzip” content, so there is no way to get something else than “gzip”.
It’s not standard WEB site behavior, but this is what I have to deal with.
I’ve tried adding “Accept-Encoding”: identity or just not existing compression values like:
“Accept-Encoding”: zahlabut and the result was always the same, which is “gzip” no matter of sent value.

#################################################################
[ashtempl@ashtempl stam]$ wget -r --no-parent --header "Accept-Encoding:identity" https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e9a/692822/8/check/neutron-tempest-plugin-bgpvpn-bagpipe/e9ade00/ -o wget.log
[ashtempl@ashtempl stam]$
[ashtempl@ashtempl stam]$
[ashtempl@ashtempl stam]$
[ashtempl@ashtempl stam]$
[ashtempl@ashtempl stam]$ tree
.
├── storage.bhs1.cloud.ovh.net
│   └── v1
│       └── AUTH_dcaab5e32b234d56b626f72581e3644c
│           └── zuul_opendev_logs_e9a
│               └── 692822
│                   └── 8
│                       └── check
│                           └── neutron-tempest-plugin-bgpvpn-bagpipe
│                               └── e9ade00
│                                   └── index.html
└── wget.log

9 directories, 2 files
[ashtempl@ashtempl stam]$ file storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e9a/692822/8/check/neutron-tempest-plugin-bgpvpn-bagpipe/e9ade00/index.html
storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e9a/692822/8/check/neutron-tempest-plugin-bgpvpn-bagpipe/e9ade00/index.html: gzip compressed data, last modified: Sun Dec  8 12:31:40 2019, max compression
#################################################################

I guess that WGET fails to read received HTML once it’s saved locally as compressed file and that is why “recursive mode” is actually fails (stoped being executed).
I think that WGET should have the ability (Data Detection) to try and detect the type of received content (just like Linux “file” command does).
So once the received data will be detected as “gzip” it will be uncompressed using “gzip” and will read it’s content in order to continue with “recursive” mode (getting all URLs e.t.c).


### whet.log file content ###
--2019-12-15 11:51:48--  https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e9a/692822/8/check/neutron-tempest-plugin-bgpvpn-bagpipe/e9ade00/
Resolving storage.bhs1.cloud.ovh.net (storage.bhs1.cloud.ovh.net)... 142.44.140.9
Connecting to storage.bhs1.cloud.ovh.net (storage.bhs1.cloud.ovh.net)|142.44.140.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 701 [text/html]
Saving to: ‘storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e9a/692822/8/check/neutron-tempest-plugin-bgpvpn-bagpipe/e9ade00/index.html’

     0K                                                       100% 60.7M=0s

2019-12-15 11:51:49 (60.7 MB/s) - ‘storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e9a/692822/8/check/neutron-tempest-plugin-bgpvpn-bagpipe/e9ade00/index.html’ saved [701/701]

FINISHED --2019-12-15 11:51:49--
Total wall clock time: 0.9s
Downloaded: 1 files, 701 in 0s (60.7 MB/s)

zahlabut <zahlabut>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by zahlabut (Submitted the item)
  • -email is unavailable- added by zahlabut
  •  

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-12-15 rockdaboot StatusNone Fixed
        Assigned toNone rockdaboot
        Open/ClosedOpen Closed
        Fixed ReleaseNone 1.20
    2019-12-15 zahlabut Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-bb6a.
    Corresponding source code