bugGNU Wget - Bugs: bug #53968, Decompressed data is written to...


bug #53968: Decompressed data is written to WARC file when using --compression=gzip

Submitted by:  None
Submitted on:  Thu 24 May 2018 12:50:50 PM UTC  
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  Tim Ruehsen <rockdaboot>
Originator Name:  William Prescott Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  1.19.5
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  trunk Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Tue 29 May 2018 10:19:24 AM UTC, comment #2: 

Thanks for fixing it. I applied it as a patch to the tar release of 1.19.5 and it appears to work correctly.

Tue 29 May 2018 08:56:00 AM UTC, comment #1: 

Thanks for reporting.

I made a potential fix (latest master branch). Could you test it or tell us how to easily reproduce, please.

If the fix doesn't work out, we reopen the issue.

Tim Ruehsen <rockdaboot>
Project AdministratorIn charge of this item.
Thu 24 May 2018 12:50:50 PM UTC, original submission:  

When using the "--compression=gzip" option, Wget will write decompressed data into "response" WARC records instead of recording the response exactly as it was received.
This also prevents one from correctly decoding data in the WARC file that had been transferred using chunked encoding.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #44208:  example.warc.gz added by None (5KiB - application/gzip - WARC and terminal output from running "wget -d --compression=gzip --warc-file=example 'http://lists.gnu.org/'")
file #44209:  wgetOutput.txt added by None (2KiB - text/plain - WARC and terminal output from running "wget -d --compression=gzip --warc-file=example 'http://lists.gnu.org/'")


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)



    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-05-29 rockdaboot StatusNone => Fixed
        Assigned toNone => rockdaboot
        Open/ClosedOpen => Closed
        Fixed ReleaseNone => trunk
    2018-05-24 None Attached File- => Added example.warc.gz, #44208
        Attached File- => Added wgetOutput.txt, #44209

    Back to the top

    Powered by Savane 3.5