bugGNU Wget - Bugs: bug #60246, When writing stdout into a pipe,...


bug #60246: When writing stdout into a pipe, wget never terminates

Submitter:  None
Submitted:  Tue 16 Mar 2021 08:21:20 PM UTC
Category:  Program Logic Severity:  4 - Important
Priority:  5 - Normal Status:  In Progress
Privacy:  Public Assigned to:  darnir
Originator Name:  U.W. Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  trunk
Operating System:  Others (Please Specify) Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Mon 11 Oct 2021 08:39:44 PM UTC, comment #2: 

Okay, this is a crazy issue, and I'm surprised no one has reported it in so many years! I think this issue has likely existed for 15 years if not more.

The problem is simple:
We pass the file descriptor of the file to which the downloaded document is saved in order to parse it for any links. So, when you combine -p with -O-, what you get is that Wget waits indefinitely on a read() call on stdout. Which will NEVER return anything.

In order to handle -p and -O properly, I think we should download to a temporary file before parsing and then append it to the output file. This is especially important when using -O-. When writing to a real file, we could probably optimize the process by writing directly to the file and then calling fseek() to only process the newly downloaded part of the file.

I would wager this is also an issue with -r -O-.

Darshit Shah <darnir>
Group administrator
Mon 11 Oct 2021 08:15:55 PM UTC, comment #1: 

I can verify that this is an issue on all platforms. However, the issue is not when writing to pipes, but rather when writing to a pipe and using --page-requisites

Darshit Shah <darnir>
Group administrator
Tue 16 Mar 2021 08:21:20 PM UTC, original submission:  

When writing to a terminal as stdout, wget termintesnormally:

wget -O - -p https://de.xhamster.com/photos/gallery/scarlett-johansson-15779909/

works like a charm.
Piping this output into some other program is not so successful:

wget -O - -p https://de.xhamster.com/photos/gallery/scarlett-johansson-15779909/ | wc

never really finishes and just hangs.

Cygwin under W10. Version is 1.20.3



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by darnir (Posted a comment)

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-10-11 darnir Severity3 - Normal 4 - Important
        StatusConfirmed In Progress
        Assigned toNone darnir
    2021-10-11 darnir StatusNone Confirmed
        Release1.20 trunk

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code