bugGNU Wget - Bugs: bug #51181, Unexpected "Redirecting...

 
 

bug #51181: Unexpected "Redirecting output to 'wget-log'."

Submitter:  Peter Wu <lekensteyn>
Submitted:  Sun 04 Jun 2017 06:45:12 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.19.5 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  1.19.6 Regression:  Yes
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 17 Jun 2018 08:58:48 PM UTC, comment #15: 

Hi Tim
I could not get your code to compile.


My simpler C version of my test case below

What I woudl suggest is, could this be added to wget testsuite? some additional tests

gcc -O2 -Wall -Wextra -Wpedantic -o main wget_main.c
#include <stdlib.h>
int main (void)
{
    const char * str = "timeout -k 26s 25s wget http://bbc.com/";
    int result = system(str);
 
    return result;
}



J <now3d>
Sun 17 Jun 2018 04:34:49 PM UTC, comment #14: 

Sorry, I forgot to mention that I used J's system() call.

> Given the special behavior of "timeout", isn't this "working as intended"?


I see wget-log being created with J's C code though --output-file was given (-o dump.html):

#include <stdlib.h>
int main (void) {
    int result = system("timeout 25 src/wget -o dump.html http://google.com/");
    return result;
}

$ gcc x.c -o x
$ ./x

Redirecting output to ‘wget-log.1’.


Tim Ruehsen <rockdaboot>
Group administrator
Sun 17 Jun 2018 11:22:34 AM UTC, comment #13: 

I can only reproduce it with fork before execve and when using the "timeout" command:

#include <unistd.h>
#include <stdio.h>
#include <sys/wait.h>

int main() {
    char *args[] = {
        "timeout", "-k", "26s", "25s",
        "wget", "-O", "test.html", "http://example.com", NULL
    };
    int pid = fork();
    if (pid < 0) {
        perror("fork");
        return 1;
    } else if (pid == 0) {
        execvp(args[0], args);
        perror("execvp");
        return 1;
    } else {
        wait(NULL);
        perror("wait");
    }
    return 0;
}

It appears that "timeout" is creating a new process group:

977           if (foreground_pgrp != -1 && foreground_pgrp != getpgrp ())
(gdb) p foreground_pgrp
$4 = 12905   # pidof wrapper
(gdb) p (int)getpgrp()
$5 = 12906   # pidof timeout

Suggested workarounds:
- Use "timeout --foreground" or,
- Add the "-o-" option to "wget" to force logging to stdout.

Given the special behavior of "timeout", isn't this "working as intended"?

Peter Wu <lekensteyn>
Sat 16 Jun 2018 09:55:51 PM UTC, comment #12: 

Actually, I am unable to reproduce the problem.

`$ timeout -k 26s 25s wget example.com`

does not put Wget in the background. The entire task runs in the foreground.

And even when wget does run in the background, I don't see how the manual is incorrect. It says, wget will download to `wget-log`, but if the local file already exists, due to no-clobbering, Wget will create a unique filename by appending a counter.

I just don't see what is wrong here

Darshit Shah <darnir>
Group administrator
Sat 16 Jun 2018 06:50:51 PM UTC, comment #11: 

The 'timeout' command puts wget into background.

If you either leave it away or use --foreground with 'timeout', you won't see wget-log. You can use --timeout=25 for wget if you need a timeout.

But that's a work-around. The issue is that wget behaves contrary to the manual which says "--background: If no output file is specified via the -o, output is redirected to wget-log".

I re-open the issue.

Tim Ruehsen <rockdaboot>
Group administrator
Sat 16 Jun 2018 06:05:59 PM UTC, comment #10: 

Hi Peter

Ok, sorry my example was bad. This is my actual example below.

This program reproduces the issue

jonny@asus:~/code$ g++ -O2 -Wall -Wextra -Wpedantic -o main main3.cpp
jonny@asus:~/code$ ./main

Redirecting output to ‘wget-log.2’.
jonny@asus:~/code$





//g++ -O2 -Wall -Wextra -Wpedantic -o main main.c

#include <string>
#include <stdio.h>
#include <stdlib.h>

int main (void)
{
    std::string str = "timeout -k 26s 25s wget --output-document dump.html http://TajInternational.com/";

    int result = system(str.c_str());

    return result;
}

J <now3d>
Sat 16 Jun 2018 05:34:29 PM UTC, comment #9: 

"&" is a special shell character which causes a program to go to the background. When you execute

    wget https://example.com/dpp&key

it will actually be interpreted as:

    wget https://example.com/dpp &
    key

which will execute "wget https://example.com/foo" to download that URL in the background (due to "&"). After that it will execute the command "key" (which it cannot find in your case).

To have the intended effect of downloading that particular URL, quote your URL instead such that the special shell characters are not intepreted:

    wget "https://example.com/dpp&key"

See also https://www.tldp.org/LDP/abs/html/quoting.html

Peter Wu <lekensteyn>
Sat 16 Jun 2018 04:39:10 PM UTC, comment #8: 

Hi Peter Wu

I'm using Latest ubuntu LTS, which shows version 1.19.4. Logs below. Is it something to do with '&' ampersand in a URL?



$ dpkg -l |grep wget
ii  wget                          1.19.4-1ubuntu2.1

$ wget --version
GNU Wget 1.19.4 built on linux-gnu.


$ wget https://uk.godaddy.com/dpp&key
[1] 5080

Redirecting output to ‘wget-log.3’.

Command 'key' not found, but can be installed with:

sudo apt install donkey

jonny@asus:~/test$



J <now3d>
Fri 15 Jun 2018 07:33:39 PM UTC, comment #7: 

This was already fixed in wget 1.19.3, are you sure that you are using this (or a newer) version?

You really have to provide more details on your environment and the steps to reproduce. See for example the original description.

Peter Wu <lekensteyn>
Fri 15 Jun 2018 05:32:15 PM UTC, comment #6: 

Issue reproduced 1.19.4 on Ubuntu

Any news when fix will be released please?

J <now3d>
Sun 31 Dec 2017 12:04:53 PM UTC, comment #5: 

Thanks, applied.

Tim Ruehsen <rockdaboot>
Group administrator
Wed 27 Dec 2017 01:08:00 PM UTC, comment #4: 

I have investigated the issue, found the issue and attached a patch for it.

In the provided test case, file descriptors std{in,out,err} (/proc/self/fd/{0,1,2}) seem to refer to /dev/console. This happens even without chroot.

Peter Wu <lekensteyn>
Fri 27 Oct 2017 08:29:34 PM UTC, comment #3: 

From the mailing list (21./22.5.2017):

If you use an explicit logfile (-o / --output-file), that code for
creating 'wget-log' isn't triggered.

It is a documented behavior, see 'man wget':
"       -b
       --background
           Go to background immediately after startup.  If no output
file is specified via the -o, output is redirected to wget-log.
"

Maybe the implementation was buggy and didn't match the docs - and since 1.19 it has been fixed ?

Anyways, we want to keep backward compatibility and if that change breaks existing scripts, we should revert the change and change the docs (I am sure people dislike that as well ;-)).

Tim Ruehsen <rockdaboot>
Group administrator
Fri 27 Oct 2017 05:35:16 PM UTC, comment #2: 

This affects me, too.  Please fix.

Anonymous
Sun 17 Sep 2017 07:39:45 PM UTC, comment #1: 

an additional comment from a Debian user:

--8<--
This has hit many and is already reported in several distributions, but not in Debian yet.

Since 1.19, when run in the background, even with --quiet, wget creates a log file wget-log in the current directory, which is normally empty.
If wget-log exists, it creates wget-log.1, and so on.

The workaround is to use -o /dev/null, but this changed behaviour breaks existing scripts and is undocumented.
--8<--
https://bugs.debian.org/874590

Noël Köthe <nok>
Sun 04 Jun 2017 06:45:12 PM UTC, original submission:  

Since upgrading to wget 1.19.1 from wget 1.18 on Arch Linux, the normal progress output is omitted and instead a "wget-log" file is created in some circumstances.

Not sure how exactly this is triggered, I can only reproduce this when starting wget in an initrd.

Steps to reproduce:
1. Create an initrd with "/init" consisting of the script below.
2. Boot the initrd, for example with:
qemu-system-x86_64 -m 1G -M pc,accel=kvm -kernel /boot/vmlinuz-linux -initrd initrd.gz -nographic -serial stdio -monitor none -append console=ttyS0

Expected output:
A status message such as:
--2017-06-04 20:38:57--  http://127.0.0.1/
Connecting to 127.0.0.1:80... failed: Connection refused.

Actual output:
Redirecting output to 'wget-log'.

Other information:
I triggered this problem in my bootstrap script https://github.com/Lekensteyn/archdir. Another affected user report: https://unix.stackexchange.com/q/363765/8250
It could be a regression from commit v1.18-84-gdd5c549f


#!/bin/sh
export PATH=/usr/bin:/usr/sbin
/bin/busybox --install -s
ip link set lo up
mkdir /new
mount -t tmpfs none /new
cp -a /lib* /*bin /usr /etc /new/
chroot /new /usr/bin/wget -O - 127.0.0.1 || :

# Power down
mkdir /proc
mount -t proc none /proc
echo > /proc/sysrq-trigger o
echo should not be reached

Peter Wu <lekensteyn>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #42734:  0001-Avoid-redirecting-output-to-file-when-tcgetpgrp-fail.patch added by lekensteyn (1KiB - text/x-patch - [PATCH] Avoid redirecting output to file when tcgetpgrp fails)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by darnir (Posted a comment)
  • -email is unavailable- added by now3d (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by nok (Posted a comment)
  • -email is unavailable- added by nok
  • -email is unavailable- added by lekensteyn (Submitted the item)
  • -email is unavailable- added by lekensteyn
  •  

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-06-16 rockdaboot StatusFixed Confirmed
        Open/ClosedClosed Open
        Release1.19.1 1.19.5
        Planned Release1.19.3 1.19.6
    2017-12-31 rockdaboot StatusConfirmed Fixed
        Open/ClosedOpen Closed
        Planned ReleaseNone 1.19.3
    2017-12-27 lekensteyn Attached File- Added 0001-Avoid-redirecting-output-to-file-when-tcgetpgrp-fail.patch, #42734
    2017-10-27 rockdaboot StatusNone Confirmed
    2017-09-17 nok Carbon-Copy- Added -email is unavailable-
    2017-06-04 lekensteyn Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code