bugGNU Wget - Bugs: bug #66248, Sending wget spider to the...

 
 

bug #66248: Sending wget spider to the background will avoid issue with Bad file descriptor

Submitter:  None
Submitted:  Tue 24 Sep 2024 02:40:29 PM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  freddieventura Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  trunk
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 24 Sep 2024 02:40:29 PM UTC, original submission:  

Hi,

I was just doing a simple webcrawl, trying to gather a list of all the urls in a .txt , doing a spider check first.

I am following the first response on this thread.
https://stackoverflow.com/questions/52610592/wget-spider-a-website-to-collect-all-links

But I am doing

```
wget --spider --force-html --span-hosts -np --limit-rate=20k -e robots=off --wait=3 --random-wait -r -l2 https://developers.google.com -o wget.log &
```

It works. But I just wanted to run it on the foreground (not send it to the bacground)
So I am just doing this

```
wget --spider --force-html --span-hosts -np --limit-rate=20k -e robots=off --wait=3 --random-wait -r -l2 https://developers.google.com -o wget.log
```

This last doesnt work , it takes 3 seconds to exit `wget` giving some lines on the log like thisone


```
developers.google.com: No such file or directory
developers.google.com/index.html.tmp.tmp: Bad file descriptor
Cannot write to ‘developers.google.com/index.html.tmp.tmp’ (Bad file descriptor).
Found no broken links.
```

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by None (Submitted the item)
  •  

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.14-6db8.
    Corresponding source code