bugGNU Wget - Bugs: bug #49458, Please search txt files named...

 
 

bug #49458: Please search txt files named ".ram" for links

Submitter:  None
Submitted:  Thu 27 Oct 2016 04:11:13 PM UTC
   
 
Category:  Feature Request Severity:  3 - Normal
Priority:  5 - Normal Status:  Wont Fix
Privacy:  Public Assigned to:  None
Originator Name:  powerKitten Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  1.18
Operating System:  None Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  No Work Required:  None
Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 28 Oct 2016 07:36:44 PM UTC, comment #4: 

Really, this opens up the can of worms of making which files are scanned for URLs configurable.

Ideally, the user could supply a list of extensions/MIME types and provide corresponding programs that would extract the URLs from the files.

But you would also want to implement --convert-links for these files, and that would be considerably more complicated.

Dale Worley <worley>
Fri 28 Oct 2016 07:53:53 AM UTC, comment #3: 

This URL(s) within the .ram/.ra file likely points to some audio stream - it might be a radio station with changing (and endless !) content.
'Downloading' the contents could easily lead to a 'hanging' wget. That is perhaps not what you want.
--spider could at least 'check' that URL.
And the ICY-Metadata HTTP header makes a difference since the remote side likely is a shoutcast server.

BTW, there are lot's of other file formats that contain URLs.
E.g. Wget2 reads URLs from sitemaps and sitemap catalogs (compressed or uncompressed), from Atom and RSS feeds.
There is also examples/getstream.c that 'downloads' URLs (streams) from .ram, .m3u, .wax, .asx, .pls, .xspf (audio/x-mpegurl, audio/x-pn-realaudio, application/pls+xml, audio/x-scpls, application/xspf+xml).

Tim Ruehsen <rockdaboot>
Group administrator
Thu 27 Oct 2016 05:03:10 PM UTC, comment #2: 

Sorry, I didn't really word it that well. The idea was for it to scan .ram text files for resources when mirroring a site using the -m flag, as right now I have to make two scans when mirroring a site that uses .ram assets to a WARC.

Patrick Alan Marshall <powerkitten>
Thu 27 Oct 2016 04:55:34 PM UTC, comment #1: 

I don't exactly know what you're asking, but I don't think it would be a good fit for wget given that you could easily automate it with a shell script.

The closest enhancement I can come across is to make -i accept multiple files, which, to the best of my knowledge, does not yet.

E.g. wget -i file1.ram,file2.ram,foo.txt,bar.html

What do you all think?

Ander Juaristi <ajuaristi>
Thu 27 Oct 2016 04:11:13 PM UTC, original submission:  

The .ram "format" is simply an ANSI encoded text file. It consists of links to RealAudio resources. I think that it would benefit from this feature, as currently I have to go into each .ram file, copy the links into a list of urls, and then fetch them using the -i flag. I've attached a sample .ram file which can also be found at the (archived) address: http://web.archive.org/web/20161027160920/http://www.theorderoftime.com/art/music/ra-popular/theimportance.ram
or the original address:
http://www.theorderoftime.com/art/music/ra-popular/theimportance.ram

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #38823:  theimportance.ram added by None (64B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by worley (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by powerkitten (Posted a comment)
  • -email is unavailable- added by ajuaristi (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-11-23 rockdaboot StatusNone Wont Fix
        Open/ClosedOpen Closed
    2016-10-28 rockdaboot Release1.17.1 1.18
        Operating SystemMicrosoft Windows None
    2016-10-27 None Attached File- Added theimportance.ram, #38823

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code