bugGNU Wget - Bugs: bug #20484, Should wget attempt to do...

 
 

bug #20484: Should wget attempt to do conversions when interrupted (Control-C)?

Submitter:  Micah Cowan <micahcowan>
Submitted:  Sat 14 Jul 2007 09:02:11 PM UTC
   
 
Category:  Feature Request Severity:  1 - Wish
Priority:  7 - High Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  1 - Days Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 20 Jul 2007 07:27:06 AM UTC, comment #9: 

After discussion with Tony, one idea was to do a greatly simplified version of what he suggested; response to an interrupt would result in a prompt such as:

(skip, exit)?

A second interrupt at the prompt, or after exit has been chosen, will cause an immediate exit; otherwise, "exit" will cause conversions to be done, and then exit. Skip will resume downloading with the next file (possibly deleting the one that had been in progress?).

The prompt would be preceded with a status message, describing some or all of the following information:

How many files have been fetched
How many files are still queued to fetch
Mean file size
How long wget has been running (interrupts such as this one should not count towards this total).

Micah Cowan <micahcowan>
Tue 17 Jul 2007 10:10:54 PM UTC, comment #8: 


> > Because of their unreliability, I think using signals to
> > modify wget's behaviour and configuration on the fly is a
> > very bad idea.
>
> In what way are signals unreliable? Using interrupt signals to > halt whatever a program is doing, and bring up a command prompt > is not exactly unusual (though typically, whatever the program > was doing previously is abandoned).


ah ok, i see. you're talking about switching wget's operating mode upon receival of user-generated SIGINT. that could work, in my opinion.

i thought you guys were planning to add complex forms of coordinations with other non-interactive external programs via unix signals, e.g., SIGUSR1 stops downloading process, SIGUSR2 resumes  downloading process, SIGINT skips the current download and starts the next download, etc. this sounds like a nightmare to me.

BTW, unix signals are unreliable in the sense that they don't get queued. as a result, SIGX can be generated N > 1 times and delivered just once.


> > I think we should consider a dynamic language for plugins.
> > Ruby would be my first choice, but Python is an excellent
> > alternative.
>
> I see no reason why it couldn't be both, plus Perl and whatever
> else. We just provide the API, and the rest is just glue.


right.


> I'm not sure how plugins would help with dynamic, interactive
> reconfiguration, though.


of course, wget should be changed to support dynamic reconfiguration. but on top of that, we could add plugin support to make dynamic reconfiguration easily exploitable from external, higher level code.


> > As Micah says, having interactive console would also be
> > great. Maybe that could be implemented with a plugin as well.
>
> Actually, that was Tony's idea :)


sorry, guys. i skimmed through your emails in a hurry, and i totally misunderstood what tony was suggesting.

my proposal to implement interactive console via hooks and plugins is still valid, though.

Mauro Tortonesi <mtortonesi>
Group Member
Tue 17 Jul 2007 06:42:49 PM UTC, comment #7: 


> Because of their unreliability, I think using signals to modify
> wget's behaviour and configuration on the fly is a very bad idea.


In what way are signals unreliable? Using interrupt signals to halt whatever a program is doing, and bring up a command prompt is not exactly unusual (though typically, whatever the program was doing previously is abandoned).

> I think we should consider a dynamic language for plugins. Ruby
> would be my first choice, but Python is an excellent alternative.


I see no reason why it couldn't be both, plus Perl and whatever else. We just provide the API, and the rest is just glue.

I'm not sure how plugins would help with dynamic, interactive reconfiguration, though.

> As Micah says, having interactive console would also be great.
> Maybe that could be implemented with a plugin as well.


Actually, that was Tony's idea :)

Micah Cowan <micahcowan>
Tue 17 Jul 2007 12:35:22 PM UTC, comment #6: 

Because of their unreliability, I think using signals to modify wget's behaviour and configuration on the fly is a very bad idea.

Instead, I believe we can obtain the same result by introducing support for plugins and have wget call external code from well-defined places. For instance, we could define pre-download and post-download hooks, and let plugins interact w/ wget by appending new urls to the download queue or cancelling the download of urls.

I think we should consider a dynamic language for plugins. Ruby would be my first choice, but Python is an excellent alternative.

Notice that this would allow us to easily write new parsers to extract urls from currently unsupported mime types, e.g., css or javascript.

As Micah says, having interactive console would also be great. Maybe that could be implemented with a plugin as well.

Mauro Tortonesi <mtortonesi>
Group Member
Sun 15 Jul 2007 10:23:58 PM UTC, comment #5: 

So you'd like to restrict the implemention of this feature to (abort,exit) portion of my suggestion and have the default be that a single interrupt causes the exit behavior and a second one causes the abort behavior. OK.

As for whether this makes wget an interactive program, I don't think it does. It allows the user to temporarily interact with wget in an interactive mode, but as soon as the user chooses to continue, it is back in its normal batch mode.

If you really think we need a --give-me-control-on-interrupt switch, I guess that's fine. If we did that, I'd just add the switch to my wget alias in .bashrc.

Tony Lewis <tdlewis77>
Sun 15 Jul 2007 09:34:43 PM UTC, comment #4: 

If you mean that an interrupt signal should send it to an interactive mode, that could work, but I think "batch mode" should be the default; that is, you should specifically give an option to enable interactive mode on interrupt (as, according to descriptions that exist in various places, wget is a "non-interactive" utility, and I'm not convinced it should become an interactive one, at least in default invocations).

Another idea would be to have a separate program that communicates with the first, by means of a pipe, to provide equivalent functionality; but perhaps this would complicate these things overly much.

Still, it'd be a heluva lot of work, and I'm not sure I'd be interested in doing it, myself. If it's something you'd like to work on, Tony... :)

However, I'd prefer to reserve this bug for the original proposition; let's submit a new bug for the interactive features.

Micah Cowan <micahcowan>
Sun 15 Jul 2007 06:14:18 PM UTC, comment #3: 

Changing the options with the interrupt signal would be a great feature, but this sounds like a huge possibility of bugs all throughout the program IMHO.. we'd have to be extremely cautious about it.

You've raised a lot of interesting ideas. I think we should start working on this. It'd be a great feature for v2.

John Doe <yurimxpxman>
Sun 15 Jul 2007 05:12:44 PM UTC, comment #2: 

I have long wished that wget had an intelligent interrupt signal handler (perhaps disabled by a command option for batch jobs and such). Some options it could offer:

  • help: explain all these options :-)
  • continue: resume the job
  • skip: stop downloading the current file and continue the job
  • next: finish downloading the current/next file and return to the interrupt prompt
  • abort: stop the job and leave downloaded files unchanged (current behavior)
  • exit: stop the job, but convert the files downloaded so far
  • suspend: save the job state in a file for later resumption
  • status: report on what's left to be done
  • options: allow the user to enter (some) command-line options to change the ongoing behavior of the job


For the "options" case, some command-line options clearly make no sense to change once the job is running.

Tony Lewis <tdlewis77>
Sun 15 Jul 2007 04:04:02 AM UTC, comment #1: 

That sounds like an excellent idea to me.

John Doe <yurimxpxman>
Sat 14 Jul 2007 09:02:11 PM UTC, original submission:  

I'm wondering if wget should attempt to convert the files it has downloaded, after encountering an interrupt signal. A second interrupt could cause it to exit immediately.

Micah Cowan <micahcowan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtortonesi (Posted a comment)
  • -email is unavailable- added by tdlewis77 (Posted a comment)
  • -email is unavailable- added by yurimxpxman (Posted a comment)
  • -email is unavailable- added by micahcowan (Submitted the item)
  •  

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-08-21 micahcowan Planned Release1.15 1.14
    2008-01-27 micahcowan Planned Release1.12 1.15
    2007-08-01 micahcowan Priority5 - Normal 7 - High
    2007-07-20 micahcowan StatusNone Confirmed
        Planned ReleaseNone 1.12

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code