Gneural Network - Tasks: task #14204, Implement RProp training for nnet
You are not allowed to post comments on this tracker with your current authentication level.
task #14204: Implement RProp training for nnet
Submitter: | Ray Dillinger <rayd> | ||
Submitted: | Sun 30 Oct 2016 05:30:23 PM UTC | ||
Should Start On: | Sun 30 Oct 2016 07:00:00 AM UTC | Should be Finished on: | Fri 30 Dec 2016 08:00:00 AM UTC |
Category: | None | Priority: | 5 - Normal |
Status: | None | Privacy: | Public |
Assigned to: | None | Percent Complete: | 0% |
Open/Closed: | Open | Effort: | 0.00 |
No files currently attached
Depends on the following items: None found
Items that depend on this one: None found
Carbon-Copy List
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
No changes have been made to this item
Powered by Savane 3.13-02a9.
Corresponding source code
Rprop training is gradient descent, but isn't necessarily in the steepest direction. Instead the speed in each dimension (weight change) increases subject to a maximum for as long as the error signal for that connection is decreasing, then the most recent change is subtracted instead of added and a smaller change tried. This continues until a change leading to a smaller error but having an opposite gradient direction is discovered - and then the weight change starts in the opposite direction.
The virtue of this method is that it more or less ignores 'vanishing and exploding gradients' problems, quickly traversing large distances in the weight space with a consistent error gradient, no matter how tiny that error gradient may be. As such it is a semi-required technique when error feedback has to reach across more than two or three layers in deep networks.
An altered version of this may also work well for forward propagation of errors under DBN training; at least that's an interesting experiment to do.