taskGneural Network - Tasks: task #14204, Implement RProp training for nnet

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #14204: Implement RProp training for nnet

Submitter:  Ray Dillinger <rayd>
Submitted:  Sun 30 Oct 2016 05:30:23 PM UTC
   
 
Should Start On:  Sun 30 Oct 2016 07:00:00 AM UTC Should be Finished on:  Fri 30 Dec 2016 08:00:00 AM UTC
Category:  None Priority:  5 - Normal
Status:  None Privacy:  Public
Assigned to:  None Percent Complete:  0%
Open/Closed:  Open Effort:  0.00

Sun 30 Oct 2016 05:30:23 PM UTC, original submission:  


Rprop training is gradient descent, but isn't necessarily in the steepest direction.  Instead the speed in each dimension (weight change) increases subject to a maximum for as long as the error signal for that connection is decreasing, then the most recent change is subtracted instead of added and a smaller change tried.  This continues until a change leading to a smaller error but having an opposite gradient direction is discovered - and then the weight change starts in the opposite direction. 

The virtue of this method is that it more or less ignores 'vanishing and exploding gradients' problems, quickly traversing large distances in the weight space with a consistent error gradient, no matter how tiny that error gradient may be.  As such it is a semi-required technique when error feedback has to reach across more than two or three layers in deep networks.

An altered version of this may also work well for forward propagation of errors under DBN training; at least that's an interesting experiment to do.

Ray Dillinger <rayd>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rayd (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code