taskGneural Network - Tasks: task #14201, Implement L0, L1, L2, L3 training...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #14201: Implement L0, L1, L2, L3 training regularization strategies for nnet.

Submitter:  Ray Dillinger <rayd>
Submitted:  Sun 30 Oct 2016 05:00:47 PM UTC
   
 
Should Start On:  Sun 30 Oct 2016 07:00:00 AM UTC Should be Finished on:  Fri 30 Dec 2016 08:00:00 AM UTC
Category:  None Priority:  6
Status:  None Privacy:  Public
Assigned to:  None Percent Complete:  0%
Open/Closed:  Open Effort:  0.00

Sun 30 Oct 2016 05:00:47 PM UTC, original submission:  


L0 - subtract a teensy constant from all weights after each round of training.  Has the virtue of driving weights in "unnecessary" connections to zero, which facilitates network pruning.

L1 - subtract a teensy percentage of the weights' current value after each round of training.  Has the virtue of being least intrusive/disruptive to training.

L2 - subtract a teensy percentage of the square of the weights' current values after each round of training.  Has the virtue of being more resistant to overfitting than L1.

L3 - subtract a teensy percentage of the cube of the weights' current values after each round of training.  Has the virtue of optimally distributing/minimizing/equalizing weights, but if the topology is suboptimal (too many or not enough weights in some layer relative to others) it may unduly limit search space or training accuracy.

In all cases "teensy" needs to be an adaptive coefficient.

It's probably worthwhile to treat all these as special cases of L(real number) regularization, both to minimize the code needed and in consideration that the optimal for a given problem may lie between these integers rather than among them.  Subject to the caveat that exponentiation with real numbers may be much more compute expensive, or may not parallelize on some hardware.

Ray Dillinger <rayd>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rayd (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code