Gneural Network - Tasks: task #14201, Implement L0, L1, L2, L3 training...
You are not allowed to post comments on this tracker with your current authentication level.
task #14201: Implement L0, L1, L2, L3 training regularization strategies for nnet.
Submitter: | Ray Dillinger <rayd> | ||
Submitted: | Sun 30 Oct 2016 05:00:47 PM UTC | ||
Should Start On: | Sun 30 Oct 2016 07:00:00 AM UTC | Should be Finished on: | Fri 30 Dec 2016 08:00:00 AM UTC |
Category: | None | Priority: | 6 |
Status: | None | Privacy: | Public |
Assigned to: | None | Percent Complete: | 0% |
Open/Closed: | Open | Effort: | 0.00 |
No files currently attached
Depends on the following items: None found
Items that depend on this one: None found
Carbon-Copy List
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
No changes have been made to this item
Powered by Savane 3.13-4448.
Corresponding source code
L0 - subtract a teensy constant from all weights after each round of training. Has the virtue of driving weights in "unnecessary" connections to zero, which facilitates network pruning.
L1 - subtract a teensy percentage of the weights' current value after each round of training. Has the virtue of being least intrusive/disruptive to training.
L2 - subtract a teensy percentage of the square of the weights' current values after each round of training. Has the virtue of being more resistant to overfitting than L1.
L3 - subtract a teensy percentage of the cube of the weights' current values after each round of training. Has the virtue of optimally distributing/minimizing/equalizing weights, but if the topology is suboptimal (too many or not enough weights in some layer relative to others) it may unduly limit search space or training accuracy.
In all cases "teensy" needs to be an adaptive coefficient.
It's probably worthwhile to treat all these as special cases of L(real number) regularization, both to minimize the code needed and in consideration that the optimal for a given problem may lie between these integers rather than among them. Subject to the caveat that exponentiation with real numbers may be much more compute expensive, or may not parallelize on some hardware.