Weight matrix-based regularization techniques for deep learning

datasets.py

descriptors.py

ignite_callbacks.py

ignite_main.py

networks.py

timit_prep.py

utilities.py

visualize.py

WMM.md

Weight Matrix Modification - a regularization approach

General description:

Author: Patrik Reizinger

The project contains the source files (without the datasets) which implement WMM (Weight Matrix Modification,) a weight matrix-based regularization technique for Deep Neural Networks. In the following the proposed methods are shortly introduced, including the evalutaion framework.

Weight shuffling

Weight shuffling is based on the assumption that locally the coefficients of a weight matrix are correlated. Based on this, I hypothesize that shuffling the weight within a rectangular window - which is under the beforementioned assumption a way of adding correlated noise to the weights - may help reduce overfitting.

Weight reinitialization

Weight reinitialization aims to reduce overfitting while partially reinitializing the weight matrix, thus in the case of a non-representative training set it may reduce the over-/underestimation of the significance regarding specific input data.

License: GNU LESSER GENERAL PUBLIC LICENSE (Version 3, 29 June 2007)

wmm.py