Training very deep neural networks

VeryDeepLearning.md

Very deep neural networks (May 2016)

• Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, Deep Residual Learning for Image Recognition (2015)
  • Uses identity shortcuts connections that skip one or more layers and merge back by adding to the output of the last layer that has been skipped. The point of such netoworks is to be able to train deeper networks without the known gradient vanishing problem. They show that residual networs are easier to optimize and can achieve better accuracy with the depth increase. Same architecture used for classification, feature extraction, object detection and segmentation tasks with success.
• Sergey Ioffe, Christian Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shif (2015)
  • An apporach to reduce the internal covariance shift by fixing the input layer distribution for each layer, thus allowing for a much faster learning without vanishing/exploding gradients problem.
• fast R-CNNc