Classic Networks
LeNet-5: LeCun et al, "Gradient-Based Learning Applied to Document Recognition": http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf
AlexNet: Krizhevsky et al, "ImageNet Classification with Deep Convolutional Neural Networks": https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
VGG-16: Simonyan et al, "Very Deep Convolutional Networks for Large-Scale Image Recognition": https://arxiv.org/pdf/1409.1556.pdf
Resnets
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, "Deep Residual Learning for Image Recognition": https://arxiv.org/abs/1512.03385
Networks in Networks and 1x1 Convolutions
Min Lin, Qiang Chen, Shuicheng Yan, "Network In Network": https://arxiv.org/abs/1312.4400
Inception Networks
Christian Szegedy, and lots of others, "Going Deeper with Convolutions": https://arxiv.org/abs/1409.4842
Convolutional Implementation of Sliding Windows
Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, Yann LeCun, "OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks": https://arxiv.org/abs/1312.6229
Bounding Box Predictions
Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection": https://arxiv.org/abs/1506.02640
Region Proposals
Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation": https://arxiv.org/abs/1311.2524
Siamese Network
Taigman et al, "DeepFace: Closing the Gap to Human-Level Performance in Face Verification": https://www.cs.toronto.edu/~ranzato/publications/taigman_cvpr14.pdf
Triplet Loss
Florian Schroff, Dmitry Kalenichenko, James Philbin, "FaceNet: A Unified Embedding for Face Recognition and Clustering": https://arxiv.org/abs/1503.03832
What are deep ConvNets learning?
Matthew D Zeiler, Rob Fergus, "Visualizing and Understanding Convolutional Networks": https://arxiv.org/abs/1311.2901
Neural Style
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, "A Neural Algorithm of Artistic Style": https://arxiv.org/abs/1508.06576
=== Directly related to the course ===
Dropout
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15 , 1929–1958.
Initialization
- .He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. arXiv preprint arXiv:1502.01852 .
- .Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In AISTATS'2010 .
Optimization
-
.RMSprop: http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
-
.Kingma, D. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .
-
.Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y. (2014). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In NIPS'2014 .
Hyperparameter tuning
- .Bergstra, J. and Bengio, Y. (2012). Random search for hyper-parameter optimization. J. Machine Learning Res., 13 , 281–305.
- .Bergstra, J, et. al. Algorithms for Hyper-Parameter Optimization. Advances in Neural Information Processing Systems (pp. 2546-2554).
Batch norm
- .Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift.
=== Additional materials ===
- .Kaggle tutorials using examples for MNIST
- . This online book - which makes use of javascript to make things intuitive
- .Youtube channels - Siraj Raval , Two Minute Papers, etc
- .Applying dropout units for image recognition competition: Going Deeper with Convolutions
- . ai - Making neural nets uncool again
Ian Goodfellow's Generative Adversarial Models : https://arxiv.org/abs/1406.2661
Geoffrey Hinton's Capsule Networks (Article Explanation) : Part 1, Part 2, Part 3