Some useful links I find here and there

Machine_Learning.md

About Gist https://www.labnol.org/internet/github-gist-tutorial/28499/

(blog)SVMs
http://blog.hackerearth.com/simple-tutorial-svm-parameter-tuning-python-r

Background subtraction
https://github.com/stgstg27/Background-Subtraction

==Visualizaion==

Basic data visualization maps
https://mubaris.com/2017/09/26/introduction-to-data-visualizations-using-python/

More visualisation
https://github.com/ContextLab/storytelling-with-data/blob/master/data-stories/education/tutorial.ipynb

Visualisation with pandas
https://www.kaggle.com/residentmario/univariate-plotting-with-pandas/notebook

Average Precision as AU-PR curve
https://sanchom.wordpress.com/tag/average-precision/

https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173

==TSNE==

Misreading TSNE plots
https://distill.pub/2016/misread-tsne/

==CNN/DL==

SGD >> Adam for Generalisation:
https://arxiv.org/abs/1705.08292

https://shaoanlu.wordpress.com/2017/05/29/sgd-all-which-one-is-the-best-optimizer-dogs-vs-cats-toy-experiment/

CS231n Gradient check http://cs231n.github.io/neural-networks-3/#gradcheck

Open Images dataset maker https://github.com/aferriss/openImageDownloader

DL Mistakes http://ppwwyyxx.com/2017/Unawareness-Of-Deep-Learning-Mistakes/#more

Training classification network- kaggle10th https://towardsdatascience.com/image-classification-challenge-using-transfer-learning-and-deep-learning-studio-2e89c3189fcf

Kaggle4th classification tips https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/45733

Kaggle #1 classification tips https://medium.com/neuralspace/kaggle-1-winning-approach-for-image-classification-challenge-9c1188157a86

Warm restarts paper https://arxiv.org/pdf/1608.03983.pdf

Optimal Learning rate https://towardsdatascience.com/estimating-optimal-learning-rate-for-a-deep-neural-network-ce32f2556ce0

Hyperparams https://towardsdatascience.com/artificial-intelligence-hyperparameters-48fa29daa516

On Convolutional NN

(blog)About CNN developments through the years- https://adeshpande3.github.io/adeshpande3.github.io/The-9-Deep-Learning-Papers-You-Need-To-Know-About.html

Understanding Convolutional operation in CNN- https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

VIsualising MNIST & understanding Dimensionality Reduction http://colah.github.io/posts/2014-10-Visualizing-MNIST/

Lao ML notes http://claoudml.strikingly.com

Tfrecords https://planspace.org/20170323-tfrecords_for_humans/

https://planspace.org/20170403-images_and_tfrecords/

Google ML crash course https://developers.google.com/machine-learning/crash-course/ml-intro

(blog)R-CNN to Mask R-CNN https://blog.athelas.com/a-brief-history-of-cnns-in-image-segmentation-from-r-cnn-to-mask-r-cnn-34ea83205de4

(blog)Fast, Faster R-CNN https://tryolabs.com/blog/2018/01/18/faster-r-cnn-down-the-rabbit-hole-of-modern-object-detection/

https://jhui.github.io/2017/03/15/Fast-R-CNN-and-Faster-R-CNN/

(blog)RoI pooling https://blog.deepsense.ai/region-of-interest-pooling-explained/

(blog)Receptive field arithmetic https://medium.com/mlreview/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807

(SO answer) Anchors and faster-RCNN https://stats.stackexchange.com/questions/265875/anchoring-faster-rcnn

(SO Answer) Cnn filter weights initialization https://stats.stackexchange.com/questions/200513/how-to-initialize-the-elements-of-the-filter-matrix

(blog)cross entropy http://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/

(article)Cross Entropy losses - categorical, focal https://gombru.github.io/2018/05/23/cross_entropy_loss/

(article)Transfer Learning/Fine tune CNN http://cs231n.github.io/transfer-learning/

(coursera)CNN/NMS/Object Detection https://www.coursera.org/learn/convolutional-neural-networks/lecture/dvrjH/non-max-suppression

(medium article)Image augmentation with tf https://medium.com/ymedialabs-innovation/data-augmentation-techniques-in-cnn-using-tensorflow-371ae43d5be9

https://towardsdatascience.com/image-augmentation-for-deep-learning-using-keras-and-histogram-equalization-9329f6ae5085

Also an augmentor library https://github.com/mdbloice/Augmentor

On Bounding Box Regression https://pdxscholar.library.pdx.edu/cgi/viewcontent.cgi?article=4949&context=open_access_etds

Siamese Network Image similarity

Tensorflow series https://blog.metaflow.fr/tensorflow-a-primer-4b3fa0978be3

==PYTHON==

Python Tutorial https://www.python-course.eu/

Guide to import https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html

Amazing things about python https://nedbatchelder.com/text/names.html https://stackoverflow.com/questions/5131538/slicing-a-list-in-python-without-generating-a-copy

Why self is here to stay http://neopythonic.blogspot.in/2008/10/why-explicit-self-has-to-stay.html

The init self confusion https://stackoverflow.com/questions/625083/python-init-and-self-what-do-they-do

(website)Learn Python http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html

==KAGGLE==

kaggle ensemble guide https://mlwave.com/kaggle-ensembling-guide/

(blog)Kaggle Zoo Solution http://benanne.github.io/2014/04/05/galaxy-zoo.html

Setting up the computer https://www.kaggle.com/c/allstate-claims-severity/discussion/26423#150025

https://www.columnfivemedia.com/25-tips-to-upgrade-your-data-visualization-design

https://tensorspace.org/index.html

karpathy.github.io

pca and eigenvalues
https://www.math.hmc.edu/calculus/tutorials/eigenstuff/

5 Clustering Algorithms
https://towardsdatascience.com/the-5-clustering-algorithms-data-scientists-need-to-know-a36d136ef68

image hashing algos
https://content-blockchain.org/research/testing-different-image-hash-functions/

https://shreevatsa.wordpress.com/2008/05/16/premature-optimization-is-the-root-of-all-evil/

http://www.paulgraham.com/knuth.html

A Short Introduction to Entropy, Cross-Entropy and KL-Divergence
https://www.youtube.com/watch?v=ErfnhcEV1O8

https://towardsdatascience.com/a-big-of-tricks-for-image-classification-fec41eb28e01

python oop resources
https://www.reddit.com/r/Python/comments/226ahl/some_links_about_python_oop/

dropout explanation
https://pgaleone.eu/deep-learning/regularization/2017/01/10/anaysis-of-dropout/

a comment in defense of Keras
https://www.reddit.com/r/KerasML/comments/7mlxzj/what_can_straight_tensorflow_do_that_keras_cant_do/

https://jovianlin.io/keras-models-sequential-vs-functional/

probability book
http://www.med.mcgill.ca/epidemiology/hanley/bios601/GaussianModel/JaynesProbabilityTheory.pdf

https://blog.juliobiason.net/thoughts/things-i-learnt-the-hard-way/?utm_source=hackernewsletter&utm_medium=email&utm_term=fav

BN discussions

1. FYI apparently batch normalization works better in practice after the activation function
2. Interesting. Just to follow up, if you continue to read on in that summary, it says that their best model [GoogLeNet128_BN_lim0606] actually has the BN layer BEFORE the ReLU. So while BN after Activation might improve accuracy in an isolated case, when the whole model is constructed, before performed best. Likely it's possible that placing BN after Activation could improve accuracy, but is likely problem dependent.
3. Batch normalization works best after the activation function, and here or here is why: it was developed to prevent internal covariate shift. Internal covariate shift occurs when the distribution of the activations of a layer shifts significantly throughout training. Batch normalization is used so that the distribution of the inputs (and these inputs are literally the result of an activation function) to a specific layer doesn't change over time due to parameter updates from each batch (or at least, allows it to change in an advantageous way). It uses batch statistics to do the normalizing, and then uses the batch normalization parameters (gamma and beta in the original paper) "to make sure that the transformation inserted in the network can represent the identity transform" (quote from original paper). But the point is that we're trying to normalize the inputs to a layer, so it should always go immediately before the next layer in the network. Whether or not that's after an activation function is dependent on the architecture in question.
4. I just saw in the deeplearning.ai class that Andrew Ng says that there is a debate on this in the Deep Learning community. He prefers applying batch normalization before the non-linearity
5. Original paper: ""We add the BN transform immediately before the nonlinearity, by normalizing x = Wu+b. We could have also normalized the layer inputs u, but since u is likely the output of another nonlinearity, the shape of its distribution is likely to change during training, and constraining its first and second moments would not eliminate the covariate shift. In contrast, Wu + b is more likely to have a symmetric, non-sparse distribution, that is “more Gaussian” (Hyv¨arinen & Oja, 2000); normalizing it is likely to produce activations with a stable distribution.""

Visual information theory
https://colah.github.io/posts/2015-09-Visual-Information/

Putting stuff on pypi
https://hynek.me/articles/sharing-your-labor-of-love-pypi-quick-and-dirty/

Tsne vs PCA
https://stats.stackexchange.com/questions/238538/are-there-cases-where-pca-is-more-suitable-than-t-sne

tsne clustering doesnt work
https://stats.stackexchange.com/questions/263539/clustering-on-the-output-of-t-sne

Data viz - avoid overplotting
https://python-graph-gallery.com/134-how-to-avoid-overplotting-with-python/

Quantization - tensorflow - blog by Pete Warden
https://petewarden.com/2016/05/03/how-to-quantize-neural-networks-with-tensorflow/

https://github.com/Kayzaks/HackingNeuralNetworks
https://mlwhiz.com/blog/2019/01/06/pytorch_keras_conversion/

https://medium.com/modern-nlp/productionizing-nlp-models-9a2b8a0c7d14

Pytorch:
https://towardsdatascience.com/getting-started-with-pytorch-part-1-understanding-how-automatic-differentiation-works-5008282073ec
https://mc.ai/how-pytorch-backward-function-works/

PCA whitening, covariance, mahalanobis distance
https://mccormickml.com/2014/07/22/mahalanobis-distance/

https://www.youtube.com/watch?v=0frKXR-2PBY

Depthwise separable conv
https://towardsdatascience.com/a-basic-introduction-to-separable-convolutions-b99ec3102728

We can't always convert a matrix to spatially separable form. eg. prime numbers like [[1 2] [3 5]]

Interesting thing about fully conv nets like Retinanet - resizing image during preprocessing (default 800x1333) -
check min side, if min size < 800 then upsample to 800 and note the scale, then scale max size with same scale, if this exceeds 1333 then scale to 1333 and note this new scale as final scale- finally resize image both sides with this final scale.
if min size > 800 then downsample to 800 and rest same procedure.

smallest_side = min(rows, cols)

# rescale the image so the smallest side is min_side
scale = min_side / smallest_side

# check if the largest side is now greater than max_side, which can happen
# when images have a large aspect ratio
largest_side = max(rows, cols)
if largest_side * scale > max_side:
    scale = max_side / largest_side

Basically, all images are squashed to this size

On AI companies
https://a16z.com/2020/02/16/the-new-business-of-ai-and-how-its-different-from-traditional-software/