Skip to content

Instantly share code, notes, and snippets.

@nofreewill42

nofreewill42/trainSmall_TransferBig.ipynb

Created March 13, 2020 10:52
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nofreewill42/d7f8f5301760ea98fd62a86604d7c544 to your computer and use it in GitHub Desktop.
Save nofreewill42/d7f8f5301760ea98fd62a86604d7c544 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
trainSmall_TransferBig.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@nofreewill42
Copy link
Author

I have found the source of the accuracy drop present in cells 30-34.
With the 5x5 kernel size, we are adding 5x5=25 numbers together.
With the 9x9 kernel size, we are adding 9x9=81 numbers together.
The numbers in both case have the same mean and std, so the outputs of the 9x9 kernel will be 81/25 times the outputs of the 5x5 kernel.
This is important because there are biases in the network and ReLUs can go from inactive to active and vice versa.

So, all we have to do is to divide the weights of the 9x9 kernel with (81/25).
image
Here is the corrected version of the 9x9 kernel without any "fine-tuning". (This was a fresh training, without this "scaling", I got the same "accuracy drop" as before.)
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment