subodh-malgonde/transfer_learning.MD

## transfer_learning.MD

      
    Raw
  

              transfer_learning.MD
            
          
    Assumptions


We have a pre-trained network and want to perform transfer learning using it.


Lets say we keep initial 5 layers from the pre-trained network and add our own layers on top of it.


Output of layer number 5 was given the name "layer5_out" when building the network


We can perform transfer learning on this in 2 ways:
1. Initialize the first 5 layers using the weights of the pre-trained network and freeze them during training.
2. Initialize the first 5 layers using the weights of the pre-trained network and train them i.e. update them during training.


1. Restore the network

Restore the network and get the tensor for the output of the 5th layer. We are assuming the model was saved using the tf.saved_model API: tf.saved_model.builder.SavedModelBuilder and tf.saved_model.loader.load.
tf.saved_model.loader.load(sess, tags, export_directory)
graph = tf.get_default_graph()
layer5_out = graph.get_tensor_by_name("layer5_out")

2. Prevent computation of gradients

When you are building the graph, call tf.stop_gradient on layer from which you want to prevent the backpropagation of gradients. It is layer number 5 in our network.
layer5_out = tf.stop_gradient(layer5_out)

Skip this if you want to train the layers of pre-trained network as well.
3. Add subsequent layers in a scope

with tf.variable_scope("trainable_section"):
    layer6_out = tf.layers.conv2d(layer5_out, ....)
    # add more layers as required

4. Get the list of trainable variables

# get the variables declared in the scope "trainable_section"
trainable_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, "trainable_section")

Skip this if you want to train the layers of pre-trained network as well.
5. Pass this list to optimizer.minimize

optimizer.minimize(loss, var_list=trainable_vars)

Skip this if you want to train the layers of pre-trained network as well.
6. Initialize only weights for decoder layers

trainable_variable_initializers = [var.initializer for var in trainable_vars]
sess.run(trainable_variable_initializers)

This step is required for both ways of performing transfer learning (see the assumptions above).