Create a gist now

Instantly share code, notes, and snippets.

Fine-tuning a Keras model. Updated to the Keras 2.0 API.
'''This script goes along the blog post
"Building powerful image classification models using very little data"
from blog.keras.io.
It uses data that can be downloaded at:
https://www.kaggle.com/c/dogs-vs-cats/data
In our setup, we:
- created a data/ folder
- created train/ and validation/ subfolders inside data/
- created cats/ and dogs/ subfolders inside train/ and validation/
- put the cat pictures index 0-999 in data/train/cats
- put the cat pictures index 1000-1400 in data/validation/cats
- put the dogs pictures index 12500-13499 in data/train/dogs
- put the dog pictures index 13500-13900 in data/validation/dogs
So that we have 1000 training examples for each class, and 400 validation examples for each class.
In summary, this is our directory structure:
```
data/
train/
dogs/
dog001.jpg
dog002.jpg
...
cats/
cat001.jpg
cat002.jpg
...
validation/
dogs/
dog001.jpg
dog002.jpg
...
cats/
cat001.jpg
cat002.jpg
...
```
'''
from keras import applications
from keras.preprocessing.image import ImageDataGenerator
from keras import optimizers
from keras.models import Sequential
from keras.layers import Dropout, Flatten, Dense
# path to the model weights files.
weights_path = '../keras/examples/vgg16_weights.h5'
top_model_weights_path = 'fc_model.h5'
# dimensions of our images.
img_width, img_height = 150, 150
train_data_dir = 'cats_and_dogs_small/train'
validation_data_dir = 'cats_and_dogs_small/validation'
nb_train_samples = 2000
nb_validation_samples = 800
epochs = 50
batch_size = 16
# build the VGG16 network
model = applications.VGG16(weights='imagenet', include_top=False)
print('Model loaded.')
# build a classifier model to put on top of the convolutional model
top_model = Sequential()
top_model.add(Flatten(input_shape=model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))
# note that it is necessary to start with a fully-trained
# classifier, including the top classifier,
# in order to successfully do fine-tuning
top_model.load_weights(top_model_weights_path)
# add the model on top of the convolutional base
model.add(top_model)
# set the first 25 layers (up to the last conv block)
# to non-trainable (weights will not be updated)
for layer in model.layers[:25]:
layer.trainable = False
# compile the model with a SGD/momentum optimizer
# and a very slow learning rate.
model.compile(loss='binary_crossentropy',
optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
metrics=['accuracy'])
# prepare data augmentation configuration
train_datagen = ImageDataGenerator(
rescale=1. / 255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1. / 255)
train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='binary')
validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='binary')
# fine-tune the model
model.fit_generator(
train_generator,
samples_per_epoch=nb_train_samples,
epochs=epochs,
validation_data=validation_generator,
nb_val_samples=nb_validation_samples)
@SchumErik

Great example, thanks for sharing Francois!! Aside from the comment from yurkor regarding the proper VGG image pre-processing (which I agree with), I have a few questions about the general procedure:

  1. Would it not be better to train the top model with different samples than you will later on use to fine tune your model? In the cat/dog example that would mean put aside 10-20% as validation set and then use 50% of the remainder for creating the top model and the other 50% for fine tuning. Any views?
  2. Do you have any insights regarding the best size of the top model when your final goal is a segmentation into 10 classes rather than 2? I tried to go back to the original VGG16 architecture, experimenting with two fully connected layers of 1024 and 512, but it appears there still is substantial overfitting. Any advice?
@junfenglx

@SchumErik

  1. I think this approach need more data
  2. again more data if you train large network. You can use regularization to defeat overfitting

By the way, if you have data size like ImageNet, you even can train VGG16 from scratch, not use fine tune at all.

@Arsey
Arsey commented Sep 11, 2016 edited

@SchumErik, about the second question, my experiments shows that the top model from the original VGG16 works better than model with decreased dimension. Here I classify 102 classes https://github.com/Arsey/keras-transfer-learning-for-oxford102

@alinagithub

Hi Francois, first of all, thanks for all this great material.
Quick question... is there a place from which we can download the fc_model.h5 ?
Many thanks,
AL.

@vishsangale

@alinagithub - You have to train this model before fine tuning. There is one more tutorial from Francois about how to generate that model.

@bayraktare

If we want to categorize 10 or 20 classes, is it necessary just to fine tune the same code over 10 or 20 classes by adding them as sub-folders into the train and validation data?

@alinagithub

@vishsangale - many thanks!

@AakashKumarNain

I implemented the above model and everything worked fine. Now I have a sample of 1000 images in my test_data named folder which I want to predict using the model.predict() method. I first generated the test features as we generated the train_data above. This is shown below:
generator = datagen.flow_from_directory(test_dir, batch_size = 32, target_size =(img_rows, img_columns), classes = None, shuffle =False)

test_data_features = model.predict_generator(generator, 1000)
np.save(open('test_data_features.npy','wb'), test_data_features)
test_data = np.load(open('test_data_features.npy', 'rb'))

Now when I make predictions like model.predict(test_data), I get a numpy array of predictions. How can I test what each prediction is represting? The predictions are like this:
array([[ 9.99770999e-01], [ 7.65574304e-03], [ 2.06944350e-07], [ 9.96615469e-01], [ 4.59789817e-07], [ 9.93980138e-05], [ 5.27667798e-05],..........)

How can I compare my predictions to my images?

@anujshah1003

you can try model.predict_classes(test_data)

@bayraktare

In addition to my yesterday's question, I would like to know that if we want to classify an object which is not belong to any of the categories in ImageNet what I have to do?

@Arsey
Arsey commented Oct 3, 2016 edited

I'm trying to adopt this approach for multiclass classification. And the first step when we're training the top model from bottlenecks works fine and I have 76% accuracy for 102 classes. But when I proceed to fine-tune step, the model with each next epoch decreases in accuracy. So if at the end of 1st epoch of fine-tune I have about 45% of accuracy, on the 2nd epoch I'm getting 40%, etc. Here's repo with the code https://github.com/Arsey/keras-oxford102. Can anyone help? I'm trying to find the issue for a few weeks but no results :(

@apapiu
apapiu commented Oct 5, 2016

Great Post, I am learning a lot. I do have a question about the top model. If I understand correctly I need to have the weights properly tuned already before sticking it to the convolutional layers. But how do I compute the weights to begin with?

To be more specific: In this line: top_model_weights_path = 'fc_model.h5' how did you compute the weights for the top model?

@CharlesNord

@alinagithub Did you find the tutorial about how to train fc_model.h5 ? I can't find it.

@Golly
Golly commented Oct 8, 2016

@Arsey Same problem with fine tuning. Im trying now and model decreased in accuracy from 87% to 50%.

@Euphemiasama
Euphemiasama commented Oct 9, 2016 edited

@apapiu @CharlesNord You can compute the weights to get the 'fc_model.h5' file by running part 2 of this tutorial entitled 'Using the bottleneck features of a pre-trained network: 90% accuracy in a minute' it creates a file named 'bottleneck_fc_model.h5' you rename it to 'fc_model.h5' and run the code.

@jamescfli

@fchollet
In line 156 and 162, target_size=(img_height, img_width)
which should go first 'img_height' or 'img_width'?
In classifier_from_little_data_script_2.py, it is in reverse order. I am confused here.

@CharlesNord

@Euphemiasama Thank you very much. Another question, has any one reached 95% accuracy by using the fine tuning method? Before fine tuning, my accuracy is about 90%, and after fine tuning, it reaches 92% and I cannot make the result better anymore

@aquibjaved

How do I predict_classes on a new a Image(which is not in data base),
well I am trying the code below to load model and load a new Image pass through the predict_classes but I am getting an error:

import keras
from keras.models import load_model
from keras.models import Sequential
import cv2
import numpy as np 
from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img

model = Sequential()

model =load_model('firstmodel.h5')
model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

img = cv2.imread('cat.jpg',0).astype(np.float32) / 255
img = cv2.resize(img, (150, 150))
#img = prep_data(img)
#img = img.flatten()
img = np.expand_dims(img, axis=0)
# img1 = np.expand_dims(img, axis=0)
# img = cv2.imread('cat.jpg')
# img = cv2.resize(img,(150,150))
# x = img_to_array(img)  # this is a Numpy array with shape (3, 150, 150)
classes = model.predict_classes(img)
print classes

I am getting error:

using Theano backend.
Traceback (most recent call last):
  File "detect.py", line 28, in <module>
    classes = model.predict_classes(img)
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 780, in predict_classes
    proba = self.predict(x, batch_size=batch_size, verbose=verbose)
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 672, in predict
    return self.model.predict(x, batch_size=batch_size, verbose=verbose)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1174, in predict
    check_batch_dim=False)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 100, in standardize_input_data
    str(array.shape))
Exception: Error when checking : expected convolution2d_input_1 to have 4 dimensions, but got array with shape (1, 150, 150)

@nathmo
nathmo commented Dec 14, 2016 edited

hi, i made a derrived script from your code and get the following error :
i'm running tensor flow 0.12.0-rc1 With python 2.7
i have installed the latest version of keras from git.
What i have done wrong ?

Source code : http://pastebin.com/JEqcZe5x
Full Error : http://pastebin.com/yEFVHWEK

"ValueError: Error when checking model input: expected convolution2d_input_1 to have shape (None, 3, 32, 32) but got array with shape (32, 32, 32, 3)"

Thank in advance

Nathann

@Abusnina
Abusnina commented Jan 5, 2017 edited

How can we use the modified pre-trained model to predict test data? I compiled the model successfully, but when I try to predict my test data i get the following error:
Error when checking : expected flatten_input_1 to have shape (None, 4, 4, 512) but got array with shape (1, 3, 150, 150)
Any advice on this?

@austinchencym

@Golly
@Arsey

Same problem, before fine-tuning my model for 5 classes reached 98% accuracy but the first epoch of fine-tuning dropped to 20%. Did you or does anyone work it out for multi-class problem? I guess we need more train data to feed our model

@varunagrawal

Is the slice of :25 correct for setting the non-trainable parameter? If you do model.layers[:26] you see that the last layer is also a Conv layer.

@KamalOthman

Hi everyone,
I am following this example on GRAYSCALE images, which I converted them from color scale using PIL convert method. I expected the input_sahpe in the model should be (1, w, h). but I get this error:
Exception: Error when checking model input: expected convolution2d_input_1 to have shape (None, 1, 150, 150) but got array with shape (32, 3, 150, 150)
When I checked the image shape, I got only (w,h)
However, It still works with input_shape=(3, w, h). Isn't it strange?

My goal to compare the learning on gray images to color images. But I do not know how to expand the dimension of gray images within flow_from_directory.

Anyone can help please?

Thanks
Kamal

@cobir
cobir commented Jan 30, 2017

Hi,

The input shape of the model is (3, img_width, img_height). Does this mean that we can only work with theano as backend when working with this pre-trained VGG16 model?

Thanks,
OC

@austinchencym

@cobir

I used Tensorflow backend it works fine.

@biswatherockstar

I get the following error when fine tuning a 8 class multi class classification. Any idea anyone please help

nb_val_samples=nb_validation_samples)

File "C:\Users\bgsingh\Anaconda2\lib\site-packages\keras\models.py", line 935, in fit_generator
initial_epoch=initial_epoch)
File "C:\Users\bgsingh\Anaconda2\lib\site-packages\keras\engine\training.py", line 1553, in fit_generator
class_weight=class_weight)
File "C:\Users\bgsingh\Anaconda2\lib\site-packages\keras\engine\training.py", line 1310, in train_on_batch
check_batch_axis=True)
File "C:\Users\bgsingh\Anaconda2\lib\site-packages\keras\engine\training.py", line 1034, in _standardize_user_data
exception_prefix='model target')
File "C:\Users\bgsingh\Anaconda2\lib\site-packages\keras\engine\training.py", line 124, in standardize_input_data
str(array.shape))
ValueError: Error when checking model target: expected sequential_2 to have shape (None, 1) but got array with shape (32L, 8L)

Thanks,
Biswa

@a-ozbek
a-ozbek commented Feb 5, 2017 edited

Excuse me if this issue was brought up before about this script. I couldn't find a resolution to this in the comments.

In this script, during the fine-tuning, the "train_generator" and "validation_generator" do not seem to do VGG16 pre-processing which is

      # 'RGB'->'BGR'  
        x = x[:, :, :, ::-1]  
        # Zero-center by mean pixel  
        x[:, :, :, 0] -= 103.939  
        x[:, :, :, 1] -= 116.779  
        x[:, :, :, 2] -= 123.68  

Isn't it wrong to do fine-tuning of VGG16 without this pre-processing step?

@aidiary
aidiary commented Feb 16, 2017

@a-ozbek

I have the same question.
I have experimented with and without this pre-processing and get the slightly better result in the case of without this pre-processing...

without pre-processing => val_acc = 93.5%@ epoch 50
with pre-processing      => val_acc = 92.8% @ epoch 50

I think this pre-processing is essential to use keras.applications.vgg16..
Does anyone know why?

# with pre-precessing version
# Use keras 1.2.2 for preprocessing_function

# x = 3D tensor version
def preprocess_input(x):
    # 'RGB'->'BGR'
    x = x[:, :, ::-1]
    # Zero-center by mean pixel
    x[:, :, 0] -= 103.939
    x[:, :, 1] -= 116.779
    x[:, :, 2] -= 123.68
    return x

train_datagen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(
    preprocessing_function=preprocess_input)
@jdelange

@CharlesNord: same here, at best 91.75%. Also tried the VGG16 net included in keras\applications, but that gave similar results.

@embanner
embanner commented Mar 5, 2017

See this notebook for an example of fine-tuning a keras.applications.vgg16.VGG16 that hooks together keras.preprocessing.image.ImageDataGenerator withkeras.applications.vgg16.preprocess_input() for image preprocessing.

Note, I'm using the theano backend.

Original gist at https://gist.github.com/embanner/6149bba89c174af3bfd69537b72bca74.

@cswwp
cswwp commented Mar 7, 2017

@biswatherockstar I also have the same problem, do you solove it ?

@Irtza
Irtza commented Mar 15, 2017

The applications.VGG16 model is defined using the Functional API .. when I'm trying to model.'add' the instance returned by VGG16 base class on imagenet weights. I get Attribute error: that metaclass keras.engine.training.Model has no arrtibute add? has there been a change ? or am I missing something?

@sampathweb
sampathweb commented Mar 15, 2017 edited

@Irtza Yes, keras.applications.vgg16 uses Functional API. You can only use the "add" method to a Sequential API. Functional API is actually more flexible and you can build out a graph. For an example of how to add your own layers on top, checkout this notebook posted by @embanner couple of posts above here.

@fwahhab89

I am trying to finetune the VGG16 for catsVSdogs datasets by exactly replicating the gist above. I am using tensorflow as my backend.
But I am getting the error at line:
top_model.add(Flatten(input_shape=model.output_shape[1:]))

The error is:
ValueError: The shape of the input to "Flatten" is not fully defined (got (None, None, 512). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.

I am already up to date with the Keras 2 API. Please help me out here.

@rikenmehta03

While trying above code, I am getting this error when adding top layers.
AttributeError: 'Model' object has no attribute 'add'

@telesphore
telesphore commented Mar 18, 2017 edited

@fwahhab89

I ran into a similar issue. I think you may need to specify the input shape for your VGG16 model. I'm not doing "cats and dogs" here but for my case I did.

model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))

Remember tensorflow vs theano order

@kevinpatricksmith
kevinpatricksmith commented Mar 19, 2017 edited

I am working through this set of three tutorials with tensorflow 1.0 on GPU and keras 2

When I try to create the top_model and as a first add step call Flatten, I get the following error:

ValueError: The shape of the input to "Flatten" is not fully defined (got (None, None, 512). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model

When fix as per @fwahhab89 with input-shape=(150,150,3), I then get the same error as rikenmehta03:
model.add(top_model)
AttributeError: 'Model' object has no attribute 'add'

Sequential model has an add function, but VGG16 is based on Model.

@Omegamon

Got error like this:

 base_model.add(top_model)
AttributeError: 'Model' object has no attribute 'add'
@kevinpatricksmith
kevinpatricksmith commented Mar 19, 2017 edited

This "AttributeError: 'Model' object has no attribute 'add'" issue seems to be related to this issue: fchollet/keras#3465

Their approach would work if we weren't trying to reload the top model weights from part 2 of the tutorial

Should it perhaps be something like:
model = Model(input=model.input, output=top_model)

@Omegamon
Omegamon commented Mar 20, 2017 edited

@kevinpatricksmith
Thanks!
Fixed by this :

  input_tensor = Input(shape=(150,150,3))
  base_model = VGG16(weights='imagenet',include_top= False,input_tensor=input_tensor)
  top_model = Sequential()
  top_model.add(Flatten(input_shape=base_model.output_shape[1:]))
  top_model.add(Dense(256, activation='relu'))
  top_model.add(Dropout(0.5))
  top_model.add(Dense(1, activation='sigmoid'))
  top_model.load_weights('bootlneck_fc_model.h5')
  model = Model(input= base_model.input, output= top_model(base_model.output))
@hiroyachiba

I guess this part needs to be updated for Keras 2 API.

# fine-tune the model
model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)
@kevinpatricksmith
kevinpatricksmith commented Mar 21, 2017 edited

@Omegamon and @hiroyachiba
Thanks.
I used both your suggestions and can now get training to complete.

I am now getting accuracy of 92% and validation set accuracy of almost 90%. However it does not really improve much from epoch to epoch.

When I use model.summary() to print out the model, I am seeing only one additional layer after the last set of VGG16 convolution/pooling. It is called Sequential_1. I don't know if the top_model was added correctly:

model.summary()

Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 150, 150, 3)       0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 150, 150, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 150, 150, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 75, 75, 64)        0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 75, 75, 128)       73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 75, 75, 128)       147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 37, 37, 128)       0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 37, 37, 256)       295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 37, 37, 256)       590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 37, 37, 256)       590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 18, 18, 256)       0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 18, 18, 512)       1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 18, 18, 512)       2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 18, 18, 512)       2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 9, 9, 512)         0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 9, 9, 512)         2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 9, 9, 512)         2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 9, 9, 512)         2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 4, 4, 512)         0         
_________________________________________________________________
sequential_1 (Sequential)    (None, 1)                 2097665   
=================================================================
@shnayder

This code and https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html says to freeze the first 25 layers, but applications.VGG16 doesn't have that many! Pretty sure you need to freeze the first 15, not 25: here's my model.layers, including the top dense sequential model):

[<keras.engine.topology.InputLayer at 0x11aa49290>,
 <keras.layers.convolutional.Conv2D at 0x11aa49610>,
 <keras.layers.convolutional.Conv2D at 0x11a710490>,
 <keras.layers.pooling.MaxPooling2D at 0x11aa6c5d0>,
 <keras.layers.convolutional.Conv2D at 0x11aa49310>,
 <keras.layers.convolutional.Conv2D at 0x11aa78310>,
 <keras.layers.pooling.MaxPooling2D at 0x119c18fd0>,
 <keras.layers.convolutional.Conv2D at 0x119ca7fd0>,
 <keras.layers.convolutional.Conv2D at 0x11aa13510>,
 <keras.layers.convolutional.Conv2D at 0x11ac1da90>,
 <keras.layers.pooling.MaxPooling2D at 0x11b4b6f90>,
 <keras.layers.convolutional.Conv2D at 0x11b4c2c90>,
 <keras.layers.convolutional.Conv2D at 0x11b535990>,
 <keras.layers.convolutional.Conv2D at 0x11b518cd0>,
 <keras.layers.pooling.MaxPooling2D at 0x11b57dd50>, # want to freeze up to and including this layer
 <keras.layers.convolutional.Conv2D at 0x11b5a9950>,
 <keras.layers.convolutional.Conv2D at 0x11b84f990>,
 <keras.layers.convolutional.Conv2D at 0x11b58af90>,
 <keras.layers.pooling.MaxPooling2D at 0x11bb59bd0>,
 <keras.models.Sequential at 0x1179c58d0>]
@Omegamon

@kevinpatricksmith
My model structure is the same as yours. I froze the first 15 layers by:

for layer in model.layers[:15]:
    layer.trainable = False

After 100 epoch, I got accuracy of 99.8% and validation set accuracy of 93.75%. After 50 epoch, the validation set accuracy
stabilized in the range of 93% ~ 94%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment