Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
Fine-tuning a Keras model. Updated to the Keras 2.0 API.
'''This script goes along the blog post
"Building powerful image classification models using very little data"
from blog.keras.io.
It uses data that can be downloaded at:
https://www.kaggle.com/c/dogs-vs-cats/data
In our setup, we:
- created a data/ folder
- created train/ and validation/ subfolders inside data/
- created cats/ and dogs/ subfolders inside train/ and validation/
- put the cat pictures index 0-999 in data/train/cats
- put the cat pictures index 1000-1400 in data/validation/cats
- put the dogs pictures index 12500-13499 in data/train/dogs
- put the dog pictures index 13500-13900 in data/validation/dogs
So that we have 1000 training examples for each class, and 400 validation examples for each class.
In summary, this is our directory structure:
```
data/
train/
dogs/
dog001.jpg
dog002.jpg
...
cats/
cat001.jpg
cat002.jpg
...
validation/
dogs/
dog001.jpg
dog002.jpg
...
cats/
cat001.jpg
cat002.jpg
...
```
'''
from keras import applications
from keras.preprocessing.image import ImageDataGenerator
from keras import optimizers
from keras.models import Sequential
from keras.layers import Dropout, Flatten, Dense
# path to the model weights files.
weights_path = '../keras/examples/vgg16_weights.h5'
top_model_weights_path = 'fc_model.h5'
# dimensions of our images.
img_width, img_height = 150, 150
train_data_dir = 'cats_and_dogs_small/train'
validation_data_dir = 'cats_and_dogs_small/validation'
nb_train_samples = 2000
nb_validation_samples = 800
epochs = 50
batch_size = 16
# build the VGG16 network
model = applications.VGG16(weights='imagenet', include_top=False)
print('Model loaded.')
# build a classifier model to put on top of the convolutional model
top_model = Sequential()
top_model.add(Flatten(input_shape=model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))
# note that it is necessary to start with a fully-trained
# classifier, including the top classifier,
# in order to successfully do fine-tuning
top_model.load_weights(top_model_weights_path)
# add the model on top of the convolutional base
model.add(top_model)
# set the first 25 layers (up to the last conv block)
# to non-trainable (weights will not be updated)
for layer in model.layers[:25]:
layer.trainable = False
# compile the model with a SGD/momentum optimizer
# and a very slow learning rate.
model.compile(loss='binary_crossentropy',
optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
metrics=['accuracy'])
# prepare data augmentation configuration
train_datagen = ImageDataGenerator(
rescale=1. / 255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1. / 255)
train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='binary')
validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='binary')
# fine-tune the model
model.fit_generator(
train_generator,
samples_per_epoch=nb_train_samples,
epochs=epochs,
validation_data=validation_generator,
nb_val_samples=nb_validation_samples)

Great example, thanks for sharing Francois!! Aside from the comment from yurkor regarding the proper VGG image pre-processing (which I agree with), I have a few questions about the general procedure:

  1. Would it not be better to train the top model with different samples than you will later on use to fine tune your model? In the cat/dog example that would mean put aside 10-20% as validation set and then use 50% of the remainder for creating the top model and the other 50% for fine tuning. Any views?
  2. Do you have any insights regarding the best size of the top model when your final goal is a segmentation into 10 classes rather than 2? I tried to go back to the original VGG16 architecture, experimenting with two fully connected layers of 1024 and 512, but it appears there still is substantial overfitting. Any advice?

@SchumErik

  1. I think this approach need more data
  2. again more data if you train large network. You can use regularization to defeat overfitting

By the way, if you have data size like ImageNet, you even can train VGG16 from scratch, not use fine tune at all.

Arsey commented Sep 11, 2016

@SchumErik, about the second question, my experiments shows that the top model from the original VGG16 works better than model with decreased dimension. Here I classify 102 classes https://github.com/Arsey/keras-transfer-learning-for-oxford102

Hi Francois, first of all, thanks for all this great material.
Quick question... is there a place from which we can download the fc_model.h5 ?
Many thanks,
AL.

@alinagithub - You have to train this model before fine tuning. There is one more tutorial from Francois about how to generate that model.

If we want to categorize 10 or 20 classes, is it necessary just to fine tune the same code over 10 or 20 classes by adding them as sub-folders into the train and validation data?

@vishsangale - many thanks!

I implemented the above model and everything worked fine. Now I have a sample of 1000 images in my test_data named folder which I want to predict using the model.predict() method. I first generated the test features as we generated the train_data above. This is shown below:
generator = datagen.flow_from_directory(test_dir, batch_size = 32, target_size =(img_rows, img_columns), classes = None, shuffle =False)

test_data_features = model.predict_generator(generator, 1000)
np.save(open('test_data_features.npy','wb'), test_data_features)
test_data = np.load(open('test_data_features.npy', 'rb'))

Now when I make predictions like model.predict(test_data), I get a numpy array of predictions. How can I test what each prediction is represting? The predictions are like this:
array([[ 9.99770999e-01], [ 7.65574304e-03], [ 2.06944350e-07], [ 9.96615469e-01], [ 4.59789817e-07], [ 9.93980138e-05], [ 5.27667798e-05],..........)

How can I compare my predictions to my images?

you can try model.predict_classes(test_data)

In addition to my yesterday's question, I would like to know that if we want to classify an object which is not belong to any of the categories in ImageNet what I have to do?

Arsey commented Oct 3, 2016

I'm trying to adopt this approach for multiclass classification. And the first step when we're training the top model from bottlenecks works fine and I have 76% accuracy for 102 classes. But when I proceed to fine-tune step, the model with each next epoch decreases in accuracy. So if at the end of 1st epoch of fine-tune I have about 45% of accuracy, on the 2nd epoch I'm getting 40%, etc. Here's repo with the code https://github.com/Arsey/keras-oxford102. Can anyone help? I'm trying to find the issue for a few weeks but no results :(

apapiu commented Oct 5, 2016

Great Post, I am learning a lot. I do have a question about the top model. If I understand correctly I need to have the weights properly tuned already before sticking it to the convolutional layers. But how do I compute the weights to begin with?

To be more specific: In this line: top_model_weights_path = 'fc_model.h5' how did you compute the weights for the top model?

@alinagithub Did you find the tutorial about how to train fc_model.h5 ? I can't find it.

Golly commented Oct 8, 2016

@Arsey Same problem with fine tuning. Im trying now and model decreased in accuracy from 87% to 50%.

Euphemiasama commented Oct 9, 2016

@apapiu @CharlesNord You can compute the weights to get the 'fc_model.h5' file by running part 2 of this tutorial entitled 'Using the bottleneck features of a pre-trained network: 90% accuracy in a minute' it creates a file named 'bottleneck_fc_model.h5' you rename it to 'fc_model.h5' and run the code.

@fchollet
In line 156 and 162, target_size=(img_height, img_width)
which should go first 'img_height' or 'img_width'?
In classifier_from_little_data_script_2.py, it is in reverse order. I am confused here.

@Euphemiasama Thank you very much. Another question, has any one reached 95% accuracy by using the fine tuning method? Before fine tuning, my accuracy is about 90%, and after fine tuning, it reaches 92% and I cannot make the result better anymore

How do I predict_classes on a new a Image(which is not in data base),
well I am trying the code below to load model and load a new Image pass through the predict_classes but I am getting an error:

import keras
from keras.models import load_model
from keras.models import Sequential
import cv2
import numpy as np 
from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img

model = Sequential()

model =load_model('firstmodel.h5')
model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

img = cv2.imread('cat.jpg',0).astype(np.float32) / 255
img = cv2.resize(img, (150, 150))
#img = prep_data(img)
#img = img.flatten()
img = np.expand_dims(img, axis=0)
# img1 = np.expand_dims(img, axis=0)
# img = cv2.imread('cat.jpg')
# img = cv2.resize(img,(150,150))
# x = img_to_array(img)  # this is a Numpy array with shape (3, 150, 150)
classes = model.predict_classes(img)
print classes

I am getting error:

using Theano backend.
Traceback (most recent call last):
  File "detect.py", line 28, in <module>
    classes = model.predict_classes(img)
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 780, in predict_classes
    proba = self.predict(x, batch_size=batch_size, verbose=verbose)
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 672, in predict
    return self.model.predict(x, batch_size=batch_size, verbose=verbose)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1174, in predict
    check_batch_dim=False)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 100, in standardize_input_data
    str(array.shape))
Exception: Error when checking : expected convolution2d_input_1 to have 4 dimensions, but got array with shape (1, 150, 150)

nathmo commented Dec 14, 2016

hi, i made a derrived script from your code and get the following error :
i'm running tensor flow 0.12.0-rc1 With python 2.7
i have installed the latest version of keras from git.
What i have done wrong ?

Source code : http://pastebin.com/JEqcZe5x
Full Error : http://pastebin.com/yEFVHWEK

"ValueError: Error when checking model input: expected convolution2d_input_1 to have shape (None, 3, 32, 32) but got array with shape (32, 32, 32, 3)"

Thank in advance

Nathann

Abusnina commented Jan 5, 2017

How can we use the modified pre-trained model to predict test data? I compiled the model successfully, but when I try to predict my test data i get the following error:
Error when checking : expected flatten_input_1 to have shape (None, 4, 4, 512) but got array with shape (1, 3, 150, 150)
Any advice on this?

@Golly
@Arsey

Same problem, before fine-tuning my model for 5 classes reached 98% accuracy but the first epoch of fine-tuning dropped to 20%. Did you or does anyone work it out for multi-class problem? I guess we need more train data to feed our model

Is the slice of :25 correct for setting the non-trainable parameter? If you do model.layers[:26] you see that the last layer is also a Conv layer.

Hi everyone,
I am following this example on GRAYSCALE images, which I converted them from color scale using PIL convert method. I expected the input_sahpe in the model should be (1, w, h). but I get this error:
Exception: Error when checking model input: expected convolution2d_input_1 to have shape (None, 1, 150, 150) but got array with shape (32, 3, 150, 150)
When I checked the image shape, I got only (w,h)
However, It still works with input_shape=(3, w, h). Isn't it strange?

My goal to compare the learning on gray images to color images. But I do not know how to expand the dimension of gray images within flow_from_directory.

Anyone can help please?

Thanks
Kamal

cobir commented Jan 30, 2017

Hi,

The input shape of the model is (3, img_width, img_height). Does this mean that we can only work with theano as backend when working with this pre-trained VGG16 model?

Thanks,
OC

@cobir

I used Tensorflow backend it works fine.

I get the following error when fine tuning a 8 class multi class classification. Any idea anyone please help

nb_val_samples=nb_validation_samples)

File "C:\Users\bgsingh\Anaconda2\lib\site-packages\keras\models.py", line 935, in fit_generator
initial_epoch=initial_epoch)
File "C:\Users\bgsingh\Anaconda2\lib\site-packages\keras\engine\training.py", line 1553, in fit_generator
class_weight=class_weight)
File "C:\Users\bgsingh\Anaconda2\lib\site-packages\keras\engine\training.py", line 1310, in train_on_batch
check_batch_axis=True)
File "C:\Users\bgsingh\Anaconda2\lib\site-packages\keras\engine\training.py", line 1034, in _standardize_user_data
exception_prefix='model target')
File "C:\Users\bgsingh\Anaconda2\lib\site-packages\keras\engine\training.py", line 124, in standardize_input_data
str(array.shape))
ValueError: Error when checking model target: expected sequential_2 to have shape (None, 1) but got array with shape (32L, 8L)

Thanks,
Biswa

a-ozbek commented Feb 5, 2017

Excuse me if this issue was brought up before about this script. I couldn't find a resolution to this in the comments.

In this script, during the fine-tuning, the "train_generator" and "validation_generator" do not seem to do VGG16 pre-processing which is

      # 'RGB'->'BGR'  
        x = x[:, :, :, ::-1]  
        # Zero-center by mean pixel  
        x[:, :, :, 0] -= 103.939  
        x[:, :, :, 1] -= 116.779  
        x[:, :, :, 2] -= 123.68  

Isn't it wrong to do fine-tuning of VGG16 without this pre-processing step?

aidiary commented Feb 16, 2017

@a-ozbek

I have the same question.
I have experimented with and without this pre-processing and get the slightly better result in the case of without this pre-processing...

without pre-processing => val_acc = 93.5%@ epoch 50
with pre-processing      => val_acc = 92.8% @ epoch 50

I think this pre-processing is essential to use keras.applications.vgg16..
Does anyone know why?

# with pre-precessing version
# Use keras 1.2.2 for preprocessing_function

# x = 3D tensor version
def preprocess_input(x):
    # 'RGB'->'BGR'
    x = x[:, :, ::-1]
    # Zero-center by mean pixel
    x[:, :, 0] -= 103.939
    x[:, :, 1] -= 116.779
    x[:, :, 2] -= 123.68
    return x

train_datagen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(
    preprocessing_function=preprocess_input)

@CharlesNord: same here, at best 91.75%. Also tried the VGG16 net included in keras\applications, but that gave similar results.

embanner commented Mar 5, 2017

See this notebook for an example of fine-tuning a keras.applications.vgg16.VGG16 that hooks together keras.preprocessing.image.ImageDataGenerator withkeras.applications.vgg16.preprocess_input() for image preprocessing.

Note, I'm using the theano backend.

Original gist at https://gist.github.com/embanner/6149bba89c174af3bfd69537b72bca74.

cswwp commented Mar 7, 2017

@biswatherockstar I also have the same problem, do you solove it ?

Irtza commented Mar 15, 2017

The applications.VGG16 model is defined using the Functional API .. when I'm trying to model.'add' the instance returned by VGG16 base class on imagenet weights. I get Attribute error: that metaclass keras.engine.training.Model has no arrtibute add? has there been a change ? or am I missing something?

sampathweb commented Mar 15, 2017

@Irtza Yes, keras.applications.vgg16 uses Functional API. You can only use the "add" method to a Sequential API. Functional API is actually more flexible and you can build out a graph. For an example of how to add your own layers on top, checkout this notebook posted by @embanner couple of posts above here.

I am trying to finetune the VGG16 for catsVSdogs datasets by exactly replicating the gist above. I am using tensorflow as my backend.
But I am getting the error at line:
top_model.add(Flatten(input_shape=model.output_shape[1:]))

The error is:
ValueError: The shape of the input to "Flatten" is not fully defined (got (None, None, 512). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.

I am already up to date with the Keras 2 API. Please help me out here.

While trying above code, I am getting this error when adding top layers.
AttributeError: 'Model' object has no attribute 'add'

telesphore commented Mar 18, 2017

@fwahhab89

I ran into a similar issue. I think you may need to specify the input shape for your VGG16 model. I'm not doing "cats and dogs" here but for my case I did.

model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))

Remember tensorflow vs theano order

kevinpatricksmith commented Mar 19, 2017

I am working through this set of three tutorials with tensorflow 1.0 on GPU and keras 2

When I try to create the top_model and as a first add step call Flatten, I get the following error:

ValueError: The shape of the input to "Flatten" is not fully defined (got (None, None, 512). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model

When fix as per @fwahhab89 with input-shape=(150,150,3), I then get the same error as rikenmehta03:
model.add(top_model)
AttributeError: 'Model' object has no attribute 'add'

Sequential model has an add function, but VGG16 is based on Model.

Got error like this:

 base_model.add(top_model)
AttributeError: 'Model' object has no attribute 'add'

kevinpatricksmith commented Mar 19, 2017

This "AttributeError: 'Model' object has no attribute 'add'" issue seems to be related to this issue: fchollet/keras#3465

Their approach would work if we weren't trying to reload the top model weights from part 2 of the tutorial

Should it perhaps be something like:
model = Model(input=model.input, output=top_model)

Omegamon commented Mar 20, 2017

@kevinpatricksmith
Thanks!
Fixed by this :

  input_tensor = Input(shape=(150,150,3))
  base_model = VGG16(weights='imagenet',include_top= False,input_tensor=input_tensor)
  top_model = Sequential()
  top_model.add(Flatten(input_shape=base_model.output_shape[1:]))
  top_model.add(Dense(256, activation='relu'))
  top_model.add(Dropout(0.5))
  top_model.add(Dense(1, activation='sigmoid'))
  top_model.load_weights('bootlneck_fc_model.h5')
  model = Model(input= base_model.input, output= top_model(base_model.output))

I guess this part needs to be updated for Keras 2 API.

# fine-tune the model
model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

kevinpatricksmith commented Mar 21, 2017

@Omegamon and @hiroyachiba
Thanks.
I used both your suggestions and can now get training to complete.

I am now getting accuracy of 92% and validation set accuracy of almost 90%. However it does not really improve much from epoch to epoch.

When I use model.summary() to print out the model, I am seeing only one additional layer after the last set of VGG16 convolution/pooling. It is called Sequential_1. I don't know if the top_model was added correctly:

model.summary()

Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 150, 150, 3)       0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 150, 150, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 150, 150, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 75, 75, 64)        0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 75, 75, 128)       73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 75, 75, 128)       147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 37, 37, 128)       0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 37, 37, 256)       295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 37, 37, 256)       590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 37, 37, 256)       590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 18, 18, 256)       0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 18, 18, 512)       1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 18, 18, 512)       2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 18, 18, 512)       2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 9, 9, 512)         0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 9, 9, 512)         2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 9, 9, 512)         2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 9, 9, 512)         2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 4, 4, 512)         0         
_________________________________________________________________
sequential_1 (Sequential)    (None, 1)                 2097665   
=================================================================

This code and https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html says to freeze the first 25 layers, but applications.VGG16 doesn't have that many! Pretty sure you need to freeze the first 15, not 25: here's my model.layers, including the top dense sequential model):

[<keras.engine.topology.InputLayer at 0x11aa49290>,
 <keras.layers.convolutional.Conv2D at 0x11aa49610>,
 <keras.layers.convolutional.Conv2D at 0x11a710490>,
 <keras.layers.pooling.MaxPooling2D at 0x11aa6c5d0>,
 <keras.layers.convolutional.Conv2D at 0x11aa49310>,
 <keras.layers.convolutional.Conv2D at 0x11aa78310>,
 <keras.layers.pooling.MaxPooling2D at 0x119c18fd0>,
 <keras.layers.convolutional.Conv2D at 0x119ca7fd0>,
 <keras.layers.convolutional.Conv2D at 0x11aa13510>,
 <keras.layers.convolutional.Conv2D at 0x11ac1da90>,
 <keras.layers.pooling.MaxPooling2D at 0x11b4b6f90>,
 <keras.layers.convolutional.Conv2D at 0x11b4c2c90>,
 <keras.layers.convolutional.Conv2D at 0x11b535990>,
 <keras.layers.convolutional.Conv2D at 0x11b518cd0>,
 <keras.layers.pooling.MaxPooling2D at 0x11b57dd50>, # want to freeze up to and including this layer
 <keras.layers.convolutional.Conv2D at 0x11b5a9950>,
 <keras.layers.convolutional.Conv2D at 0x11b84f990>,
 <keras.layers.convolutional.Conv2D at 0x11b58af90>,
 <keras.layers.pooling.MaxPooling2D at 0x11bb59bd0>,
 <keras.models.Sequential at 0x1179c58d0>]

@kevinpatricksmith
My model structure is the same as yours. I froze the first 15 layers by:

for layer in model.layers[:15]:
    layer.trainable = False

After 100 epoch, I got accuracy of 99.8% and validation set accuracy of 93.75%. After 50 epoch, the validation set accuracy
stabilized in the range of 93% ~ 94%

codemukul95 commented Mar 25, 2017

Hi! If we look at the VGG16 source code (https://github.com/fchollet/keras/blob/master/keras/applications/vgg16.py) number of layers till last convolutional block are 18. they why learning rate of first 25 layers is set to false in the above code.

for layer in model.layers[:25]:
 layer.trainable = False 

Please clarify it ASAP.
Thanks in anticipation.

liangxiao05 commented Apr 1, 2017

for layer in model.layers[:25]
[:25] shoule be [:15]

block5_conv1 (Convolution2D) (None, 512, 9, 9) 2359808 block4_pool[0][0]


block5_conv2 (Convolution2D) (None, 512, 9, 9) 2359808 block5_conv1[0][0]


block5_conv3 (Convolution2D) (None, 512, 9, 9) 2359808 block5_conv2[0][0]


block5_pool (MaxPooling2D) (None, 512, 4, 4) 0 block5_conv3[0][0]


sequential_1 (Sequential) (None, 1) 2097665 block5_pool[0][0]


Total params: 16,812,353
Trainable params: 9,177,089 ----------------#9177089=2359808*3+2097665
Non-trainable params: 7,635,264

@Abusnina How did you solve this problem?

What if we want to use multi-gpu?

CP121 commented Apr 8, 2017

What is the Keras API 2.0 version of this:

model = Model(input=base_model.input, output=top_model(base_model.output))

This line produces the user warning:

UserWarning: Update yourModelcall to the Keras 2 API:Model(outputs=Tensor("se..., inputs=Tensor("in...)

lusob commented Apr 16, 2017

@jwgs1 Use plurals in the args:

model = Model(inputs=base_model.input, outputs=top_model(base_model.output))

It was quite a trouble running this. Finally did it with your inputs @Omegamon @hiroyachiba @liangxiao05 :)

saulthu commented Apr 19, 2017

Working code (below) was updated as per comments to run with Keras 2.

from keras import applications
from keras.preprocessing.image import ImageDataGenerator
from keras import optimizers
from keras.models import Sequential
from keras.models import Model
from keras.layers import Dropout, Flatten, Dense

# path to the model weights files.
weights_path = '../keras/examples/vgg16_weights.h5'
top_model_weights_path = 'bottleneck_fc_model.h5'
# dimensions of our images.
img_width, img_height = 150, 150

train_data_dir = 'data/train'
validation_data_dir = 'data/validation'
nb_train_samples = 2000
nb_validation_samples = 800
epochs = 50
batch_size = 16

# build the VGG16 network
base_model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(150,150,3))
print('Model loaded.')

# build a classifier model to put on top of the convolutional model
top_model = Sequential()
top_model.add(Flatten(input_shape=base_model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))

# note that it is necessary to start with a fully-trained
# classifier, including the top classifier,
# in order to successfully do fine-tuning
top_model.load_weights(top_model_weights_path)

# add the model on top of the convolutional base
# model.add(top_model)
model = Model(inputs=base_model.input, outputs=top_model(base_model.output))

# set the first 25 layers (up to the last conv block)
# to non-trainable (weights will not be updated)
for layer in model.layers[:15]:
    layer.trainable = False

# compile the model with a SGD/momentum optimizer
# and a very slow learning rate.
model.compile(loss='binary_crossentropy',
              optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])

# prepare data augmentation configuration
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='binary')

model.summary()

# fine-tune the model
model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size,
    verbose=2)

Can any one tell me what is the "predict_class" equivalent in keras 2.0.3?
model.predict(x) gives me array of probas

Thank you.

@bis-carbon Believe it's predict_classes now. If you use anaconda, you can take a look at models.py under ~/anaconda2/lib/python2.7/site-packages/keras/

raaju-shiv commented Apr 27, 2017

Hi, I have a dimension mismatch error on running the code:
`import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dropout, Flatten, Dense
from keras import applications
from keras import backend as K
K.set_image_dim_ordering('tf')
from keras import optimizers
from keras.models import Model

path to the model weights files.

weights_path = '../keras/examples/vgg16_weights.h5'
top_model_weights_path = 'bottleneck_fc_model.h5'
img_width, img_height = 150, 150
train_data_dir = 'data4/train'
validation_data_dir = 'data4/validation'
nb_train_samples = 20
nb_validation_samples = 20
epochs = 5
batch_size = 1
base_model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(150,150,3))
print('Model loaded.')
top_model = Sequential()
top_model.add(Flatten(input_shape=base_model.output_shape[1:]))
top_model.add(Dense(128, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))
top_model.load_weights('bottleneck_fc_model.h5')
model = Model(input= base_model.input, output= top_model(base_model.output))
for layer in model.layers[:15]:
layer.trainable = False
model.compile(loss='binary_crossentropy',
optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
metrics=['accuracy'])

prepare data augmentation configuration

train_datagen = ImageDataGenerator(
rescale=1. / 255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1. / 255)
train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='binary')
validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='binary')
model.summary()

fine-tune the model

model.fit_generator(
train_generator,
steps_per_epoch=nb_train_samples // batch_size,
epochs=epochs,
validation_data=validation_generator,
validation_steps=nb_validation_samples // batch_size,
verbose=2)`

The error is: Dimension 0 in both shapes must be equal, but are 8192 and 115200 for 'Assign_734' (op: 'Assign') with input shapes: [8192,128], [115200,128].
The error occurs when I try to execute the comment:
top_model.load_weights('bottleneck_fc_model.h5').

Kindly help.

skoch9 commented Apr 28, 2017

I run a slightly modified version of this example which only fine tunes the top layers (with Keras 2.0.3/Tensorflow on Ubuntu). This looks like the following:

img_width, img_height = 150, 150
train_data_dir = 'data/train_s'
validation_data_dir = 'data/val_s'
nb_train_samples = 2000
nb_validation_samples = 800
epochs = 10
batch_size = 16

base_model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_width, img_height, 3))

top_model = Sequential()
top_model.add(Flatten(input_shape=base_model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dense(1, activation='sigmoid'))

model = Model(inputs=base_model.input, outputs=top_model(base_model.output))
model.compile(loss='binary_crossentropy', optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])

train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='binary', shuffle=False)

model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size,
    verbose=2, workers=12)

score = model.evaluate_generator(validation_generator, nb_validation_samples/batch_size, workers=12)

scores = model.predict_generator(validation_generator, nb_validation_samples/batch_size, workers=12)

correct = 0
for i, n in enumerate(validation_generator.filenames):
    if n.startswith("cats") and scores[i][0] <= 0.5:
        correct += 1
    if n.startswith("dogs") and scores[i][0] > 0.5:
        correct += 1

print("Correct:", correct, " Total: ", len(validation_generator.filenames))
print("Loss: ", score[0], "Accuracy: ", score[1])

With this, I get arbitrary validation accuracy results. For example, predict_generator predicts 640 out of 800 (80%) classes correctly whereas evaluate_generator produces an accuracy score of 95%. Someone in #3477 suggests to remove the rescale=1. / 255 parameter from the validation generator, then I get results of 365/800=45% and 89% from evaluate_generator.

Is there something wrong with my evaluation or is this due to a bug? There are many similar issues (e.g. #3849, #6245) where the stated accuracy (during training and afterwards) doesn't match the actual predictions. Could someone experienced maybe shine some light onto this problem?

@Golly
@Arsey
@austinchencym

Same problem, before fine-tuning my model for 10 classes reached 68% accuracy but the first epoch of fine-tuning dropped to 30%. Did you or does anyone work out for multi-class problem? Thanks!

sakvaua commented May 3, 2017

Why do you guys not de-mean images but rescale=1. / 255, while pretrained VGG models require only de-meaning?

chairath commented May 5, 2017

after I finetuned already,but How can I extract feature from the added layer (sequentail_1 > dense)

what is the equivalent of predict_classes in keras 2.0

Hello there :)

1- I would like to know (please) what changes to do in order to make this work on python 3.5.

2- Also, i would like to know why in the code of tutorial 2 (for the bottleneck pre-trained features), we use put input shape as 'train_data.shape' inside the "flatten" layer, whereas in this code we choose the input shape to be 'model.output_shape' (see the line below for the particular 2 pieces of codes that i am talking about):

"model.add(Flatten(input_shape=train_data.shape[1:]))" versus " top_model.add(Flatten(input_shape=model.output_shape[1:]))"

Thank you for your time :)

@saulthu thank you for sharing your working code :) u da man!!
and yeah i don't understand why in the initial posted code all 24 layers were set to non trainable when we wanted to keep the last Conv Block free.

@saulthu
@akaashagarwal
Hi Guys,
What do you feel is the best way to fine-tune the Keras Xception network?

# set the first 100 layers 
for layer in model.layers[:100]:
layer.trainable = False

Can I do like this or do I have to begin the fine-tuning specifically from where the exit_flow starts and not in the entry and middle_flow?
Looking forward...

@raaju-shiv , i ran into the same error, "Dimension 0 in both shapes must be equal, but are 25088 and 8192 for 'Assign_26'"
do you have solved it?

jroberayalas commented May 20, 2017

Quick questions:

  • Once you trained your model and you want to try it with new images, is there a fast way to convert it to the corresponding input dimensions, i.e. (150, 150, 3)? Can we use the image generator somehow to achieve this?
  • model.predict(x) computes the class for a given input x. Is there a way to get the probability of belonging to such class?

kai06046 commented Jun 4, 2017

Hi there, can anyone tell me why start with a fully-trained classifier is necessary?

top_model.load_weights(top_model_weights_path)

oxydron commented Jun 6, 2017

Line 75 doesn't work for application ResNet50. Any tips?

How do you add auc computation to this code?

HodaGH commented Jun 29, 2017

I really couldn't find any solution to get rid of this error: Error when checking : expected flatten_input_1 to have shape (None, 4, 4, 512) but got array with shape (1, 3, 150, 150). Any ideas ? appreciate any help :)

I forked and updated this gist for Keras 1.0.5

I still get some warnings but, hey, they're not errors (yet!)

classifier_from_little_data_script_3.py:75: UserWarning: Update your 'Model' call to the Keras 2 API: 'Model(outputs=Tensor("se..., inputs=Tensor("in...)'
model = Model(input= base_model.input, output= top_model(base_model.output))
Model loaded.
Found 20000 images belonging to 2 classes.
Found 5000 images belonging to 2 classes.
classifier_from_little_data_script_3.py:117: UserWarning: Update your 'fit_generator' call to the Keras 2 API: 'fit_generator(<keras.pre..., validation_data=<keras.pre..., steps_per_epoch=125, epochs=50, validation_steps=800)' nb_val_samples=nb_validation_samples)

How do we create the 'bottleneck_fc_model.h5'?

Is that the same model we save by transfer-learning in Step 2 of the blog post? https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

mahernadar commented Jul 6, 2017

Dear @fchollet, @saulthu, (or others :) )

Could you please explain to me why we do not need the training and validation labels in this example?

Is it because the labeling is done automatically through the sub-folders within the database folder? If so, why did we use labeling in tutorial of the bottleneck features?

Thanks

@radhakrishnancegit yes it is the same :)

ardianumam commented Jul 10, 2017

@hiroyachiba @kevinpatricksmith : For mine, by setting "steps_per_epoch=nb_train_samples // batch_size" where nb_train_samples = 1998 and batch_size = 18, I get number per epoch = 111 // 18 = 6 instead. When I set "steps_per_epoch=nb_train_samples", I get number per epoch = 111. Don't know why. But, we should expect 111 (from: 1998//18) in this case, right? I use Keras 2.0.2 with tensorFlow 1.2.0.

nb_train_samples = 1998
nb_validation_samples = 800
epochs = 50
batch_size = 18
#......
#......
#......
# fine-tune the model
model.fit_generator(
    train_generator,
    samples_per_epoch=nb_train_samples,
    epochs=epochs,
    validation_data=validation_generator,
    nb_val_samples=nb_validation_samples)

@Golly, @Arsey and @austinchencym,

I also am experimenting the same issue. When i applied the BottleNeck example on my 5 class project, i get respectable accuracies around 90%.

But when i switch to the fine-tuning of the 15th layer and above (as this example prescribes), i start with an accuracy of around 30% in the 1st epoch, and throughout this 1st epoch, it keeps going down while going through.

Anyone of you guys figured out the solution for this weird behavior?

Thanks :)

GitHubKay commented Jul 19, 2017

Hello,

i struggle getting this code to run properly. I followed the suggestions and modified my code to reduce the errors step by step.
However, now i have no idea how to proceed. Any help would be really appreciated :)

Setup

  • Win10
  • Python 3.6
  • Keras 2.0.5/ Theano 0.9.0
  • tensorflow-gpu 1.2.0

Actual Code

from keras import applications
from keras.preprocessing.image import ImageDataGenerator
from keras import optimizers
from keras.models import Sequential
from keras.models import Model
from keras.layers import Dropout, Flatten, Dense

# path to the model weights files.
weights_path = 'vgg16_weights.h5'
top_model_weights_path = 'fc_model.h5'
# dimensions of our images.
img_width, img_height = 150, 150

train_data_dir = 'data/sample/train'
validation_data_dir = 'data/sample/valid'
nb_train_samples = 2000
nb_validation_samples = 800
epochs = 50
batch_size = 16

# build the VGG16 network
model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(150,150,3))
#model = applications.VGG16(weights='imagenet', include_top=False)
print('Model loaded.')

# build a classifier model to put on top of the convolutional model
input_tensor = Input(shape=(150,150,3))
base_model = VGG16(weights='imagenet',include_top= False,input_tensor=input_tensor)
#top_model = Sequential()
#top_model.add(Flatten(input_shape=base_model.output_shape[1:]))
#top_model.add(Dense(256, activation='relu'))
#top_model.add(Dropout(0.5))
#top_model.add(Dense(1, activation='sigmoid'))
#top_model.load_weights('bootlneck_fc_model.h5')
#model = Model(input= base_model.input, output= top_model(base_model.output))
top_model = Sequential()
top_model.add(Flatten(input_shape=base_model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))



# note that it is necessary to start with a fully-trained
# classifier, including the top classifier,
# in order to successfully do fine-tuning
top_model.load_weights(top_model_weights_path)

# add the model on top of the convolutional base
#model.add(top_model)
model = Model(inputs=base_model.input, outputs=top_model(base_model.output))

# set the first 15 layers (up to the last conv block)
# to non-trainable (weights will not be updated)
for layer in model.layers[:15]:
    layer.trainable = False

# compile the model with a SGD/momentum optimizer
# and a very slow learning rate.
model.compile(loss='binary_crossentropy',
              optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])

# prepare data augmentation configuration
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='binary')

model.sumary()

# fine-tune the model
#model.fit_generator(
#    train_generator,
#    samples_per_epoch=nb_train_samples,
#    epochs=epochs,
#    validation_data=validation_generator,
#    nb_val_samples=nb_validation_samples)

model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size, verbose=2)

Error

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-2-234711fb2433> in <module>()
     26 
     27 # build a classifier model to put on top of the convolutional model
---> 28 input_tensor = Input(shape=(150,150,3))
     29 base_model = VGG16(weights='imagenet',include_top= False,input_tensor=input_tensor)
     30 #top_model = Sequential()

NameError: name 'Input' is not defined

GitHubKay commented Jul 20, 2017

@skoch9 i tried your code now, but it seems like my GPU run out of memory even though it says:

  • The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
  • Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_TIMEOUT

The Process starts and breaks down saying, that python is down.

Using TensorFlow backend.
Found 2000 images belonging to 2 classes.
Found 800 images belonging to 2 classes.
Epoch 1/10

Here the full Report:

name: GeForce GTX 950
major: 5 minor: 2 memoryClockRate (GHz) 1.2785
pciBusID 0000:04:00.0
Total memory: 2.00GiB
Free memory: 1.64GiB
2017-07-20 10:24:50.315152: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:961] DMA: 0
2017-07-20 10:24:50.317399: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0:   Y
2017-07-20 10:24:50.317852: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 950, pci bus id: 0000:04:00.0)
2017-07-20 10:24:53.389471: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.52GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-07-20 10:24:53.500072: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.14GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-07-20 10:24:53.605247: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.29GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-07-20 10:24:53.696293: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.10GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-07-20 10:24:53.795242: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.19GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-07-20 10:24:53.911425: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.10GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-07-20 10:24:54.011991: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.20GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-07-20 10:24:54.012182: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.14GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-07-20 10:24:54.095455: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.14GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-07-20 10:24:54.574779: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.14GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
[I 10:26:18.720 NotebookApp] Saving file at /Masterarbeit/KerasCNN/Different Finetuning.ipynb
2017-07-20 10:28:09.415272: E c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\stream_executor\cuda\cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_TIMEOUT
2017-07-20 10:28:09.415378: F c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\gpu\gpu_event_mgr.cc:203] Unexpected Event status: 1
[I 10:28:31.838 NotebookApp] KernelRestarter: restarting kernel (1/5)
WARNING:root:kernel 2ccf2624-b385-4814-aa25-44a53a86dd02 restarted

Thx guys!

fadam commented Aug 1, 2017

@GitHubKay

NameError: name 'Input' is not defined

Seems like you forgot to import Input, add this
from keras.models import Input

@fadam
Yeah! I proceeded today with my files and now everything is running fine. After all theses issues i couldn't find the small ones anymore! Anyway, i split up the code in small batches within jupyter and could manage to run the code successfully!
Thx to everybody who shared his code here!!

any suggestions where i could find more tuning tips for little data! I've my own landscape pictures and would like to classify them after my own preferences in bad and good pictures. But i only have 800 good ones.

any help is appreciated!

@sakvaua
I'm trying to work that out as well, looks like demeaning doesn't help but re-scaling does in testing for me so far, but not sure why that would be the case if it was the standard weights.

S1M0N38 commented Aug 10, 2017

I'm stuck on this problem

top_model = Sequential()
top_model.add(Flatten(input_shape=model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))


ValueError Traceback (most recent call last)
in ()
1 top_model = Sequential()
----> 2 top_model.add(Flatten(input_shape=model.output_shape[1:]))
3 top_model.add(Dense(256, activation='relu'))
4 top_model.add(Dropout(0.5))
5 top_model.add(Dense(1, activation='sigmoid'))

/Users/Simo/.conda/envs/main/lib/python3.5/site-packages/keras/models.py in add(self, layer)
434 # and create the node connecting the current layer
435 # to the input layer we just created.
--> 436 layer(x)
437
438 if len(layer.inbound_nodes) != 1:

/Users/Simo/.conda/envs/main/lib/python3.5/site-packages/keras/engine/topology.py in call(self, inputs, **kwargs)
613 # Infering the output shape is only relevant for Theano.
614 if all([s is not None for s in _to_list(input_shape)]):
--> 615 output_shape = self.compute_output_shape(input_shape)
616 else:
617 if isinstance(input_shape, list):

/Users/Simo/.conda/envs/main/lib/python3.5/site-packages/keras/layers/core.py in compute_output_shape(self, input_shape)
475 raise ValueError('The shape of the input to "Flatten" '
476 'is not fully defined '
--> 477 '(got ' + str(input_shape[1:]) + '. '
478 'Make sure to pass a complete "input_shape" '
479 'or "batch_input_shape" argument to the first '

ValueError: The shape of the input to "Flatten" is not fully defined (got (None, None, 512). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.

edumucelli commented Aug 12, 2017

@S1M0N38 your problem is that you should set the input_shape parameter when using VGG16 with include_top=false. Take a look into the code here where it is explained.

@skoch9 be aware that as you are not including the top layers, but replacing by a way simpler layer than the original VGG16. Just for comparison's sake take the number of parameters from summary of your model and the original VGG16.

Your model's top is like:

block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         
_________________________________________________________________
sequential_2 (Sequential)    (None, 1)                 6423041   
=================================================================
Total params: 21,137,729
Trainable params: 21,137,729
Non-trainable params: 0

While original VGG16 is

block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0         
_________________________________________________________________
fc1 (Dense)                  (None, 4096)              102764544 
_________________________________________________________________
fc2 (Dense)                  (None, 4096)              16781312  
_________________________________________________________________
predictions (Dense)          (None, 1000)              4097000   
=================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0

You are not fine-tuning, but simplyfing the model.

Can I use callbacks (e.g. early stopping) with model.fit_generator as I have used with model.fit?

who can point out how to implment like this:
data/
train/
dogs1/
dog001.jpg
dog002.jpg
...
dogs2/
dog001.jpg
dog002.jpg
...
validation/
dogs/
dog001.jpg
dog002.jpg
...
dogs2/
dog001.jpg
dog002.jpg
...
and set the name of dog1 is dd,set the name of dog2 ff
then when i put an image it will show the result like this > this is a dog named dd&ff

Hi guys:
I have a question, when i train the model Under Windows using tensorflow as backend, I hava 89% accuracy for 2 classes. But when i train the model Under Linux, the model decreased in accuracy from 89% to 15%, And then i use the weights which is trained under the Windows to fine-tuning the model under Linux, I got the 91% accuracy , It seems that the 'save_weights' is different between Windows and Linux, but i hava no idea.
Any advice on this?

nimpy commented Aug 21, 2017

@edumucelli are you sure @skoch9 isn't including the top layers?
I thought the same thing, so I used this code instead to create a model:

base_model = applications.VGG16(weights='imagenet', include_top=False, input_shape=input_shape)

x = Flatten()(base_model.output)
x = Dense(256, activation='relu')(x)
x = Dropout(0.5)(x)
predictions = Dense(1, activation = 'sigmoid')(x)

model = Model(input = base_model.input, output = predictions)

and when you look at the model.summary(), the last layers:

flatten_8 (Flatten)          (None, 25088)             0         
_________________________________________________________________
dense_15 (Dense)             (None, 256)               6422784   
_________________________________________________________________
dropout_8 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_16 (Dense)             (None, 1)                 257            

have the same number of parameters (in total) as in @skoch9 's model's last layer:

sequential_1 (Sequential)    (None, 1)                 6423041   

I believe it's the same thing, but please correct me if I'm wrong.

all3xfx commented Sep 5, 2017

Thanks @saulthu

Thanks for your providing helpful source codes like this. But I need your help now. I'm trying binary classification through cats and dogs dataset. The code is running well without any code errors. But acc and loss does not improve at all. I need you guys' powerful brain.

Keras 2.0.7 (backend tensorflow)
tensorflow-gpu 1.2.1

dataset
data/
train/
dogs/
dog001.jpg
dog002.jpg
...
cats/
cat001.jpg
cat002.jpg
...
validation/
dogs/
dog001.jpg
dog002.jpg
...
cats/
cat001.jpg
cat002.jpg
...

The code
`from keras import applications
from keras.preprocessing.image import ImageDataGenerator
from keras import optimizers
from keras.models import Sequential, Model
from keras.layers import Dropout, Flatten, Dense

img_width, img_height, img_channel = 150, 150, 3

train_data_dir = './data/train'
validation_data_dir = './data/validation'
nb_train_samples = 2000
nb_validation_samples = 800
epochs = 50
batch_size = 16

build the VGG16 network

base_model = applications.VGG16(weights='imagenet',
input_shape=(img_width, img_height, img_channel),
include_top=False)
print('Model loaded.')

build a classifier model to put on top of the convolutional model

vgg16_output = base_model.output
flatten = Flatten(name='flatten')(vgg16_output)
dense = Dense(256, activation='relu', kernel_initializer='he_normal', name='fc1')(flatten)
dense = Dense(256, activation='relu', kernel_initializer='he_normal', name='fc2')(dense)
pred = Dense(units=1, activation='softmax', kernel_initializer='he_normal', name='prediction')(dense)

new_model = Model(input=base_model.input, output=pred)
new_model.summary()

set the first 25 layers (up to the last conv block)

to non-trainable (weights will not be updated)

for layer in new_model.layers:
if layer.name in ['fc1', 'fc2', 'prediction']:
continue
layer.trainable = False

compile the model with a SGD/momentum optimizer

and a very slow learning rate.

new_model.compile(loss='binary_crossentropy',
optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
metrics=['accuracy'])
new_model.summary()

prepare data augmentation configuration

train_datagen = ImageDataGenerator(
rescale=1. / 255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='binary')

fine-tune the model

new_model.fit_generator(
train_generator,
steps_per_epoch=nb_train_samples//batch_size,
epochs=epochs,
validation_data=validation_generator,
nb_val_samples=nb_validation_samples//batch_size)`

Hi, I cannot donwload the data, can someone help me please? thanks.

Hi just wan to share that the number of layers to freeze is really 25 instead of 15.
It is not as simple as count the layers listed in model.summary(), please look at VGG16 @ https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3, and count the layers (pay attention to those ZeroPadding2D layers).

Thanks and have a nice day! Ciao.

@taozhiiq I see same results - 89% accuracy using saved weights when using keras 2.0 with TF backend on Windows.
Something is wrong.

JennyLuciaVg commented Oct 23, 2017

Hi everyone. After run this part of the tutorial, i get this error:
Traceback (most recent call last): File "dogandcats6.py", line 34, in <module> top_model.load_weights(top_model_weights_path) File "C:\Users\USUARIO\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py", line 719, in load_weights topology.load_weights_from_hdf5_group(f, layers) File "C:\Users\USUARIO\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\topology.py", line 3095, in load_weights_from_hdf5_group K.batch_set_value(weight_value_tuples) File "C:\Users\USUARIO\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 2188, in batch_set_value assign_op = x.assign(assign_placeholder) File "C:\Users\USUARIO\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\variables.py", line 522, in assign return state_ops.assign(self._variable, value, use_locking=use_locking) File "C:\Users\USUARIO\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 47, in assign use_locking=use_locking, name=name) File "C:\Users\USUARIO\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 763, in apply_op op_def=op_def) File "C:\Users\USUARIO\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2329, in create_op set_shapes_for_outputs(ret) File "C:\Users\USUARIO\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1717, in set_shapes_for_outputs shapes = shape_func(op) File "C:\Users\USUARIO\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1667, in call_with_requiring return call_cpp_shape_fn(op, require_shape_fn=True) File "C:\Users\USUARIO\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 610, in call_cpp_shape_fn debug_python_shape_fn, require_shape_fn) File "C:\Users\USUARIO\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 676, in _call_cpp_shape_fn_impl raise ValueError(err.message) ValueError: Dimension 1 in both shapes must be equal, but are 1 and 2 for 'Assign_28' (op: 'Assign') with input shapes: [256,1], [256,2].

This is the code:
`from keras import applications
from keras.preprocessing.image import ImageDataGenerator
from keras import optimizers
from keras.models import Sequential
from keras.layers import Dropout, Flatten, Dense

weights_path = '../keras/examples/vgg16_weights.h5'
top_model_weights_path = 'bottleneck_fc_model.h5'

img_width, img_height = 350,350

train_data_dir = 'data/train'
validation_data_dir = 'data/validation'
nb_train_samples = 100
nb_validation_samples = 20
epochs = 50
batch_size = 16

model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(350, 350, 3))
print('Model loaded.')

top_model = Sequential()
top_model.add(Flatten(input_shape=model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))

top_model.load_weights(top_model_weights_path)

model.add(top_model)

for layer in model.layers[:25]:
layer.trainable = False

model.compile(loss='binary_crossentropy',
optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
metrics=['accuracy'])

train_datagen = ImageDataGenerator(
rescale=1. / 255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='binary')

model.fit_generator(
train_generator,
samples_per_epoch=nb_train_samples,
epochs=epochs,
validation_data=validation_generator,
nb_val_samples=nb_validation_samples)`

Please, help me!

@GitHubKay How did you fix the memory problem? I think I'm getting the same error. Jupyter says "Dead kernel" when I'm running the last part: model.fit_generator(...)

mandliya commented Nov 7, 2017

Getting the exact same error as @MathiasKahlen . Any help please.

@saulthu Excellent! Works like a charm.

@fchollet thanks for the example! I've trained and saved a model with 1 epoch. I intend to load the model back up to train for the 2nd epoch. Any tips on how I would approach it with finetuning? Cheers :)

DaniRuiz92 commented Nov 17, 2017

@pebbleshx thanks for your comment but I'm not sure that what you say is correct. If you look at this code, the vgg16 is composed of less than 25 layers. I suppose that it is possible to use different implementations of the VGG16.
In fact, when you load the VGG16 model from the keras.applications repository model = applications.VGG16(weights='imagenet', include_top=False, input_shape = (img_width, img_height, 3)) and you look the structure of the layers model.layers, you will find only 20 layers. When you put the flag trainable = False, you do it in the layers of model.layers, so if you choose to freeze 25 layers you will freeze all layers. You can check this by doing a model.summary and checking the number of trainable weigths.

Thats only my interpretation of how the freezing should be done.

Probably @fchollet can clarify this.

Moondra commented Nov 19, 2017

I'm running this script for the first time, and It's downloading the VGG weights or model structure(I'm assuming). It's taking forever. Is this normal behavior?

Also these lines:

weights_path = '../keras/examples/vgg16_weights.h5'
top_model_weights_path = 'fc_model.h5'

Are these the same as these weights:
vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5

If so I have these weights in a different directory, should I change the weights_path ?

I did follow the tutorial and I was successfully able to create updated weights. I save the model and weights using below command:
model_json = model.to_json()
with open('model_def_gen1.1.json','w') as json_file:
json_file.write(model_json)
model.save_weights(top_model_weights_path)

However, when I predict I get below error : You are trying to load a weight file containing 14 layers into a model with 2 layers.
Below is my code for prediction:
#reading the json file created above
with open('model_def_gen1.json') as json_file:
model_json = json_file.read()
mymodel = keras.models.model_from_json(model_json)
#loading the weights created at the end
mymodel.load_weights('bottleneck_fc_model1.h5')
img_array1 =[]
#path to my test images directory
img_path_test="//test-images//*.jpg"
for img1 in glob.glob(img_path_test):
img_array1.append(img1)
img_w=150
img_h=150
preds_gender_img =[]
#creating my base model for extracting features
base_model = applications.VGG16(weights='imagenet', include_top=False )
#accessing each image in my test folder
for i2 in img_array1:
l=load_img(i2,target_size=(img_w,img_h))
x_test_img=image.img_to_array(l)
x_test = np.expand_dims(x_test, axis=0)
x_test=x_test/255
#extracting features using vgg16
features = base_model.predict(x_test)
#using my weights to predict the class, I am doing binary classification
preds_gender_img.append(mymodel.predict_classes(features))

Can anyone please tell me what I am doing wrong? I followed another GitHub link to predict classes.#6408 fchollet/keras#6408
@KamalOthman can you please tell me what is wrong here?

@AakashKumarNain can you tell me how did you load the model and the weights

Ksen17 commented Nov 21, 2017

After performing this tutorial accuracy falls down to 0.0195 from 0.7089
Can someone help me with this problem
Also, I want to make input shape (150, 150, 3) to convert this model to CoreML

Any help would be great

savinay commented Nov 26, 2017

Can anybody please tell me how to predict the class for a new image. I have a multiclass problem and I am running the following code:

`model = load_model('my_model.h5')

model.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])

img=array(Image.open('yalefaces/subject01.glasses'))

img = cv2.resize(img,(150,150), interpolation = cv2.INTER_AREA)

img = np.reshape(img,[1,150,150,3])

classes = model.predict_classes(img)

print classes`

But I get the following error:
ValueError: cannot reshape array of size 22500 into shape (1,150,150,3)

Any help would be appreciated.

rmis7170 commented Nov 29, 2017

chiragyeole commented Dec 2, 2017

@Golly
@Arsey
@austinchencym
@JinnyZhao
@mahernadar

For Bottleneck features accuracy is ~90%. But after fine tuning its 50%
Are you guys able to figure out what was the issue?

Is it just me or is the weights_path never used? Does this cause the confusion of freezing 15 or 25 layers? When i print all layers in the VGG16 model there isn't actually 25? The keras.applications implementation seem to have less layers than the implementation in this code: https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3. I'm truly confused.

@nimpy If you're doing that don't you then skip the bottleneck and training of top_model parts and go directly to the finetuning part? And if so, what is the difference between the two methods?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment