Create a gist now

Instantly share code, notes, and snippets.

@GilLevi /README.md
Last active Apr 27, 2017

What would you like to do?
Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns

Gil Levi and Tal Hassner, Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns

Convolutional neural networks for emotion classification from facial images as described in the following work:

Gil Levi and Tal Hassner, Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns, Proc. ACM International Conference on Multimodal Interaction (ICMI), Seattle, Nov. 2015

Project page: http://www.openu.ac.il/home/hassner/projects/cnn_emotions/

If you find our models useful, please add suitable reference to our paper in your work.

gist_id: 54aee1b8b0397721aa4b

###Emotion Classification CNN - RGB

caffemodel: VGG_S_rgb/EmotiW_VGG_S.caffemodel

caffemodel_url: https://dl.dropboxusercontent.com/u/38822310/demodir/VGG_S_rgb/EmotiW_VGG_S.caffemodel

mean_file_proto: https://dl.dropboxusercontent.com/u/38822310/demodir/VGG_S_rgb/mean.binaryproto

###Emotion Classification CNN - LBP

caffemodel: VGG_S_lbp/EmotiW_VGG_S.caffemodel

caffemodel_url: https://dl.dropboxusercontent.com/u/38822310/demodir/VGG_S_lbp/EmotiW_VGG_S.caffemodel

mean_file_proto: https://dl.dropboxusercontent.com/u/38822310/demodir/VGG_S_lbp/mean.binaryproto

###Emotion Classification CNN - Cyclic LBP

caffemodel: VGG_S_cyclic_lbp/EmotiW_VGG_S.caffemodel

caffemodel_url: https://dl.dropboxusercontent.com/u/38822310/demodir/VGG_S_cyclic_lbp/EmotiW_VGG_S.caffemodel

mean_file_proto: https://dl.dropboxusercontent.com/u/38822310/demodir/VGG_S_cyclic_lbp/mean.binaryproto

###Emotion Classification CNN - Cyclic LBP-5

caffemodel: VGG_S_cyclic_lbp_5/EmotiW_VGG_S.caffemodel

caffemodel_url: https://dl.dropboxusercontent.com/u/38822310/demodir/VGG_S_cyclic_lbp_5/EmotiW_VGG_S.caffemodel

mean_file_proto: https://dl.dropboxusercontent.com/u/38822310/demodir/VGG_S_cyclic_lbp_5/mean.binaryproto

###Emotion Classification CNN - Cyclic LBP-10

caffemodel: VGG_S_cyclic_lbp_10/EmotiW_VGG_S.caffemodel

caffemodel_url: https://dl.dropboxusercontent.com/u/38822310/demodir/VGG_S_cyclic_lbp_10/EmotiW_VGG_S.caffemodel

mean_file_proto: https://dl.dropboxusercontent.com/u/38822310/demodir/VGG_S_cyclic_lbp_10/mean.binaryproto


Copyright 2015, Gil Levi and Tal Hassner

The SOFTWARE provided in this page is provided "as is", without any guarantee made as to its suitability or fitness for any particular use. It may contain bugs, so use of this tool is at your own risk. We take no responsibility for any damage of any sort that may unintentionally be caused through its use.

name: "CaffeNet"
input: "data"
input_dim: 1
input_dim: 3
input_dim: 224
input_dim: 224
layers {
name: "conv1"
type: CONVOLUTION
bottom: "data"
top: "conv1"
convolution_param {
num_output: 96
kernel_size: 7
stride: 2
}
}
layers {
name: "relu1"
type: RELU
bottom: "conv1"
top: "conv1"
}
layers {
name: "norm1"
type: LRN
bottom: "conv1"
top: "norm1"
lrn_param {
local_size: 5
alpha: 0.0005
beta: 0.75
}
}
layers {
name: "pool1"
type: POOLING
bottom: "norm1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 3
stride: 3
}
}
layers {
name: "conv2"
type: CONVOLUTION
bottom: "pool1"
top: "conv2"
convolution_param {
num_output: 256
pad: 2
kernel_size: 5
}
}
layers {
name: "relu2"
type: RELU
bottom: "conv2"
top: "conv2"
}
layers {
name: "pool2"
type: POOLING
bottom: "conv2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layers {
name: "conv3"
type: CONVOLUTION
bottom: "pool2"
top: "conv3"
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
}
}
layers {
name: "relu3"
type: RELU
bottom: "conv3"
top: "conv3"
}
layers {
name: "conv4"
type: CONVOLUTION
bottom: "conv3"
top: "conv4"
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
}
}
layers {
name: "relu4"
type: RELU
bottom: "conv4"
top: "conv4"
}
layers {
name: "conv5"
type: CONVOLUTION
bottom: "conv4"
top: "conv5"
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
}
}
layers {
name: "relu5"
type: RELU
bottom: "conv5"
top: "conv5"
}
layers {
name: "pool5"
type: POOLING
bottom: "conv5"
top: "pool5"
pooling_param {
pool: MAX
kernel_size: 3
stride: 3
}
}
layers {
name: "fc6"
type: INNER_PRODUCT
bottom: "pool5"
top: "fc6"
inner_product_param {
num_output: 4048
}
}
layers {
name: "relu6"
type: RELU
bottom: "fc6"
top: "fc6"
}
layers {
name: "drop6"
type: DROPOUT
bottom: "fc6"
top: "fc6"
dropout_param {
dropout_ratio: 0.5
}
}
layers {
name: "fc7"
type: INNER_PRODUCT
bottom: "fc6"
top: "fc7"
inner_product_param {
num_output: 4048
}
}
layers {
name: "relu7"
type: RELU
bottom: "fc7"
top: "fc7"
}
layers {
name: "drop7"
type: DROPOUT
bottom: "fc7"
top: "fc7"
dropout_param {
dropout_ratio: 0.5
}
}
layers {
name: "fc8_cat"
type: INNER_PRODUCT
bottom: "fc7"
top: "fc8"
inner_product_param {
num_output: 7
}
}
layers {
name: "prob"
type: SOFTMAX
bottom: "fc8"
top: "prob"
}
name: "CaffeNet"
layers {
name: "data"
type: DATA
top: "data"
top: "label"
data_param {
source: "/home/ubuntu/EmotiW/lmdb/train_lmdb"
backend: LMDB
batch_size: 30
}
transform_param {
crop_size: 224
mean_file: "/home/ubuntu/EmotiW/mean_image/mean.binaryproto"
mirror: true
}
include: { phase: TRAIN }
}
layers {
name: "data"
type: DATA
top: "data"
top: "label"
data_param {
source: "/home/ubuntu/EmotiW/lmdb/val_lmdb"
backend: LMDB
batch_size: 20
}
transform_param {
crop_size: 224
mean_file: "/home/ubuntu/EmotiW/mean_image/mean.binaryproto"
mirror: false
}
include: { phase: TEST }
}
layers {
name: "conv1"
type: CONVOLUTION
bottom: "data"
top: "conv1"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 96
kernel_size: 7
stride: 2
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
}
}
layers {
name: "relu1"
type: RELU
bottom: "conv1"
top: "conv1"
}
layers {
name: "norm1"
type: LRN
bottom: "conv1"
top: "norm1"
lrn_param {
local_size: 5
alpha: 0.0005
beta: 0.75
}
}
layers {
name: "pool1"
type: POOLING
bottom: "norm1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 3
stride: 3
}
}
layers {
name: "conv2"
type: CONVOLUTION
bottom: "pool1"
top: "conv2"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 256
pad: 2
kernel_size: 5
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu2"
type: RELU
bottom: "conv2"
top: "conv2"
}
layers {
name: "pool2"
type: POOLING
bottom: "conv2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layers {
name: "conv3"
type: CONVOLUTION
bottom: "pool2"
top: "conv3"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
}
}
layers {
name: "relu3"
type: RELU
bottom: "conv3"
top: "conv3"
}
layers {
name: "conv4"
type: CONVOLUTION
bottom: "conv3"
top: "conv4"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu4"
type: RELU
bottom: "conv4"
top: "conv4"
}
layers {
name: "conv5"
type: CONVOLUTION
bottom: "conv4"
top: "conv5"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu5"
type: RELU
bottom: "conv5"
top: "conv5"
}
layers {
name: "pool5"
type: POOLING
bottom: "conv5"
top: "pool5"
pooling_param {
pool: MAX
kernel_size: 3
stride: 3
}
}
layers {
name: "fc6"
type: INNER_PRODUCT
bottom: "pool5"
top: "fc6"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
inner_product_param {
num_output: 4048
weight_filler {
type: "gaussian"
std: 0.005
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu6"
type: RELU
bottom: "fc6"
top: "fc6"
}
layers {
name: "drop6"
type: DROPOUT
bottom: "fc6"
top: "fc6"
dropout_param {
dropout_ratio: 0.7
}
}
layers {
name: "fc7"
type: INNER_PRODUCT
bottom: "fc6"
top: "fc7"
blobs_lr: 9
blobs_lr: 18
weight_decay: 1
weight_decay: 0
inner_product_param {
num_output: 4048
weight_filler {
type: "gaussian"
std: 0.005
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu7"
type: RELU
bottom: "fc7"
top: "fc7"
}
layers {
name: "drop7"
type: DROPOUT
bottom: "fc7"
top: "fc7"
dropout_param {
dropout_ratio: 0.7
}
}
layers {
name: "fc8_cat"
type: INNER_PRODUCT
bottom: "fc7"
top: "fc8"
blobs_lr: 12
blobs_lr: 24
weight_decay: 1
weight_decay: 0
inner_product_param {
num_output: 7
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
}
}
layers {
name: "accuracy"
type: ACCURACY
bottom: "fc8"
bottom: "label"
top: "accuracy"
include: { phase: TEST }
}
layers {
name: "loss"
type: SOFTMAX_LOSS
bottom: "fc8"
bottom: "label"
top: "loss"
}
Gzzgz commented Jun 16, 2016 edited

---OK I has finished it.

Can you help me ? TKS
image

Gzzgz commented Jun 25, 2016

Other problem. I finded the deploy file is not fit to the CNN-LBP CNN-Cyclic-LBP CNN-Cyclic-LBP-5 and CNN-Cyclic-LBP-10 . Can you help me ? THX

Gil Levi, your jupyter notebook gives categories = [ 'Angry' , 'Disgust' , 'Fear' , 'Happy' , 'Neutral' , 'Sad' , 'Surprise']. Are these the same for each model you have published here? I am using VGG_S_rgb/EmotiW_VGG_S.caffemodel above. I wanted to confirm what labels are being used here. Would appreciate if you put the labels along with it.

Owner
GilLevi commented Jul 29, 2016

Hi Gzzg, sorry for the late response (I didn't get an email about your comment). The deploy file attached should work with all models. What seems to be the problem?

Owner
GilLevi commented Jul 29, 2016

Hi singarajus , indeed the labels are the same for all models.

Hi Gil

I downloaded the demo dir and when I go through the demo script even i see the error reported above:

I0830 17:48:38.695304 32182 net.cpp:761] Ignoring source layer loss
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nutanix/workspace/caffe/python/caffe/classifier.py", line 34, in __init__
    self.transformer.set_mean(in_, mean)
  File "/home/nutanix/workspace/caffe/python/caffe/io.py", line 259, in set_mean
    raise ValueError('Mean shape incompatible with input shape.')
ValueError: Mean shape incompatible with input shape.

any ideas how can I fix this?

Thanks!
Aditya

NoorShaker commented Sep 13, 2016 edited

Hi Aditya,

Try

mean = caffe.io.blobproto_to_array(a)[0]
mean=mean.mean(1).mean(1)

This worked for me.

Good luck.

Owner
GilLevi commented Oct 24, 2016

Hi Aditya,

The issue is explained and fixed here (update from July 15th, 2015):
http://www.openu.ac.il/home/hassner/projects/cnn_agegender/

ilyamaslo commented Oct 31, 2016 edited

Hi
I tried this attached deploy.prototxt for all LBP models and had same problem as Gzzgz (kernel died). It seems logical since RGB model and LBP models had different size.

from jupyter log:
F1031 17:28:42.965265 7498 net.cpp:765] Cannot copy param 0 weights from layer 'fc6'; shape mismatch. Source param shape is 4096 18432 (75497472); target param shape is 4048 25088 (101556224). To learn this layer's parameters from scratch rather than copying from a saved net, rename the layer.
*** Check failure stack trace: ***

Hi, GilLevi
Does the align step same as http://www.openu.ac.il/home/hassner/projects/cnn_agegender/?
Thanks.

Owner
GilLevi commented Jan 3, 2017

Sorry for the late response, I didn't get notifications for some reason.

@ilyamaslo , try replacing the size of fc6 from 4048 to 4096.

@GuitarZhang , no. In this project the aligned faces were provided by the challenge authors.

Hey @GilLevi,

I downloaded your RGB caffemodel, along with the deploy.prototxt that you've attached, and it's working really well!

I had a few requests, if you don't mind:

  1. Could you please provide the labels.txt? I don't know the exact order of the labels, so it'll be great if you could provide that.
  2. Could you also provide the solver.prototxt file? I'd like to fine-tune the caffemodel to my own dataset, if you don't mind.

Thanks in advance! :)

Hi @GilLevi,
I downloaded RGB caffemodel, and execute it as you showed at python notebook example. I works really good at happy, fear and disgusted images but it can't predict sad images. I used KDEF database as a testing database. It can detect all emotions except sad and suprise.
Can you give an example with any sad or suprise person picture like the angry one on the python notebook example?
Thanks.

Owner
GilLevi commented Feb 11, 2017 edited

Hi @rahulkulhalli,

Thank you for your interest in our work.

The solver is:
net: "/home/ubuntu/EmotiW/VGG_S_David/train_val.prototxt"
test_iter: 1000
test_interval: 1000
base_lr: 0.001
lr_policy: "step"
gamma: 0.1
#stepsize: 20000
stepsize: 2000000
display: 100
max_iter: 400000
momentum: 0.9
weight_decay: 0.0005
snapshot: 1000
snapshot_prefix: "EmotiW_VGG_S"
solver_mode: GPU
device_id:0

There is no labels file, the labels are just :[ 'Angry' , 'Disgust' , 'Fear' , 'Happy' , 'Neutral' , 'Sad' , 'Surprise'] in that order.

Best,
Gil

Owner
GilLevi commented Feb 11, 2017

Hi @user65432,

Thank you for your interest in our work.

I've attached an example of an angry face and a sad face which are classified correctly by our model. Perhaps you can get better results by finetuning our network on the KDEF dataset.

alexemma_003716360_00000040

airheads_000519240_00000005

@user65432,can you share me KDEF dataset? I download it always 0kb.

@zhanglaplace, sure. I tried to share it from there but i can't because it's bigger than 10MB.

@user65432,my email is lapcace@gmail.com,can you email me? thx

shinchanyox commented Feb 15, 2017 edited

Hi GilLevi ,
I am using the Emotion Classification CNN - RGB model configured on EC2 AWS server(preconfigured with caffe and CUDA).The model is giving the same emotion for every input image.
categories[prediction.argmax()] always gives categories[0] and so I always get the 0th index emotion for every input.

https://cloud.githubusercontent.com/assets/6024900/22856133/3a70dcec-f094-11e6-9bf2-7dd793ffa605.png
This is the image we used as our input.The dimensions of the image we used was 256*256.

Owner
GilLevi commented Feb 17, 2017

Hi @shinchanyox, can you pleas attache the images that you used and gave the same emotion each time?

Best,
Gil

sir , i was using incorrect images , now it is working fine . Can you please suggest a method to connect to your model on ec2 via android application ?

Owner
GilLevi commented Feb 20, 2017

Hi @shinchanyox, I have no experience in android programming so I really don't know.

Best,
Gil

Hi GilLevi,
I'm still trying to fine tune your CNN :). Could you provide "train_val.prototxt" , if you don't mind ?
Thanks.

Owner
GilLevi commented Mar 19, 2017

Hi @user65432,

Thanks you for your interest in our project, I uploaded the "train_val.prototxt".

Best,
Gil

Hi GilLevi,
Thanks for sharing the models and prototxt files. To compare the results using different pretrained models, I tried VGG_S_rgb and VGG_S_cyclic_lbp_10. While the first one finished finetuning with the new data, the latter model shows the following error:

F0425 11:27:24.064520 62685 net.cpp:774] Cannot copy param 0 weights from layer 'fc6'; shape mismatch. Source param shape is 4096 18432 (75497472); target param shape is 4048 25088 (101556224). To learn this layer's parameters from scratch rather than copying from a saved net, rename the layer.
I looked up online and found that that the difference in feature map size could be the reason (link)
Would you please let me know what are the differences between the above two models?
Thank you again.

dayinji commented Apr 27, 2017

Hi GilLevi,
The deploy.txt above is not suit for [ Emotion Classification CNN - RGB ], even I change the [ num_output ] of [ fc6 ] from 4048 to 4096. I hope you can upload a correct deploy.txt that suit for [ Emotion Classification CNN - RGB ]. Anyway, Thank you for sharing this nice work! Hope for your response. : )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment