Skip to content

Instantly share code, notes, and snippets.

@GilLevi
Last active June 17, 2023 20:58
Show Gist options
  • Save GilLevi/54aee1b8b0397721aa4b to your computer and use it in GitHub Desktop.
Save GilLevi/54aee1b8b0397721aa4b to your computer and use it in GitHub Desktop.
Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns

Gil Levi and Tal Hassner, Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns

Convolutional neural networks for emotion classification from facial images as described in the following work:

Gil Levi and Tal Hassner, Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns, Proc. ACM International Conference on Multimodal Interaction (ICMI), Seattle, Nov. 2015

Project page: http://www.openu.ac.il/home/hassner/projects/cnn_emotions/

If you find our models useful, please add suitable reference to our paper in your work.

gist_id: 54aee1b8b0397721aa4b

Emotion Classification CNN - RGB

caffemodel: VGG_S_rgb/EmotiW_VGG_S.caffemodel

caffemodel_url: https://drive.google.com/open?id=0BydFau0VP3XSNVYtWnNPMU1TOGM

mean_file_proto: https://drive.google.com/open?id=0BydFau0VP3XSTEZpN3ItN1c0LUE

deploy_file: https://drive.google.com/open?id=0BydFau0VP3XSOFp4Ri1ITzZuUkk

Emotion Classification CNN - LBP

caffemodel: VGG_S_lbp/EmotiW_VGG_S.caffemodel

caffemodel_url: https://drive.google.com/open?id=0BydFau0VP3XScTFlWkszazA0b0E

mean_file_proto: https://drive.google.com/open?id=0BydFau0VP3XSNXBRVnlZS0FsdzQ

deploy_file: https://drive.google.com/open?id=0BydFau0VP3XSckxVTmN5YUlfUGM

Emotion Classification CNN - Cyclic LBP

caffemodel: VGG_S_cyclic_lbp/EmotiW_VGG_S.caffemodel

caffemodel_url: https://drive.google.com/open?id=0BydFau0VP3XSNUFVSDZydS0yc2c

mean_file_proto: https://drive.google.com/open?id=0BydFau0VP3XSWXJ0UnFHaHROQUU

deploy_file: https://drive.google.com/open?id=0BydFau0VP3XSbHdoU1N0LWtFRG8

Emotion Classification CNN - Cyclic LBP-5

caffemodel: VGG_S_cyclic_lbp_5/EmotiW_VGG_S.caffemodel

caffemodel_url: https://drive.google.com/open?id=0BydFau0VP3XSYjVxbGJZYVdveWM

mean_file_proto: https://drive.google.com/open?id=0BydFau0VP3XSa0pXXzJ1ZGNPYTg

deploy_file: https://drive.google.com/open?id=0BydFau0VP3XSdndLMnpDMWItLTA

Emotion Classification CNN - Cyclic LBP-10

caffemodel: VGG_S_cyclic_lbp_10/EmotiW_VGG_S.caffemodel

caffemodel_url: https://drive.google.com/open?id=0BydFau0VP3XSOWlZekZXWGdySEU

mean_file_proto: https://drive.google.com/open?id=0BydFau0VP3XSenRYa2Q1S3E0Q2s

deploy_file: https://drive.google.com/open?id=0BydFau0VP3XSMGkxMTgtU0U1dVU


Copyright 2015, Gil Levi and Tal Hassner

The SOFTWARE provided in this page is provided "as is", without any guarantee made as to its suitability or fitness for any particular use. It may contain bugs, so use of this tool is at your own risk. We take no responsibility for any damage of any sort that may unintentionally be caused through its use.

name: "CaffeNet"
layers {
name: "data"
type: DATA
top: "data"
top: "label"
data_param {
source: "/home/ubuntu/EmotiW/lmdb/train_lmdb"
backend: LMDB
batch_size: 30
}
transform_param {
crop_size: 224
mean_file: "/home/ubuntu/EmotiW/mean_image/mean.binaryproto"
mirror: true
}
include: { phase: TRAIN }
}
layers {
name: "data"
type: DATA
top: "data"
top: "label"
data_param {
source: "/home/ubuntu/EmotiW/lmdb/val_lmdb"
backend: LMDB
batch_size: 20
}
transform_param {
crop_size: 224
mean_file: "/home/ubuntu/EmotiW/mean_image/mean.binaryproto"
mirror: false
}
include: { phase: TEST }
}
layers {
name: "conv1"
type: CONVOLUTION
bottom: "data"
top: "conv1"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 96
kernel_size: 7
stride: 2
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
}
}
layers {
name: "relu1"
type: RELU
bottom: "conv1"
top: "conv1"
}
layers {
name: "norm1"
type: LRN
bottom: "conv1"
top: "norm1"
lrn_param {
local_size: 5
alpha: 0.0005
beta: 0.75
}
}
layers {
name: "pool1"
type: POOLING
bottom: "norm1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 3
stride: 3
}
}
layers {
name: "conv2"
type: CONVOLUTION
bottom: "pool1"
top: "conv2"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 256
pad: 2
kernel_size: 5
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu2"
type: RELU
bottom: "conv2"
top: "conv2"
}
layers {
name: "pool2"
type: POOLING
bottom: "conv2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layers {
name: "conv3"
type: CONVOLUTION
bottom: "pool2"
top: "conv3"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
}
}
layers {
name: "relu3"
type: RELU
bottom: "conv3"
top: "conv3"
}
layers {
name: "conv4"
type: CONVOLUTION
bottom: "conv3"
top: "conv4"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu4"
type: RELU
bottom: "conv4"
top: "conv4"
}
layers {
name: "conv5"
type: CONVOLUTION
bottom: "conv4"
top: "conv5"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu5"
type: RELU
bottom: "conv5"
top: "conv5"
}
layers {
name: "pool5"
type: POOLING
bottom: "conv5"
top: "pool5"
pooling_param {
pool: MAX
kernel_size: 3
stride: 3
}
}
layers {
name: "fc6"
type: INNER_PRODUCT
bottom: "pool5"
top: "fc6"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
inner_product_param {
num_output: 4048
weight_filler {
type: "gaussian"
std: 0.005
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu6"
type: RELU
bottom: "fc6"
top: "fc6"
}
layers {
name: "drop6"
type: DROPOUT
bottom: "fc6"
top: "fc6"
dropout_param {
dropout_ratio: 0.7
}
}
layers {
name: "fc7"
type: INNER_PRODUCT
bottom: "fc6"
top: "fc7"
blobs_lr: 9
blobs_lr: 18
weight_decay: 1
weight_decay: 0
inner_product_param {
num_output: 4048
weight_filler {
type: "gaussian"
std: 0.005
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu7"
type: RELU
bottom: "fc7"
top: "fc7"
}
layers {
name: "drop7"
type: DROPOUT
bottom: "fc7"
top: "fc7"
dropout_param {
dropout_ratio: 0.7
}
}
layers {
name: "fc8_cat"
type: INNER_PRODUCT
bottom: "fc7"
top: "fc8"
blobs_lr: 12
blobs_lr: 24
weight_decay: 1
weight_decay: 0
inner_product_param {
num_output: 7
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
}
}
layers {
name: "accuracy"
type: ACCURACY
bottom: "fc8"
bottom: "label"
top: "accuracy"
include: { phase: TEST }
}
layers {
name: "loss"
type: SOFTMAX_LOSS
bottom: "fc8"
bottom: "label"
top: "loss"
}
@GilLevi
Copy link
Author

GilLevi commented Jul 24, 2017

Hi @vinnitu,

From your code I see that you are using the lbp network. The lbp network requires specific preprocessing to the images that you haven't done (according to the images you have attached). It requires running our mapped LBP code in Matlab. Try the RGB network.

Best,
Gil

@GilLevi
Copy link
Author

GilLevi commented Jul 24, 2017

Hi @0xPr0xy,

Thank you for your interest in our work.

Are you trying to run the RGB network? did you try the deploy.txt file I attached in the gits?

Best,
Gil

@vinnitu
Copy link

vinnitu commented Jul 25, 2017

Hi @GilLevi, I trying with rgb model and have such result

screenshot from 2017-07-25 11-32-27

Do you think it is ok?
Wich is the best from all models? How can I prepare photos for lbp usage?

Thanks for reply

@GilLevi
Copy link
Author

GilLevi commented Jul 26, 2017

Hi @vinnitu,

Can you please share with me the code you used to get those results and the images as well? You can mail me at gil.levi100@gmail.com.

I want to run it on my machine to reproduce the results and try to debug it or see if it's ok.

Best,
Gil

@vinnitu
Copy link

vinnitu commented Jul 26, 2017

only this line

cur_net_dir = 'VGG_S_rgb'

@GilLevi
Copy link
Author

GilLevi commented Jul 31, 2017

Hi @vinnitu,

Can you pleas mail me the images so I can debug it on my laptop? my mail is gil.levi100@gmail.com

Best,
Gil

@vinnitu
Copy link

vinnitu commented Aug 3, 2017

Hi @GilLevi,

Do you receive my mail with images?

@GilLevi
Copy link
Author

GilLevi commented Aug 3, 2017

Yes, @vinnitu, I'll debug it in the weekend,

Best,
Gil

@vinnitu
Copy link

vinnitu commented Aug 4, 2017

@GilLevi thanks

@vinnitu
Copy link

vinnitu commented Aug 15, 2017

@GilLevi, any news?.. sorry

@chest3x
Copy link

chest3x commented Aug 25, 2017

UPDATE: I have found the solution, there are 2 more differences (between RGB and Cyclic5-LBP train_val):

  1. add k: 2 to lrn_param in norm1
  2. remove pad: 2 from convolution_param in conv2

INITIAL POST:
Hi, I am trying to finetune the Cyclic-5 LBP model with my data.
I have used Matlab code to convert my pictures using lbp_mapping_code.m and writing out_im into an image file using imwrite. Then I processed them into lmdb using this convert_imageset --resize_height=256 --resize_width=256 --shuffle <> <> <>
However, I am still getting this:
Cannot copy param 0 weights from layer 'fc6'; shape mismatch. Source param shape is 4096 18432 (75497472); target param shape is 4096 25088 (102760448). To learn this layer's parameters from scratch rather than copying from a saved net, rename the layer.

I have used solver and train_val from this page and modified 4048 to 4096. (Here it is: https://pastebin.com/LFUAT2Wz)

Is there anything else that I am doing wrong?

Thank you.

@Zumbalamambo
Copy link

Do you have the pretrained model for tensorfow ?

@Red-Night-Aria
Copy link

The urls of caffemodel are broken. Can anyone give me a copy?
Thx.

@xiaochehe
Copy link

The urls of caffemodel are broken. Can anyone give me a copy?
Thx.

@GarryWan
Copy link

The urls caffemodel are borken ,Can anyone give me copy?

@GilLevi
Copy link
Author

GilLevi commented Sep 18, 2017

Hi @Red-Night-Aria, @xiaochehe and @GarryWan,

Thank you for your interest in our work and for letting us know the links are broken. I updated the links, please check now.

Thanks!
Gil

@GilLevi
Copy link
Author

GilLevi commented Sep 18, 2017

Hi @Zumbalamambo,

Thank you for interest in our work. I'm afraid we never converted the models to TF.

Best,
Gil

@GilLevi
Copy link
Author

GilLevi commented Sep 18, 2017

Hi @chest3x,

Thanks for posing the solution.

btw, I just uploaded the various deploy files. There are indeed some small differences.

Best,
Gil

@GinYM
Copy link

GinYM commented Oct 2, 2017

The url of python notebook is broken in http://www.openu.ac.il/home/hassner/projects/cnn_emotions/.
Could you please check it? Thank you vary much.

@GilLevi
Copy link
Author

GilLevi commented Oct 27, 2017

Hi @GinYM,

Thank you for your interest in our work and thank you for notifying us the url is broken. The ipython notebook is now available in the following url: https://github.com/GilLevi/AgeGenderDeepLearning/blob/master/EmotiW_Demo.ipynb

Best,
Gil

@AishRatsh
Copy link

AishRatsh commented Nov 13, 2017

The emotion predictions runs on CPU, but when I change to GPU, it always breaks and give a out of memory error. Should I do something other than adding caffe.set_mode_gpu() (Python)? Please help.

@YiChunLin105062518
Copy link

Hi, I am new to caffe.
I am using the Emotion Classification CNN - RGB model configured. I want to use your model test on other datasets (ex: FER2013)
Which mean_pixel I would subtract (1.mean_file_proto you provide or 2.calculate FER training set mean_pixel)?
and if i want to fine tune on other dataset (ex:FER2013),which mean_pixel I would subtract?
Thank you for your help.

@Purpleyu
Copy link

Purpleyu commented Dec 5, 2017

Could you please share the training model on CASIA WEBFACE Database?

@aishwarya-singh25
Copy link

I am new to this, so kindly help me out. After having a look at the train as well as the deploy prototxt, I see that there's a missing link for the fc8 layer. Can you please provide more clarification on this? fc8 is being used as top and bottom layer for some other layers but this layer isn't defined anywhere. There is a layer fc8_cat and this isn't being used with any other layer as top or bottom. So, is fc8 and fc8_cat the same?

@GilLevi
Copy link
Author

GilLevi commented May 6, 2018

@AishRatsh, that shouldn't happen. Which GPU are you using?

@YiChunLin105062518, when finetuning you need to use the "mean_image" from the target dataset.

@Purpleyu, I'm afraid I do not have that model.

@aishwarya-singh25, the later is named fc8_cat, but is connected to fc7 and produces an output named fc8 that is fed to the softmax. So yes, fc8_cat is in fact fc8, I should have renamed that for clarity.

@LarryAndCode
Copy link

Hi GilLevi, I have apply your model in my ios application but the result is so bad.
Have you ever test your model with FER2013 dataset and what the accuracy is it?

@GilLevi
Copy link
Author

GilLevi commented Aug 26, 2018

Hi @LarryAndCode,

I have never tested my code on FER. Did you apply cropping prior to running the network?

@sd961960
Copy link

Hi Gil

Im using the RGB model...and i keep getting the "Sad" emotion being chosen with about 60 to 80 percent almost all of the time...it never picks any other emotion...is there an update to this model that i am missing?
Thanks for the help

Steve

@sd961960
Copy link

Hi Gil
Any chance i can get your image data so i can train my own model?...im having trouble converting some of your models to a mlmodel....thanks for the help.

Steve

@qqqafter
Copy link

qqqafter commented Nov 3, 2021

Hi Gil, I wanted to use your model in my project, but I can't download the model, because Google says that access to the file is denied, there is some way to get to the model. I try dowload Emotion Classification CNN - RGB(https://drive.google.com/open?id=0BydFau0VP3XSNVYtWnNPMU1TOGM).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment