Student: Saiteja Talluri
Mentor: Satya Mallick
Link to the accomplished work:
- PR in the opencv_contrib repository: opencv_contrib/pull/2197
- Tutorial Code for the facemark API in both C++ and python: code
Hello everyone, this is Saiteja Talluri. I worked on adding python support to the facial landmark API of OpenCV as a part of GSoC 2019. (Link to Project)
The main goal of the project was to change the inheritance structure of the current facemark API comprising of three models LBF, AAM and Kazemi and implement the python bindings to the facial landmark API. This helps the python users to train custom models and also use the existing models for the facial landmark detection.
Some other goals were to add a pytorch model for 3d facial alignment to OpenCV model zoo and also make a 5 point landmark detection model available for the users.
-
The inheritance structure for the facemark Kazemi has been updated so that the python bindigs work. Moreover some functions have been modified with the base intact so that the inheritance change avoids function conflicts.
-
Added some new functions like setParams to set the parameters specific to each model from python.
The demonstration of the facemark API in python for both training and inference can be found below.
Snippet for training the LBF model with your custom dataset.
facemark = cv2.face.createFacemarkLBF()
status, images_train, landmarks_train = cv2.face.loadDatasetList(args.training_images, args.training_annotations)
facemark.setParams(args.face_cascade,args.lbf_model, None, None)
for i in range(len(images_train)):
img = cv2.imread(images_train[i])
status, facial_points = cv2.face.loadFacePoints(landmarks_train[i])
facemark.addTrainingSample(img,facial_points)
facemark.training()
Snippet for inference using the LBF model.
face_cascade = cv2.CascadeClassifier(args.face_cascade)
facemark = cv2.face.createFacemarkLBF()
facemark.loadModel(args.lbf_model)
status, images_test, _ = cv2.face.loadDatasetList(args.test_images, args.test_images)
for image in images_test:
frame = cv2.imread(image)
gray_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray_frame, 1.3, 5)
if len(faces) > 0:
status, landmarks = facemark.fit(frame, faces)
for f in range(len(landmarks)):
cv2.face.drawFacemarks(frame, landmarks[f])
else:
print("No Face Detected")
cv2.imshow('Landmark_Window', frame)
Snippet for training the AAM model with your custom dataset.
facemark = cv2.face.createFacemarkAAM()
status, images_train, landmarks_train = cv2.face.loadDatasetList(args.training_images, args.training_annotations)
scale = np.array([2.0, 4.0])
facemark.setParams(args.face_cascade, args.aam_model, None, scale)
for i in range(len(images_train)):
img = cv2.imread(images_train[i])
status, facial_points = cv2.face.loadFacePoints(landmarks_train[i])
facemark.addTrainingSample(img,facial_points)
facemark.training()
Snippet for inference using the AAM model.
face_cascade = cv2.CascadeClassifier(args.face_cascade)
facemark = cv2.face.createFacemarkAAM()
facemark.loadModel(args.aam_model)
status, images_test, _ = cv2.face.loadDatasetList(args.test_images, args.test_images)
scale = np.array([2.0, 4.0])
facemark.setParams(args.face_cascade, args.aam_model, None, scale)
for image in images_test:
frame = cv2.imread(image)
gray_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray_frame, 1.3, 5)
if len(faces) > 0:
status, landmarks = facemark.fit(frame, faces)
for f in range(len(landmarks)):
cv2.face.drawFacemarks(frame, landmarks[f])
else:
print("No Face Detected")
cv2.imshow('Landmark_Window', frame)
Snippet for training the Kazemi model with your custom dataset.
facemark = cv2.face.createFacemarkKazemi()
status, images_train, landmarks_train = cv2.face.loadDatasetList(args.training_images, args.training_annotations)
scale = np.array([460.0, 460.0])
facemark.setParams(args.face_cascade,args.kazemi_model,args.kazemi_config,scale)
for i in range(len(images_train)):
img = cv2.imread(images_train[i])
status, facial_points = cv2.face.loadFacePoints(landmarks_train[i])
facemark.addTrainingSample(img,facial_points)
facemark.training()
Snippet for inference using the Kazemi model.
face_cascade = cv2.CascadeClassifier(args.face_cascade)
facemark = cv2.face.createFacemarkKazemi()
facemark.loadModel(args.kazemi_model)
gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray_img, 1.3, 5)
if len(faces) > 0:
status, landmarks = facemark.fit(frame, faces)
for f in range(len(landmarks)):
cv2.face.drawFacemarks(frame, landmarks[f])
else:
print("No Face Detected")
cv2.imshow('Landmark_Window', frame)
You can find the youtube video for this project here
-
There are three models (both in pytorch and torch7) in Adrian Bulat implementation of the 2D and 3D facial alignment - namely 2D-FAN,3D-FAN and Resnet Depth models. I converted all the three models in pytorch to onnx using a fixed size placeholder. Following were some issues I faced:
a. I tried using readNetfromTorch() to import torch7 model, it succeded in the case of Resnet Depth model but the FAN models were implemented using nngraph which doesn't have support in dnn yet. Issue is already reported here
b. Since implementing layers is relatively easy compared to changing the whole setup to implement nngraph, so I resorted to using onnx models. Importing the onnx using readnetFromOnnx() worked for ResnetDepth but failed for other two FAN models owing to some weird issue which didn't yet have a fix.
-
Implement face stabilization using optical flow as a part of the API itself and also provide an option to train models with custom number of points as landmarks.
[1] I would like to thank Laksono Kurnianggoro for his wonderful implementation of LBF and AAM in OpenCV Facemark API. [2] I would like to thank Sukhad Anand for his wonderful implementation of Kazemi in the OpenCV Facemark API.
what opencv-contrib version is used?
I get AttributeError: 'cv2.face_Facemark' object has no attribute 'setParams'