It is really amazing that a machine can determine the gesture of our hands. There are many applications based on gesture detection like you can make a remote control which works when a particular gesture was detected,you can have a gesture controlled game etc. Hand gesture detection can be done in many ways. we can use CNN to train models and use them for gesture detection.We will be using OpenCV and Convexity defects for detecting hand gestures. Gestures such as best of luck,thumbs down,1,2,3,4,5 etc
Any deviation of the contour from its convex hull is known as the convexity defect. Let’s understand this with the help of the above image.
Here, the red line shows the convex hull, the grey line represents the contour and the black arrow shows the deviation of the hull from the contour (convexity defect). So, it’s obvious that the convex curve has no convexity defects. Now, let’s discuss how to find the convexity defects using OpenCV-Python.
OpenCV provides a function cv2.convexityDefects() for finding the convexity defects of a contour. This takes as input the contour and its corresponding hull indices and returns an array containing the convexity defects as output. The basic syntax is given below.
The output contains an array where each row contains 4 values [start point, endpoint, farthest point, approximate distance to the farthest point]. Note that the first three values returned are indices of contour.
we will start the code by capturing video using opencv then we will flip the frame about Y axis as in web-cam frame needs to rotate about y axis to see image clearly. After this we will define kernel for dilation and erosion and then we will define the region of interest. Dilation and is done to increase the object area.
import cv2
import numpy as np
import math
cap = cv2.VideoCapture(0)
print("started")
while(1):
try: #an error comes if it does not find anything in window as it cannot find contour of max area
#therefore this try error statement
ret, frame = cap.read()
frame=cv2.flip(frame,1)
kernel = np.ones((3,3),np.uint8)
#define region of interest
roi=frame[100:300, 100:300]
After defining region of interst we will draw a rectangle and convert color from BGR to HSV.After this steps define upper and lower range of skin to form mask.so anything that is in given range will be considered as white and remaining things will be considered as black. After forming masks perform dilation and blur the image.Image is blurred to reduce the noise.
cv2.rectangle(frame,(100,100),(300,300),(0,255,0),0)
hsv = cv2.cvtColor(roi, cv2.COLOR_BGR2HSV)
# define range of skin color in HSV
lower_skin = np.array([0,20,70], dtype=np.uint8)
upper_skin = np.array([20,255,255], dtype=np.uint8)
#extract skin color image
mask = cv2.inRange(hsv, lower_skin, upper_skin)
#extrapolate the hand to fill dark spots within
mask = cv2.dilate(mask,kernel,iterations = 4)
#blur the image
mask = cv2.GaussianBlur(mask,(5,5),100)
Now,we need to find the contours and after that we will find the contour of max area that is our hand which we will be using further for classifying gestures.After this we need to do approximation of contours to get good shape.After this we form a convex hull and define the area of convexhull and contour which is a major part of our gesture detection algorithm. After this step define arearatio.On basis of arearatio we will be classifying gestures.
#find contours
contours,hierarchy= cv2.findContours(mask,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
#find contour of max area(hand)
cnt = max(contours, key = lambda x: cv2.contourArea(x))
#approx the contour a little
epsilon = 0.0005*cv2.arcLength(cnt,True)
approx= cv2.approxPolyDP(cnt,epsilon,True)
#make convex hull around hand
hull = cv2.convexHull(cnt)
#define area of hull and area of hand
areahull = cv2.contourArea(hull)
areacnt = cv2.contourArea(cnt)
#find the percentage of area not covered by hand in convex hull
arearatio=((areahull-areacnt)/areacnt)*100
Now,we will use convexity defects algorithm and cosine rule for finding the defects
We only want certain defects and for that we will eliminate other defects by appying two conditions. If angle is greater than 90 degrees then we will eliminate that defect as angle between fingers will be acute,also there should be a minimum distance from convex hull.If both this conditions are satisfied then that defect will be considered.In below images you can see that highlited part forms a convexity defect but is ignored by our if condition.
#find the defects in convex hull with respect to hand
hull = cv2.convexHull(approx, returnPoints=False)
defects = cv2.convexityDefects(approx, hull)
# l = no. of defects
l=0
#code for finding no. of defects due to fingers
for i in range(defects.shape[0]):
s,e,f,d = defects[i,0]
start = tuple(approx[s][0])
end = tuple(approx[e][0])
far = tuple(approx[f][0])
pt= (100,180)
# find length of all sides of triangle
a = math.sqrt((end[0] - start[0])**2 + (end[1] - start[1])**2)
b = math.sqrt((far[0] - start[0])**2 + (far[1] - start[1])**2)
c = math.sqrt((end[0] - far[0])**2 + (end[1] - far[1])**2)
s = (a+b+c)/2
ar = math.sqrt(s*(s-a)*(s-b)*(s-c))
#distance between point and convex hull
d=(2*ar)/a
# apply cosine rule here
angle = math.acos((b**2 + c**2 - a**2)/(2*b*c)) * 57After checking conditions we will draw a circle which you can think of as your convexity defect and draw lines around hand.Now the main part will be no of defects that we found. If no of defects are 0 then we have options like fist vertical , best of luck , thumbs down and 1. Now on the basis of arearatio we will detect the gestures.similarly if we have no of defects as 1 then available options are 2 and L. similary we will proceed and determine all the gestures.
# ignore angles > 90 and ignore points very close to convex hull(they generally come due to noise)
if angle <= 90 and d>30:
l += 1
cv2.circle(roi, far, 3, [0,0,255], -1)
#draw lines around hand
cv2.line(roi,start, end, [0,255,0], 2)
l+=1
#print corresponding gestures which are in their ranges
font = cv2.FONT_HERSHEY_SIMPLEX
if l==1:
if areacnt<2000:
cv2.putText(frame,'Put hand in the box',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)
else:
if arearatio<10:
cv2.putText(frame,'fist vertical',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)
elif arearatio<18:
cv2.putText(frame,'thumbs down',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)
elif arearatio<27:
cv2.putText(frame,'Best of luck',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)
else:
cv2.putText(frame,'1',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)elif l==2:
if arearatio<40:
cv2.putText(frame,'2',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)
else:
cv2.putText(frame,'L',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)
elif l==3:
if arearatio<27:
cv2.putText(frame,'3',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)
else:
cv2.putText(frame,'ok',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)
elif l==4:
cv2.putText(frame,'4',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)
elif l==5:
cv2.putText(frame,'5',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)
elif l==6:
cv2.putText(frame,'reposition',(0,50), font, 2, (255,0,0), 3, cv2.LINE_AA)
else :
cv2.putText(frame,'reposition',(10,50), font, 2, (255,0,0), 3, cv2.LINE_AA)Now finally we will display the frame and mask.
#show the windows
cv2.imshow('mask',mask)
cv2.imshow('frame',frame)
except:
pass
k = cv2.waitKey(5) & 0xFF
if k == 27:
break
cv2.destroyAllWindows()
cap.release() You can also refer to the flowchart shown below for step by step process.
Finally we are done with our code and now you can run the code and detect your hand gestures.Also there are some references given below for concepts used and also link to the code is provided.