Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Simple Hand Mouvement Recognition Code - Hand tracking - Mediapipe

Simple Hand Mouvement Recognition Code - Hand tracking - Mediapipe

Goal of this gist is to recognize some simple hand mouvements like Scrolling, Zoom in/out and Slide left/right (see 08-hand-mouvement.gif below).

The whole code for that can be found here: hand-mouvement-recognition-calculator.cc.

You can clone my forked version of mediapipe here: https://github.com/TheJLifeX/mediapipe. I have already commited all code in that repository in the "hand-mouvement-recognition" branch (https://github.com/TheJLifeX/mediapipe/tree/hand-mouvement-recognition).

If you want to know how to recognize hand gesture like ONE, TWO, TREE, FOUR, FIVE, SIX, YEAH, ROCK, SPIDERMAN and OK. You can read this gist: Simple Hand Gesture Recognition.

Prerequisite: You kwon how to run the hand tracking example.

  1. Get Started with mediapipe
  2. Hand Tracking on Desktop

Hand Mouvement: Scrolling

There are four defined scroll directions (up, right, down and left). To determine the scroll direction we have to calculate the angle between the current center of the hand closed (red) rectangle, the previous center and the x-axis.

  • Scroll up: angle between 135° and 45°
  • Scroll right: angle between 45° and -45°
  • Scroll down: angle between -45° and -135°
  • Scroll left: angle between -135° and 135°

Code for the hand mouvement "Scrolling" recogniton can be found here

Hand Mouvement: Zoom in/out

The idea here is that when the hand closed (red) rectangle becomes bigger that will be recognized as "Zoom in", and when smaller as "Zoom out". I used only the height of the hand closed rectangle but the width or the height and width can be used.

Code for the hand mouvement "Zoom in/out" recogniton can be found here

Hand Mouvement: Slide left/right

The idea here is that to slide left or the right the hand must be first upright. For that we define a interavall for the angle between the hand (wirst and MCP) and the x-axis. For the start position this angle is between 80° and 100°. We define this rule for a simple reason. After a "Slide right" for example the hand mouvement to the left should not be as "Slide left" recognized.

Code for the hand mouvement "Slide left/right" recogniton can be found here

This file has been truncated, but you can view the full file.
@SaddamBInSyed
Copy link

SaddamBInSyed commented Jun 9, 2020

@TheJLifeX, Thanks for your gesture code.

Could you please advise whether your gesture (slidig, zoom in out) code available in python.

since I am new to c++ I don't understand the flow.

highly appreciated your advice,

@SaddamBInSyed
Copy link

SaddamBInSyed commented Jun 10, 2020

@TheJLifeX,

I come across the below line in your code here,

float threshold = 0.01;
float closeFingerThreshold = 0.01;

float pseudoFixKeyPoint = landmarkList.landmark(2).x();
if (landmarkList.landmark(3).x() + threshold < pseudoFixKeyPoint && landmarkList.landmark(4).x() + threshold < landmarkList.landmark(3).x())
{
    thumbState = FingerState::OPEN;
}

else if (pseudoFixKeyPoint + threshold (<--SEE HERE) < landmarkList.landmark(3).x() && landmarkList.landmark(3).x() + threshold (<--SEE HERE) < landmarkList.landmark(4).x())
{
    thumbState = FingerState::CLOSE;
}

you used a threshold variable instead of "closeFingerThreshold " for both open and close cases.

is this behavior RIGHT?

please advise

@TheJLifeX
Copy link
Author

TheJLifeX commented Jun 15, 2020

Hi @SaddamBInSyed, it isn't available in python.

you used a threshold variable instead of "closeFingerThreshold " for both open and close cases.

It wasn't the right behavior. I have fixed it. Thank you!

@TibbersBao
Copy link

TibbersBao commented Sep 25, 2020

Hi, i am not familiar with C++, that's hard for me to read the hand_Mouvement Recognition Code, Could u share a python version code for this code.

@kishorkuttan
Copy link

kishorkuttan commented Sep 30, 2020

AnnotationOverlayCalculator: ; cc->Inputs().HasTag(recognizedHandMouvementScrollingTag)_overlay_calculator.cc:176)
Got this error while running the code. Any solution

@kishorkuttan
Copy link

kishorkuttan commented Sep 30, 2020

your gesture code works well, but movement one, throwing this error

@kishorkuttan
Copy link

kishorkuttan commented Sep 30, 2020

I0930 10:06:46.818581 5139 demo_run_graph_main.cc:53] Initialize the calculator graph.
E0930 10:06:46.820902 5139 demo_run_graph_main.cc:146] Failed to run the graph: ValidatedGraphConfig Initialization failed.
AnnotationOverlayCalculator: ; cc->Inputs().HasTag(recognizedHandMouvementScrollingTag)_overlay_calculator.cc:176)

@kishorkuttan
Copy link

kishorkuttan commented Sep 30, 2020

running script: GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment