Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save sayaleepote/c46a645d41caed5741fa880086e95e78 to your computer and use it in GitHub Desktop.
Save sayaleepote/c46a645d41caed5741fa880086e95e78 to your computer and use it in GitHub Desktop.
// MARK: - AVCaptureVideoDataOutputSampleBufferDelegate
/* This delegate is fired periodically every time a new video frame is written.
It is called on the dispatch queue specified while setting up the capture session.
*/
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
/* Initialise CVPixelBuffer from sample buffer
CVPixelBuffer is the input type we will feed our coremlmodel .
*/
guard let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
/* Initialise Core ML model
We create a model container to be used with VNCoreMLRequest based on our HandSigns Core ML model.
*/
guard let handSignsModel = try? VNCoreMLModel(for: HandSigns().model) else { return }
/* Create a Core ML Vision request
The completion block will execute when the request finishes execution and fetches a response.
*/
let request = VNCoreMLRequest(model: handSignsModel) { (finishedRequest, err) in
/* Dealing with the result of the Core ML Vision request
The request's result is an array of VNClassificationObservation object which holds
identifier - The prediction tag we had defined in our Custom Vision model - FiveHand, FistHand, VictoryHand, NoHand
confidence - The confidence on the prediction made by the model on a scale of 0 to 1
*/
guard let results = finishedRequest.results as? [VNClassificationObservation] else { return }
/* Results array holds predictions iwth decreasing level of confidence.
Thus we choose the first one with highest confidence. */
guard let firstResult = results.first else { return }
var predictionString = ""
/* Depending on the identifier we set the UILabel text with it's confidence.
We update UI on the main queue. */
DispatchQueue.main.async {
switch firstResult.identifier {
case HandSign.fistHand.rawValue:
predictionString = "Fist👊🏽"
case HandSign.victoryHand.rawValue:
predictionString = "Victory✌🏽"
case HandSign.fiveHand.rawValue:
predictionString = "High Five🖐🏽"
case HandSign.noHand.rawValue:
predictionString = "No Hand ❎"
default:
break
}
self.predictionLabel.text = predictionString + "(\(firstResult.confidence))"
}
}
/* Perform the above request using Vision Image Request Handler
We input our CVPixelbuffer to this handler along with the request declared above.
*/
try? VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:]).perform([request])
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment