Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Fast reading from the raspberry camera with Python, Numpy, and OpenCV. See the comments for more details.
# Fast reading from the raspberry camera with Python, Numpy, and OpenCV
# Allows to process grayscale video up to 124 FPS (tested in Raspberry Zero Wifi with V2.1 camera)
#
# Made by @CarlosGS in May 2017
# Club de Robotica - Universidad Autonoma de Madrid
# http://crm.ii.uam.es/
# License: Public Domain, attribution appreciated
import cv2
import numpy as np
import subprocess as sp
import time
import atexit
frames = [] # stores the video sequence for the demo
max_frames = 300
N_frames = 0
# Video capture parameters
(w,h) = (640,240)
bytesPerFrame = w * h
fps = 250 # setting to 250 will request the maximum framerate possible
# "raspividyuv" is the command that provides camera frames in YUV format
# "--output -" specifies stdout as the output
# "--timeout 0" specifies continuous video
# "--luma" discards chroma channels, only luminance is sent through the pipeline
# see "raspividyuv --help" for more information on the parameters
videoCmd = "raspividyuv -w "+str(w)+" -h "+str(h)+" --output - --timeout 0 --framerate "+str(fps)+" --luma --nopreview"
videoCmd = videoCmd.split() # Popen requires that each parameter is a separate string
cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPE) # start the camera
atexit.register(cameraProcess.terminate) # this closes the camera process in case the python scripts exits unexpectedly
# wait for the first frame and discard it (only done to measure time more accurately)
rawStream = cameraProcess.stdout.read(bytesPerFrame)
print("Recording...")
start_time = time.time()
while True:
cameraProcess.stdout.flush() # discard any frames that we were not able to process in time
# Parse the raw stream into a numpy array
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
if frame.size != bytesPerFrame:
print("Error: Camera stream closed unexpectedly")
break
frame.shape = (h,w) # set the correct dimensions for the numpy array
# The frame can be processed here using any function in the OpenCV library.
# Full image processing will slow down the pipeline, so the requested FPS should be set accordingly.
#frame = cv2.Canny(frame, 50,150)
# For instance, in this example you can enable the Canny edge function above.
# You will see that the frame rate drops to ~35fps and video playback is erratic.
# If you then set fps = 30 at the beginning of the script, there will be enough cycle time between frames to provide accurate video.
# One optimization could be to work with a decimated (downscaled) version of the image: deci = frame[::2, ::2]
frames.append(frame) # save the frame (for the demo)
#del frame # free the allocated memory
N_frames += 1
if N_frames > max_frames: break
end_time = time.time()
cameraProcess.terminate() # stop the camera
elapsed_seconds = end_time-start_time
print("Done! Result: "+str(N_frames/elapsed_seconds)+" fps")
print("Writing frames to disk...")
out = cv2.VideoWriter("slow_motion.avi", cv2.cv.CV_FOURCC(*"MJPG"), 30, (w,h))
for n in range(N_frames):
#cv2.imwrite("frame"+str(n)+".png", frames[n]) # save frame as a PNG image
frame_rgb = cv2.cvtColor(frames[n],cv2.COLOR_GRAY2RGB) # video codec requires RGB image
out.write(frame_rgb)
out.release()
print("Display frames with OpenCV...")
for frame in frames:
cv2.imshow("Slow Motion", frame)
cv2.waitKey(1) # request maximum refresh rate
cv2.destroyAllWindows()
@CarlosGS

This comment has been minimized.

Copy link
Owner Author

CarlosGS commented May 27, 2017

Introduction

For some reason every Raspberry Camera tutorial specifies the maximum camera frame rate at 90 FPS. This is not true!
With the proof-of-concept script above, the camera has been tested to work up to at least 120FPS. Now it is possible to have real time, ultra-low-latency video processing even in the Raspberry Zero. The form factor of this board makes it ideal for robotics.

These high FPS are possible thanks to the great work at Raspividyuv. The above script is the first one to connect Raspividyuv and Python to achieve ultra low latency results. I hope this example code can enable many people to integrate efficient computer vision algorithms into many kinds of robots. The scripts receives grayscale video only, though it could be extended to fetch color as well.

Maximum framerates at multiple resolutions

raspi_camerav2_fpsresolution
Download full spreadsheet here: https://www.dropbox.com/s/28vq8d9qx0tm81i/raspi_cameraV2_fpsResolution.ods

Example images at each resolution: https://www.dropbox.com/s/k0gzpt15jj0qbqd/raspberry_cameraV2_quality_FPS_comparison.zip (useful to compare the field-of-view in every mode). Note some of the large modes produce corrupt frames, this needs to be studied.

Videos (click to open)

Slow motion video example (~120fps played at 30fps, captured with the python script:
Watch video

Real time Canny edge detection (30fps):
Watch video

The videos are not only recorded, but also processed in real time in the Python script.

Now it is possible to have low cost vision for fast robots! Even if you don't actually use 120FPS, the lower time between frames will give you more cycle time to process each image.

Please share your progress too, so we can all learn :-)

@MakerVisioneer

This comment has been minimized.

Copy link

MakerVisioneer commented Nov 1, 2017

Thank you! Needed "VideoWriter_fourcc" vs "cv.CV_FOURCC" to get this working for OpenCV3.3.0 on Pi3
Ex. out = cv2.VideoWriter("slow_motion.avi", cv2.VideoWriter_fourcc(*"MJPG"), 30, (w,h))

@CarlosGS

This comment has been minimized.

Copy link
Owner Author

CarlosGS commented Nov 13, 2017

Sweet! Thanks for sharing how to run it on OpenCV 3.3, will come in handy for the future :)

@rokartnaz

This comment has been minimized.

Copy link

rokartnaz commented Mar 26, 2018

Thank you! it's amazing. But how could I get colorful video?

@CarlosGS

This comment has been minimized.

Copy link
Owner Author

CarlosGS commented Apr 25, 2018

Currently I have no plans to implement color version. To do this, it is necessary to remove the --luma option from raspividyuv and extract & combine the new chroma channels.

@MyraBaba

This comment has been minimized.

Copy link

MyraBaba commented Sep 14, 2018

Hi,

Is there any c++ code for same speed ? or can we get more with c++ ?

Best

@maliksyria

This comment has been minimized.

Copy link

maliksyria commented Mar 1, 2019

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

@realizator

This comment has been minimized.

Copy link

realizator commented Oct 14, 2019

Hello
thank you for sharing
I have got the following error :
frame = np.fromfile(cameraProcess.stdout, count=bytesPerFrame, dtype=np.uint8)
OSError: obtaining file position failed

This error appears because in Python 3 pipe is buffered but unseekable (details).
So as a patch you need to add "bufsize=0" option, that is:

cameraProcess = sp.Popen(videoCmd, stdout=sp.PIPEm, bufsize=0) # start the camera

Also in my case image has been distorted, and I changed resolution to 640x480 instead of 640x240.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.