PYNQ v1.3 capture HDMI and display in notebook and output to HDMI

capture_hdmi_display_notebook_hdmi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PYNQ-Z1 Video input/output example\n",
    "----\n",
    "\n",
    "## Contents\n",
    "\n",
    "* [Setup the board](#Setup-the-board)\n",
    "* [Capture and display a frame](#Capture-and-display-a-frame)\n",
    "* [Setup the HDMI output](#Setup-the-HDMI-output)\n",
    "* [Connecting input to output](#Connecting-input-to-output)\n",
    "* [Setup and capture input from Webcam](#Setup-and-capture-input-from-Webcam)\n",
    "* [Setup the HDMI and display on the output](#Setup-the-HDMI-and-display-on-the-output)\n",
    "----\n",
    "\n",
    "\n",
    "## Introduction\n",
    "\n",
    "The first part of this notebook will show how to use the HDMI input, and the HDMI output on the PYNQ-Z1 board. The base overlay contains a HDMI input controller, and HDMI output controller. \n",
    "\n",
    "The PYNQ-Z1 board has 512 MB external DDR memory connected to a DDR memory controller in the Zynq PS. There is a direct path from the PL to DDR memory.\n",
    "\n",
    "The HDMI input controller can stream data to DDR memory for image capture using a DMA. Similarly, the HDMI output can stream data from DDR memory for output display using a seperate DMA. \n",
    "\n",
    "The PYNQ-Z1 board also has a USB port connected to the Zynq PS. A USB webcam can be connected and frame data can be streamed to DDR memory in a similar way to the HDMI. The second part of this notebook will show how to use a USB webcam. \n",
    "\n",
    "The base overlay also supports writing to three framebuffers in DDR memory. [Double buffering and triple buffering](#https://en.wikipedia.org/wiki/Multiple_buffering) are common techniques for smooth display of graphics, and can be used with the base overlay, but will not be demonstrated in this notebook. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: HDMI\n",
    "## Setup the board"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the _base_ overlay:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from pynq.pl import Overlay\n",
    "Overlay(\"base.bit\").download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup the HDMI input\n",
    "\n",
    "Import the HDMI package, and create an instance of the HDMI input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pynq.drivers.video import HDMI\n",
    "hdmi_in = HDMI('in')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start the HDMI input controller:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdmi_in.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the status of the controller:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "state = hdmi_in.state()\n",
    "print(\"HDMI input status is \" + str(state) ) # 1 means streaming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Capture and display a frame\n",
    "Capture a frame from the HDMI input. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "frame = hdmi_in.frame_raw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The frame will be saved as a NumPy array. _NumPy_ is a Python package for scientific commputing, which provides support for large, multi-dimensional arrays. It also includes a high level libarary of optimized operations for manipulating arrays. Converting the frame to a NumPy array allows it to be manipulated more easily in Python. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np_frame= (np.frombuffer(frame, dtype=np.uint8)).reshape(1080,1920,3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently the input is captured in BGR format. The input frame will be displayed in the notebook, but needs to be converted to RGB first. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np_frame_rgb = np_frame[:,:,[2,1,0]] # reorder RGB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display the image\n",
    "_Matplotlib_ is a Python package for 2D plotting. Matplotlib will be used to display the input image in the notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline \n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "fig1 = plt.figure(figsize = (14,14)) # Set figure size\n",
    "ax1 = fig1.add_subplot(111) # add plot as subplot\n",
    "ax1.axes.get_xaxis().set_visible(False) # Hide axis\n",
    "ax1.axes.get_yaxis().set_visible(False)\n",
    "ax1.imshow(np_frame_rgb, interpolation='none') # display image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This ends the HDMI input capture section. The HDMI can be stopped, and the objects cleaned at this point. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hdmi_in.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup the HDMI output\n",
    "\n",
    "Import the HDMI package (not necessary if the cells above were run), and create an instance of the HDMI output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pynq.drivers.video import HDMI\n",
    "hdmi_out = HDMI('out')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the HDMI output mode to 1920 x 1080:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "HDMI_OUT_1920_1080 = 4\n",
    "\n",
    "hdmi_out.mode(HDMI_OUT_1920_1080)\n",
    "\n",
    "hdmi_out.start() # Start the HDMI out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the original frame on the HDMI output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdmi_out.frame_raw(frame)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This ends this section, and the objects can be cleaned up. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hdmi_in.stop()\n",
    "hdmi_out.stop()\n",
    "del hdmi_out, hdmi_in, np_frame, np_frame_rgb, fig1, ax1, frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connecting input to output\n",
    "Import the HDMI package, and create instances of the HDMI input and HDMI output controllers. Connect the HDMI output framebuffer to the HDMI input framebuffer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pynq.drivers.video import HDMI\n",
    "hdmi_out = HDMI('out')\n",
    "hdmi_in = HDMI('in', hdmi_out.frame_buffer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the mode and start the two HDMI controllers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "HDMI_OUT_1920_1080 = 4\n",
    "\n",
    "hdmi_out.mode(HDMI_OUT_1920_1080)\n",
    "\n",
    "hdmi_in.start()\n",
    "hdmi_out.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: USB webcam\n",
    "\n",
    "## Setup and capture input from Webcam\n",
    "A USB webcam can be connected to the USB port of the Zynq PS, and can be used to capture image data to memory in a similar way to the HDMI input.\n",
    "To capture a frame, the OpenCV package is used. OpenCV has been preinstalled as part of the PYNQ-Z1 image. Note that OpenCV is only used hear to read a frame from the webcam and save it to memory. \n",
    "\n",
    "Data will be streamed from the webcam to DDR, and from DDR to the HDMI output. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup frame sizes \n",
    "Define some constants for the input (webcam) and output (HDMI output) frame sizes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# HDMI Out configuration\n",
    "HDMI_OUT_1920_1080 = 4\n",
    "\n",
    "# HDMI Output frame buffer size\n",
    "frame_out_w = 1920\n",
    "frame_out_h = 1080\n",
    "\n",
    "# camera (input) configuration\n",
    "frame_in_w = 640\n",
    "frame_in_h = 480"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup camera\n",
    "Initialize camera from OpenCV (set requested image size)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "videoIn = cv2.VideoCapture(0)\n",
    "videoIn.set(cv2.CAP_PROP_FRAME_WIDTH, frame_in_w);\n",
    "videoIn.set(cv2.CAP_PROP_FRAME_HEIGHT, frame_in_h);\n",
    "print(\"capture device is open: \" + str(videoIn.isOpened())) ## Check the webcam is available"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Capture Image\n",
    "Capture data from the webcam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "readError = 0\n",
    "# check actual input resolution\n",
    "ret, frame = videoIn.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the actual resolution of the frame received"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame_w = frame.shape[1]\n",
    "frame_h = frame.shape[0]\n",
    "print(\"Webcam resolution: \", frame_w, \"x\", frame_h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup the HDMI and display on the output\n",
    "\n",
    "### Setup HDMI output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from pynq.drivers.video import Frame, HDMI\n",
    "# create HDMI output object\n",
    "hdmi_out=HDMI('out')\n",
    "\n",
    "# create frame \n",
    "hdmi_out.mode(HDMI_OUT_1920_1080)\n",
    "frameOut = hdmi_out.frame(0)\n",
    "# start\n",
    "hdmi_out.frame_index(0)\n",
    "hdmi_out.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the state of the HDMI output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "state = hdmi_out.state()\n",
    "print(\"output state is \" + str(state) )  # 1 means streaming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stream Data\n",
    "Stream 100 frames from the webcam to the HDMI output (via DDR memory) and calculate the frame rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import time\n",
    "start = time.time()\n",
    "for NumOfFrames in range (50): \n",
    "    # read next image\n",
    "    ret, frame = videoIn.read()\n",
    "    frame.resize(frame_w*frame_h*3)\n",
    "    \n",
    "    if (ret):\n",
    "        for y in range (frame_h):\n",
    "            frameOut.frame[y*frame_out_w*3:y*frame_out_w*3+frame_w*3] = frame.data[y*frame_w*3:(y+1)*frame_w*3]\n",
    "        # copy to frame buffer / show on monitor\n",
    "        hdmi_out.frame(0, frameOut)   \n",
    "    else:\n",
    "        readError = readError + 1\n",
    "\n",
    "end = time.time()\n",
    "print(\"Frames per second:     \" + str(50 / (end - start)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change resolution\n",
    "Change the webcam resolution to 1280 x 800 and repeat the frame streaming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# camera (input) configuration\n",
    "frame_in_w = 1280\n",
    "frame_in_h = 720\n",
    "\n",
    "# initialize camera from OpenCV\n",
    "videoIn = cv2.VideoCapture(0)\n",
    "videoIn.set(cv2.CAP_PROP_FRAME_WIDTH, frame_in_w);\n",
    "videoIn.set(cv2.CAP_PROP_FRAME_HEIGHT, frame_in_h);\n",
    "print(\"capture device is open: \" + str(videoIn.isOpened()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the actual webcam resolution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# initialize support variables\n",
    "readError = 0\n",
    "# check input resolution\n",
    "ret, frame = videoIn.read()\n",
    "frame_w = frame.shape[1]\n",
    "frame_h = frame.shape[0]\n",
    "print(\"Webcam Resolution:\", frame_w, \"x\", frame_h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repeat data stremaing\n",
    "Repeat the streaming of 100 frames with the new resolution and calculate the framerate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "for NumOfFrames in range (50):\n",
    "            \n",
    "    # read next image\n",
    "    ret, frame = videoIn.read()\n",
    "    frame.resize(frame_w*frame_h*3)\n",
    "    \n",
    "    if (ret):\n",
    "        for y in range (frame_h):\n",
    "            frameOut.frame[y*frame_out_w*3:y*frame_out_w*3+frame_w*3] = frame.data[y*frame_w*3:(y+1)*frame_w*3]\n",
    "        # copy to frame buffer / show on monitor\n",
    "        hdmi_out.frame(0, frameOut)   \n",
    "    else:\n",
    "        readError = readError + 1\n",
    "\n",
    "end = time.time()\n",
    "print(\"Frames per second:     \" + str(50 / (end - start)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}