The goal was to capture video using the ZED on the TX2 without also capturing depth sensing information. The ZED sdk does not allow you to seperately capture video from the ZED, so gstreamer was used. I was able to capture 30fps @1080p from the ZED.
After recieving the Jetson TX2 I connected it to power with the AC adapter, plugged it into an hdmi display TV and powered it on. I also connected a usb hub to the TX2 and on the usb hub connected a mouse, keyboard. The antennae are neccessary, otherwise the TX2 will have poor wifi reception. Upon booting up the TX2, connecting onto my home network, and rebooting, I get only a blue screen from the TX2.
I followed the instructions from https://devtalk.nvidia.com/default/topic/1000569/jetson-tx2/tx2-blue-screen-after-booting/. Basically, get yourself a linux host computer with Ubuntu 14.04 on it, download Jetpack from here https://developer.nvidia.com/embedded/jetpack-notes. Open a terminal and cd into the directory with Jetpack-L4T-3.0-linux-x64.run in it and run:
chmod +x Jetpack-L4T-3.0-linux-x64.run
./Jetpack-L4T-3.0-linux-x64.run
Pain through the lengthy download and installation process and once it is done, a TX2 flavored Ubuntu should be booted up. Connect the ZED to the TX2 via usb. You can install the ZED sdk from here https://www.stereolabs.com/developers/release/2.0/ although you won't need it. It will install a bunch tools you can play around with in the /usr/local/zed/tools directory.
For some reason, sound via hdmi doesn't automatically work upon flashing the TX2. From reading https://devtalk.nvidia.com/default/topic/1004298/jetson-tx2/no-sound-via-hdmi/ , I ran these two commands:
export DISPLAY=:0
pactl load-module module-alsa-sink device=hw:0,7
and went to system settings then sound and changed the output from Analog output to tegra-hda. Sound should work now.
Two of the 6 CPUs are disabled by default. Your going to want to turn those on to get better video framerate from the ZED. You can read more from this link: https://devtalk.nvidia.com/default/topic/1000345/two-cores-disabled-/ and run these commands in the terminal:
$ sudo su
$ echo 1 > /sys/devices/system/cpu/cpu1/online
$ echo 1 > /sys/devices/system/cpu/cpu2/online
$ exit
To maximize clock governor frequencies:
$ sudo /home/ubuntu/jetson_clocks.sh --store
$ sudo ./jetson_clocks.sh
And return to default state:
$ sudo /home/ubuntu/jetson_clocks.sh --restore
And to switch between the different performance modes on the TX2 use:
$ sudo nvpmodel -m [mode]
where modes range from 0 - 4.
To check the CPU usage open system moniter and click on resources. To check GPU usage run:
$ sudo ./tegrastats
The GPU stats are the GR3D ones. 1st number is GPU usage and the second (after @) is the current GPU frequency. You read more here: https://devtalk.nvidia.com/default/topic/933774/how-to-monitor-gpu-utilization-on-jetson-tk1/.
You can read up more from http://www.jetsonhacks.com/2017/03/25/nvpmodel-nvidia-jetson-tx2-development-kit/.
Trying to record video on the ZED using OpenCV's videocapture did not work. After reading about the problems OpenCV has with the TX2, I moved to using gstreamer. Run:
gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-raw,framerate=30/1,width=3840,height=1080 ! xvimagesink
and gstreamer should start streaming video from the ZED camera and playing it back on a window. Make sure to run the earlier commands on enabling the cpus and maximizing clock frequencies or the video quality will suck. Make sure to install the gstreamer good, bad, and ugly plugins. You can find them on the terminal by running:
apt-cache search libgst
Run:
gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-raw,framerate=30/1,width=3840,height=1080 ! fpsdisplaysink name=fpssink text-overlay=false video-sink=xvimagesink
to view the framerate as gstreamer is playing back the video. Finally run:
gst-launch-1.0 v4l2src device=/dev/video1 do-timestamp=true blocksize=3110400 ! 'video/x-raw, width=3840, height=1080, framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=3840, height=1080, format=I420' ! omxh264enc ! h264parse ! qtmux ! filesink location=tx2_zed_sample.mp4 sync=false -e -vvv
To save the streaming video as an mp4 file as tx2_zed_sample.mp4 in your current directory. You can play it back using gstreamer with:
gst-launch-1.0 playbin uri=file:/home/nvidia/tx2_zed_sample20.mp4
You can get better quality video by increasing the bitrate parameter on the omxh264enc element of the pipeline. The bitrate defaults to 4000000 and ranges from 0 to ~2 billion.
gst-launch-1.0 v4l2src device=/dev/video1 do-timestamp=true blocksize=3110400 ! 'video/x-raw, width=3840, height=1080, framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=3840, height=1080, format=I420' ! omxh264enc bitrate=40000000! h264parse ! qtmux ! filesink location=tx2_zed_sample.mp4 sync=false -e -vvv
Note that increasing the bitrate from the default 4000000 to 40000000, the size of the mp4 file increases ~10x.
I was able to get 1080p video @30fps, averaging 29.5fps using these commands.
http://www.pixeltools.com/rate_control_paper.html
The bitrate on a video controls the number of bits alotted to a given group of frames in the video. So when there is a lot going on in a group of frames, there is more variance in colors and thus more bits will need to be encoded. But if a maximum bitrate is set, the mpe4 compression algorithm will need to compromise. This is why you will see more 'pixelation' during more active parts of the video. To not exceed the maximum bitrate, pixels of varied but similar shades of color get converted to one color, which is why you get 'blocky' areas in the frames.
By increasing the maximum bitrate during video encoding, more bits can be alotted to each group of frames in the video and thus quality is better preserved.
gstreamer acts as a middle man for digital media between sources (over http, v4l2, or a file) and destinations (a video window, output file). gst-launch-1.0 is the terminal command. Gstreamer creates a pipeline bewteen the sources and sinks or destination. In our above commands v4l2 is a linux program (source) that plays back video from a camera accessible from /dev/video1. We tell it to capture 1080p @30fps using video/x-raw,framerate=30/1,width=3840,height=1080 (even though it is 1920 X 1080 we have to use 3840 as the width). The video is then piped to a window on our screen using xvimagesink (sink).
Instead of piping the video to a window on our screen we can convert the video from a raw stream to our desired 1080p @ 30fps by piping our stream into nvidia's proprietary gstreamer plugin nvvidconv and then saving the video by encoding it from I420 format to h264 codec and then saving it as a file using the filesink location=/path/to/file gst element.
Right now the raw video is converted froom I420 format to h264 format so that is compatible with an mp4 container. In the future we may want to capture the video as the original YUY2 format because the color quality is worsened when doing the conversion.
You can read more up on gstreamer from here:
https://gstreamer.freedesktop.org/documentation/