hhackbarth/jetson_hardware_accelerated_opencv_4.5.3_with_ffmpeg_4.2.4.md

## jetson_hardware_accelerated_opencv_4.5.3_with_ffmpeg_4.2.4.md

      
    Raw
  

              jetson_hardware_accelerated_opencv_4.5.3_with_ffmpeg_4.2.4.md
            
          
    OpenCV 4.5.3 and FFMPEG 4.2.4 (with SRT support) hardware accelerated on NVidia Jetson

Unfortunately, NVidia Jetpack 4.5.x for Jetson Nano, Xavier etc. comes with OpenCV 4.1.1 and FFMPEG 3.4.8 compiled without any hardware (GPU, CUDA) acceleration and without SRT support.
Retrieving current OpenCV build information:
$ python3
import cv2
print(cv2.getBuildInformation())
exit()

In the output you can see, if your build supports FFMPEG, GStreamer, CUDA etc.
If you meanwhile installed another version of OpenCV, you should uninstall it now:
$ python3 -m pip uninstall opencv-python
$ sudo apt-get --purge autoremove python-opencv
If you build OpenCV from scratch now, you will propably end up without FFMPEG support. Reason is that during the FFMPEG detection, CMake tries a test compile with FFMPEG libraries, which will fail. (for details, see https://github.com/opencv/opencv/blob/4.5.0/modules/videoio/cmake/detect_ffmpeg.cmake#L88)
As far as I could see, this is caused by some static libraries in the FFMPEG build. According to my tests, also the FFMPEG 4.2.2 version with hardware decoding support, that NVidia provides for Jetson (see description here: https://forums.developer.nvidia.com/t/jetson-tx2-and-ffmpeg-cant-initialize-nvrm-channel/165629/5), results in an OpenCV build without FFMPEG support.
According to my tests, it was possible to get a successful OpenCV build, when you use a patched FFMPEG 4.2.4 build using the patch from jocover, as described here: https://github.com/jocover/jetson-ffmpeg. Instead of the 4.2 version used there, you can use the 4.2.4 version. The patch will also work with that release (but not with 4.2.5). You can download the FFMPEG 4.2.4 release from here: http://www.ffmpeg.org/download.html#releases. Please only apply the patch but DO NOT YET CONFIGURE AND MAKE the ffmpeg compilation.
Before you build FFMPEG from scratch, you should remove earlier installations with
$ sudo apt --purge autoremove ffmpeg
If you already tried a FFMPEG build from scratch before, you should go to the folder containing the build and uninstall it from there:
$ sudo make uninstall
Assuming, you are in the folder with the patched FFMPEG 4.2(.4) version, you can proceed as follows:
First you most propably will have to download and compile SRT (libsrt as shared library). You can do this below the current ffmpeg folder:
git clone https://github.com/Haivision/srt.git
cd srt
git checkout -b work v1.4.1
sudo apt-get install tclsh pkg-config cmake libssl-dev build-essential
PATH="$HOME/bin:$PATH" ./configure --prefix=/usr/local --disable-static --enable-shared
make -j$(nproc)
sudo make install
sudo ldconfig
cd ..

Now we should be back in the ffmpeg folder and can configure the CMake build to generate a version with shared libraries and package configuration files:
./configure --enable-shared --disable-static --prefix=/usr/local --enable-nvmpi --extra-libs='-L/usr/lib/aarch64-linux-gnu/tegra -lnvbuf_utils' --extra-cflags='-I /usr/src/jetson_multimedia_api/include/' --enable-libx264 --enable-libx265 --enable-protocol=libsrt --enable-libsrt --enable-gpl --pkgconfigdir=/usr/local/lib/pkgconfig
Afterwards (still in the ffmpeg-4.2.4 folder):
$ make -j$(nproc)
$ sudo make install
Now make sure to update the ld config:
$ sudo gedit /etc/ld.so.conf.d/ffmpeg.conf
This will propably show an empty file, here you add the lines:
/usr/local/cuda-10.2/lib64
/usr/local/lib/

Apply the update:
$ sudo ldconfig
General guides for compiling FFMpeg from scratch are given here: https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu
You can check if ffmpeg is installed with hardware encoding now with the following command:
$ ffmpeg -codecs | grep nvmpi
The output should contain h264_nvmpi and hevc_nvmpi. If you do not see these encoder but instead see a warning like "WARNING: library configuration mismatch", there are still some other avcodec, avutil, avformat libraries on your system. This will happen, when you also installed the "bad plugins" for GStreamer, e.g. with sudo apt install gstreamer1.0-plugins-bad
If you want to remove them in order to make use of ffmpeg hardware codecs, you will break GStreamer pipelines (e.g. h264parse and h265parse elements will not be found anymore). So you will have to decide here, if you want GStreamer or FFmpeg...
You can clean up the library configuration mismatch using these commands:
sudo apt remove
sudo apt remove libavcodec58
sudo apt remove libavutil56
sudo apt remove libavutil55
sudo apt autoremove

Afterwards (still from the ffmpeg folder where you have been building) you should again run:
sudo make install
sudo ldconfig

Before you can start the OpenCV build, you will have to set / update some aditional environment variables:
$ export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
$ export PATH=/usr/local/bin:$PATH
$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
$ export OPENCV_VERSION=4.5.3

Now you can build OpenCV with CUDA support and including the FFMPEG, we just built before. There are many build scripts out there (e.g. from JetsonHacks: https://github.com/JetsonHacksNano/buildOpenCV), here I give a brief step by step description. It is given for a Jetson Xavier NX and OpenCV 4.5.3. On a Jetson Nano you must replace 7.2 for the CUDA_ARCH_BIN version with 5.3
# Repository setup
sudo apt-add-repository universe
sudo apt update

# Download dependencies
# we will NOT download FFMPEG libraries because we built them from scratch just before:
#     libavcodec-dev \
#     libavformat-dev \
#     libavutil-dev \
#     libswscale-dev \

sudo apt-get install -y build-essential cmake curl python3-pip libeigen3-dev libglew-dev libgtk2.0-dev libgtk-3-dev libjpeg-dev libpng-dev libpostproc-dev libtbb-dev libtiff5-dev libv4l-dev v4l-utils libxvidcore-dev libx264-dev qt5-default zlib1g-dev libatlas-base-dev gfortran pkg-config

# Python 2.7
sudo apt-get install -y python-dev  python-numpy  python-py  python-pytest
# Python 3.6
sudo apt-get install -y python3-dev python3-numpy python3-py python3-pytest

# GStreamer support
sudo apt-get install -y libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev

# we install opencv in the user's home directory:
cd $HOME
curl -L https://github.com/opencv/opencv/archive/${OPENCV_VERSION}.zip -o opencv-4.5.3.zip
curl -L https://github.com/opencv/opencv_contrib/archive/${OPENCV_VERSION}.zip -o opencv_contrib-4.5.3.zip
unzip opencv-4.5.3.zip
unzip opencv_contrib-4.5.3.zip
# remove opencv folder that might have been created earlier
rm -r opencv/
mv opencv-4.5.3 opencv
# remove opencv-contrib folder that might have been created earlier
rm -r opencv_contrib/
mv opencv_contrib-4.5.3 opencv_contrib
rm opencv-4.5.3.zip
rm opencv_contrib-4.5.3.zip

# Go into the downloaded opencv dir and create the build directory and start cmake
cd opencv
mkdir build
cd build

# in the following cmake configuration command set CUDA_ARCH_BIN to 5.3 for Jetson Nano and 7.2 for Jetson Xavier NX
cmake -D CMAKE_BUILD_TYPE=RELEASE -D PYTHON_DEFAULT_EXECUTABLE=/usr/bin/python3 -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_CUDA=ON -D CUDA_ARCH_BIN=7.2 -D CUDA_ARCH_PTX="" -D ENABLE_FAST_MATH=ON -D CUDA_FAST_MATH=ON -D WITH_CUDNN=ON -D OPENCV_DNN_CUDA=ON -D WITH_CUBLAS=ON -D WITH_TBB=ON -D WITH_EIGEN=ON -D WITH_LIBV4L=ON -D WITH_V4L=ON -D WITH_FFMPEG=ON -D FFMPEG_PREFIX=/usr/local -D WITH_GSTREAMER=ON -D WITH_GSTREAMER_0_10=OFF -D OPENCV_ENABLE_NONFREE=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D BUILD_opencv_python3=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules -D CPACK_BINARY_DEB=ON ..

# make with all processors available. On a Jetson Nano you may use make -j1 instead to save memory
make -j$(nproc)
sudo make install
sudo ldconfig

# if you want to generate a package:
# sudo make package -j$(nproc)
Have fun!