Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Get video udp h264 with gstreamer and opencv
/**
* Based on:
* https://stackoverflow.com/questions/10403588/adding-opencv-processing-to-gstreamer-application
*/
// Include atomic std library
#include <atomic>
// Include gstreamer library
#include <gst/gst.h>
#include <gst/app/app.h>
// Include OpenCV library
#include <opencv.hpp>
// Share frame between main loop and gstreamer callback
std::atomic<cv::Mat*> atomicFrame;
/**
* @brief Check preroll to get a new frame using callback
* https://gstreamer.freedesktop.org/documentation/design/preroll.html
* @return GstFlowReturn
*/
GstFlowReturn new_preroll(GstAppSink* /*appsink*/, gpointer /*data*/)
{
return GST_FLOW_OK;
}
/**
* @brief This is a callback that get a new frame when a preroll exist
*
* @param appsink
* @return GstFlowReturn
*/
GstFlowReturn new_sample(GstAppSink *appsink, gpointer /*data*/)
{
static int framecount = 0;
// Get caps and frame
GstSample *sample = gst_app_sink_pull_sample(appsink);
GstCaps *caps = gst_sample_get_caps(sample);
GstBuffer *buffer = gst_sample_get_buffer(sample);
GstStructure *structure = gst_caps_get_structure(caps, 0);
const int width = g_value_get_int(gst_structure_get_value(structure, "width"));
const int height = g_value_get_int(gst_structure_get_value(structure, "height"));
// Print dot every 30 frames
if(!(framecount%30)) {
g_print(".");
}
// Show caps on first frame
if(!framecount) {
g_print("caps: %s\n", gst_caps_to_string(caps));
}
framecount++;
// Get frame data
GstMapInfo map;
gst_buffer_map(buffer, &map, GST_MAP_READ);
// Convert gstreamer data to OpenCV Mat
cv::Mat* prevFrame;
prevFrame = atomicFrame.exchange(new cv::Mat(cv::Size(width, height), CV_8UC3, (char*)map.data, cv::Mat::AUTO_STEP));
if(prevFrame) {
delete prevFrame;
}
gst_buffer_unmap(buffer, &map);
gst_sample_unref(sample);
return GST_FLOW_OK;
}
/**
* @brief Bus callback
* Print important messages
*
* @param bus
* @param message
* @param data
* @return gboolean
*/
static gboolean my_bus_callback(GstBus *bus, GstMessage *message, gpointer data)
{
// Debug message
//g_print("Got %s message\n", GST_MESSAGE_TYPE_NAME(message));
switch(GST_MESSAGE_TYPE(message)) {
case GST_MESSAGE_ERROR: {
GError *err;
gchar *debug;
gst_message_parse_error(message, &err, &debug);
g_print("Error: %s\n", err->message);
g_error_free(err);
g_free(debug);
break;
}
case GST_MESSAGE_EOS:
/* end-of-stream */
break;
default:
/* unhandled message */
break;
}
/* we want to be notified again the next time there is a message
* on the bus, so returning TRUE (FALSE means we want to stop watching
* for messages on the bus and our callback should not be called again)
*/
return true;
}
int main(int argc, char *argv[]) {
gst_init(&argc, &argv);
gchar *descr = g_strdup(
"udpsrc port=5600 "
"! application/x-rtp, payload=96 ! rtph264depay ! h264parse ! avdec_h264 "
"! decodebin ! videoconvert ! video/x-raw,format=(string)BGR ! videoconvert "
"! appsink name=sink emit-signals=true sync=false max-buffers=1 drop=true"
);
// Check pipeline
GError *error = nullptr;
GstElement *pipeline = gst_parse_launch(descr, &error);
if(error) {
g_print("could not construct pipeline: %s\n", error->message);
g_error_free(error);
exit(-1);
}
// Get sink
GstElement *sink = gst_bin_get_by_name(GST_BIN(pipeline), "sink");
/**
* @brief Get sink signals and check for a preroll
* If preroll exists, we do have a new frame
*/
gst_app_sink_set_emit_signals((GstAppSink*)sink, true);
gst_app_sink_set_drop((GstAppSink*)sink, true);
gst_app_sink_set_max_buffers((GstAppSink*)sink, 1);
GstAppSinkCallbacks callbacks = { nullptr, new_preroll, new_sample };
gst_app_sink_set_callbacks(GST_APP_SINK(sink), &callbacks, nullptr, nullptr);
// Declare bus
GstBus *bus;
bus = gst_pipeline_get_bus(GST_PIPELINE(pipeline));
gst_bus_add_watch(bus, my_bus_callback, nullptr);
gst_object_unref(bus);
gst_element_set_state(GST_ELEMENT(pipeline), GST_STATE_PLAYING);
// Main loop
while(1) {
g_main_iteration(false);
cv::Mat* frame = atomicFrame.load();
if(frame) {
cv::imshow("Frame", atomicFrame.load()[0]);
cv::waitKey(30);
}
}
gst_element_set_state(GST_ELEMENT(pipeline), GST_STATE_NULL);
gst_object_unref(GST_OBJECT(pipeline));
return 0;
}
@pdeman

This comment has been minimized.

Copy link

@pdeman pdeman commented Jul 3, 2020

I am trying your code, but when I run it I get "no element "udpsrc". I tried adding gstudp.dll in the same folder as my .exe but i get the same. for some reason it doesn't link with it

@patrickelectric

This comment has been minimized.

Copy link
Owner Author

@patrickelectric patrickelectric commented Jul 3, 2020

@pdeman make sure that you have all necessary gst plugins installed, you can see if it's installed or accessible with gst-inspect, E.g: gst-inspect-1.0 udpsrc

@pdeman

This comment has been minimized.

Copy link

@pdeman pdeman commented Jul 3, 2020

using gst-launch.exe etc ... everything works. but not using c++ sdk (on windows 10, visual studio 2017).

here is the output of gst-inspect

WARNING: no real random source present!
Factory Details:
Rank none (0)
Long-name UDP packet receiver
Klass Source/Network
Description Receive data over the network via UDP
Author Wim Taymans wim@fluendo.com, Thijs Vermeir thijs.vermeir@barco.com

Plugin Details:
Name udp
Description transfer data via UDP
Filename G:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\gstudp.dll
Version 1.16.2
License LGPL
Source module gst-plugins-good
Binary package GStreamer Good Plug-ins source release
Origin URL Unknown package origin

GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSrc
+----GstPushSrc
+----GstUDPSrc

Implemented Interfaces:
GstURIHandler

Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
ANY

Element has no clocking capabilities.

URI handling capabilities:
Element can act as source.
Supported URI protocols:
udp

Pads:
SRC: 'src'
Pad Template: 'src'

Element Properties:
address : Address to receive packets for. This is equivalent to the multicast-group property for now
flags: readable, writable
String. Default: "0.0.0.0"
auto-multicast : Automatically join/leave multicast groups
flags: readable, writable
Boolean. Default: true
blocksize : Size in bytes to read per buffer (-1 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
buffer-size : Size of the kernel receive buffer in bytes, 0=default
flags: readable, writable
Integer. Range: 0 - 2147483647 Default: 0
caps : The caps of the source pad
flags: readable, writable
Caps (NULL)
close-socket : Close socket if passed as property on state change
flags: readable, writable
Boolean. Default: true
do-timestamp : Apply current stream time to buffers
flags: readable, writable
Boolean. Default: true
loop : Used for setting the multicast loop parameter. TRUE = enable, FALSE = disable
flags: readable, writable
Boolean. Default: true
mtu : Maximum expected packet size. This directly defines the allocationsize of the receive buffer pool.
flags: readable, writable
Unsigned Integer. Range: 0 - 2147483647 Default: 1492
multicast-group : The Address of multicast group to join. (DEPRECATED: Use address property instead)
flags: readable, writable, deprecated
String. Default: "0.0.0.0"
multicast-iface : The network interface on which to join the multicast group.This allows multiple interfaces seperated by comma. ("eth0,eth1")
flags: readable, writable
String. Default: null
name : The name of the object
flags: readable, writable
String. Default: "udpsrc0"
num-buffers : Number of buffers to output before sending EOS (-1 = unlimited)
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
port : The port to receive the packets from, 0=allocate
flags: readable, writable
Integer. Range: 0 - 65535 Default: 5004
retrieve-sender-address: Whether to retrieve the sender address and add it to buffers as meta. Disabling this might result in minor performance improvements in certain scenarios
flags: readable, writable
Boolean. Default: true
reuse : Enable reuse of the port
flags: readable, writable
Boolean. Default: true
skip-first-bytes : number of bytes to skip for each udp packet
flags: readable, writable
Integer. Range: 0 - 2147483647 Default: 0
socket : Socket to use for UDP reception. (NULL == allocate)
flags: readable, writable
Object of type "GSocket"
timeout : Post a message after timeout nanoseconds (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
typefind : Run typefind before negotiating (deprecated, non-functional)
flags: readable, writable, deprecated
Boolean. Default: false
uri : URI in the form of udp://multicast_group:port
flags: readable, writable
String. Default: "udp://0.0.0.0:5004"
used-socket : Socket currently in use for UDP reception. (NULL = no socket)
flags: readable
Object of type "GSocket"

@patrickelectric

This comment has been minimized.

Copy link
Owner Author

@patrickelectric patrickelectric commented Jul 3, 2020

Hi @pdeman,

The last tip that I can give, since I'm not a windows developer, is to make sure that you have all plugins accessible under GST_PLUGIN_PATH environment variable, If that does not help, I would recommend to seek suggestions in the official gstreamer communication channel for windows.

@pdeman

This comment has been minimized.

Copy link

@pdeman pdeman commented Jul 3, 2020

GST_PLUGIN_PATH is set to 👍 G:\gstreamer\1.0\x86_64\lib
where I have
G:.
├───gio
│ └───modules
├───glib-2.0
│ └───include
├───graphene-1.0
│ └───include
├───gst-validate-launcher
│ └───python
│ └───launcher
│ ├───apps
│ └───testsuites
├───gstreamer-1.0
│ └───include
│ └───gst
│ └───gl
└───pkgconfig
etc ...
if I use

udpfactory = gst_element_factory_find("udpsrc");
g_return_if_fail(udpfactory != NULL);

udp = gst_element_factory_make("udpsrc", "udp");

I get the following error:

(TestGstreamerCpp.exe:11028): GStreamer-WARNING **: 16:14:00.989: Failed to load plugin 'G:\gstreamer\1.0\x86_64\lib\gio\modules\giognutls.dll': 'G:\gstreamer\1.0\x86_64\lib\gio\modules\giognutls.dll': The specified module could not be found.

(TestGstreamerCpp.exe:11028): GStreamer-WARNING **: 16:17:26.514: Failed to load plugin 'G:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\gstudp.dll': 'G:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\gstudp.dll': The specified module could not be found.

while the dll are exactly at these locations.

@patrickelectric

This comment has been minimized.

Copy link
Owner Author

@patrickelectric patrickelectric commented Jul 3, 2020

Maybe you are compiling with the 32 bits and linked with the 64 bits or vice-versa ?

@pdeman

This comment has been minimized.

Copy link

@pdeman pdeman commented Jul 3, 2020

no I checked. I just uninstall and reinstall gstreamer. and I can't even compile in 32bits.

@pdeman

This comment has been minimized.

Copy link

@pdeman pdeman commented Jul 6, 2020

seems to work now. I added another env variable.
I have now gst_pluging_path, gstreamer_dir, gstreamer_1_0_ROOT_X86_64 and in PATH I added the path of gstreamer bin. quite painful this need to add env variables.

@pdeman

This comment has been minimized.

Copy link

@pdeman pdeman commented Jul 7, 2020

any chance you have a similar code to make an udp sink for opencv frame ?

@pdeman

This comment has been minimized.

Copy link

@pdeman pdeman commented Jul 7, 2020

in your code, I don't understand the buffer.
you set

    gst_app_sink_set_drop((GstAppSink*)sink, true);
    gst_app_sink_set_max_buffers((GstAppSink*)sink, 1);

so I thought that there was no buffer. and if I put a delay at reception (let's say a sleep of 5s), I would lose 5s of frame.

but actually not.
it waits 5 seconds, but I don't lose any frame. it seems there still is a buffer.

@patrickelectric

This comment has been minimized.

Copy link
Owner Author

@patrickelectric patrickelectric commented Jul 7, 2020

It's just a pipeline configuration, this part of code only runs once, this only enables drop frames and creates a buffer of 1 single frame.

@patrickelectric

This comment has been minimized.

Copy link
Owner Author

@patrickelectric patrickelectric commented Jul 7, 2020

any chance you have a similar code to make an udp sink for opencv frame ?

Can you explain what you want to do ?

@tranducanhbk

This comment has been minimized.

Copy link

@tranducanhbk tranducanhbk commented Jul 8, 2020

Dear Patrickelectric,

I run your source code but no frame appears in window.
And please tell me how use source with rtps stream with link like: rtsp://192.168.1.1:554/stream
I am looked forward to your reply
Many thanks

@pdeman

This comment has been minimized.

Copy link

@pdeman pdeman commented Jul 8, 2020

It's just a pipeline configuration, this part of code only runs once, this only enables drop frames and creates a buffer of 1 single frame.

I know, but then if I have a buffer of 1 single frame and drops frames. why does it "bufferize" ? I mean that if I slow down the computing time to make the program not able to deal with frames at the same rates as the video. like a video at 15 fps and I put a "sleep" in new_sample:

GstFlowReturn new_sample(GstAppSink *appsink, gpointer /*data*/)
{
	static int framecount = 0;
	std::this_thread::sleep_for(std::chrono::milliseconds(2000));

in main I did this change in the while true loop:

	cv::Mat* predFrame = NULL;
	while (1) {
		g_main_iteration(false);

		cv::Mat* frame = atomicFrame.load();
		 
		if (frame) {
			if (predFrame != NULL)
			{
				if (cv::countNonZero(predFrame - frame)!=0)
				{
					std::cout << "we have a frame 2" << std::endl;
					cv::imwrite("G:\\gstreamer\\testWrite\\frame" + std::to_string(i) + ".jpg", *frame);
					i++;
					predFrame = frame;
				}

			}

I would have expect that it doesn't write every frame of the video, but instead one frame every 2 secondes of the video.

but it writes every frame of the video, and if I stop the video, it continue writing until it empty a queue/buffer.
(here is the command line I am using to send the video received by video_udp.cpp : gst-launch-1.0.exe -v filesrc location=G:\gstreamer\Gravity.mp4 ! decodebin ! videoconvert ! openh264enc ! rtph264pay name=pay0 pt=96 config-interval=1 ! udpsink host=10.231.220.199 port=5000)
my goal is a real time "application", so I prefer to lose frames that having a delay

@patrickelectric

This comment has been minimized.

Copy link
Owner Author

@patrickelectric patrickelectric commented Jul 8, 2020

For my application it was necessary to have a single frame, the buffer was not important for me, this is just a minimal example and for different usages the code may be changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment