我使用Open cv在Jetson Nano上解码h.264。
我使用这个代码：
import cv2
try:
cap = cv2.VideoCapture('udp://234.0.0.0:46002', cv2.CAP_FFMPEG)
print(f"cap = {cap}")
except Exception as e:
print(f"Error: {e}")
if not cap.isOpened():
print('VideoCapture not opened')
exit(-1)
while True:
ret, frame = cap.read()
# print(f"frame = {frame}")
try:
cv2.imshow('Image', frame)
except Exception as e:
print(e)
if cv2.waitKey(1) & 0XFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
一切都很好。现在我不会试图通过使用GPU解码来优化我的代码，我的问题是我该怎么做？我看到这个选项：
cap = cv2.VideoCapture('filesrc location=sample2.mp4 ! qtdemux ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw,format=BGRx ! queue ! videoconvert ! queue ! video/x-raw, format=BGR ! appsink', cv2.CAP_GSTREAMER)
但我的来源是URL。
我很乐意提供任何关于如何使用GPU从python中的URL解码h.264的帮助。
我在电脑上使用FFmpeg命令来获取有关视频的信息，我得到了这个图：
ffmpeg -hide_banner -loglevel debug -i udp://127.0.0.0:46002 -f xv display
Splitting the commandline.
Reading option '-hide_banner' ... matched as option 'hide_banner' (do not
show program banner) with argument '1'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging
level) with argument 'debug'.
Reading option '-i' ... matched as input url with argument
'udp://127.0.0.0:46002'.
Reading option '-f' ... matched as option 'f' (force format) with argument
'xv'.
Reading option 'display' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option hide_banner (do not show program banner) with argument 1.
Applying option loglevel (set logging level) with argument debug.
Successfully parsed a group of options.
Parsing a group of options: input url udp://127.0.0.0:46002.
Successfully parsed a group of options.
Opening an input file: udp://127.0.0.0:46002.
[NULL @ 0000020a7c5ded80] Opening 'udp://127.0.0.0:46002' for reading
[udp @ 0000020a7c5cb700] No default whitelist set
[udp @ 0000020a7c5cb700] end receive buffer size reported is 393216
[h264 @ 0000020a7c5ded80] Format h264 probed with size=32768 and score=51
[h264 @ 0000020a7c5ded80] Before avformat_find_stream_info() pos: 0 bytes
read:33339 seeks:0 nb_streams:1
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[extract_extradata @ 0000020a7c60eec0] nal_unit_type: 1(Coded slice of a
non-IDR picture), nal_ref_idc: 2
Last message repeated 1 times
[h264 @ 0000020a7c631340] nal_unit_type: 1(Coded slice of a non-IDR
picture), nal_ref_idc: 2
Last message repeated 1 times
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[h264 @ 0000020a7c631340] decode_slice_header error
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[h264 @ 0000020a7c631340] decode_slice_header error
[h264 @ 0000020a7c631340] no frame!
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[extract_extradata @ 0000020a7c60eec0] nal_unit_type: 1(Coded slice of a
non-IDR picture), nal_ref_idc: 2
Last message repeated 1 times
[h264 @ 0000020a7c631340] nal_unit_type: 1(Coded slice of a non-IDR
picture), nal_ref_idc: 2
Last message repeated 1 times
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[h264 @ 0000020a7c631340] decode_slice_header error
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[h264 @ 0000020a7c631340] decode_slice_header error
[h264 @ 0000020a7c631340] no frame!
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[extract_extradata @ 0000020a7c60eec0] nal_unit_type: 1(Coded slice of a
non-IDR picture), nal_ref_idc: 2
Last message repeated 1 times
[h264 @ 0000020a7c631340] nal_unit_type: 1(Coded slice of a non-IDR
picture), nal_ref_idc: 2
Last message repeated 1 times
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[h264 @ 0000020a7c631340] decode_slice_header error
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[h264 @ 0000020a7c631340] decode_slice_header error
[h264 @ 0000020a7c631340] no frame!
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[extract_extradata @ 0000020a7c60eec0] nal_unit_type: 1(Coded slice of a
non-IDR picture), nal_ref_idc: 2
Last message repeated 1 times
[h264 @ 0000020a7c631340] nal_unit_type: 1(Coded slice of a non-IDR
picture), nal_ref_idc: 2
Last message repeated 1 times
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[h264 @ 0000020a7c631340] decode_slice_header error
[h264 @ 0000020a7c631340] non-existing PPS 0 referenced
[h264 @ 0000020a7c631340] decode_slice_header error
[h264 @ 0000020a7c631340] no frame!
[extract_extradata @ 0000020a7c60eec0] nal_unit_type: 7(SPS), nal_ref_idc:3
[extract_extradata @ 0000020a7c60eec0] nal_unit_type: 8(PPS), nal_ref_idc:3
[extract_extradata @ 0000020a7c60eec0] nal_unit_type: 5(IDR), nal_ref_idc:3
Last message repeated 1 times
[h264 @ 0000020a7c631340] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0000020a7c631340] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0000020a7c631340] nal_unit_type: 5(IDR), nal_ref_idc: 3
Last message repeated 1 times
[h264 @ 0000020a7c631340] Format yuv420p chosen by get_format().
[h264 @ 0000020a7c631340] Reinit context to 720x576, pix_fmt: yuv420p
[h264 @ 0000020a7c631340] nal_unit_type: 1(Coded slice of a non-IDR
picture), nal_ref_idc: 2
Last message repeated 11 times
[h264 @ 0000020a7c5ded80] max_analyze_duration 5000000 reached at 5000000
microseconds st:0
[h264 @ 0000020a7c5ded80] After avformat_find_stream_info() pos: 971047
bytes read:971495 seeks:0 frames:128
Input #0, h264, from 'udp://127.0.0.0:46002':
Duration: N/A, bitrate: N/A
Stream #0:0, 128, 1/1200000: Video: h264 (Constrained Baseline), 1
reference frame, yuv420p(progressive, left), 720x576, 0/1, 25 fps, 25 tbr,
1200k tbn, 50 tbc
Successfully opened the file.
Parsing a group of options: output url display.
Applying option f (force format) with argument xv.
Successfully parsed a group of options.
Opening an output file: display.
[NULL @ 0000020a7ce73000] Requested output format 'xv' is not a suitable
output format
display: Invalid argument
[AVIOContext @ 0000020a7c610300] Statistics: 971495 bytes read, 0 seeks
您可以使用uridecodebin来解码各种类型的url、容器、协议和编解码器。
对于Jetson，uridecodebin为h264选择的解码器将是nvv4l2decoder，它不使用GPU，而是更好的专用硬件解码器NVDEC。
nvv4l2decoder以NV12格式输出到NVMM存储器，而opencv-appsink期望在系统存储器中使用BGR格式。因此，您将使用硬件转换器nvvidconv来转换和复制到系统内存中。不幸的是，nvvidconv不支持BGR格式，因此首先使用nvvidconv转换为支持的BGRx格式，最后使用CPU插件videoconvert进行BGRx->BGR转换，如：
pipeline='uridecodebin uri=rtsp://127.0.0.1:8554/test ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1'
cap = cv2.VideoCapture(pipeline, cv2.CAP_GSTREAMER)
这是一般的方式。
不过，对于某些流媒体协议来说，它可能并不那么简单。
对于RTP-H264/UDP，ffmpeg后端可能只能使用SDP文件。对于gstreamer后端，您将使用管道，例如：
pipeline='udpsrc port=46002 multicast-group=234.0.0.0 ! application/x-rtp,encoding-name=H264 ! rtpjitterbuffer latency=500 ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1'
由于您可以使用FFMPEG，我推测接收流使用的是RTP-MP2T。所以你可以试试：
# Using NVDEC, but this may fail depending on sender side's codec:
cap = cv2.VideoCapture('udpsrc multicast-group=234.0.0.0 port=46002 ! application/x-rtp,media=video,encoding-name=MP2T,clock-rate=90000,payload=33 ! rtpjitterbuffer latency=300 ! rtpmp2tdepay ! tsdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1', cv2.CAP_GSTREAMER)
# Or using CPU (may not support high pixel rate with Nano):
cap = cv2.VideoCapture('udpsrc multicast-group=234.0.0.0 port=46002 ! application/x-rtp,media=video,encoding-name=MP2T,clock-rate=90000,payload=33 ! rtpjitterbuffer latency=300 ! rtpmp2tdepay ! tsdemux ! h264parse ! avdec_h264 ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1', cv2.CAP_GSTREAMER)
[注意，我不熟悉234.0.0.0，所以不确定是否应该像我一样使用多播组]。
如果这不起作用，您可以尝试获取有关接收流的更多信息。您可以尝试使用ffmpeg，例如：
ffmpeg -hide_banner -loglevel debug -i udp://234.0.0.0:46002 -f xv display
如果你看到：
Stream #0:0, 133, 1/1200000: Video: h264 (Constrained Baseline), 1 reference frame, yuv420p(progressive, left), 720x576, 0/1, 25 fps, 25 tbr, 1200k tbn, 50 tbc
您可能需要将时钟速率更改为1200000(默认值为90000(：
application/x-rtp,media=video,encoding-name=MP2T,clock-rate=1200000
这是假设流是mpeg2 ts。在这种情况下，第一行显示：
...
Opening an input file: udp://127.0.0.1:5002.
[NULL @ 0x55761c4690] Opening 'udp://127.0.0.1:5002' for reading
[udp @ 0x55761a27c0] No default whitelist set
[udp @ 0x55761a27c0] end receive buffer size reported is 131072
[mpegts @ 0x55761c4690] Format mpegts probed with size=2048 and score=47
[mpegts @ 0x55761c4690] stream=0 stream_type=1b pid=41 prog_reg_desc=HDMV
[mpegts @ 0x55761c4690] Before avformat_find_stream_info() pos: 0 bytes read:26560 seeks:0 nb_streams:1
...
ffmpeg试图猜测，在这里发现流是mpegts格式的。你可以在你的案例中查看ffmpeg的发现。请注意，第一个猜测可能不正确，您必须检查整个日志，看看它发现什么有效。
另一种推测是，您的流不是RTP，而是原始h264流。在这种情况下，你可以用类似的东西解码：
gst-launch-1.0 udpsrc port=46002 multicast-group=234.0.0.0 ! h264parse ! nvv4l2decoder ! autovideosink
如果这样做有效，对于opencv，您将使用：
pipeline='udpsrc port=46002 multicast-group=234.0.0.0 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1'