我正在处理一个通过USB输出原始h.264视频流的实时视频源(无人机无线视频接收器(。我的目标是将它集成到Android中的QGroundStation中，该站点有一个GStreamer管道。
我已经将接收到的USB数据的一部分转储到一个文件中，该文件可以使用以下命令与vlc一起完美播放：
vlc -c dump.bin --demux h264
然而，如果我使用这个GStreamer管道播放，播放速度太高(比如x10(
gst-launch-1.0.exe filesrc location="dump.bin" ! h264parse ! avdec_h264 ! autovideosink
我正在使用appsrc将USB数据推送到QGroundControl管道中。视频播放，但许多帧被丢弃，gstreamer抱怨由于帧太晚而丢弃了数据包。
[...]
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.404134207)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.403025291)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.401385832)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.400435290)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.399607540)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.398911040)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.398131998)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.397308623)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.396620290)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.395761040)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.395125498)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.394197123)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.393461831)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.392803831)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.391983373)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.391033998)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.389664914)
W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.388862831)
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 38639 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 37460 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 46566 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 36055 will be dropped
W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 43397 will be dropped
[...]
在仔细检查了我的转储后，我意识到流缺少pts和dts信息(这在基线h.264流中似乎很常见(
ffprobe -show_frames dump.bin
[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=48000
duration_time=0.040000
convergence_duration=N/A
convergence_duration_time=N/A
size=1843
pos=0
flags=K_
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=48000
duration_time=0.040000
convergence_duration=N/A
convergence_duration_time=N/A
size=16851
pos=1843
flags=K_
[/PACKET]
但显然，持续时间信息是存在的。
USB端点读取512字节的块(由于大容量端点的USB高速最大有效负载大小(，并且一些传输较小(400+字节长(。我没有办法检测NAL的开始/结束，因为它是一个不透明的连续字节流。(视频/x-h264，流格式=(字符串(字节流，对齐=无(
因此，我构建了一个appsrc来将视频流推送到管道，并尝试像这样盲目地给缓冲区加上时间戳：
void _startFeed(GstElement *source, guint size, gpointer pContext) {
(void)pContext;
GstBuffer *pBuffer;
GstFlowReturn ret;
GstMapInfo map;
guint8 *pRaw;
GstClock *pClk = gst_element_get_clock(gPipeline);
GstClockTime absolute = gst_clock_get_time(pClk);
GstClockTime base = gst_element_get_base_time(gPipeline);
gst_object_unref(pClk);
pBuffer = gst_buffer_new_and_alloc(bufferSize);
// Timestamp the buffers
GST_BUFFER_PTS(pBuffer) = absolute - base;
GST_BUFFER_DTS(pBuffer) = GST_BUFFER_PTS(pBuffer);
gst_buffer_map(pBuffer, &map, GST_MAP_WRITE);
pRaw = (guint8 *)map.data;
fifo_pull(pRaw, size); // This pulls up to 'size' bytes from the USB FIFO and copies it to pRaw
gst_buffer_unmap(pBuffer, &map);
g_signal_emit_by_name(gAppSrc, "push-buffer", pBuffer, &ret);
gst_buffer_unref(pBuffer);
}
但仍然没有运气。。。
我使用以下管道将h.264流编码为RTP有效载荷，然后使用指定目标帧速率的caps过滤器对其进行解码，这一成功有限：
gst-launch-1.0.exe filesrc location="dump.bin" ! video/x-h264, stream-format=(string)byte-stream, alignment=none ! h264parse ! rtph264pay ! rtpjitterbuffer ! rtph264depay ! video/x-h264, stream-format=byte-stream, alignment=nal, framerate=(fraction)30/1 ! h264parse ! avdec_h264 ! autovideosink
我可以在C++中将其构建到QGroundControl中，但我认为这不是正确的方法，我不应该对目标帧速率做出任何假设，因为在这种情况下，它是30帧/秒，但它可能会动态变化。
所以，我的问题是：
- 在没有任何掉帧的情况下，以正确的速度播放视频的正确方法是什么
- 要求GStreamer使用标准流水线根据数据包的持续时间信息生成PTS/DTS(没有B帧，所以PTS应该等于DTS(是否合理或可能
更新：：
- 我尝试了一种已知的从DTS插入PTS的解决方法+这里描述的持续时间，但在我的情况下，既不是PTS也不是DTS在流中可用，所以没有任何区别
- 此特定视频转储的正确帧速率应为29.97 fps(NTSC(。我在视频图像中有一个运行计数器来验证它
- 正如你在我的ffprobe输出中看到的，duration_time信息显示为0.04秒，相当于25帧/秒，这让我很困惑
- 令人惊讶的是，VLC以正确的速度播放视频，并正确猜测帧速率为29.97 fps(根据按Ctrl+J得到的视频编解码器信息窗口(
- 我注意到rtph264pay会生成丢失的时间戳，但猜测的帧速率是25帧/秒，正如数据包的duration_time字段所示
我使用了一个h.264流分析器，并意识到这个特定流在SPS NAL中没有VUI信息(因此没有分析器可以用来计算PTS的time_tick或time_scale信息(。
对我起作用的是；sync"=false，以便在帧到达时立即渲染帧。不理想，但我可以接受。