Qt+GStreamer:如何在播放实时视频流时拍摄快照

我开发了一个基于Qt和QtGstreamer的视频播放器。它用于播放实时流（RTSP）。我必须为用户添加在播放实时流时拍摄快照的可能性，而不会干扰视频播放。

这是我制作的管道图：

                                -->queue-->autovideosink
uridecodebin-->videoflip-->tee--|
            |                   -->queue->videoconvert-->pngenc-->filesink
            |
            |->audioconvert-->autoaudiosink

我使用来自uridecodebin的pad-added信号将我的元素动态添加并链接到管道，这是接收上限的功能。

void Player::onPadAdded(const QGst::PadPtr &pad)
{
    QGst::CapsPtr caps = pad->currentCaps();
    if (caps->toString().startsWith("video/x-raw")) {
        qDebug("Received 'video/x-raw' caps");
        handleNewVideoPad(pad);
    }
    else if (caps->toString().startsWith("audio/x-raw")) {
        qDebug("Received 'audio/x-raw' caps");
        if (!m_audioEnabled) {
            qDebug("Audio is disabled in the player. Ignoring...");
            return;
        }
        handleNewAudioPad(pad);
    }
    else {
        qWarning("Unsuported caps, arborting ...!");
        return;
    }
}
[...]
void Player::handleNewVideoPad(QGst::PadPtr pad)
{
    m_player->videoTeeVideoSrcPad = m_player->videoTee->getRequestPad("src_%u");
    // Add video elements
    m_player->pipeline->add(m_player->videoFlip);
    m_player->pipeline->add(m_player->videoTee);
    m_player->pipeline->add(m_player->videoQueue);
    m_player->pipeline->add(m_player->videoSink);
    // Add snap elements
    m_player->pipeline->add(m_player->snapQueue);
    m_player->pipeline->add(m_player->snapConverter);
    m_player->pipeline->add(m_player->snapEncoder);
    m_player->pipeline->add(m_player->snapSink);
    // Link video elements
    m_player->videoFlip->link(m_player->videoTee);
    m_player->videoQueue->link(m_player->videoSink);
    // Link snap elements
    m_player->snapQueue->link(m_player->snapConverter);
    m_player->snapConverter->link(m_player->snapEncoder);
    m_player->snapEncoder->link(m_player->snapSink);
    // Lock snap elements
    m_player->snapQueue->setStateLocked(true);
    m_player->snapConverter->setStateLocked(true);
    m_player->snapEncoder->setStateLocked(true);
    m_player->snapSink->setStateLocked(true);
    m_player->videoFlip->setState(QGst::StatePlaying);
    m_player->videoTee->setState(QGst::StatePlaying);
    m_player->videoQueue->setState(QGst::StatePlaying);
    m_player->videoSink->setState(QGst::StatePlaying);
    // Link pads
    m_player->videoTeeVideoSrcPad->link(m_player->videoQueue->getStaticPad("sink"));
    pad->link(m_player->videoSinkPad);
    m_player->videoLinked = true;
}

拍摄快照的方法：

void Player::takeSnapshot()
{
    QDateTime dateTime = QDateTime::currentDateTime();
    QString snapLocation = QString("/%1/snap_%2.png").arg(m_snapDir).arg(dateTime.toString(Qt::ISODate));
    m_player->inSnapshotCaputre = true;
    if (m_player->videoTeeSnapSrcPad) {
        m_player->videoTee->releaseRequestPad(m_player->videoTeeSnapSrcPad);
        m_player->videoTeeSnapSrcPad.clear();
    }
    m_player->videoTeeSnapSrcPad = m_player->videoTee->getRequestPad("src_%u");
    // Stop the snapshot branch
    m_player->snapQueue->setState(QGst::StateNull);
    m_player->snapConverter->setState(QGst::StateNull);
    m_player->snapEncoder->setState(QGst::StateNull);
    m_player->snapSink->setState(QGst::StateNull);
    // Link Tee src pad to snap queue sink pad
    m_player->videoTeeSnapSrcPad->link(m_player->snapQueue->getStaticPad("sink"));
    // Set the snapshot location property
    m_player->snapSink->setProperty("location", snapLocation);
    // Unlock snapshot branch
    m_player->snapQueue->setStateLocked(false);
    m_player->snapConverter->setStateLocked(false);
    m_player->snapEncoder->setStateLocked(false);
    m_player->snapSink->setStateLocked(false);
    m_player->videoTeeSnapSrcPad->setActive(true);
    // Synch snapshot branch state with parent
    m_player->snapQueue->syncStateWithParent();
    m_player->snapConverter->syncStateWithParent();
    m_player->snapEncoder->syncStateWithParent();
    m_player->snapSink->syncStateWithParent();
}

总线消息回调：

void Player::onBusMessage(const QGst::MessagePtr & message)
{
    QGst::ElementPtr source = message->source().staticCast<QGst::Element>();
    switch (message->type()) {
    case QGst::MessageEos: { //End of stream. We reached the end of the file.
        qDebug("Message End Off Stream");
        if (m_player->inSnapshotCaputre) {
            blockSignals(true);
            pause();
            play();
            blockSignals(false);
            m_player->inSnapshotCaputre = false;
        }
        else {
            m_eos = true;
            stop();
        }
        break;
    }
    [...]
}

问题是：

当我将 snapshot 属性设置为 pngenc 元素的 true 时，我收到停止管道的 EOS 事件，因此我需要重新启动它，这会冻结视频播放大约半秒，这在我的情况下是不可接受的。
当我将 snapshot 属性设置为 pngenc 元素的 false 时，我没有管道扰动，但我的 png 文件不断增长，直到我再次调用 Player::takeSnapshot() 方法。

我错在哪里？有没有更好的方法可以做到这一点？我尝试为我的snapshot分支创建QGst::Bin元素失败。垫式探头呢？

提前致谢

可以在任何接收器（例如视频接收器）上获取最后一个示例属性。它包含一个 GstSample，其中包含一个包含最新视频帧的缓冲区。您可以将其作为快照，例如使用 gst_video_convert_sample（）或它的异步变体，将其转换为 PNG/JPG/其他任何东西。

查看 https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-libs/html/GstBaseSink.html#GstBaseSink--last-sample 和 https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-gstvideo.html#gst-video-convert-sample

或者，您必须在第一帧之后关闭 filesink 快照管道。例如，通过有一个焊盘探针来知道第一帧何时发生，然后注入EOS事件以防止将更多的PNG帧附加到同一文件中。

多亏了@sebastian-droge的回答，我找到了解决方案，使用gst_video_convert_sample和视频接收器的last-sample属性。

我实施的解决方案是：

void Player::takeSnapshot()
{
    QDateTime currentDate = QDateTime::currentDateTime();
    QString location = QString("%1/snap_%2.png").arg(QDir::homePath()).arg(currentDate.toString(Qt::ISODate));
    QImage snapShot;
    QImage::Format snapFormat;
    QGlib::Value val = m_videoSink->property("last-sample");
    GstSample *videoSample = (GstSample *)g_value_get_boxed(val);
    QGst::SamplePtr sample = QGst::SamplePtr::wrap(videoSample);
    QGst::SamplePtr convertedSample;
    QGst::BufferPtr buffer;
    QGst::CapsPtr caps = sample->caps();
    QGst::MapInfo mapInfo;
    GError *err = NULL;
    GstCaps * capsTo = NULL;
    const QGst::StructurePtr structure = caps->internalStructure(0);
    int width, height;
    width = structure.data()->value("width").get<int>();
    height = structure.data()->value("height").get<int>();
    qDebug() << "Sample caps:" << structure.data()->toString();
    /*
     * { QImage::Format_RGBX8888, GST_VIDEO_FORMAT_RGBx  },
     * { QImage::Format_RGBA8888, GST_VIDEO_FORMAT_RGBA  },
     * { QImage::Format_RGB888  , GST_VIDEO_FORMAT_RGB   },
     * { QImage::Format_RGB16   , GST_VIDEO_FORMAT_RGB16 }
     */
    snapFormat = QImage::Format_RGB888;
    capsTo = gst_caps_new_simple("video/x-raw",
                                 "format", G_TYPE_STRING, "RGB",
                                 "width", G_TYPE_INT, width,
                                 "height", G_TYPE_INT, height,
                                 NULL);
    convertedSample = QGst::SamplePtr::wrap(gst_video_convert_sample(videoSample, capsTo, GST_SECOND, &err));
    if (convertedSample.isNull()) {
        qWarning() << "gst_video_convert_sample Failed:" << err->message;
    }
    else {
        qDebug() << "Converted sample caps:" << convertedSample->caps()->toString();
        buffer = convertedSample->buffer();
        buffer->map(mapInfo, QGst::MapRead);
        snapShot = QImage((const uchar *)mapInfo.data(),
                          width,
                          height,
                          snapFormat);
        qDebug() << "Saving snap to" << location;
        snapShot.save(location);
        buffer->unmap(mapInfo);
    }
    val.clear();
    sample.clear();
    convertedSample.clear();
    buffer.clear();
    caps.clear();
    g_clear_error(&err);
    if (capsTo)
        gst_caps_unref(capsTo);
}

我创建了一个简单的测试应用程序，用于实现此解决方案。该代码在我的 Github 上可用

相关内容

最新更新

热门标签：