用Xugler解码音频/视频时音频断断续续

Question

所以我正在为一个在libGDX中工作的预先存在的视频解码器编写一个音频解码器。问题是，当音频代码没有线程化时，音频和视频会断断续续。音频会播放一个大块，然后视频会播放一块。

我的解决方案是做一些多线程处理，让视频的东西来完成它的工作（因为libGDX渲染线程不是线程安全的，破坏它们会导致糟糕的事情）。自然的选择是使用线程处理来进行音频处理。

这修复了视频的抖动，但不仅音频仍然抖动，而且到处都是伪影。

这是我第一次尝试严肃的音频编程，所以请记住，我可能不知道一些基本的东西。executor服务是一个SingleThreadExecutor，其思想是音频需要按顺序解码和写入。

以下是更新方法：

public boolean update(float dtSeconds) {
    if(playState != PlayState.PLAYING) return false;
    long dtMilliseconds = (long)(dtSeconds * 1000);
    playTimeMilliseconds += dtMilliseconds;
    sleepTimeoutMilliseconds = (long) Math.max(0, sleepTimeoutMilliseconds - dtMilliseconds);
    if(sleepTimeoutMilliseconds > 0) {
        // The playhead is still ahead of the current frame - do nothing
        return false;
    }

    while(true) {
        int packet_read_result = container.readNextPacket(packet);
        if(packet_read_result < 0) {
            // Got bad packet - we've reached end of the video stream
            stop();
            return true;
        }

        if(packet.getStreamIndex() == videoStreamId) 
        {
            // We have a valid packet from our stream
            // Allocate a new picture to get the data out of Xuggler
            IVideoPicture picture = IVideoPicture.make(
                videoCoder.getPixelType(),
                videoCoder.getWidth(),
                videoCoder.getHeight()
            );
            // Attempt to read the entire packet
            int offset = 0;
            while(offset < packet.getSize()) {
                // Decode the video, checking for any errors
                int bytesDecoded = videoCoder.decodeVideo(picture, packet, offset);
                if (bytesDecoded < 0) {
                    throw new RuntimeException("Got error decoding video");
                }
                offset += bytesDecoded;
                /* Some decoders will consume data in a packet, but will not
                 * be able to construct a full video picture yet. Therefore
                 * you should always check if you got a complete picture
                 * from the decoder
                 */
                if (picture.isComplete()) {
                    // We've read the entire packet
                    IVideoPicture newPic = picture;
                    // Timestamps are stored in microseconds - convert to milli
                    long absoluteFrameTimestampMilliseconds = picture.getTimeStamp() / 1000;
                    long relativeFrameTimestampMilliseconds = (absoluteFrameTimestampMilliseconds - firstTimestampMilliseconds);
                    long frameTimeDelta = relativeFrameTimestampMilliseconds - playTimeMilliseconds;
                    if(frameTimeDelta > 0) {
                        // The video is ahead of the playhead, don't read any more frames until it catches up
                        sleepTimeoutMilliseconds = frameTimeDelta + sleepTolleranceMilliseconds;
                        return false;
                    }
                    /* If the resampler is not null, that means we didn't get the video in
                     * BGR24 format and need to convert it into BGR24 format
                     */
                    if (resampler != null) {
                        // Resample the frame
                        newPic = IVideoPicture.make(
                            resampler.getOutputPixelFormat(),
                            picture.getWidth(), picture.getHeight()
                        );
                        if (resampler.resample(newPic, picture) < 0) {
                            throw new RuntimeException("Could not resample video");
                        }
                    }
                    if (newPic.getPixelType() != IPixelFormat.Type.BGR24) {
                        throw new RuntimeException("Could not decode video" + " as BGR 24 bit data");
                    }
                    // And finally, convert the BGR24 to an Java buffered image
                    BufferedImage javaImage = Utils.videoPictureToImage(newPic);
                    // Update the current texture
                    updateTexture(javaImage);
                    // Let the caller know the texture has changed
                    return true;
                }
            }
        }
        else if(packet.getStreamIndex() == this.audioStreamId)
        {
            IAudioSamples samples = IAudioSamples.make(1024, audioCoder.getChannels());
            Thread thread = new Thread(new DecodeSoundRunnable(samples));
            thread.setPriority(Thread.MAX_PRIORITY);
            this.decodeThreadPool.execute(thread);
        }
    }

这是音频线程：

private class DecodeSoundRunnable implements Runnable
    {
        IAudioSamples samples;
        int offset = 0;
        IStreamCoder coder;
        public DecodeSoundRunnable(IAudioSamples samples)
        {
            this.samples = samples.copyReference();
            this.coder = audioCoder.copyReference();
        }
        @Override
        public void run() {
            while(offset < packet.getSize())
            {
                 int bytesDecoded = this.coder.decodeAudio(samples, packet, offset);
                 if (bytesDecoded < 0)
                    break;//throw new RuntimeException("got error decoding audio in: " + videoPath);
                 offset += bytesDecoded;                  
            }
            playJavaSound(samples, 0);
            //writeOutThreadPool.execute(new WriteOutSoundRunnable(samples, 0));
        }
    }

Answer 1

通过创建一个专用线程来解决这个问题，该线程仅写入音频数据。这是因为mLine.write（byte[]字节）在写入数据时会阻塞。

private class WriteOutSoundBytes implements Runnable
    {
        byte[] rawByte;
        public WriteOutSoundBytes(byte[] rawBytes)
        {
            rawByte = rawBytes;
        }
        @Override
        public void run() 
        {
            mLine.write(rawByte, 0, rawByte.length);
        }

    }