使用 ML 工具包(使用 CMSampleBuffer)从实时视频流中识别文本

Question

我正在尝试修改Google在此处提供的设备上文本识别示例，以使其与实时摄像机馈送一起使用。

当将相机放在文本上(适用于图像示例)时，我的控制台会在流中生成以下内容，然后最终耗尽内存：

2018-05-16 10:48:22.129901+1200 TextRecognition[32138:5593533] An empty result returned from from GMVDetector for VisionTextDetector.

这是我的视频捕获方法：

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
if let textDetector = self.textDetector {
let visionImage = VisionImage(buffer: sampleBuffer)
let metadata = VisionImageMetadata()
metadata.orientation = .rightTop
visionImage.metadata = metadata
textDetector.detect(in: visionImage) { (features, error) in
guard error == nil, let features = features, !features.isEmpty else {
// Error. You should also check the console for error messages.
// ...
return
}
// Recognized and extracted text
print("Detected text has: (features.count) blocks")
// ...
}
}
}

这是正确的方法吗？

Answer 1

ML Kit 早已从 Firebase 迁移而出，成为独立的 SDK(迁移指南)。

Swift 中的快速入门示例应用程序展示了如何使用 ML Kit(带有 CMSampleBuffer)从实时视频流中识别文本，现在可在此处获得：

https://github.com/googlesamples/mlkit/tree/master/ios/quickstarts/textrecognition/TextRecognitionExample

实时馈送在 CameraViewController 中实现.swift：

https://github.com/googlesamples/mlkit/blob/master/ios/quickstarts/textrecognition/TextRecognitionExample/CameraViewController.swift

Answer 2

ML Kit 仍在向 Firebase Quick Start 添加使用 CMSampleBuffer 的示例代码。

同时，下面的代码适用于CMSampleBuffer。

设置 AV Capture (使用 kCVPixelFormatType_32BGRA for kCVPixelBufferPixelFormatTypeKey)：

@property(nonatomic, strong) AVCaptureSession *session;
@property(nonatomic, strong) AVCaptureVideoDataOutput *videoDataOutput;
- (void)setupVideoProcessing {
self.videoDataOutput = [[AVCaptureVideoDataOutput alloc] init];
NSDictionary *rgbOutputSettings = @{
(__bridge NSString*)kCVPixelBufferPixelFormatTypeKey :  @(kCVPixelFormatType_32BGRA)
};
[self.videoDataOutput setVideoSettings:rgbOutputSettings];
if (![self.session canAddOutput:self.videoDataOutput]) {
[self cleanupVideoProcessing];
NSLog(@"Failed to setup video output");
return;
}
[self.videoDataOutput setAlwaysDiscardsLateVideoFrames:YES];
[self.videoDataOutput setSampleBufferDelegate:self queue:self.videoDataOutputQueue];
[self.session addOutput:self.videoDataOutput];
}

使用 CMSampleBuffer 并运行检测：

- (void)runDetection:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection {
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
size_t imageWidth = CVPixelBufferGetWidth(imageBuffer);
size_t imageHeight = CVPixelBufferGetHeight(imageBuffer);
AVCaptureDevicePosition devicePosition = self.isUsingFrontCamera ? AVCaptureDevicePositionFront : AVCaptureDevicePositionBack;
// Calculate the image orientation.
UIDeviceOrientation deviceOrientation = [[UIDevice currentDevice] orientation];
ImageOrientation orientation =
[ImageUtility imageOrientationFromOrientation:deviceOrientation
withCaptureDevicePosition:devicePosition
defaultDeviceOrientation:[self deviceOrientationFromInterfaceOrientation]];
// Invoke text detection.
FIRVisionImage *image = [[FIRVisionImage alloc] initWithBuffer:sampleBuffer];
FIRVisionImageMetadata *metadata = [[FIRVisionImageMetadata alloc] init];
metadata.orientation = orientation;
image.metadata = metadata;
FIRVisionTextDetectionCallback callback =
^(NSArray<id<FIRVisionText>> *_Nullable features, NSError *_Nullable error) {
...
};
[self.textDetector detectInImage:image completion:callback];
}

上面用于确定方向的 ImageUtility 的辅助函数：

+ (FIRVisionDetectorImageOrientation)imageOrientationFromOrientation:(UIDeviceOrientation)deviceOrientation
withCaptureDevicePosition:(AVCaptureDevicePosition)position
defaultDeviceOrientation:(UIDeviceOrientation)defaultOrientation {
if (deviceOrientation == UIDeviceOrientationFaceDown ||
deviceOrientation == UIDeviceOrientationFaceUp ||
deviceOrientation == UIDeviceOrientationUnknown) {
deviceOrientation = defaultOrientation;
}
FIRVisionDetectorImageOrientation orientation = FIRVisionDetectorImageOrientationTopLeft;
switch (deviceOrientation) {
case UIDeviceOrientationPortrait:
if (position == AVCaptureDevicePositionFront) {
orientation = FIRVisionDetectorImageOrientationLeftTop;
} else {
orientation = FIRVisionDetectorImageOrientationRightTop;
}
break;
case UIDeviceOrientationLandscapeLeft:
if (position == AVCaptureDevicePositionFront) {
orientation = FIRVisionDetectorImageOrientationBottomLeft;
} else {
orientation = FIRVisionDetectorImageOrientationTopLeft;
}
break;
case UIDeviceOrientationPortraitUpsideDown:
if (position == AVCaptureDevicePositionFront) {
orientation = FIRVisionDetectorImageOrientationRightBottom;
} else {
orientation = FIRVisionDetectorImageOrientationLeftBottom;
}
break;
case UIDeviceOrientationLandscapeRight:
if (position == AVCaptureDevicePositionFront) {
orientation = FIRVisionDetectorImageOrientationTopRight;
} else {
orientation = FIRVisionDetectorImageOrientationBottomRight;
}
break;
default:
orientation = FIRVisionDetectorImageOrientationTopLeft;
break;
}
return orientation;
}