MediaFoundation将四个摄像机整合到一个框架中

Question

我正在尝试使用MediaFoundation获取HoloLens 2的深度摄像头。然后，此线程与UWP和MediaFoundation相关。

使用此初始化 C# 代码(我有一些C++代码来处理 MediaFoundation 管道(，我能够获得以下捕获的图像，我使用 gimp 对其进行修改以强调低值(即粘贴的图像更亮(。

/// <summary>
/// Asynchronously create a Hand Detector with the first depth camera found
/// </summary>
/// <param name="id">The ID of the device to look for if needed. If NULL, a device with "depth" capabilities will be randomly choose.</param>
/// <returns>The asynchronous task</returns>
public static async Task<HandDetector> CreateAsync(String id=null)
{
Debug.WriteLine("Initialize the hand detector");
//Search for the correct media frame source
MediaFrameSourceGroup selectedFrameSourceGroup = null;
MediaFrameSourceInfo  selectedFrameSourceInfo  = null;
IReadOnlyList<MediaFrameSourceGroup> allFrameSourceGroups = await MediaFrameSourceGroup.FindAllAsync();
Debug.WriteLine($"Found {allFrameSourceGroups.Count} frame sources...");
foreach (MediaFrameSourceGroup group in allFrameSourceGroups)
{
Debug.WriteLine($"Group: {group.DisplayName}");
Debug.WriteLine($"Found {group.SourceInfos.Count} source infos...");
foreach(MediaFrameSourceInfo info in group.SourceInfos)
{
//Debug.WriteLine($"{info.SourceKind} : {info.MediaStreamType} -> {info.DeviceInformation.EnclosureLocation.Panel}");
//If an ID is given
if ((id == null || info.DeviceInformation.Id == id) && (info.MediaStreamType == MediaStreamType.VideoPreview || info.MediaStreamType == MediaStreamType.VideoRecord))
{
//Check the depth capabilities
if (info.SourceKind == MediaFrameSourceKind.Depth)
{                            
selectedFrameSourceGroup = group;
selectedFrameSourceInfo = info;
Debug.WriteLine($"Found Device : {info.DeviceInformation.Name}:{info.DeviceInformation.Id}");
}
}
if (selectedFrameSourceGroup != null)
break;
}
if(selectedFrameSourceGroup != null)
break;
}
if (selectedFrameSourceGroup == null)
{
Debug.WriteLine("No frame source available found");
return null;
}
HandDetector HandDetector = new HandDetector(selectedFrameSourceGroup, selectedFrameSourceInfo);
return HandDetector;
}
/// <summary>
/// Creates asynchronously the Media Capture which will process the depth stream images
/// </summary>
/// <param name="clbk">The Callback object to call when the hand detection status changes.</param>
/// <returns>The asynchronous task</returns>
public async Task InitializeAsync(IHDMediaSinkClbk clbk)
{
//Create the media capture
Debug.WriteLine("Creating a media capture...");
m_mediaCapture = new MediaCapture();
await m_mediaCapture.InitializeAsync(new MediaCaptureInitializationSettings()
{
SourceGroup             = m_mediaGroup,
SharingMode             = MediaCaptureSharingMode.SharedReadOnly,
MemoryPreference        = MediaCaptureMemoryPreference.Auto,   //For the Hololens, MediaCaptureMemoryPreference.CPU does not work
StreamingCaptureMode    = StreamingCaptureMode.Video
});
//Find a correct video profile with the best capabilities (resolution)
Debug.WriteLine("Search a video profile...");
VideoEncodingProperties videoProfile = null;
var mediaProperties = m_mediaCapture.VideoDeviceController.GetAvailableMediaStreamProperties(MediaStreamType.VideoPreview);
UInt32 maxHeight = 0;
foreach (var mediaProp in mediaProperties)
{
VideoEncodingProperties videoProp = mediaProp as VideoEncodingProperties;
Debug.WriteLine($"VideoProp : {videoProp.Type}:{videoProp.Subtype} {videoProp.Width}x{videoProp.Height}");
if(videoProp.Subtype == "ARGB32" || videoProp.Subtype == "L8" || videoProp.Subtype == "D16" || videoProp.Subtype == "D8" || videoProp.Subtype == "L16" || videoProp.Subtype == "RGB24")
{ 
if(maxHeight < videoProp.Height)
{ 
videoProfile = videoProp;
maxHeight = videoProp.Height;
}
}
}
if (videoProfile == null)
{
Debug.WriteLine("No video profile found...");
await Task.FromResult<Windows.Foundation.IAsyncAction>(null);
}
else
{
Debug.WriteLine($"Starting to preview {m_mediaInfo.DeviceInformation.Name} : {m_mediaInfo.DeviceInformation.Id} at {videoProfile.Width}x{videoProfile.Height}: {videoProfile.Subtype}");
//Create the video encoding
MediaEncodingProfile profile = new MediaEncodingProfile();
profile.Video = videoProfile;
profile.Audio = null;
//Create and start preview in the MediaSink
Debug.WriteLine(m_mediaInfo.DeviceInformation.Name);
m_mediaSink = new HDMediaSinkProxy();
IMediaExtension ext = await m_mediaSink.InitializeAsync(clbk, profile.Video);
await m_mediaCapture.StartPreviewToCustomSinkAsync(profile, ext);
}
Debug.WriteLine("End of Create media capture async");
}

正如你们所看到的，当我只想要该帧中的深度值时，我正在为给定的相机帧(一个深度、一个红外摄像头和两个未知摄像头(捕获四种模式。使用HoloLens1，我只能获得具有完全相同代码的深度相机，而无需所有不太可能难以解析的剩余图像(因为我没有找到任何元数据来帮助我裁剪图像......如果可能的话，我想避免硬编码数值。

你们知道如何自动提取顶部图像吗？

谢谢！

Answer 1

根据文档，您可以设置帧源的首选格式

var colorFrameSource = mediaCapture.FrameSources[colorSourceInfo.Id];
var preferredFormat = colorFrameSource.SupportedFormats.Where(format =>
{
return format.VideoFormat.Width >= 1080
&& format.Subtype == MediaEncodingSubtypes.Argb32;
}).FirstOrDefault();
if (preferredFormat == null)
{
// Our desired format is not supported
return;
}
await colorFrameSource.SetFormatAsync(preferredFormat);

您需要更改值才能获得深度。

MediaFrameSource.SetFormatAsync

此方法是 MediaCapture.SetEncodingPropertiesAsync 的替代方法，但与 SetEncodingPropertiesAsync 不同，此方法可用于任何流，而不仅仅是彩色相机预览、录制和照片捕获流。