在FireStore-asp.net c#中的一个集合中获取大量文档时出错

Question

我试图从FireStore获取大约数十万个数据，但我遇到了这种错误。我能在一次执行中达到阅读文件的极限吗？

这是错误声明

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="The datastore operation timed out, or the data was temporarily unavailable.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1648103323.710000000","description":"Error received from peer ipv4:34.101.5.42:443","file":"......srccorelibsurfacecall.cc","file_line":1067,"grpc_message":"The datastore operation timed out, or the data was temporarily unavailable.","grpc_status":14}")
at Grpc.Core.Internal.ClientResponseStream`2.MoveNext(CancellationToken token)
at Google.Api.Gax.Grpc.AsyncResponseStream`1.MoveNextAsync(CancellationToken cancellationToken)
at System.Linq.AsyncEnumerable.<ForEachAsync>g__Core|295_0[TSource](IAsyncEnumerable`1 source, Action`1 action, CancellationToken cancellationToken) in d:a1sIx.NETSourceSystem.Linq.AsyncSystemLinqOperatorsForEach.cs:line 31
at System.Linq.AsyncEnumerable.<ForEachAsync>g__Core|295_0[TSource](IAsyncEnumerable`1 source, Action`1 action, CancellationToken cancellationToken) in d:a1sIx.NETSourceSystem.Linq.AsyncSystemLinqOperatorsForEach.cs:line 31
at Google.Cloud.Firestore.Query.GetSnapshotAsync(ByteString transactionId, CancellationToken cancellationToken)
at vgr_data.Services.PesertaService.GetWithFilterFaskesExternalAsync(PesertaFilter filter) in D:ProjectsVGR-Data-APIvgr-datavgr-dataServicesPesertaService.cs:line 994
at vgr_data.Controllers.PesertaController.ExportFaskesExternal(PesertaFilter filter) in D:ProjectsVGR-Data-APIvgr-datavgr-dataControllersPesertaController.cs:line 2980
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

我试图用OFFSET量的chunk[[query.OFFSET((.GetsnapshotAsync((]]来获取这个数据，起初它有效，但由于数据仍在增长，但现在出现了错误。

int pageCount = 0;
int dataCount = 1;
List<MyModel> listItem = new List<MyModel>();

while (dataCount > 0)
{
dataCount = 0;
QuerySnapshot querySnapshot = await _query.Offset(40000 * pageCount).Limit(40000).GetSnapshotAsync();
dataCount = querySnapshot.Count();
foreach (DocumentSnapshot documentSnapshot in querySnapshot.Documents)
{
if (documentSnapshot.Exists)
{
Dictionary<string, object> obj = documentSnapshot.ToDictionary();
MyModel item = _mapper.Map<MyModel>(obj);
item.Id = documentSnapshot.Id;
listItem.Add(item);
}
}
System.Diagnostics.Debug.WriteLine("Total List : "+ listItem.Count());
pageCount += 1;
}

但最后，在达到大约400k的数据后，我仍然会得到错误

Answer 1

Firestore查询分页基于查询游标基于查询游标，文档建议不要使用偏移量：

不要使用偏移量。相反，请使用光标。使用偏移量只能避免将跳过的文档返回到应用程序，但这些文档仍然是在内部检索的。跳过的文档会影响查询的延迟，并且您的应用程序会为检索这些文档所需的读取操作付费。

看起来您的读取操作在Firestore一侧超时，我怀疑可能涉及此offset调用：

_query.Offset(40000 * pageCount).Limit(40000)

我建议重写您的代码，以便使用基于光标的查询进行分页。