Firebase Cloud 功能，Firestore 回归"Deadline Exceeded"

Question

我从Firestore文档中获取了一个示例函数，并能够在本地firebase环境中成功运行它。然而，一旦我部署到firebase服务器，该功能就完成了，但在firestore数据库中没有任何条目。firebase函数日志显示"超过截止日期"。我有点困惑。有人知道为什么会发生这种情况以及如何解决吗？

以下是示例函数：

exports.testingFunction = functions.https.onRequest((request, response) => {
var data = {
name: 'Los Angeles',
state: 'CA',
country: 'USA'
};
// Add a new document in collection "cities" with ID 'DC'
var db = admin.firestore();
var setDoc = db.collection('cities').doc('LA').set(data);
response.status(200).send();
});

Answer 1

Firestore有限制。

"超过最后期限"可能是因为它的局限性。

看看这个。https://firebase.google.com/docs/firestore/quotas

每秒对文档的最大写入速率1

https://groups.google.com/forum/#！消息/谷歌云消防商店讨论/tGaZpTWQ7tQ/NdaDGRAzBgAJ

Answer 2

根据我自己的经验，当你试图使用糟糕的互联网连接编写文档时，也会发生这个问题。

我使用了一个类似于Jurgen建议的解决方案，一次插入小于500的批量文档，如果我使用的是不太稳定的wifi连接，就会出现这个错误。当我插入电缆时，具有相同数据的相同脚本运行时不会出现错误。

Answer 3

我写了这个小脚本，它使用批处理写入(最大500)，并且只写一个批接一个批。

通过首先创建batchWorkerlet batch: any = new FbBatchWorker(db);来使用它然后向工作者batch.set(ref.doc(docId), MyObject);添加任何内容。并通过batch.commit()完成。该api与普通的FirestoreBatch相同(https://firebase.google.com/docs/firestore/manage-data/transactions#batched-写入)但是，目前它只支持set。

import { firestore } from "firebase-admin";
class FBWorker {
callback: Function;
constructor(callback: Function) {
this.callback = callback;
}
work(data: {
type: "SET" | "DELETE";
ref: FirebaseFirestore.DocumentReference;
data?: any;
options?: FirebaseFirestore.SetOptions;
}) {
if (data.type === "SET") {
// tslint:disable-next-line: no-floating-promises
data.ref.set(data.data, data.options).then(() => {
this.callback();
});
} else if (data.type === "DELETE") {
// tslint:disable-next-line: no-floating-promises
data.ref.delete().then(() => {
this.callback();
});
} else {
this.callback();
}
}
}
export class FbBatchWorker {
db: firestore.Firestore;
batchList2: {
type: "SET" | "DELETE";
ref: FirebaseFirestore.DocumentReference;
data?: any;
options?: FirebaseFirestore.SetOptions;
}[] = [];
elemCount: number = 0;
private _maxBatchSize: number = 490;
public get maxBatchSize(): number {
return this._maxBatchSize;
}
public set maxBatchSize(size: number) {
if (size < 1) {
throw new Error("Size must be positive");
}
if (size > 490) {
throw new Error("Size must not be larger then 490");
}
this._maxBatchSize = size;
}
constructor(db: firestore.Firestore) {
this.db = db;
}
async commit(): Promise<any> {
const workerProms: Promise<any>[] = [];
const maxWorker = this.batchList2.length > this.maxBatchSize ? this.maxBatchSize : this.batchList2.length;
for (let w = 0; w < maxWorker; w++) {
workerProms.push(
new Promise((resolve) => {
const A = new FBWorker(() => {
if (this.batchList2.length > 0) {
A.work(this.batchList2.pop());
} else {
resolve();
}
});
// tslint:disable-next-line: no-floating-promises
A.work(this.batchList2.pop());
}),
);
}
return Promise.all(workerProms);
}
set(dbref: FirebaseFirestore.DocumentReference, data: any, options?: FirebaseFirestore.SetOptions): void {
this.batchList2.push({
type: "SET",
ref: dbref,
data,
options,
});
}
delete(dbref: FirebaseFirestore.DocumentReference) {
this.batchList2.push({
type: "DELETE",
ref: dbref,
});
}
}

Answer 4

我测试了这一点，让15个并发的AWS Lambda函数将10000个请求写入数据库，写入不同的集合/文档毫秒部分。我没有得到DEADLINE_EXCEEDED错误。

请参阅有关firebase的文档。

'deadline-exceeded'：在操作完成之前截止日期已过期。对于更改系统状态的操作，即使操作已成功完成，也可能返回此错误。例如，来自服务器的成功响应可能被延迟了足够长的时间，以至于截止日期到期。

在我们的案例中，我们只写少量数据，大多数时候都能正常工作，但丢失数据是不可接受的。我还没有总结出Firestore为什么不能写入简单的小数据。

解决方案：

我使用的是一个使用SQS事件触发器的AWS Lambda函数。

# This function receives requests from the queue and handles them
# by persisting the survey answers for the respective users.
QuizAnswerQueueReceiver:
handler: app/lambdas/quizAnswerQueueReceiver.handler
timeout: 180 # The SQS visibility timeout should always be greater than the Lambda function’s timeout.
reservedConcurrency: 1 # optional, reserved concurrency limit for this function. By default, AWS uses account concurrency limit    
events:
- sqs:
batchSize: 10 # Wait for 10 messages before processing.
maximumBatchingWindow: 60 # The maximum amount of time in seconds to gather records before invoking the function
arn:
Fn::GetAtt:
- SurveyAnswerReceiverQueue
- Arn
environment:
NODE_ENV: ${self:custom.myStage}

我正在使用一个死信队列，该队列连接到失败事件的主队列。

Resources:
QuizAnswerReceiverQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: ${self:provider.environment.QUIZ_ANSWER_RECEIVER_QUEUE}
# VisibilityTimeout MUST be greater than the lambda functions timeout https://lumigo.io/blog/sqs-and-lambda-the-missing-guide-on-failure-modes/
# The length of time during which a message will be unavailable after a message is delivered from the queue.
# This blocks other components from receiving the same message and gives the initial component time to process and delete the message from the queue.
VisibilityTimeout: 900 # The SQS visibility timeout should always be greater than the Lambda function’s timeout.
# The number of seconds that Amazon SQS retains a message. You can specify an integer value from 60 seconds (1 minute) to 1,209,600 seconds (14 days).
MessageRetentionPeriod: 345600  # The number of seconds that Amazon SQS retains a message. 
RedrivePolicy:
deadLetterTargetArn:
"Fn::GetAtt":
- QuizAnswerReceiverQueueDLQ
- Arn
maxReceiveCount: 5 # The number of times a message is delivered to the source queue before being moved to the dead-letter queue.
QuizAnswerReceiverQueueDLQ:
Type: "AWS::SQS::Queue"
Properties:
QueueName: "${self:provider.environment.QUIZ_ANSWER_RECEIVER_QUEUE}DLQ"
MessageRetentionPeriod: 1209600 # 14 days in seconds

Answer 5

如果在大约10秒后生成错误，可能不是您的互联网连接，可能是您的函数没有返回任何承诺。根据我的经验，我之所以出现错误，只是因为我在另一个promise中封装了一个firebase集操作(返回promise)。你可以做这个

return db.collection("COL_NAME").doc("DOC_NAME").set(attribs).then(ref => {
var SuccessResponse = {
"code": "200"
}
var resp = JSON.stringify(SuccessResponse);
return resp;
}).catch(err => {
console.log('Quiz Error OCCURED ', err);
var FailureResponse = {
"code": "400",
}
var resp = JSON.stringify(FailureResponse);
return resp;
});

而不是

return new Promise((resolve,reject)=>{ 
db.collection("COL_NAME").doc("DOC_NAME").set(attribs).then(ref => {
var SuccessResponse = {
"code": "200"
}
var resp = JSON.stringify(SuccessResponse);
return resp;
}).catch(err => {
console.log('Quiz Error OCCURED ', err);
var FailureResponse = {
"code": "400",
}
var resp = JSON.stringify(FailureResponse);
return resp;
});
});