S3对象在成功删除它和删除标记后仍在bucket中

Question

我构建了这个上下文管理器，以便在S3中创建一个临时对象用于测试：

from types import TracebackType
from typing import BinaryIO, Optional, Type
import boto3

class S3Object:
def __init__(self, file_object: BinaryIO, bucket_name: str, key: str):
self.file_object = file_object
self.bucket_name = bucket_name
self.key = key
self.s3 = boto3.client("s3")
def __enter__(self) -> "S3Object":
self.s3.upload_fileobj(self.file_object, self.bucket_name, self.key)
return self
def __exit__(
self,
exc_type: Optional[Type[BaseException]],
exc_val: Optional[BaseException],
exc_tb: Optional[TracebackType],
) -> bool:
delete_object_response = self.s3.delete_object(Bucket=self.bucket_name, Key=self.key)
# Delete version marker
self.s3.delete_object(
Bucket=self.bucket_name, Key=self.key, VersionId=delete_object_response["VersionId"]
)
print("Deleted object")  # debugging
return False  # Propagate exception

我在一个测试中这样使用它：

with S3Object(BytesIO(), bucket_name, filename) as s3_object:
…

奇怪的是，即使测试输出包括"；删除的对象"；调试输出，原始项仍在bucket中。来自cdk destroy输出：

您尝试删除的bucket不为空。您必须删除存储桶中的所有版本。

它绝对是正确的文件，因为我根据测试名称生成了一个唯一的文件名。这似乎也不是时间问题，因为这是100%可复制的。

我可以手动删除文件，所以似乎也没有任何特殊的保护措施。

在修改代码以打印两个响应之后，我想知道我是否只是删除了两次删除标记？可读性的缩写：

{'ResponseMetadata': {'HTTPStatusCode': 204, 'HTTPHeaders': {'x-amz-version-id': 'd08uv05t7lU7YKle4kMSURa_jrMU.GSL', 'x-amz-delete-marker': 'true'}}, 'DeleteMarker': True, 'VersionId': 'd08uv05t7lU7YKle4kMSURa_jrMU.GSL'}
{'ResponseMetadata': {'HTTPStatusCode': 204, 'HTTPHeaders': {'x-amz-version-id': 'd08uv05t7lU7YKle4kMSURa_jrMU.GSL', 'x-amz-delete-marker': 'true'}}, 'DeleteMarker': True, 'VersionId': 'd08uv05t7lU7YKle4kMSURa_jrMU.GSL'}

Answer 1

根据评论和另一个答案，我得出了以下结论：

from types import TracebackType
from typing import BinaryIO, Optional, Type
import boto3
DELETE_OBJECTS_MAX_KEYS = 1000

class S3Object:
def __init__(self, file_object: BinaryIO, bucket_name: str, key: str):
self.file_object = file_object
self.bucket_name = bucket_name
self.key = key
self.s3 = boto3.client("s3")
def __enter__(self) -> "S3Object":
self.s3.upload_fileobj(self.file_object, self.bucket_name, self.key)
return self
def __exit__(
self,
exc_type: Optional[Type[BaseException]],
exc_val: Optional[BaseException],
exc_tb: Optional[TracebackType],
) -> bool:
version_list = []
object_versions_paginator = self.s3.get_paginator("list_object_versions")
for object_versions_page in object_versions_paginator.paginate(Bucket=self.bucket_name):
for marker in object_versions_page.get("DeleteMarkers", []):
if marker["Key"] == self.key:
version_list.append({"Key": self.key, "VersionId": marker["VersionId"]})
for version in object_versions_page.get("Versions", []):
if version["Key"] == self.key:
version_list.append({"Key": self.key, "VersionId": version["VersionId"]})
for index in range(0, len(version_list), DELETE_OBJECTS_MAX_KEYS):
response = self.s3.delete_objects(
Bucket=self.bucket_name,
Delete={
"Objects": version_list[index : index + DELETE_OBJECTS_MAX_KEYS],
},
)
print(response)
return False  # Propagate exception