使用预先签名的URL进行S3 complete_multipart_upload操作

Question

我使用的是带有预签名URL的AWS S3多部分上传。

上传带有预签名URL的部件(upload_part操作(效果良好。由于技术原因，我还必须使用预先签名的URL(complete_multipart_upload操作(完成多部分上传。

在我的python代码中，我生成如下URL：

url = s3_client.generate_presigned_url(
'complete_multipart_upload',
Params={
'Bucket': self.bucket_name, 'Key': key, 'UploadId': upload_id,
'MultipartUpload': {'Parts': parts_param}
},
ExpiresIn=PRESIGNED_URLS_EXPIRATION_SECONDS
)

生成的URL似乎包含Bucket、Key和UploadId，但关键是缺少包含部件列表的MultipartUpload参数。我检查了parts_param的格式是否正确——我以前使用它时没有预先签名的URL。

https://<bucket>.s3.amazonaws.com/<key>?uploadId=<upload_id>&AWSAccessKeyId=<upload_id>&Signature=<signature>&Expires=<expires>

预签名的URL是否不支持传递MultipartUpload字典？

Answer 1

如果您查看API引用(https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html(，"CompleteMultipartUpload"不是参数，而是正文的一部分。因此，它不应该包含在generate_presigned_url调用中。

有了这个代码(MultipartUpload从Params中删除+在正文中传递的CompleteMultipartUploads(，它应该可以工作：

signed_url = s3.generate_presigned_url(ClientMethod='complete_multipart_upload',
Params={'Bucket': self.bkt, 'Key':key,
'UploadId': upload_id},
ExpiresIn=3600,
HttpMethod='POST')
def to_xml(multi_parts):
s = '<CompleteMultipartUpload>n'
for part in multi_parts:
s += '  <Part>n'
s += '    <PartNumber>%d</PartNumber>n' % part["PartNumber"]
s += '    <ETag>%s</ETag>n' % part["ETag"]
s += '  </Part>n'
s += '</CompleteMultipartUpload>'
return s
res = requests.post(signed_url, data=to_xml(parts)

Answer 2

例如，引用github时，似乎需要首先调用create_multipart_upload。https://github.com/boto/boto3/issues/2305.

import boto3
import requests
s3 = boto3.client('s3')
max_size = 5 * 1024 * 1024 #you can define your own size
res = s3.create_multipart_upload(Bucket=bucket_name, Key=key)
upload_id = res['UploadId']
# please note this is for only 1 part of the file, you have to do it for all parts and store all the etag, partnumber in a list 
parts=[]
signed_url = s3.generate_presigned_url(ClientMethod='upload_part',Params={'Bucket': bucket_name, 'Key': key, 'UploadId': upload_id, 'PartNumber': part_no})
with target_file.open('rb') as f:
file_data = f.read(max_size) #here reading content of only 1 part of file 
res = requests.put(signed_url, data=file_data)
etag = res.headers['ETag']
parts.append({'ETag': etag, 'PartNumber': part}) #you have to append etag and partnumber of each parts  
#After completing for all parts, you will use complete_multipart_upload api which requires that parts list 
res = s3.complete_multipart_upload(Bucket=bucket_name, Key=key, MultipartUpload={'Parts': parts},UploadId=upload_id)