S3 - 传输前自动压缩

Question

我想知道：

1. 如果默认情况下有任何可用的工具，请压缩和传输文件 到 S3

2. 或工具可以选择压缩文件并将其传输到S3

3. 或者我必须调用python库并压缩，然后转移到S3

Answer 1

如果你愿意，你可以使用简单的python代码。

import gzip
import json
import boto3

# To compress
# Data to compress
data = [{'name': 'test'}, {'name': 'test2'}]
# Converting data to string
json_str = json.dumps(data) + "n"
# Converting to bytes
json_bytes = json_str.encode('utf-8')
jsonfilename = "s3_compressed_file.json.gz"
# Compressing to gzip
with gzip.GzipFile(jsonfilename, 'w') as f:
f.write(json_bytes)

# Upload to S3
s3BucketName = 'mybucket'
s3_resource = boto3.resource('s3')
# if you want to rename file while uploading
file_name = 's3_compressed_file-1.json.gz'
# '/current_dir_path/' + filename, '<bucket-name>', 's3_folder/{}'.format(filename)
s3_response = s3_resource.meta.client.upload_file('source_dir/' + 's3_compressed_file.gz', s3BucketName,
'destination_dir/{}'.format(file_name))