谷歌云存储传输与一个安全的tsv网址列表文件

Question

我正在尝试传输GCS存储桶上的内容或公开可用url。

为此，我使用谷歌云存储传输api，它需要我执行两个步骤：

1. 创建一个.tsv文件，该文件包含我的公共URLS列表
2. 若要，请创建传输(此处使用python api(

要启动脚本，我使用一个服务帐户，该帐户对包含transfer.tsv文件的存储桶和接收器存储桶都具有存储对象管理权限。

只有当transfer.tsv文件上传到互联网上打开的bucket时，我才能使其工作。

你知道是否可以将is放在一个安全的bucket上，并向创建转账的服务帐户授予权限吗？

到目前为止，我所有的尝试都产生了以下错误。

错误

PERMISSION_DENIED   1   
https://storage.googleapis.com/my-private-bucket/transfer.tsv   
Received HTTP error code 403.   

传输.tsv

TsvHttpData-1.0
https://image.shutterstock.com/image-photo/portrait-surprised-cat-scottish-straight-260nw-499196506.jpg

python脚本

from google.cloud import storage_transfer
from datetime import datetime
def create_one_time_http_transfer(
project_id: str,
description: str,
list_url: str,
sink_bucket: str,
):
"""Creates a one-time transfer job from Amazon S3 to Google Cloud
Storage."""
client = storage_transfer.StorageTransferServiceClient()
# the same time creates a one-time transfer
one_time_schedule = {"day": now.day, "month": now.month, "year": now.year}
transfer_job_request = storage_transfer.CreateTransferJobRequest(
{
"transfer_job": {
"project_id": project_id,
"description": description,
"status": storage_transfer.TransferJob.Status.ENABLED,
"schedule": {
"schedule_start_date": one_time_schedule,
"schedule_end_date": one_time_schedule,
},
"transfer_spec": {
"http_data_source": storage_transfer.HttpData(list_url=list_url),
"gcs_data_sink": {
"bucket_name": sink_bucket,
},
},
}
}
)
result = client.create_transfer_job(transfer_job_request)
print(f"Created transferJob: {result.name}")

我把这个函数叫做

create_one_time_http_transfer(
project_id="my-project-id",
description="first transfer",
list_url=tsv_url,
sink_bucket="my-destination-bucket",
)

Answer 1

问题可能是storage_transfer.StorageTransferServiceClient()中的权限。创建一个访问存储的角色，并将其附加到运行Python脚本的服务帐户。或者将您的凭据放在此处storage_transfer.StorageTransferServiceClient(credentials=XXXX.json)

Answer 2

找到了一种使其工作的方法。

当我将transfer.tsv文件上传到存储器时，我返回签名的url，而不是公共url

from datetime import datetime
from google.cloud import storage
def upload_to_storage(
file_input_path: str, file_output_path: str, bucket_name: str
) -> str:
gcs = storage.Client()
# # Get the bucket that the file will be uploaded to.
bucket = gcs.get_bucket(bucket_name)
# # Create a new blob and upload the file's content.
blob = bucket.blob(file_output_path)
blob.upload_from_filename(file_input_path)
return blob.generate_signed_url(datetime.now())

然后，这个签名的url被传递到上面提到的create_one_time_http_transfer。