使用 c# 将数据传输到 AWS s3 雪球上

Question

我正在尝试使用 snowball 将所有数据导入 s3，但我需要跟踪每个文件的一些元数据。我有一些 c# 代码，可以使用此元数据将文件直接上传到存储桶，工作正常。我以为我可以使用 snowball s3 适配器连接到我的 snowball 并将数据从我的 c# 代码上传到我的 c# 代码上，因为此链接显示：

您可以将 S3 适配器与现有的 Amazon S3 接口配合使用，例如 AWS 开发工具包、AWS CLI 或您自己的自定义 Amazon S3 REST 客户。

我有适配器在 http://localhost:8090 上运行，所以我尝试通过传入一个 AmazonS3Config 对象(该终端节点作为 ServiceUrl(来将其设置为 Amazon3Client 的终端节点，但是当我尝试使用 PutObjectRequest 或 TransferUtility 上传文件时，我得到以下异常：

Amazon.Runtime.AmazonServiceException: A WebException with status
NameResolutionFailure was thrown. ---> System.Net.WebException: The
remote name could not be resolved: 'BUCKET.localhost' at
System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream() at
Amazon.Runtime.Internal.HttpRequest.GetRequestContent() at
Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.RedirectHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext
executionContext)
--- End of inner exception stack trace --- at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext
executionContext, WebException exception) at
Amazon.Runtime.Internal.ExceptionHandler`1.Handle(IExecutionContext
executionContext, Exception exception) at
Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext
executionContext, Exception exception) at
Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.Signer.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.CredentialsRetriever.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.S3.Internal.AmazonS3KmsHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.EndpointResolver.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.S3.Internal.AmazonS3PostMarshallHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.Marshaller.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.S3.Internal.AmazonS3PreMarshallHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext
executionContext) at
Amazon.Runtime.AmazonServiceClient.Invoke(TRequest request,
IMarshaller`2 marshaller, ResponseUnmarshaller unmarshaller) at
Amazon.S3.AmazonS3Client.PutObject(PutObjectRequest request)

我尝试使用我的本地IP地址以及127.0.0.1，但都遇到了相同的错误。当我使用 amazon cli 将文档上传到 snowball 时，使用--endpoint更改终端节点，它有效(使用 localhost 或我的本地 ip(，所以显然这只是关于 .net sdk 的东西。但我真的非常愿意使用我现有的 c# 代码，而不是编写命令行程序来执行此操作 - 从 cli 来看，它不会那么简单。

编辑：这是不起作用的 c# 代码：

var config = new AmazonS3Config
{
ServiceURL = "http://localhost:8090"
};
var file = new FileInfo(FILENAME);
var pRequest = new PutObjectRequest
{
BucketName = BUCKET,
Key = S3FILENAME,
StorageClass = S3StorageClass.StandardInfrequentAccess,
FilePath = file.FullName
};
pRequest.Metadata.Add("x-amz-meta-local-date-modified", file.CreationTime.ToFileTimeUtc().ToString());
using (var client = new AmazonS3Client(config))
{
client.PutObject(pRequest);
}

这是有效的命令行语句：

aws s3 cp FILENAME s3://BUCKET/S3FILENAME --endpoint http://localhost:8090

谁能帮我？

谢谢

Answer 1

问题是，默认情况下，AWS 开发工具包会将您的存储桶名称附加到 ServiceURL 前面，因此您最终会得到一个无效的 URL：http://BUCKET.localhost:8090。

您需要使用ForcePathStyle参数，以防止将存储桶名称附加到 ServiceURL 前面：

var config = new AmazonS3Config
{
ServiceURL = "http://localhost:8090",
ForcePathStyle = true
};