我无法将大于 20 M 的文件上传到我的 S3 存储桶

  • 本文关键字:S3 存储 文件 大于 s3fs
  • 更新时间 :
  • 英文 :


我最近在Scaleway创建了一个S3存储桶。我使用s3fs安装它,没有明显的问题。我上传一些"中等大小"的文件时遇到问题。如果大小小于20 M,可以,但对于较大的文件(50 M及以上(,复制失败,并显示消息"无法写入文件,权限被拒绝"。我联系了scaleway支持,但他们说这与我的s3fs客户端有关。我在调试模式下安装了bucket,使用:

$ sudo s3fs tellurix /mnt/scaleway/ -o passwd_file=${HOME}/.passwd-s3fs,url=https://s3.fr-par.scw.cloud,allow_other -o use_path_request_style,noatime -o dbglevel=info -f -o curldbg

我复制/粘贴日志的最后100行,因为我看不出错误在哪里。非常感谢的帮助

* SSL_write() returned SYSCALL, errno = 32
* Closing connection 6
[ERR] curl.cpp:RequestPerform(2546): ### CURLE_SEND_ERROR
* SSL_write() returned SYSCALL, errno = 32
* Closing connection 5
[ERR] curl.cpp:RequestPerform(2546): ### CURLE_SEND_ERROR
[INF] curl.cpp:RequestPerform(2621): ### retrying...
[INF]       curl.cpp:RemakeHandle(2248): Retry request. [type=9][url=https://s3.fr-par.scw.cloud/tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=5&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1][path=/ant/MyHome automation guide 72488.pdf]
* Hostname s3.fr-par.scw.cloud was found in DNS cache
*   Trying 2001:bc8:1002::30:443...
* TCP_NODELAY set
* Connected to s3.fr-par.scw.cloud (2001:bc8:1002::30) port 443 (#6)
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL re-using session ID
* SSL_write() returned SYSCALL, errno = 32
* Closing connection 5
[ERR] curl.cpp:RequestPerform(2546): ### CURLE_SEND_ERROR
* old SSL session ID is stale, removing
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* Server certificate:
*  subject: CN=s3.fr-par.scw.cloud
*  start date: Feb 10 23:20:22 2020 GMT
*  expire date: May 10 23:20:22 2020 GMT
*  subjectAltName: host "s3.fr-par.scw.cloud" matched cert's "s3.fr-par.scw.cloud"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> PUT /tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=5&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1 HTTP/1.1
Host: s3.fr-par.scw.cloud
User-Agent: s3fs/1.86 (commit hash 005a684; OpenSSL)
Accept: */*
Content-Length: 10485760
Expect: 100-continue
* SSL_write() returned SYSCALL, errno = 32
* Closing connection 6
[ERR] curl.cpp:RequestPerform(2546): ### CURLE_SEND_ERROR
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< x-amz-id-2: tx97bf2f1b3ccd47c4a5f91-005eaa999a
< x-amz-request-id: tx97bf2f1b3ccd47c4a5f91-005eaa999a
< Content-Type: application/xml
< Date: Thu, 30 Apr 2020 09:25:46 GMT
< Transfer-Encoding: chunked
* HTTP error before end of send, keep sending
< 
[INF] curl.cpp:RequestPerform(2621): ### retrying...
[INF]       curl.cpp:RemakeHandle(2248): Retry request. [type=9][url=https://s3.fr-par.scw.cloud/tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=2&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1][path=/ant/MyHome automation guide 72488.pdf]
[ERR] curl.cpp:RequestPerform(2639): ### giving up
[WAN] curl.cpp:MultiPerform(4340): thread failed - rc(-5)
[INF]       curl.cpp:insertV4Headers(2797): computing signature [PUT] [/ant/MyHome automation guide 72488.pdf] [partNumber=6&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1] [34ec149b334729973e407bada5e11b96774acfd1375b8009f789474ecb9bb2bb]
[INF]       curl.cpp:url_to_host(99): url is https://s3.fr-par.scw.cloud
* Hostname s3.fr-par.scw.cloud was found in DNS cache
*   Trying 2001:bc8:1002::30:443...
* TCP_NODELAY set
* Connected to s3.fr-par.scw.cloud (2001:bc8:1002::30) port 443 (#7)
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL re-using session ID
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* Server certificate:
*  subject: CN=s3.fr-par.scw.cloud
*  start date: Feb 10 23:20:22 2020 GMT
*  expire date: May 10 23:20:22 2020 GMT
*  subjectAltName: host "s3.fr-par.scw.cloud" matched cert's "s3.fr-par.scw.cloud"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> PUT /tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=6&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1 HTTP/1.1
Host: s3.fr-par.scw.cloud
User-Agent: s3fs/1.86 (commit hash 005a684; OpenSSL)
Authorization: AWS4-HMAC-SHA256 Credential=xxxxxx/20200430/fr-par/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=91bbf50cc33a1f1d1cd3f3660fcc116e857223b4f8297b6c796e7dc32f244bac
x-amz-content-sha256: 34ec149b334729973e407bada5e11b96774acfd1375b8009f789474ecb9bb2bb
x-amz-date: 20200430T092546Z
Content-Length: 1132789
Expect: 100-continue
[INF] curl.cpp:RequestPerform(2621): ### retrying...
[INF]       curl.cpp:RemakeHandle(2248): Retry request. [type=9][url=https://s3.fr-par.scw.cloud/tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=1&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1][path=/ant/MyHome automation guide 72488.pdf]
[ERR] curl.cpp:RequestPerform(2639): ### giving up
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
* SSL_write() returned SYSCALL, errno = 32
* Closing connection 6
[ERR] curl.cpp:RequestPerform(2546): ### CURLE_SEND_ERROR
[INF] curl.cpp:RequestPerform(2621): ### retrying...
[INF]       curl.cpp:RemakeHandle(2248): Retry request. [type=9][url=https://s3.fr-par.scw.cloud/tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=3&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1][path=/ant/MyHome automation guide 72488.pdf]
[ERR] curl.cpp:RequestPerform(2639): ### giving up
[INF] curl.cpp:RequestPerform(2621): ### retrying...
[INF]       curl.cpp:RemakeHandle(2248): Retry request. [type=9][url=https://s3.fr-par.scw.cloud/tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=4&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1][path=/ant/MyHome automation guide 72488.pdf]
[ERR] curl.cpp:RequestPerform(2639): ### giving up
[INF] curl.cpp:RequestPerform(2621): ### retrying...
[INF]       curl.cpp:RemakeHandle(2248): Retry request. [type=9][url=https://s3.fr-par.scw.cloud/tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=5&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1][path=/ant/MyHome automation guide 72488.pdf]
[ERR] curl.cpp:RequestPerform(2639): ### giving up
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 0
< x-amz-id-2: tx64fa48b5fffb4985bee17-005eaa999a
< Last-Modified: Thu, 30 Apr 2020 09:25:46 GMT
< ETag: "30c5132a619a14608ff0a3d9bac63fe2"
< x-amz-request-id: tx64fa48b5fffb4985bee17-005eaa999a
< x-amz-version-id: 1588238746862950
< Content-Type: text/html; charset=UTF-8
< Date: Thu, 30 Apr 2020 09:25:59 GMT
< 
* Connection #7 to host s3.fr-par.scw.cloud left intact
[INF]       curl.cpp:RequestPerform(2455): HTTP response code 200
[WAN] curl.cpp:MultiPerform(4374): thread failed - rc(-5)
[WAN] curl.cpp:MultiPerform(4374): thread failed - rc(-5)
[WAN] curl.cpp:MultiPerform(4374): thread failed - rc(-5)
[WAN] curl.cpp:MultiPerform(4374): thread failed - rc(-5)
[WAN] curl.cpp:MultiRead(4400): error from callback function(https://s3.fr-par.scw.cloud/tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=1&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1).
[WAN] curl.cpp:MultiRead(4400): error from callback function(https://s3.fr-par.scw.cloud/tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=2&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1).
[WAN] curl.cpp:MultiRead(4400): error from callback function(https://s3.fr-par.scw.cloud/tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=3&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1).
[WAN] curl.cpp:MultiRead(4400): error from callback function(https://s3.fr-par.scw.cloud/tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=4&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1).
[WAN] curl.cpp:MultiRead(4400): error from callback function(https://s3.fr-par.scw.cloud/tellurix/ant/MyHome%20automation%20guide%2072488.pdf?partNumber=5&uploadId=YmNkMmE3MWMtMDFhYi00NDhmLTlkYWItMjEyMDA1YTM1Njk1).
[INF]       curl.cpp:CompleteMultipartPostRequest(3642): [tpath=/ant/MyHome automation guide 72488.pdf][parts=6]
[ERR] curl.cpp:CompleteMultipartPostRequest(3653): 1 file part is not finished uploading.
[INF] s3fs.cpp:s3fs_release(2358): [path=/ant/MyHome automation guide 72488.pdf][fd=11]
[INF]       cache.cpp:DelStat(582): delete stat cache entry[path=/ant/MyHome automation guide 72488.pdf]
[INF]       fdcache.cpp:GetFdEntity(2650): [path=/ant/MyHome automation guide 72488.pdf][fd=11]

我使用命令行参数成功地安装并编写了一个500 MB的文件到scaleway。考虑到CURLE_SEND_ERROR,我想知道你是否有某种网络问题?也许可以为-o parallel_count尝试一个较低的值,例如1?看见https://github.com/s3fs-fuse/s3fs-fuse/issues/1283#issuecomment-623026911。

您从哪里安装铲斗?它是你家里的电脑还是云虚拟机?您需要多长时间才能收到此错误?

我之所以这么问,是因为"SSL_write((return SYSCALL,errno=32"看起来像是有什么东西正在关闭您的连接。"发送结束前出现HTTP错误,继续发送"也指出了这种问题。可能发生超时?你和你的bucket之间有NAT网关吗?如果它不关心keepalives,这也可能导致问题,因为上传可能需要相对较长的时间。

正如s3fswiki所说,20MB是多部分上传的阈值,而不是单个请求。Scaleway的多部分上传API可能与Amazon略有不同?从s3fs维基:"一些提供商不支持完整的S3 API,例如,缺乏多部分上传。"请注意,s3fs主要用于与Amazon S3合作,正如我所看到的,Scaleway不在s3fs维基中支持的提供商名单上:https://github.com/s3fs-fuse/s3fs-fuse/wiki/Non-Amazon-S3.

最后一件事,你的libcurl版本是什么?s3fs文档中说应该是7.16或7.17。你使用的是最新版本的s3fs吗?

相关内容

  • 没有找到相关文章

最新更新