使用Go SDK 2获取Amazon S3中文件夹的大小

Question

我知道Amazon S3中没有文件夹，但是我们可以使用"/"在键名上。考虑到这一点，是否有可能使用AWS SDK for Go v2来计算文件夹的大小?或者我必须检索文件夹中的所有对象，然后逐个计算大小?

Answer 1

给出这个例子，以及这里的对象类型文档

可以计算bucket中项目占用的大小

package main
import (
"context"
"flag"
"fmt"
"log"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
var (
bucketName      string
objectPrefix    string
objectDelimiter string
maxKeys         int
)
func init() {
flag.StringVar(&bucketName, "bucket", "", "The `name` of the S3 bucket to list objects from.")
flag.StringVar(&objectPrefix, "prefix", "", "The optional `object prefix` of the S3 Object keys to list.")
flag.StringVar(&objectDelimiter, "delimiter", "",
"The optional `object key delimiter` used by S3 List objects to group object keys.")
flag.IntVar(&maxKeys, "max-keys", 0,
"The maximum number of `keys per page` to retrieve at once.")
}
// Lists all objects in a bucket using pagination
func main() {
flag.Parse()
if len(bucketName) == 0 {
flag.PrintDefaults()
log.Fatalf("invalid parameters, bucket name required")
}
// Load the SDK's configuration from environment and shared config, and
// create the client with this.
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
log.Fatalf("failed to load SDK configuration, %v", err)
}
client := s3.NewFromConfig(cfg)
// Set the parameters based on the CLI flag inputs.
params := &s3.ListObjectsV2Input{
Bucket: &bucketName,
}
if len(objectPrefix) != 0 {
params.Prefix = &objectPrefix
}
if len(objectDelimiter) != 0 {
params.Delimiter = &objectDelimiter
}
// Create the Paginator for the ListObjectsV2 operation.
p := s3.NewListObjectsV2Paginator(client, params, func(o *s3.ListObjectsV2PaginatorOptions) {
if v := int32(maxKeys); v != 0 {
o.Limit = v
}
})
// Iterate through the S3 object pages, printing each object returned.
var i int
var total int64
log.Println("Objects:")
for p.HasMorePages() {
i++
// Next Page takes a new context for each page retrieval. This is where
// you could add timeouts or deadlines.
page, err := p.NextPage(context.TODO())
if err != nil {
log.Fatalf("failed to get page %v, %v", i, err)
}
// Log the objects found
for _, obj := range page.Contents {
// fmt.Println("Object:", *obj.Key)
total += obj.Size
}
}
fmt.Println("total", total)
}

然后，如果我是正确的，阅读在s3。listobjectsv2输入文档，在我看来，您可以配置s3.ListObjectV2Input实例的Prefix成员来选择特定的文件夹。这个例子已经说明了如果你传入标志-prefix=...

Answer 2

不确定是否最简单的方法，但是您可以遍历您的对象列表- https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html，并在本地聚合大小。

Answer 3

另一种说法:

• 为您的桶启用AWSstorage lens-advanced metric-prefix aggregation，并将度量作为csv导出到桶
• 从桶
中的文件中获取csv数据

注:metric每24小时导出一次