从 AWS S3 中取消删除文件夹

Question

我有一个启用了版本控制的 S3 存储桶。可以取消删除文件，但如何取消删除文件夹？

我知道，S3没有文件夹...但是如何取消删除常见前缀？是否可以递归取消删除文件？

Answer 1

我创建了这个简单的 bash 脚本来恢复我删除的 S3 文件夹中的所有文件：

#!/bin/bash

recoverfiles=$(aws s3api list-object-versions --bucket MyBucketName  --prefix TheDeletedFolder/ --query "DeleteMarkers[?IsLatest && starts_with(LastModified,'yyyy-mm-dd')].{Key:Key,VersionId:VersionId}")
for row in  $(echo "${recoverfiles}" | jq -c '.[]'); do
key=$(echo "${row}" | jq -r '.Key'  )
versionId=$(echo "${row}" | jq -r '.VersionId'  )
echo aws s3api delete-object --bucket MyBucketName --key $key --version-id $versionId
done

yyyy-mm-dd= 文件夹被删除的日期

Answer 2

我在这里找到了一个令人满意的解决方案，在这里有更详细的描述。

总而言之，没有开箱即用的工具，但一个简单的 bash 脚本包装了 AWS 工具"s3api"以实现递归取消删除。

该解决方案对我有用。我发现的唯一缺点是，亚马逊似乎在大约 30.000 个文件后限制了恢复操作。

Answer 3

您无法取消删除公共前缀。您需要一次取消删除一个对象。当对象出现时，任何关联的文件夹也将重新出现。

可以通过两种方式完成取消删除：

删除

• 将撤消删除的删除标记，或
• 将对象的先前版本复制到自身，这将使最新版本比删除标记更新，因此它将重新出现。(我希望你明白这一点！

Answer 4

如果文件夹及其内容被删除，您可以使用以下脚本恢复它们，该脚本的灵感来自以前的答案

该脚本适用于事先启用版本控制的 S3 存储桶。它使用删除标记标签来还原 S3 前缀中的文件。

#!/bin/bash
# Inspired by https://www.dmuth.org/how-to-undelete-files-in-amazon-s3/
# This script can be used to undelete objects from an S3 bucket.
# When run, it will print out a list of AWS commands to undelete files, which you
# can then pipe into Bash.
#
#
# You will need the AWS CLI tool from https://aws.amazon.com/cli/ in order to run this script.
#
# Note that you must have the following permissions via IAM:
#
# Bucket permissions:
#
#   s3:ListBucket
#   s3:ListBucketVersions
#
# File permissions:
#
#   s3:PutObject
#   s3:GetObject
#   s3:DeleteObject
#   s3:DeleteObjectVersion
#
# If you want to do this in a "quick and dirty manner", you could just grant s3:* to
# the account, but I don't really recommend that.
#

# profile = company
# bucket = company-s3-bucket
# prefix = directory1/directory2/directory3/lastdirectory/
# pattern = (.*)
# USAGE
# bash undelete.sh  > recover_files.txt  | bash
read -p "Enter your aws  profile: " PROFILE
read -p "Enter your S3 bucket name: " BUCKET
read -p "Enter your S3 directory/prefix to be recovered from, leave empty for to recover all of the S3 bucket: " PREFIX
read -p "Enter the file pattern looking to recover, leave empty for all: " PATTERN
# Make sure Profile and Bucket are entered
[[ -z "$PROFILE" ]] && { echo "Profile is empty" ; exit 1; }
[[ -z "$BUCKET" ]] && { echo "Bucket is empty" ; exit 1; }
# Fill PATTERN to match all if empty
PATTERN=${PATTERN:-(.*)}
# Errors are fatal
set -e

if [ "$PREFIX" = "" ]; 
# To recover all of the S3 bucket
then
aws --profile ${PROFILE} --output text s3api list-object-versions --bucket ${BUCKET} 
| grep -i $PATTERN 
| grep -E "^DELETEMARKERS" 
| awk -v PROFILE=$PROFILE -v BUCKET=$BUCKET -v PREFIX=$PREFIX  
-F "[t]+" '{ print "aws --profile " PROFILE " s3api delete-object --bucket " BUCKET "--key ""$3"" --version-id "$5";"}' 

# To recover a directory
else
aws --profile ${PROFILE} --output text s3api list-object-versions --bucket ${BUCKET} --prefix ${PREFIX} 
| grep -E $PATTERN 
| grep -E "^DELETEMARKERS" 
| awk -v PROFILE=$PROFILE -v BUCKET=$BUCKET -v PREFIX=$PREFIX  
-F "[t]+" '{ print "aws --profile " PROFILE " s3api delete-object --bucket " BUCKET "--key ""$3"" --version-id "$5";"}' 
fi

Answer 5

如果您为存储桶启用了版本，请使用以下命令检索文件或文件夹。

对于具有尊重的文件夹： -

echo '#!/bin/bash' > undeleteScript.sh && aws --output text s3api list-object-versions --bucket bucketname --prefix path/to/retrieve| grep -E "^DELETEMARKERS" | awk '{FS = "[t]+"; print "aws s3api delete-object --bucket buckername --key 42"$3"42 --version-id "$5";"}' >> undeleteScript.sh && . undeleteScript.sh; rm -f undeleteScript.sh;

对于存储桶中的文件：-

echo '#!/bin/bash' > undeleteScript.sh && aws --output text s3api list-object-versions --bucket bucketname --prefix | grep -E "^DELETEMARKERS" | awk '{FS = "[t]+"; print "aws s3api delete-object --bucket buckername --key 42"$3"42 --version-id "$5";"}' >> undeleteScript.sh && . undeleteScript.sh; rm -f undeleteScript.sh;

Answer 6

我编写了一个优化版本来批量恢复 1000 个文件：

#!/bin/bash
if [[ "$#" -lt 3 ]]; then
echo "Run: $0 <BUCKET> <PREFIX> <PROFILE>"
exit 1
fi
BUCKET=$1
PREFIX=$2
PROFILE=$3
echo "Restore bucket PREFIX - ${BUCKET}/${PREFIX}"
while true; do
result="$(aws --profile ${PROFILE} s3api list-object-versions --max-items 1000 --bucket ${BUCKET} --prefix ${PREFIX} --query '{Objects: DeleteMarkers[0:999].{Key:Key,VersionId:VersionId}}')"

if [ "$(echo $result|jq '.Objects')" == "null" ]; then
echo "No more files to undelete."
break
fi

echo "Restoring:"
aws --profile ${PROFILE} --no-cli-pager s3api delete-objects --bucket ${BUCKET} --delete "$result" |grep Key
done