等待来自池的连接进行S3上载时超时

Question

我正试图从7台机器上传大量文件。在每台机器中，我运行6个线程上传到S3。当我在一台机器上运行上传时，它运行得很好，但当我在7台机器中运行时，它开始出现故障。

我在其余的机器中得到了低于错误的结果。

错误-AmazonClientException com.amazonaws.SdkClientException：无法执行HTTP请求：等待池连接超时

我上传到S3的小文件总数=1659328

每个线程中的记录数=276554

那么我必须关闭TransferManager吗？如果是，那么我应该如何关闭它？我的应用程序多线程。当我调用tm.shutdownNow();时，其他线程将无法使用它。

这是我要上传到S3的代码。

AWSCredentials credential = new ProfileCredentialsProvider("skjfffkjg-Prod-ServiceUser").getCredentials();
AmazonS3Client s3Client = (AmazonS3Client) AmazonS3ClientBuilder.standard().withRegion("us-east-1")
.withCredentials(new AWSStaticCredentialsProvider(credential)).withForceGlobalBucketAccessEnabled(true)
.build();
s3Client.getClientConfiguration().setMaxConnections(100);

上传到S3方法

public void uploadToToS3() {
_logger.info("Number of record to be processed in current thread: : " + records.size());

TransferManager tm = new TransferManager(s3Client);
MultipleFileUpload upload = tm.uploadFileList(bucketName, "", new File(fileLocation), records);
if (upload.isDone() == false) {
System.out.println("Transfer: " + upload.getDescription());
System.out.println("  - State: " + upload.getState());
System.out.println("  - Progress: " + upload.getProgress().getBytesTransferred());
}
try {
upload.waitForCompletion();
} catch (AmazonServiceException e1) {
_logger.error("AmazonServiceException " + e1.toString());
} catch (AmazonClientException e1) {
_logger.error("AmazonClientException " + e1.toString());
} catch (InterruptedException e1) {
_logger.error("InterruptedException " + e1.toString());
}
System.out.println("Is Upload completed Successfully ="+upload.isDone());
for (File file : records) {
try {
Files.delete(FileSystems.getDefault().getPath(file.getAbsolutePath()));
} catch (IOException e) {
_logger.error("IOException in file delete: " + e.toString());
System.exit(1);
_logger.error("IOException: " + e.toString());
}
}
_logger.info("Calling Transfer manager shutdown");
// tm.shutdownNow();
}

我必须关闭任何东西才能顺利上传吗？

Answer 1

当您拥有S3对象时，您需要中止并关闭它——就像中止打开的连接或关闭文件读取器等一样。

因此，您需要确保您的对象请求已正确关闭。顺便说一句，增加最大连接数可能不是解决这一问题的最佳方案。

有一个AWS API允许中止S3操作。我认为您需要添加"finally"块，以便在异常期间控制上传。

以下是我将使用的一个片段：

public void uploadToToS3() {
_logger.info("Number of record to be processed in current thread: : " + records.size());
TransferManager tm = new TransferManager(s3Client);
MultipleFileUpload upload = tm.uploadFileList(bucketName, "", new File(fileLocation), records);
if (upload.isDone() == false) {
System.out.println("Transfer: " + upload.getDescription());
System.out.println("  - State: " + upload.getState());
System.out.println("  - Progress: " + upload.getProgress().getBytesTransferred());
}
try {
upload.waitForCompletion();
} catch (AmazonServiceException e1) {
_logger.error("AmazonServiceException " + e1.toString());
} catch (AmazonClientException e1) {
_logger.error("AmazonClientException " + e1.toString());
} catch (InterruptedException e1) {
_logger.error("InterruptedException " + e1.toString());
} 
// ************ THIS IS THE MODIFICATION ************ 
finally {
upload.getSubTransfers().forEach(s -> s.abort());
}
// *************************************************** 
System.out.println("Is Upload completed Successfully ="+upload.isDone());
for (File file : records) {
try {
Files.delete(FileSystems.getDefault().getPath(file.getAbsolutePath()));
} catch (IOException e) {
_logger.error("IOException in file delete: " + e.toString());
System.exit(1);
_logger.error("IOException: " + e.toString());
}
}
_logger.info("Calling Transfer manager shutdown");
// tm.shutdownNow();
}