Multipart Uploads in Amazon S3 with Java – 用Java在Amazon S3中进行多部分上传

最后修改: 2018年 3月 18日

中文/混合/英文(键盘快捷键：t)

1. Overview

1.概述

In this tutorial, we’ll see how to handle multipart uploads in Amazon S3 with AWS Java SDK.

在本教程中，我们将看到如何用AWS Java SDK处理Amazon S3中的多部分上传。

Simply put, in a multipart upload, we split the content into smaller parts and upload each part individually. All parts are re-assembled when received.

简单地说，在多部分上传中，我们将内容分成小部分，并单独上传每个部分。

Multipart uploads offer the following advantages:

多部分上传具有以下优势。

  • Higher throughput – we can upload parts in parallel
  • Easier error recovery – we need to re-upload only the failed parts
  • Pause and resume uploads – we can upload parts at any point in time. The whole process can be paused and remaining parts can be uploaded later

Note that when using multipart upload with Amazon S3, each part except the last part must be at least 5 MB in size.

请注意，当使用多部分上传时，使用Amazon S3时，除最后一部分外，每个部分的大小必须至少为5MB。

2. Maven Dependencies

2.Maven的依赖性

Before we begin, we need to add the AWS SDK dependency in our project:

在我们开始之前，我们需要在我们的项目中添加AWS SDK的依赖性。

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.11.290</version>
</dependency>

To view the latest version, check out Maven Central.

要查看最新版本，请查看Maven Central。

3. Performing Multipart Upload

3.执行多部分上传

3.1. Creating Amazon S3 Client

3.1.创建Amazon S3客户端

First, we need to create a client for accessing Amazon S3. We’ll use the AmazonS3ClientBuilder for this purpose:

首先，我们需要创建一个客户端来访问Amazon S3。我们将使用AmazonS3ClientBuilder来实现这一目的。

AmazonS3 amazonS3 = AmazonS3ClientBuilder
  .standard()
  .withCredentials(new DefaultAWSCredentialsProviderChain())
  .withRegion(Regions.DEFAULT_REGION)
  .build();

This creates a client using the default credential provider chain for accessing AWS credentials.

这将创建一个使用默认凭证提供者链的客户端，用于访问AWS凭证。

For more details on how the default credential provider chain works, please see the documentation. If you’re using a region other than the default (US West-2), make sure you replace Regions.DEFAULT_REGION with that custom region.

有关默认凭证提供者链如何工作的更多细节，请参见文档。如果您使用的区域不是默认的（US West-2），请确保用该自定义区域替换Regions.DEFAULT_REGION。

3.2. Creating TransferManager for Managing Uploads

3.2.创建用于管理上传的TransferManager

We’ll use TransferManagerBuilder to create a TransferManager instance.

我们将使用TransferManagerBuilder来创建一个TransferManager实例。

This class provides simple APIs to manage uploads and downloads with Amazon S3 and manages all related tasks:

该类提供简单的API来管理Amazon S3的上传和下载并管理所有相关的任务。

TransferManager tm = TransferManagerBuilder.standard()
  .withS3Client(amazonS3)
  .withMultipartUploadThreshold((long) (5 * 1024 * 1025))
  .build();

Multipart upload threshold specifies the size, in bytes, above which the upload should be performed as multipart upload.

多部分上传阈值指定了超过多部分上传时应执行的大小，以字节为单位。

Amazon S3 imposes a minimum part size of 5 MB (for parts other than last part), so we have used 5 MB as multipart upload threshold.

亚马逊S3规定最小的部分大小为5MB（对于最后一部分以外的部分），所以我们使用5MB作为多部分上传的门槛。

3.3. Uploading Object

3.3.上传对象

To upload object using TransferManager we simply need to call its upload() function. This uploads the parts in parallel:

使用TransferManager上传对象，我们只需要调用其upload()函数。这将平行地上传各部分。

String bucketName = "baeldung-bucket";
String keyName = "my-picture.jpg";
String file = new File("documents/my-picture.jpg");
Upload upload = tm.upload(bucketName, keyName, file);

TransferManager.upload() returns an Upload object. This can be used to check the status of and manage uploads. We’ll do so in the next section.

TransferManager.upload()返回一个Upload对象。这可以用来检查和管理上传的状态。我们将在下一节中这样做。

3.4. Waiting for Upload to Complete

3.4.等待上传完成

TransferManager.upload() is a non-blocking function; it returns immediately while the upload runs in the background.

TransferManager.upload()是一个非阻塞函数；它立即返回，而上传在后台运行。

We can use the returned Upload object to wait for the upload to complete before exiting the program:

我们可以使用返回的Upload对象来等待上传完成，然后退出程序。

try {
    upload.waitForCompletion();
} catch (AmazonClientException e) {
    // ...
}

3.5. Tracking the Upload Progress

3.5.跟踪上传进度

Track the progress of the upload is quite a common requirement; we can do that with the help of a ProgressListener instance:

跟踪上传的进度是一个很常见的要求；我们可以在ProgressListener实例的帮助下做到这一点。

ProgressListener progressListener = progressEvent -> System.out.println(
  "Transferred bytes: " + progressEvent.getBytesTransferred());
PutObjectRequest request = new PutObjectRequest(
  bucketName, keyName, file);
request.setGeneralProgressListener(progressListener);
Upload upload = tm.upload(request);

The ProgressListener we created will simply continue to print the number of bytes transferred until the upload completes.

我们创建的ProgressListener将简单地继续打印传输的字节数，直到上传完成。

3.6. Controlling Upload Parallelism

3.6.控制上传并行性

By default, TransferManager uses a maximum of ten threads to perform multipart uploads.

默认情况下，TransferManager使用最多10个线程来执行多部分上传。

We can, however, control this by specifying an ExecutorService while building TransferManager:

然而，我们可以通过在构建TransferManager时指定一个ExecutorService来控制这一点。

int maxUploadThreads = 5;
TransferManager tm = TransferManagerBuilder.standard()
  .withS3Client(amazonS3)
  .withMultipartUploadThreshold((long) (5 * 1024 * 1025))
  .withExecutorFactory(() -> Executors.newFixedThreadPool(maxUploadThreads))
  .build();

Here, we used a lambda for creating a wrapper implementation of ExecutorFactory and passed it to withExecutorFactory() function.

在这里，我们使用一个lambda来创建ExecutorFactory的封装实现，并将其传递给withExecutorFactory()函数。

4. Conclusion

4.结论

In this quick article, we learned how to perform multipart uploads using AWS SDK for Java, and we saw how to control some aspects of upload and to keep track of its progress.

在这篇快速文章中，我们学习了如何使用AWS SDK for Java执行多部分上传，我们看到了如何控制上传的某些方面并跟踪其进度。

As always, the complete code of this article is available over on GitHub.

像往常一样，本文的完整代码可在GitHub上获得超过。