Amazon S3 – Upload/download large files to S3 with SpringBoot Amazon S3 MultipartFile application

Amazon S3 – Upload_download large files to S3 with SpringBoot Amazon S3 MultipartFile application

In the previous post, we had learned how to upload a file to Amazon S3 in a single operation. But when file size reaches 100 MB, we should consider using multipart uploads with the advantages: improved throughput, quick recovery from any network issues. So Today, JavaSampleApproach will guide how to upload/download large files to S3 with SpringBoot Amazon S3 MultipartFile application.

Related post:
Amazon S3 – Upload/Download files with SpringBoot Amazon S3 application


I. Technologies

– Java 8
– Maven 3.6.1
– Spring Tool Suite: Version 3.8.4.RELEASE
– Spring Boot: 1.5.4.RELEASE
– Amazon S3

II. SpringBoot Amazon S3 MultipartFile application

AWS SDK for Java exposes a high-level API TransferManager class that simplifies multipart upload/download files.

SpringBoot Amazon S3 MultipartFile - upload-download - architecture

1. Upload/Download files

– Create an instance of the TransferManager:

TransferManager tm = TransferManagerBuilder.standard()
							.withS3Client(s3client())
							.withDisableParallelDownloads(false)
							.withMinimumUploadPartSize(Long.valueOf(5 * MB))
							.withMultipartUploadThreshold(Long.valueOf(16 * MB))
							.withMultipartCopyPartSize(Long.valueOf(5 * MB))
							.withMultipartCopyThreshold(Long.valueOf(100 * MB))
							.withExecutorFactory(()->createExecutorService(20))
							.build();

-> minimumUploadPartSize & multipartUploadThreshold define the range size of each parts for uploading.
-> multipartCopyPartSize & multipartCopyThreshold define the range size of each parts for downloading.

– Use TransferManager.upload and TransferManager.download to upload/download files:

// Upload File
Upload upload = transferManager.upload(PutObjectRequest request)

// Download File
Download download = transferManager.download(GetObjectRequest request, new File(downloadFilePath))

Note: TransferManager processes all transfers asynchronously, so this call will return immediately.
For block and wait for the upload/download to finish, we use:

...
// Wait for upload completely
upload.waitForCompletion();

...

// Wait for download completely
download.waitForCompletion();

...
2. Abort Multipart Uploads

Amazon S3 will charge for all storage associated with uploaded parts, so you need to abort the failed multipart upload to remove any uploaded parts from S3.

int oneDay = 1000 * 60 * 60 * 24;
Date oneDayAgo = new Date(System.currentTimeMillis() - oneDay);

try {
	
	tm.abortMultipartUploads(bucketName, oneDayAgo);
	
} catch (AmazonClientException e) {
	logger.error("Unable to upload file, upload was aborted, reason: " + e.getMessage());
}

-> The above code will abort all uploadings which are not completely after 1 day processing.

3. Tracking Multipart Upload Progress

Amazon S3 APIs provides a listen interface, ProgressListener, to track the progress when uploading/downloading data using the TransferManager class.

...

// For Upload files
final PutObjectRequest request = new PutObjectRequest(bucketName, keyName, new File(uploadFilePath));

request.setGeneralProgressListener(new ProgressListener() {
	@Override
	public void progressChanged(ProgressEvent progressEvent) {
		String transferredBytes = "Uploaded bytes: " + progressEvent.getBytesTransferred();
		logger.info(transferredBytes);
	}
});

...

// For Download files
final GetObjectRequest request = new GetObjectRequest(bucketName, keyName);

request.setGeneralProgressListener(new ProgressListener() {
	@Override
	public void progressChanged(ProgressEvent progressEvent) {
		String transferredBytes = "Downloaded bytes: " + progressEvent.getBytesTransferred();
		logger.info(transferredBytes);
	}
});

...

III. Practice

In the tutorial, we create a SpringBoot project to upload/download file with TransferManager class.

SpringBoot Amazon S3 MultipartFile - upload-download - project structure

Step to do:
– Create SpringBoot project
– Configure TransferManager
– Implement S3 upload/download services
– Implement S3 client
– Run and check results

1. Create SpringBoot project

Using Spring Tool Suite to create a SpringBoot project. Then add aws-java-sdk dependency:



	com.amazonaws
	aws-java-sdk
	1.11.106

2. Configure TransferManager

Create a S3Config.java to config TransferManager:

package com.javasampleapproach.s3.multipartfile.config;

import static com.amazonaws.services.s3.internal.Constants.MB;

import java.util.Date;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.ThreadPoolExecutor;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import com.amazonaws.AmazonClientException;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.TransferManagerBuilder;

@Configuration
public class S3Config {
	
	@Value("${jsa.aws.access_key_id}")
	private String awsId;

	@Value("${jsa.aws.secret_access_key}")
	private String awsKey;
	
	@Value("${jsa.s3.region}")
	private String region;
	
	@Value("${jsa.s3.bucket}")
	private String bucketName;
	
	private Logger logger = LoggerFactory.getLogger(S3Config.class);

	@Bean
	public AmazonS3 s3client() {
		
		BasicAWSCredentials awsCreds = new BasicAWSCredentials(awsId, awsKey);
		
		AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
								.withRegion(Regions.fromName(region))
		                        .withCredentials(new AWSStaticCredentialsProvider(awsCreds))
		                        .build();
		
		return s3Client;
	}
	
	@Bean
	public TransferManager transferManager(){
		
		TransferManager tm = TransferManagerBuilder.standard()
									.withS3Client(s3client())
									.withDisableParallelDownloads(false)
									.withMinimumUploadPartSize(Long.valueOf(5 * MB))
									.withMultipartUploadThreshold(Long.valueOf(16 * MB))
									.withMultipartCopyPartSize(Long.valueOf(5 * MB))
									.withMultipartCopyThreshold(Long.valueOf(100 * MB))
									.withExecutorFactory(()->createExecutorService(20))
									.build();
		
		int oneDay = 1000 * 60 * 60 * 24;
		Date oneDayAgo = new Date(System.currentTimeMillis() - oneDay);
		
		try {
			
			tm.abortMultipartUploads(bucketName, oneDayAgo);
			
		} catch (AmazonClientException e) {
			logger.error("Unable to upload file, upload was aborted, reason: " + e.getMessage());
		}
		
		return tm;
	}
	
	private ThreadPoolExecutor createExecutorService(int threadNumber) {
        ThreadFactory threadFactory = new ThreadFactory() {
            private int threadCount = 1;

            public Thread newThread(Runnable r) {
                Thread thread = new Thread(r);
                thread.setName("jsa-amazon-s3-transfer-manager-worker-" + threadCount++);
                return thread;
            }
        };
        return (ThreadPoolExecutor)Executors.newFixedThreadPool(threadNumber, threadFactory);
    }
}

– Open application.properties, add configuration:

jsa.aws.access_key_id=***(change the key)
jsa.aws.secret_access_key=***(change the key)
jsa.s3.bucket=jsa-s3-bucket
jsa.s3.region=us-east-1

jsa.s3.uploadfile=C:\\s3\\upload\\upload-multipart-file-to-amazon-s3.rar
jsa.s3.downloadfile=C:\\s3\\download\\upload-multipart-file-to-amazon-s3.rar
jsa.s3.key=upload-multipart-file-to-amazon-s3.rar

logging.file=jsa-app.log

– For logging in file, under /src/main/resources folder, adding logging.file=jsa-app.log then creating a logback-spring.xml file:



    
    
    
    
        
    

3. Implement S3 upload/download services

– Create an S3Services interface:

package com.javasampleapproach.s3.multipartfile.services;

public interface S3Services {
	public void uploadFile(String keyName, String uploadFilePath);
	public void downloadFile(String keyName, String downloadFilePath);
}

– Implement S3ServicesImpl:

package com.javasampleapproach.s3.multipartfile.services.impl;

import java.io.File;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.event.ProgressEvent;
import com.amazonaws.event.ProgressListener;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.transfer.Download;
import com.amazonaws.services.s3.transfer.TransferManager;
import com.amazonaws.services.s3.transfer.Upload;
import com.javasampleapproach.s3.multipartfile.services.S3Services;

@Service
public class S3ServicesImpl implements S3Services{
	
	protected Logger logger = LoggerFactory.getLogger(S3ServicesImpl.class);
	
	@Autowired
	protected TransferManager transferManager;

	@Value("${jsa.s3.bucket}")
	protected String bucketName;

	/**
	 * UPLOAD FILE to Amazon S3
	 */
	@Override
	public void uploadFile(String keyName, String uploadFilePath) {
		final PutObjectRequest request = new PutObjectRequest(bucketName, keyName, new File(uploadFilePath));
		
		request.setGeneralProgressListener(new ProgressListener() {
			@Override
			public void progressChanged(ProgressEvent progressEvent) {
				String transferredBytes = "Uploaded bytes: " + progressEvent.getBytesTransferred();
				logger.info(transferredBytes);
			}
		});

		Upload upload = transferManager.upload(request);
		
		// Or you can block and wait for the upload to finish
		try {
			
			upload.waitForCompletion();
			
		} catch (AmazonServiceException e) {
			logger.info(e.getMessage());
		} catch (AmazonClientException e) {
			logger.info(e.getMessage());
		} catch (InterruptedException e) {
			logger.info(e.getMessage());
		}
	}
	
	/**
	 * DOWNLOAD FILE from Amazon S3
	 */
	@Override
	public void downloadFile(String keyName, String downloadFilePath) {
		final GetObjectRequest request = new GetObjectRequest(bucketName, keyName);
		
		request.setGeneralProgressListener(new ProgressListener() {
			@Override
			public void progressChanged(ProgressEvent progressEvent) {
				String transferredBytes = "Downloaded bytes: " + progressEvent.getBytesTransferred();
				logger.info(transferredBytes);
			}
		});
		
		Download download = transferManager.download(request, new File(downloadFilePath));
		
		try {
			
			download.waitForCompletion();
			
		} catch (AmazonServiceException e) {
			logger.info(e.getMessage());
		} catch (AmazonClientException e) {
			logger.info(e.getMessage());
		} catch (InterruptedException e) {
			logger.info(e.getMessage());
		}
	}
}

4. Implement S3 client

In main class of SpringBoot application, using S3Services to upload/download a file:

package com.javasampleapproach.s3.multipartfile;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ApplicationContext;

import com.javasampleapproach.s3.multipartfile.services.S3Services;

@SpringBootApplication
public class SpringS3AmazonMultipartFileApplication implements CommandLineRunner{

	
	@Autowired
	S3Services s3Services;
	
	@Value("${jsa.s3.uploadfile}")
	private String uploadFilePath;
	
	@Value("${jsa.s3.downloadfile}")
	private String downloadFilePath;
	
	@Value("${jsa.s3.key}")
	private String key;
	
	public static void main(String[] args) {
		ApplicationContext context = SpringApplication.run(SpringS3AmazonMultipartFileApplication.class, args);
        System.exit(SpringApplication.exit(context));
	}

	@Override
	public void run(String... args) throws Exception {
		System.out.println("---------------- START UPLOAD FILE ----------------");
		s3Services.uploadFile(key, uploadFilePath);
		System.out.println("---------------- START DOWNLOAD FILE ----------------");
		s3Services.downloadFile(key, downloadFilePath);
		System.out.println("DONE!");
	}
}

5. Run and check results

Prepare:
– Setup Amazon S3 by the guide.
– In folder: C:\s3\upload, prepare a upload file:

SpringBoot Amazon S3 MultipartFile - upload-download - prepare upload file

Build and run above SpringBoot application with commandlines {mvn clean install, mvn spring-boot:run}. Waiting a minutes, on console log, we got:

---------------- START UPLOAD FILE ----------------
---------------- START DOWNLOAD FILE ----------------
DONE!

-> Results:

– Upload file:

SpringBoot Amazon S3 MultipartFile - upload-download - result upload file

– When downloading, open folder C:\s3\download, we got:

SpringBoot Amazon S3 MultipartFile - upload-download - result download file in progress

– After downloading, open folder C:\s3\download, we got:

SpringBoot Amazon S3 MultipartFile - upload-download - result download done

– Open log file jsa-app.log, see:

// For Upload file
...

[java-sdk-progress-listener-callback-thread] c.j.s.m.services.impl.S3ServicesImpl     : Uploaded bytes: 8192
[java-sdk-progress-listener-callback-thread] c.j.s.m.services.impl.S3ServicesImpl     : Uploaded bytes: 8192
[java-sdk-progress-listener-callback-thread] c.j.s.m.services.impl.S3ServicesImpl     : Uploaded bytes: 8192

...

// For Download file
...

[java-sdk-progress-listener-callback-thread] c.j.s.m.services.impl.S3ServicesImpl     : Downloaded bytes: 17408
[java-sdk-progress-listener-callback-thread] c.j.s.m.services.impl.S3ServicesImpl     : Downloaded bytes: 17408
[java-sdk-progress-listener-callback-thread] c.j.s.m.services.impl.S3ServicesImpl     : Downloaded bytes: 17408

...

IV. Sourcecode

SpringS3AmazonMultipartFile

One thought on “Amazon S3 – Upload/download large files to S3 with SpringBoot Amazon S3 MultipartFile application”

  1. hi. I have implemented some code where I upload a multipart file to S3. I tried uploading a 30MB file and it takes my app around 5 minutes to complete the download, which I think is very slow.

    ObjectMetadata objectMetadata = new ObjectMetadata();
    objectMetadata.setContentLength(multipartFile.getSize());

    PutObjectRequest objectRequest = new PutObjectRequest(bucketName, fileName, multipartFile.getInputStream(), objectMetadata).withSSEAwsKeyManagementParams(new SSEAwsKeyManagementParams(kmsKey));

    s3Client.putObject(objectRequest );

    Just wanted to ask, your code above – does the file get streamed directly to S3? or does it write to a temp disk space or something before uploading?

Leave a Reply

Your email address will not be published. Required fields are marked *