Boosting File Uploads With Amazon S3 Transfer Acceleration

This is a case study of how we managed to boost file uploads with Amazon S3 Transfer Acceleration on one of our projects.

Back story

First of all, let me start by introducing the problem we had. There’s an application where we deploy PWAs for our clients from around the world. Each PWA has the ability to take a picture and sent it to Amazon S3 for further processing. Of course, while the image processes user has to wait for the response.

However, because we use centralized S3 that is located in Europe (Ireland) our India-based users encountered some problems. They have to wait much longer for the upload to finish and this was our bottleneck. After researching potential solutions we decided to use Amazon S3 Transfer Acceleration.

What is Amazon S3 Transfer Acceleration?

S3TA allows its users to accelerate data or content uploads from all over the world to a centralized S3 bucket.

It accelerates transfers by automatically routing data to the closest AWS edge location by passing that data over AWS global backbone and by using Network Protocol Optimizations.

S3 Transfer Acceleration enables fast, easy and secure transfers of files over long distances between your client and S3 bucket. It can speed up content transfers by as much as 50-500%. The solution reduces the variability in Internet routing, speeds and congestion that affect the transfers.

Common use cases:

S3 Transfer Acceleration can be used in various cases, but the most common use cases are:

• customers with web and mobile applications that have an international user base to speed up content upload or downloads
• applications that have a lot of user-generated content that create PUTs to the S3 bucket
• customers with globally distributed offices and IT infrastructure that work with files, content and images locally to have an S3 integrated application push that data to a centralized S3 bucket

Implementing the solution

Let’s move on to the process of implementing the solution. First of all, in order to introduce the changes to the application, we have to configure S3 to work with Transfer Acceleration. We can then replace all generated upload links from a region-based URL to an accelerated one, and it’s extremely easy to do so.

1. we need to enable the acceleration status on S3; we use Terraform to manage our infrastructure so all we had to do was enable acceleration_status for our S3 resource: https://www.terraform.io/docs/providers/aws/r/s3_bucket.html#acceleration_status

2. before we upload the file we ask our server for a presigned upload URL so the PWA can upload the file. The generated URL contains the S3 region we operate in, by default: s3.eu-west-1.amazonaws.com/....

   To actually use Transfer Acceleration we have to switch from a regional bucket to s3-accelerate.amazonaws.com

For a more detailed process look into the documentation: https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html

Conclusions

The entire process of migration to the production environment took place the day of implementation. It was fast, easy and generally pain-free. All the changes were transparent to the end-users.

Thanks to S3 Transfer Acceleration we were able to decrease the upload time significantly, so with little to no effort, we achieved a great success.

If you too have a problem with slow uploads or need help with AWS, feel free to contact us!