This is a case study of how we managed to boost file uploads with Amazon S3 Transfer Acceleration on one of our projects.
In response to the challenges faced by users in India who experienced slow transfer speeds to the centralized S3 bucket, we implemented a solution: Amazon S3 Transfer Acceleration (S3TA). This technology accelerates data uploads by routing data to the nearest AWS edge location, utilising AWS's global backbone and network protocol optimizations. S3 Transfer Acceleration significantly reduces the variability in internet speeds, routing, and congestion, often speeding up file transfers by 50-500%. To integrate this solution, the S3 was configured for Transfer Acceleration using Terraform, and upload links were switched from region-based URLs to accelerated ones. This resulted in a notable improvement in upload speeds for users across different geographical locations, enhancing the overall user experience.
What you will learn from this article:
- How does Amazon S3 Transfer Acceleration (S3TA) improve upload data transfer speed for global users?
- In what ways does S3 Transfer Acceleration manage to reduce the variability in internet routing, transfer speeds, and congestion?
- How does S3 Transfer Acceleration benefit applications with a high volume of user-generated content?
- What steps does we take to configure S3 to work with Transfer Acceleration?
- How do we utilize Terraform to enable transfer acceleration status on S3 resources?
- What changes were required to switch from a regional S3 bucket to using s3-accelerate.amazonaws.com for uploads?
Back story
First of all, let me start by introducing the problem we had. There’s an application where we deploy PWAs for our clients from around the world. Each PWA has the ability to take a picture and sent it to Amazon S3 for further processing. Of course, while the image processes user has to wait for the response.
However, because we use centralized S3 that is located in Europe (Ireland) our India-based users encountered some problems. They had to wait much longer for the transfer data to finish and this was our bottleneck. After researching potential solutions we decided to use Amazon S3 Transfer Acceleration.
What is Amazon S3 Transfer Acceleration?
S3TA allows its users to accelerate data or content uploads from all over the world to a centralized S3 bucket.
It accelerates transfers by automatically routing data to the closest AWS edge location by passing that data over AWS global backbone and by using Network Protocol Optimizations.
S3 Transfer Acceleration enables fast, easy and secure transfers of files over long distances between your client and S3 bucket. It can speed up content transfers by as much as 50-500%. The solution reduces the variability in Internet routing, speeds and congestion that affect the transfers.
Common use cases
S3 Transfer Acceleration can be used in various cases, but the most common use cases are:
- customers with web and mobile applications that have an international user base to speed up content upload or downloads
- applications that have a lot of user-generated content that create PUTs to the S3 bucket
- customers with globally distributed offices and IT infrastructure that work with files, content and images locally to have an S3 integrated application push that data to a centralized S3 bucket
Implementing the solution
Let’s move on to the process of implementing the solution. First of all, in order to introduce the changes to the application, we have to configure S3 to work with Transfer Acceleration. We can then replace all generated upload links from a region-based URL to an accelerated one, and it’s extremely easy to do so.
- we need to enable the acceleration status on S3; we use Terraform to manage our infrastructure so all we had to do was enable
acceleration_statusfor our S3 resource: https://www.terraform.io/docs/providers/aws/r/s3_bucket.html#acceleration_status before we upload the file we ask our server for a presigned upload URL so the PWA can upload the file. The generated URL contains the S3 region we operate in, by default:
s3.eu-west-1.amazonaws.com/....To actually use Transfer Acceleration we have to switch from a regional bucket to
s3-accelerate.amazonaws.com
For a more detailed process look into the documentation: https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html
Conclusions
The entire process of migration to the production environment took place the day of implementation. It was fast, easy and generally pain-free. All the changes were transparent to the end-users.
Thanks to S3 Transfer Acceleration we were able to decrease the full upload speed and time significantly, so with little to no effort, we achieved great success in transferring faster data transfer well.
