How to upload large files to AWS S3 without putting pressure to your backend server (NodeJS example)
Instagram
No
date
Oct 22, 2023
slug
how-to-upload-large-files-to-aws-s3-without-putting-pressure-to-backend-server
status
Published
type
Post
Website URL
tags
aws
s3
nodejs
file
file upload
backend
frontend
javascript
summary
Let’s avoid sending file to the backend apis to upload in AWS S3. We can directly upload file to AWS S3 without exposing any credentials
If you know how to use AWS S3 signed url to upload files directly from the UI, then there’s no reason to read it. You already got this.
Amazon Web Services (AWS) S3 is one of the most popular services offered by AWS. It is used to store files. it's just like a big big hard disk, where you make your folder and keep files. They offer SDKs, APIs to upload files from your applications. After working with multiple developers, I have seen a tendency that they somehow ends up involving their backend server for upload operations. First they send file to the backend server through API from the mobile or web app, it gets the buffer data, write it to the disk and then they call s3's putFile method to upload the file from the disk to S3. There is no problem with this approach except this is costly. Suppose if your application allows to upload large files, then your backend server will be responsible in handling huge amount of data which will take extra bandwidth, disk space and is kind of unnecessary as your ultimate goal is to move that file to S3. So here's the solution -
For demonstrating this particular case, I have taken a NodeJS server(express js) as our back-end server and a React JS application for front-end operations.
Step - 1 (Setup AWS SDK in Backend Server)
Open the project folder in terminal and run below command to install aws-sdk for javascript in your project. Assuming you have already installed NodeJS and have a setup of a node js project.
Initiate AWS with your configuration. You can retrieve access key and secret key from AWS Console. Sign in to your account. From top-right, click on account name > Security Credentials. Scroll to Access Keys section to create or get access key and secret key.
Let’s create a file named s3.js and put following contents inside the file -
Next step is to create a function which will be called inside a route defined in your express js application. Assuming you have already created a bucket in S3. I will come to in-depth permissions and access required in later stage. We can change it after creating a S3 bucket. So initially you can proceed with default configurations.
Let’s create a simple route in your express js application, and call this function. Remember to put async-await or create Promise based function.
Now our back-end server is ready. Now we will move to our front-end part. Where on file upload, we need to call this api and complete next steps.
Step - 2 (Calling API from front-end)
First part is to create a simple html with input type file to open file explorer on click. On selecting a file we are going to call a function, which will hold the file in a react state.
Then on clicking the Upload button we will call a function to call the get signed url api in our api server.
Now we will get the signed url inside signedURI variable. Now we will write another function which will put this file to AWS S3, using the signedURI.
Remember we will call an S3 signed url directly from our front-end application and that will be a PUT request. Also this signed url has a expiry time which is defined when we created the signed url from our backend server. So you need to use the url by that time. If you store it somewhere and use later, it might not work because of the expiry time.
Hope the process seems clear. This process gives nearly zero load to your backend server.
But to achieve this entirely you need to configure few settings in your S3 bucket which I am explaining in next section of the article
S3 Configuration
Open your bucket, navigate to Permissions tab, scroll down to Cross-origin resource sharing (CORS) section, and edit. Put following contents inside the editable text area.
For better security, it is recommended you can put your application domain name(s) inside AllowedOrigins instead of putting “*”.
Make sure Access control list (ACL) is enabled and settings look like below -
Hope you went through all the mentioned steps and achieved what you were looking for. If you face any issue, dig through your S3 settings, IAM access. There’s a chance that the access key doesn’t have necessary permission to perform this operations. Make sure your S3 region is correct.
If you want a typescript implementation of the same solution, please let me know. Mail me to hello@snath.dev
Happy Coding 😊