Logo

dev-resources.site

for different kinds of informations.

Highly scalable image storage solution with AWS Serverless at ip.labs - Part 3 Building File API for Uploads and Downloads

Published at
12/23/2024
Categories
aws
serverless
storage
scalability
Author
vkazulkin
Author
9 person written this
vkazulkin
open
Highly scalable image storage solution with AWS Serverless at ip.labs - Part 3 Building File API for Uploads and Downloads

Main author: Firdaws Aboulaye

The File API is essential for managing file uploads and downloads in our architecture. We designed a workflow that includes registration, secure uploads, and file state updates to handle file storage efficiently.

Workflow Overview

  1. Registration: Store file information (metadata, state, user or session information) in the system before the upload.
  2. Presigned URL Generation: Provide a short-lived presigned URL for secure, direct uploads to Amazon S3.
  3. File State Update: After the upload, update the file's state and verify its integrity.

Challenges and Solutions

  • Initial Approach: Using S3 Events
    • Method: Each file upload to S3 triggered an S3 Event, invoking a Lambda function per file to update the database and check for corruption.
    • Issues:
      • High Concurrency: Massive uploads led to exceeding AWS Lambda's concurrency limits.
      • Increased Errors: Overloaded Lambdas resulted in failed executions.
      • Lack of Retries: S3 Events didn't support easy reprocessing of failed events.

Trigger post upload with S3 event design

  • Improved Approach: Leveraging SQS with Batching
    • Method: Replaced S3 Events with messages sent to an SQS queue upon file upload. Configured Lambda functions to process batches of events from the queue.
    • Benefits:
      • Reduced Executions: Batch processing minimized the number of Lambda invocations.
      • Enhanced Error Handling: SQS allowed retries for failed messages with partial batch response
      • Scalability: SQS standard queue has nearly unlimited throughput.

Trigger post upload with SQS event design

This architecture ensures scalability, reliability, and efficient handling of large volumes of uploads.

Events:
  SQSEvent:
    Type: SQS
    Properties:
      Enabled: true
      Queue: !GetAtt PostUploadHandlingSQSQueue.Arn
      FunctionResponseTypes:
        - ReportBatchItemFailures
      BatchSize: 9
Enter fullscreen mode Exit fullscreen mode

This configuration ensures optimized SQS message processing and robust scalability, even during high traffic periods.

The BatchSize property in the Lambda definition, as shown below, allows customizing the maximum number of items retrieved per batch (e.g., BatchSize: 9)


Conclusion

The evolution of our File API showcases the importance of adapting architecture to meet real-world demands. By moving from direct S3 Event triggers to an SQS-based batch processing system, we overcame concurrency limits, reduced errors, and improved scalability.

In the next article, we’ll explore another domain API, diving into its unique challenges and the solutions we implemented to address them. Stay tuned for more insights into our journey of building scalable and reliable APIs!

serverless Article's
30 articles in total
Favicon
Detect Inappropriate Content with AWS Rekognition
Favicon
How can I track my order in Maruti courier?
Favicon
Getting Started with AWS Lambda: A Guide to Serverless Computing for Beginners
Favicon
Deploying Flutter Web Apps using Globe.dev
Favicon
Unlocking the Power of AWS API Gateway and AWS AppSync: Transforming API Development, Functionality, and Use Cases
Favicon
Back to MonDEV 2025
Favicon
Building a Twitter OAuth Authentication Header Generator with Vercel Serverless Functions
Favicon
The Role of Serverless Architecture in Modern Website Development: Benefits and Impact
Favicon
This Serverless Function has crashed. Your connection is working correctly. Vercel is working correctly.
Favicon
Guide to modern app-hosting without servers on Google Cloud
Favicon
Testing AppSync Subscriptions
Favicon
Amazon SES Unwrapped: Key Lessons & Testing Tips for Building Robust Email Systems
Favicon
Spring Boot 3 application on AWS Lambda - Part 14 Measuring cold and warm starts with GraalVM Native Image and memory settings
Favicon
Deploy an Express.js API on Vercel 🚀
Favicon
Serverless for greenfield projects: How data-driven architectures are revolutionizing your software development
Favicon
2024 in Review: Key Highlights in Cloud Databases
Favicon
Managing Secrets in an AWS Serverless Application
Favicon
Build a highly scalable Serverless CRUD Microservice with AWS Lambda and the Serverless Framework
Favicon
[Boost]
Favicon
Becoming an AWS (Serverless) Hero
Favicon
Creating an AWS + NextJS site for the Cloud Resume Challenge
Favicon
Building Serverless APIS with Serverless,Node.js, Typescript, DynamoDB, and Lambda.
Favicon
Highly scalable image storage solution with AWS Serverless at ip.labs - Part 3 Building File API for Uploads and Downloads
Favicon
Serverless self-service IoT certificate management - Part 2
Favicon
How to return meaningful error messages with Zod, Lambda and API Gateway in AWS CDK
Favicon
AWS CDK - aws-lambda-nodejs Module (updated)
Favicon
nestJS Modern Framework for Scalable Applications
Favicon
Serverless OAuth2/OIDC server with OpenIddict 6 and RDS Aurora v2
Favicon
Why Your System Slows Under Load (And What to Do About It)
Favicon
HubSpot offers a powerful platform for creating user interfaces (UI) and serverless functions using React, allowing users to develop highly customizable pages and forms directly within the CRM itself.

Featured ones: