AWS SQS (Simple Queue Service)

Simple Queue Service

• Amazon SQS is a highly available distributed queue system
• A queue is a temporary repository for messages awaiting for processing and acts as a buffer between the component producer and the consumer
• Amazon SQS
• offers a reliable, highly-scalable, hosted queue for storing messages in transit between computers
• provides fault tolerant, loosely coupled, flexibility of distributed components of applications to send & receive without requiring each component to be concurrently available
• helps build distributed application with decoupled components
• requires no administrative overhead and little configuration
• supports the HTTP over SSL (HTTPS) and Transport Layer Security (TLS) protocols for security
• SQS provides two types of Queues – Standard & FIFO

SQS FIFO Queue Features & Key Points

Refer to Blog Post – SQS FIFO Queues

SQS Standard Queue Features & Key Points

• Redundant infrastructure
• offers reliable and scalable hosted queues for storing messages
• is engineered to always be available and deliver messages
• provides ability to store messages in a fail safe queue
• highly concurrent access to messages
• At-Least-Once delivery
• ensures delivery of each message at least once
• stores copies of the messages on multiple servers for redundancy and high availability
• might deliver duplicate copy of messages, if the servers storing a copy of a message is unavailable when you receive or delete the message and the copy of the message is not deleted on that unavailable server
• Applications should be designed to be idempotent with the ability to handle duplicate messages and not be adversely affected if it processes the same message more than once
• Message Attributes
• SQS message can contain up to 10 metadata attributes.
• take the form of name-type-value triples
• can be used to separate the body of a message from the metadata that describes it.
• helps process and store information with greater speed and efficiency because the applications don’t have to inspect an entire message before understanding how to process it
• Message Sample
• behavior of retrieving messages from the queue depends on whether short (standard) polling, the default behavior, or long polling is used
• With short polling,

1. SQS samples only a subset of the servers (based on a weighted random distribution) and returns messages from just those servers.
2. A receive request might not return all the messages in the queue. But a subsequent receive request would return the message

• With Long polling,

1. request persists for the time specified and returns as soon as the message is available thereby reducing costs and time the message has to dwell in the queue

• Batching
• SQS allows send, receive and delete batching which helps club up to 10 messages in a single batch while charging price for a single message
• helps lower cost and also increases the throughput
• Configurable settings per queue
• All queues don’t have to be alike
• Order
• makes a best effort to preserve order in messages does not guarantee first in, first out delivery of messages 
• can be handled by placing sequencing information within the message and performing the ordering on the client side
• Loose coupling
• removes tight coupling between components
• provides the ability to move data between distributed components of the applications that perform different tasks without losing messages or requiring each component to be always available
• Multiple writers and readers
• supports multiple readers and writers interacting with the same queue as the same time
• locks the message during processing, using Visibility Timeout, preventing it to be processed by any other consumer
• Variable message size
• supports message in any format up to 256KB of text.
• messages larger than 256 KB can be managed using the S3 or DynamoDB, with SQS holding a pointer to the S3 object
• Access Control
• Access can be controlled for who can produce and consume messages to each queue
• Delay Queues
• delay queue allows the user to set a default delay on a queue such that delivery of all messages enqueued is postponed for that time duration
• Dead Letter Queues
• Dead letter queue is a queue for messages that were not able to be processed after a maximum number of attempts
• PCI Compliance
• supports the processing, storage, and transmission of credit card data by a merchant or service provider, and has been validated as being PCI-DSS (Payment Card Industry – Data Security Standard) compliant

SQS Use Cases

• Work Queues
• Decouple components of a distributed application that may not all process the same amount of work simultaneously.
• Buffer and Batch Operations
• Add scalability and reliability to the architecture and smooth out temporary volume spikes without losing messages or increasing latency
• Request Offloading
• Move slow operations off of interactive request paths by enqueueing the request.
• Fan-out
• Combine SQS with SNS to send identical copies of a message to multiple queues in parallel for simultaneous processing.
• Auto Scaling
• SQS queues can be used to determine the load on an application, and combined with Auto Scaling, the EC2 instances can be scaled in or out, depending on the volume of traffic

How SQS Queues Works

• SQS allows queues to be created, deleted and messages can be sent and received from it
• SQS queue retains messages for four days, by default.
• Queues can configured to retain messages for 1 minute to 14 days after the message has been sent.
• SQS can delete a queue without notification if one of the following actions hasn’t been performed on it for 30 consecutive days.
• SQS allows the deletion of the queue with messages in it

Queue and Message Identifiers

Queue URLs

• Queue is identified by a unique queue name within the same AWS account
• SQS assigns each queue with a Queue URL identifier 
• Queue URL is needed to perform any operation on the Queue

Message ID

• Message IDs are useful for identifying messages,
• Each message receives a system-assigned message ID that SQS returns to with the SendMessage response.
• To delete a message, the message’s receipt handle instead of the message ID is needed
• Message ID can be of is 100 characters max

Receipt Handle

• When a message is received from a queue, a receipt handle is returned with the message which is associated with the act of receiving the message rather then the message itself
• Receipt handle is required, not the message id, to delete a message or to change the message visibility
• If a message is received more than once, each time its received, a different receipt handle is assigned and the latest should be used always

Visibility timeout

• Behaviour
• SQS does not delete the message once it is received by a consumer,
• because the system is distributed, there’s no guarantee that the consumer will actually receive the message (it’s possible the connection could break or the component could fail before receiving the message)
• Consumer should explicitly delete the message from the Queue once it is received and successfully processed
• As the message is still available on the Queue, other consumers would be able to receive and process and this needs to be prevented
• SQS handles the above behavior using Visibility timeout.
• SQS blocks the visibility of the message for the Visibility timeout period, which is the time during which SQS prevents other consuming components from receiving and processing that message
• Consumer should delete the message within the Visibility timeout. If the consumer fails to delete the message before the visibility timeout expires, the message is visible again for other consumers.
• Visibility timeout considerations
• clock starts ticking once SQS returns the message
• should be large enough to take into account the processing time for each of the message
• default Visibility timeout for each Queue is 30 seconds and can be changed at the Queue level
• when receiving messages, a special visibility timeout for the returned messages can be set without changing the overall queue timeout using the receipt handle
• can be extended by the consumer, if the consumer thinks it won’t be able to process the message within the current visibility timeout period. SQS restarts the timeout period using the new value
• a message’s Visibility timeout extension applies only to that particular receipt of the message and does not affect the timeout for the queue or later receipts of the message
• SQS has an 120,000 limit for the number of inflight messages per queue i.e. message received but not yet deleted and any further messages would receive an error after reaching the limit

Message Lifecycle

1. Component 1 sends Message A to a queue, and the message is redundantly distributed across the SQS servers.
2. When Component 2 is ready to process a message, it retrieves messages from the queue, and Message A is returned. While Message A is being processed, it remains in the queue but is not returned to subsequent receive requests for the duration of the visibility timeout.
3. Component 2 deletes Message A from the queue to avoid the message being received and processed again once the visibility timeout expires.

SQS Design Patterns

Priority Queue Pattern

1. Use SQS to prepare multiple queues for the individual priority levels.
2. Place those processes to be executed immediately (job requests) in the high priority queue.
3. Prepare numbers of batch servers, for processing the job requests of the queues, depending on the priority levels.
4. Queues have a message “Delayed Send” function, which can be used to delay the time for starting a process.

SQS Job Observer Pattern

1. Enqueue job requests as SQS messages.
2. Have the batch server dequeue and process messages from SQS.
3. Set up Auto Scaling to automatically increase or decrease the number of batch servers, using the number of SQS messages, with CloudWatch, as the trigger to do so.

Case Study:

How Airbnb used AWS to expand its Services!!!

Introduction:-

Airbnb is a community marketplace that allows property owners and travelers to connect with each other for the purpose of renting unique vacation spaces around the world. The Airbnb community users’ activities are conducted on the company’s Website and through its iPhone and Android applications. The San Francisco-based Airbnb began operation in 2008 and currently has hundreds of employees across the globe supporting property rentals in nearly 25,000 cities in 192 countries.

Transition period:-

A year after Airbnb launched, the company decided to migrate nearly all of its cloud computing functions to Amazon Web Services (AWS) because of service administration challenges experienced with its original provider. Nathan Blecharczyk, Co-founder & CTO of Airbnb says, “Initially, the appeal of AWS was the ease of managing and customizing the stack. It was great to be able to ramp up more servers without having to contact anyone and without having minimum usage commitments. As our company continued to grow, so did our reliance on the AWS cloud and now, we’ve adopted almost all of the features AWS provides. AWS is the easy answer for any Internet business that wants to scale to the next level.”

Why Airbnb chose AWS:-

Airbnb has grown significantly over the last 3 years. To support demand, the company uses 200 Amazon Elastic Compute Cloud (Amazon EC2) instances for its application, memcache, and search servers. Within Amazon EC2, Airbnb is using Elastic Load Balancing, which automatically distributes incoming traffic between multiple Amazon EC2 instances. To easily process and analyze 50 Gigabytes of data daily, Airbnb uses Amazon Elastic MapReduce (Amazon EMR). Airbnb is also using Amazon Simple Storage Service (Amazon S3) to house backups and static files, including 10 terabytes of user pictures. To monitor all of its server resources, Airbnb uses Amazon CloudWatch, which allows the company to easily supervise all of its Amazon EC2 assets through the AWS Management Console, Command Line Tools, or a Web services API.

In addition, Airbnb moved its main MySQL database to Amazon Relational Database Service (Amazon RDS). Airbnb chose Amazon RDS because it simplifies much of the time-consuming administrative tasks typically associated with databases. Amazon RDS allows difficult procedures, such as replication and scaling, to be completed with a basic API call or through the AWS Management Console. Airbnb currently uses Multi-Availability Zone (Multi-AZ) deployment to further automate its database replication and augment data durability.

Airbnb was able to complete its entire database migration to Amazon RDS with only 15 minutes of downtime. This quick transition was very important to the fast-growing Airbnb because it did not want its community of users to be shut out of its marketplace for an extended period of time. Tobi Knaup, an engineer at Airbnb says, “Because of AWS, there has always been an easy answer (in terms of time required and cost) to scale our site.”

Tech stack used:

ReactJs for creating the web application using AWS AMPLIFY

Profile Management using AWS COGNITO

Static website hosting on S3 bucket

CLOUDFRONT for CDN

Code repository in CODECOMMIT

Backend API’s using Lambda functions(in Python) which will be triggered via API Gateway

AWS ElastiCache for efficient Search functionality

DynamoDB database for storing data in key-value pairs

Static files like images are kept in an S3 bucket

CloudWatch Alarms are being used for monitoring purpose

AWS SES service to send emails to customers

AWS Pinpoint and Athena for analytics purpose

Airbnb Uses Amazon EFS to Scale CI/CD Pipeline for Expanding Online Marketplace:-

Online travel marketplace Airbnb supports hundreds of critical services on its platform, making it essential to maintain a reliable source control infrastructure. The company uses GitHub Enterprise for both source control and management of its continuous integration/continuous delivery (CI/CD) processes. Airbnb has more than 1,000 engineers, who execute more than 100,000 continuous integration jobs on an average working day. GitHub Enterprise provides the engineers with a single source of truth for all code repositories.

However, source control infrastructure had become an operational headache due to the system’s scaling issues. In the previous system, each mirror instance pulled changes from GitHub Enterprise. This became challenging to maintain because the mirrors could get out of sync with each other. The system did not scale with Airbnb’s increasing Git traffic and hindered the team from focusing on higher-level problem solving and implementing new features.

Airbnb sought a solution it could use to re-architect the source code infrastructure with a simpler storage layer. The system needed to update in seconds and read traffic needed to scale.

AWS Services used:-

Amazon EFS :-

Airbnb realized it could utilize Amazon Elastic File System (Amazon EFS), a simple, scalable system for Linux-based workloads for use with AWS Cloud services and on-premises resources. “Through our research, we discovered Amazon EFS, which we could use to share a single file system mounted on all Git mirror instances,” explains Snyder. “When one Git mirror changed, every other Git mirror was guaranteed to have the same update.” In this way, Airbnb used Amazon EFS to back the real-time image of its GitHub repository data. Mirrors were continuously in sync with the production repository at the scale required for the CI automation pipeline.

Queuing Service Helps Sync Repositories:-

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. Get started with SQS in minutes using the AWS console, Command Line Interface or SDK of your choice, and three simple commands.

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Benefits:-

• Uses single file system to sync GitHub repositories
• Keeps Git mirrors in sync by using a shared file system that allows scaling of CI/CD processes
• Ensures no repository changes are lost during syncing using event-driven queuing approach
• Allows engineers to focus on building system features instead of worrying about scaling

Thank you for your precious time reading this article... Keep Learning, Keep Sharing...