Simple Queue Service by AWS

What is SQS?

• SQS stands for Simple Queue Service.
• SQS is a managed message queue service offered by Amazon Web Services (AWS).
• Amazon SQS gives us the access to a message queue that can be used to store messages while waiting for a computer to process them.
• If we got two EC2 instances which are pulling the SQS Queue. We can configure the Autoscaling Group if a number of messages go over a certain limit. Suppose the number of messages exceeds 10, then we can add additional EC2 instance to process the job faster. In this way, SQS provides elasticity.
•  SQS is an easy solution for passing messages between different parts of software systems that run in the Cloud.

How does SQS work?

SQS provides an API endpoint to submit messages and another endpoint to read messages from a queue. Each message can only be retrieved once, and we can have many clients submitting messages to and reading messages from a queue at the same time.

The messages that SQS handles can be unformatted strings, XML or JSON. Because SQS guarantees “exactly once” delivery, and because we can concurrently submit messages to and read messages from a given queue, SQS is a good option for integrating multiple independent systems.

How does SQS integrate with other AWS services?

Most interesting for Serverless developers is SQS‘s integration with Amazon Lambda: SQS can act as an AWS Lambda event source. When configured, every SQS message triggers a Lambda function run that processes a batch of SQS Messages.

Benefits of SQS

Scalability -  We don’t need to scale the queues; all the scaling and performance-at-scale aspects are taken care of by AWS.

Pay only for what we Use - Using SQS, we will only charged for the messages we read and write.

Ease of Setup -  Since SQS is a managed service, so we don’t need to set up any infrastructure. We can simply use the API to read and write messages, or use the SQS <-> Lambda integration.

Options for Standard and FIFO Queues - When creating an SQS queue, we have to choose any one queue from standard queue and a FIFO queue.

Automatic deduplication for FIFO Queues - SQS will remove any duplicate messages for us. This makes FIFO queues on SQS suitable for tasks where it’s critical to have each task done exactly once.

A Separate Queue (dead-letter) for Unprocessed Messages -This feature of SQS is useful for debugging. All messages that couldn’t be processed are sent into a "dead-letter" queue where we can inspect them.

Disadvantages of using SQS

• High Cost at Scale - In some cases, when running at scale with millions of messages processed every day, the cost of using SQS might be higher than the cost of operating our own queue system.
• Lack of Support for Broadcast Messages - With its “exactly once” delivery, SQS doesn’t support a way for multiple entities to retrieve the same message, making SQS not so useful for one-to-many broadcasts.

What is SQS used for?

• Decoupling Microservices - If microservices run in AWS, and especially if those are Serverless services, SQS is a great choice for that aspect of the communication.
• Sending tasks between different parts of your System - We can also use it in any kind of application that needs to communicate tasks to other systems.
• Distributing Workloads from a Central Node to Worker Nodes - We can frequently find messaging systems in the flows of distributed large workloads like map-reduce operations. For these kinds of operations, it’s essential to be able to maintain a queue of all the tasks that need to be processed, efficiently distribute the tasks between the machines or functions doing the work, and guarantee that every part of the work is only done once.
• Scheduling Batch Jobs - SQS is a great option for scheduling batch jobs for two reasons. First, it maintains a durable queue of all the scheduled jobs, which means we don’t need to keep track of the job status. Second, it integrates with AWS Lambda; if we’re using AWS Lambda to process the batch jobs, SQS automatically launches our Lambda functions once the data is available for them to process.

Amazon SQS limits

Queue Delay - Minimum 0 seconds, maximum 15 minutes. If we need that delay to be longer than 15 minutes, SQS can’t offer a delay that long. For this, we need to implement the delay in the system producing the messages before it pushes them to SQS.

In-Flight Messages - SQS has a maximum of 120,000 messages (standard queue) or 20,000 messages (FIFO queue) that can be in flight. In other words, messages that have been received by a consumer but not yet removed from the queue.

Message Attributes - Each message is allowed to have a maximum of 10 metadata attributes.

Message Content. Messages which are submitted to the SQS queues can be in plain text or in JSON or XML format. Other formats aren’t supported.

Message Retention - Minimum: 60 seconds. Maximum: 14 days.

Maximum Message Size - Maximum message size is 256KB.

Message Visibility Timeout - The visibility timeout is a time period during which SQS temporarily "hides" a message that has been received by a consumer to avoid its being processed by other consumers. After the visibility timeout expires, if the message hasn’t been removed from the queue by the consumer that received it, it becomes visible to other queue consumers. The default visibility timeout is 30 seconds, the minimum is 0 seconds and the maximum is 12 hours.

If we didn't have SQS, then what happened?

A web server passes the information to an application server and then application server queried an Airline service. If an Application server crashes, then a user loses its query. One of the great thing about SQS is that data is queued in the SQS even if the application server crashes, the message in the queue is marked as an invisible in a timeout interval window. When the timeout runs out, message reappears in the queue; then a new EC2 instance can use this message to perform its job. Therefore, we can say that SQS removes the application server dependency.

SNS Use Cases

Suppose the user wants to look for a package holiday and wants to look at the best possible flight. A user types a query in a browser, it then hits the EC2 instance. An EC2 instance looks "What the user is looking for?", it then puts the message in a queue to the SQS. An EC2 instance pulls queue. An EC2 instance continuously pulling the queue and looking for the jobs to do. Once it gets the job, it then processes it. It interrogates the Airline service to get all the best possible flights. It sends the result to the web server, and the web server sends back the result to the user. A User then selects the best flight according to his or her budget.

THANK YOU!!