CASE STUDY OF AWS SQS

Amazon Simple Queue Service

Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead letter queues and cost allocation tags. It provides a generic web services API that you can access using any programming language that the AWS SDK supports.Amazon SQS supports both standard and FIFO queues.

How does SQS work?

SQS provides an API endpoint to submit messages and another endpoint to read messages from a queue. Each message can only be retrieved once, and you can have many clients submitting messages to and reading messages from a queue at the same time.

The messages that SQS handles can be unformatted strings, XML or JSON. Because SQS guarantees “exactly once” delivery, and because you can concurrently submit messages to and read messages from a given queue, SQS is a good option for integrating multiple independent systems.

You might well be asking: why use SQS if you can have an internal HTTP API for each service? While HTTP APIs are an accessible way to expose software systems to external users, it’s not the most efficient mechanism when it comes to integrating purely internal systems. A messaging queue is more lightweight. In particular, SQS also handles things like automated retries, preserving queue state across multiple availability zones in AWS, and keeping track of expiration timeouts on all messages.

Why is SQS an essential part of the Serverless ecosystem?

In an optimal Serverless architecture, when using multiple microservices, you want your services to be independent from each other. This means that they use separate databases and even run in different network segments but can still communicate with each other. The distributed systems term for this kind of independence is a decoupled system. When services are decoupled, you can’t pass messages, for example, by having them both write to a shared database. The available options under this model are: implement an API in each service, or use message queues to pass information between services.

While in certain cases HTTP and gRPC APIs might be a good choice for service intercommunication, message queues tend to be a more lightweight approach and conducive to faster iteration.

And aside from being a queue system that helps you iterate faster, SQS is also fully managed. This makes it a great choice for Serverless applications running in AWS because:

• Many Serverless applications already rely on managed services to reduce maintenance load on the team developing the applications, making SQS’s managed model a good fit.
• SQS can run in the same regions in AWS as your applications, which means high network throughput and low latency. This results in fast message delivery times using SQS.
• SQS provides a simple HTTP API which is easy to implement using the AWS SDK.

These characteristics make SQS a popular choice for Serverless developers and, consequently, an important part of the Serverless ecosystem.

How does SQS integrate with other AWS services?

Most interesting for Serverless developers is SQS‘s integration with Amazon Lambda: SQS can act as an AWS Lambda event source. When configured, every SQS message triggers a Lambda function run that processes a batch of SQS messages.

Another useful integration is with SNS an SQS queue can subscriber to an SNS topic. We cover the differences between SQS and SNS below, in the When to use SQS vs SNS section.

SQS also provides standard integrations for monitoring and debugging SQS queues using Amazon CloudWatch and AWS X-Ray.

Serverless developers can manually integrate an SQS queue with any other AWS service by writing code that uses the AWS SDK to submit messages to SQS and read them from there, or by using the SQS API directly.

Benefits of using Amazon SQS

• Security – You control who can send messages to and receive messages from an Amazon SQS queue.
• Server side encryption lets you transmit sensitive data by protecting the contents of messages in queues using keys managed in AWS Key Management Service (AWS KMS).
• Durability – For the safety of your messages, Amazon SQS stores them on multiple servers. Standard queues support atleast once message delivery and FIFO queues support exactly once message processing.

• Availability – Amazon SQS uses redundent infrastructure to provide highly-concurrent access to messages and high availability for producing and consuming messages.
• Scalability – Amazon SQS can process each bufferd request independently, scaling transparently to handle any load increases or spikes without any provisioning instructions.
• Reliability – Amazon SQS locks your messages during processing, so that multiple producers can send and multiple consumers can receive messages at the same time.
• Customization – Your queues don't have to be exactly alike—for example, you can set a default delay on queue. You can store the contents of messages larger than 256 KB using Amazon simple storage service or Amazon DynamoDB, with Amazon SQS holding a pointer to the Amazon S3 object, or you can split a large message into smaller messages.

Disadvantages of using SQS

Using SQS can also create challenges for Serverless developers, as described hereafter.

High cost at scale With pay per use pricing, if the number of messages you send is very high, your SQS bill can be quite significant. Part of SQS pricing is data transfer charges, and those can add up if you send larger messages, or if you process messages from outside the main AWS region in which the queue is located. In some cases, when running at scale with millions of messages processed every day, the cost of using SQS might be higher than the cost of operating your own queue system, even including the overhead to manage your own solution.

Lack of support for broadcast messages With its “exactly once” delivery, SQS doesn’t support a way for multiple entities to retrieve the same message, making SQS not so useful for one-to-many broadcasts.

To address this, in the cases where one-to-many delivery is important, developers can use SQS alongside SNS.

Reduced control over performance When running a message queue system at scale, something you may well end up wanting to do is to fine-tune its performance to suit your needs. With SQS this isn’t an option: the service is fully managed, and you don’t get to look under the hood.

Amazon SQS limits

Amazon SQS has a few service limits that you should consider before using SQS in production:

Queue delay. Minimum 0 seconds, maximum 15 minutes. The built-in delay functionality in SQS queues exists to allow inserting a pause between when a message is produced and when it’s visible in the queue. If you need that delay to be longer than 15 minutes, SQS can’t offer a delay that long, so you’ll need to implement the delay in the system producing the messages before it pushes them to SQS.

In-flight messages. SQS has a maximum of 120,000 messages or 20,000 messages (FIFO queue) that can be in flight: in other words, messages that have been received by a consumer but not yet removed from the queue. In most well-architected systems, it would be hard to reach this number of in-flight messages unless you are in an outage scenario. Before using SQS in production, determine if this could become an issue for you, especially in edge cases and possible error states for your application.

Message attributes. Each message is allowed to have a maximum of 10 metadata attributes. If you’d like to have more metadata than that, consider including it in the message itself rather than as a field attached to the message.

Message content. Messages you submit to SQS queues can be in plain text (unformatted) or in JSON or XML format. Other formats aren’t supported.

Message retention. Minimum: 60 seconds. Maximum: 14 days. If you think you might need shorter or longer retention times, SQS might not work for you.

Maximum message size is 256KB. If you need to send larger messages, consider uploading the larger message to S3 and including a reference to the S3 object as part of the message.

Message visibility timeout. You might be wondering 'What is SQS visibility timeout?' The visibility timeout is a configurable time period during which SQS temporarily "hides" a message that has been received by a consumer to avoid its being processed by other consumers. After the visibility timeout expires, if the message hasn’t been removed from the queue by the consumer that received it, it becomes visible to other queue consumers. The default visibility timeout is 30 seconds, the minimum is 0 seconds and the maximum is 12 hours.

Consider all these limits for your specific use case, and take future growth into account, especially if you are building a growth-oriented system.

Companies Using AWS SQS

Following are the popular companies who use Amazon Simple Queue Service:

• BMW
• NASA
• EMS Driving Fuel IQ
• Capital One
• Redbus
• OyesterChange Healthcare

AWS SQS Tutorial – Queue & Functions

a. Standard Queue

• It has a benefit of supporting an ample amount of transactions per second per API action.
• As the message is delivered on at a time but at the same time, it delivers more than one copy of a message.
• It may happen that the message delivered is in the different order from the source in which they were sent.

b. FIFO Queue

• It has a high throughput which can send 300 messages per second which include 300 send, receive, and delete operation per second.
• The message is not duplicated it is stored with the customer until and unless customer deletes it.
• The messages are treated in first in first out order as the message sent and received is strictly preserved.

THANK YOU!!!