Why we choose long polling over short polling with Aws SQS?
Among the wide range of AWS services, for one of our requirement, we choose AWS SQS(Simple Queue Service) over RabbitMQ and Kafka which have similar functionality.Already our client was trustworthy with the AWS services which was one of the reason in choosing AWS SQS.
What we really want to achieve was a pub-sub mechanism, where various microservices will be publishing and we need other applications to listen and process those contents. For this, we have the SNS(Simple Notification Service) , SQS combination in place.
Discussing SNS is out of scope in this and for people who want to get a basic understanding of this combo, refer this link.
For polling the SQS queue, AWS provides two different ways.One is short polling and other is long polling. Checking the documentation for this API in AWS, you may miss some key differences which we will be pointing out here.
Short polling is a common term we used and the functionality seems to same here as well. With short polling, you can achieve the response very quickly without any delay. With AWS SQS short polling, you will get a quick response with available set of messages in the queue. The real problem here is, if the number of messages is very few(say 1000), you may or may not get any messages in the response.
Reason:
- With short polling, we have wait_time_seconds as 0 and it queries the only the limited number of servers for messages and that also will be randomly chosen.
Because of the above case if the number of messages in the queue is very less, in order to get some non-empty response you may have to make multiple requests with SQS. Also that you make a request via short polling and the response doesn't contain any messages never means that SQS queue is empty.
If your architecture is in such a way that you have a lot of messages in the queue say some 5k or more, chances of getting valid responses are high. Also if you don't want any delays in the request-response cycle short polling is an option.
AWS long polling for sqs means we will have a value greater than 0 for the configuration parameter wait_time_seconds.Different ways of setting this value are,
- Set this config directly with the SQS queue during the creation of the queue
- Set the value while calling the receive_message api for sqs which we used to retrieve the messages from sqs.Sample request is below,
require 'aws-sdk-sqs' # v2: require 'aws-sdk'
sqs = Aws::SQS::Client.new(region: 'us-west-2')
resp = sqs.receive_message(
queue_url: URL,
max_number_of_messages: 10,
wait_time_seconds: 5
)
# :wait_time_seconds (Integer) — The duration (in seconds) for which the call waits for a message to arrive in the queue before returning. If a message is available, the call returns sooner than WaitTimeSeconds. If no messages are available and the wait time expires, the call returns successfully with an empty list of messages.
Advantages you are getting with long polling is , even if the number of messages are fewer, based on the wait_time_seconds value, chances of getting non-empty responses are less.You can have a maximum value of 20 seconds for this configuration.Increasing the wait_time_seconds may affect the request-response time.
Depending on the requirement or kind of functionality you are expecting you can choose between short polling and long polling.From my experience in using SQS , i would say having a value of 5 seconds for wait_time_second which in turn makes the request to poll long(long polling) is always better.
References:-