General Amazon SQS Concepts
Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.
SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work.
Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.
SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component.
A queue is a temporary repository for messages that are awaiting processing.
The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing.
This means the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network.
This is known as decoupling / loose coupling.
Helps enable elasticity for your application.
SQS is pull-based, not push-based (like SNS).
Messages are up to 256KB in size.
Messages can be kept in the queue from 1 minute to 14 days.
Default retention period is 4 days.
SQS guarantees that your messages will be processed at least once.
There are two types of queue – standard queues and FIFO queues.
Default queue type.
Nearly unlimited transactions per second.
Guarantee that a message is delivered at least once.
Occasionally more than one copy of a message might be delivered out of order.
Provides best-effort ordering which ensures that messages are generally delivered in the same order as they are sent.
First in First Out (FIFO)
Delivers exactly-once processing.
The order in which messages are sent and received is strictly preserved and a message is delivered once and remains available until a consumer processes and deletes it.
Duplicates are not introduced into the queue.
FIFO queues also support message groups that allow multiple ordered message groups within a single queue.
Limited to 300 transactions per second (TPS), but have all the capabilities of standard queues.
Deduplication with FIFO queues:
- Provide a
MessageDeduplicationIdwith the message.
- The de-duplication interval is 5 minutes.
- Content based duplication – the
MessageDeduplicationIdis generated as the SHA-256 with the message body.
Sequencing with FIFO queues:
- To ensure strict ordering between messages, specify a
- Messages with a different Group ID may be received out of order.
- Messages with the same Group ID are delivered to one consumer at a time.
FIFO queues require the Message Group ID and Message Deduplication ID parameters to be added to messages.
Message Group ID:
- The tag that specifies that a message belongs to a specific message group. Messages that belong to the same message group are guaranteed to be processed in a FIFO manner.
Message Deduplication ID:
- The token used for deduplication of messages within the deduplication interval.
The following table provides a side-by-side comparison of standard and FIFO queues:
The amount of time a message is invisible in the queue after a reader picks it up.
Provided the job is processed before the visibility timeout expires, the message will then be deleted from the queue.
If the job is not processed within the visibility timeout, the message will become visible again and another reader will process it.
This could result in the same message being delivered twice.
The default visibility timeout is 30 seconds.
Increase it if your task takes >30 seconds.
The maximum is 12 hours.
Long polling vs Short Polling
Amazon SQS long polling is a way to retrieve messages from SQS queues.
While the regular short polling returns immediately (even if the message queue is empty), long polling doesn’t return a response until a message arrives in the message queue or the long poll times out.
Long polling can lower costs
Long polling can be enabled at the queue level or at the API level using
Long polling is in effect when the Receive Message Wait Time is a value greater than 0 seconds and up to 20 seconds.
The main task of a dead-letter queue is handling message failure.
A dead-letter queue lets you set aside and isolate messages that can’t be processed correctly to determine why their processing didn’t succeed.
It is not a queue type, it is a standard or FIFO queue that has been specified as a dead-letter queue in the configuration of another standard or FIFO queue.
Messages are moved to the dead-letter queue when the
ReceiveCount for a message exceeds the
maxReceiveCount for a queue.
Dead-letter queues should not be used with standard queues when your application will keep retrying transmission.
Dead-letter queues will break the order of messages in FIFO queues.
SQS Delay Queues
A delay queue postpones delivery of new messages to a queue for a number of seconds.
Messages sent to the Delay Queue remain invisible to consumers for the duration of the delay period.
Default delay is 0 seconds, maximum is 900 seconds (15 minutes).
For standard SQS queues, changing this setting doesn’t affect the delay of messages already in the queue, only new messages.
For FIFO queues, this affects the delay of messages already in the queue.
When to use a delay queue:
- Large distributed applications which may need to introduce a delay in processing.
- You need to apply a delay to an entire queue of messages.
- For example adding a delay of a few seconds to allow updates to sales or stock control databases before sending a notification to a customer confirming an online transaction.
DLQs must be the same type as the source.
SQS Extended Client
The maximum message size in SQS is 256 KB.
You can use Amazon S3 and the Amazon SQS Extended Client Library for Java to manage Amazon SQS messages.
Useful for storing and consuming messages up to 2 GB in size.
You can use the Amazon SQS Extended Client Library for Java library to do the following:
- Specify whether messages are always stored in Amazon S3 or only when the size of a message exceeds 256 KB.
- Send a message that references a single message object stored in an Amazon S3 bucket.
- Get the corresponding message object from an Amazon S3 bucket.
- Delete the corresponding message object from an Amazon S3 bucket.
In-flight security is provided using HTTPS.
Can enable server-side encryption (SSE) using KMS.
- Can set the CMK you want to use.
- Can set the data key reuse period.
- SSE only encrypts the message body not the message attributes.
IAM policy must allow usage of SQS.
Can also specify permissions in an SQS queue access policy:
- Providers finer grained control.
- Control over the requests that come in.
There is no VPC endpoint for SQS (must have Internet access).
Amazon SQS API’s
APIs you should know for the exam:
CreateQueue (aws sqs create-queue):
- Creates a new standard or FIFO queue. You can pass one or more attributes in the request.
DeleteQueue (aws sqs delete-queue):
- Deletes the queue specified by the QueueUrl , regardless of the queue’s contents.
- If the specified queue doesn’t exist, Amazon SQS returns a successful response.
PurgeQueue (aws sqs purge-queue):
- Deletes the messages in a queue specified by the QueueURL parameter.
SendMessage (aws sqs send-message):
- Delivers a message to the specified queue.
ReceiveMessage (aws sqs receive-messsage):
- Retrieves one or more messages (up to 10), from the specified queue.
- Using the WaitTimeSeconds parameter enables long-poll support.
DeleteMessage (aws sqs delete-message):
- Deletes the specified message from the specified queue.
- To select the message to delete, use the ReceiptHandle of the message (not the MessageId which you receive when you send the message).
ChangeMessageVisibility (aws sqs change-message-visibility):
- Changes the visibility timeout of a specified message in a queue to a new value.
- The default visibility timeout for a message is 30 seconds.
- The minimum is 0 seconds.
- The maximum is 12 hours.
To reduce costs or manipulate up to 10 messages with a single action, you can use the following actions: