AWS Application Integration Services
Please use the menu below to navigate the article sections:
The AWS Application Integration Services are a suite of services that can be used to enable communication between application components in a decoupled architecture. This includes microservices application, distributed systems, and serverless applications.
AWS Application Integration Services
The AWS application integration services are a family of services that enable decoupled communication between applications.
These services provide decoupling for microservices, distributed systems, and serverless applications.
AWS application integration services allow you to connect apps, without needing to write custom code to enable interoperability.
Decoupled applications can interoperate whilst being resilient to the failure or overload of any individual component.
The following services are involved with application integration:
|Service||What it does||Example use cases|
|Simple Queue Service (SQS)||Messaging queue; store and forward patterns||Building distributed / decoupled applications|
|Simple Notification Service (SNS)||Set up, operate, and send notifications from the cloud||Send email notification when CloudWatch alarm is triggered|
|Step Functions||Out-of-the-box coordination of AWS service components with visual workflow||Order processing workflow|
|Simple Workflow Service (SWF)||Need to support external processes or specialized execution logic||Human-enabled workflows like an order fulfilment system or for procedural requests
Note: AWS recommends that for new applications customers consider Step Functions instead of SWF
|Amazon MQ||Message broker service for Apache Active MQ and RabbitMQ||Need a message queue that supports industry standard APIs and protocols; migrate queues to AWS|
|Amazon Kinesis||Collect, process, and analyze streaming data.||Collect data from IoT devices for later processing|
Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.
The pub/sub functionality provides messaging for high-throughput, push-based, many-to-many use cases.
Amazon SNS is used for sending notifications between distributed systems, microservices, and event-driven serverless applications.
Push notifications can go to Apple, Google, Fire OS, and Windows devices as well as Android devices in China with Baidu Cloud Push.
Amazon SNS can also send notifications via SMS text message, email, SQS queues or to any HTTP endpoint.
Amazon SNS notifications can also trigger Lambda functions.
Amazon SNS is inexpensive and based on a pay-as-you-go model with no upfront costs.
SNS uses a pub-sub model whereby users or applications subscribe to SNS topics.
Amazon SNS provides decoupling of your applications so messages can be processed asynchronously.
Multiple recipients can be grouped using Topics.
A topic is an “access point” for allowing recipients to dynamically subscribe for identical copies of the same notification.
One topic can support deliveries to multiple endpoint types.
All messages are stored redundantly across multiple availability zones.
Provides instantaneous, push-based delivery.
Flexible message delivery is provided over multiple transport protocols.
SNS Subscribers and Endpoints
When subscribing to an SNS topic the following endpoint types are supported:
- Amazon Kinesis Data Firehose.
- Amazon SQS.
- AWS Lambda.
- Platform application endpoint (mobile push).
Your publisher systems can fanout messages to many subscriber systems including Amazon SQS queues, AWS Lambda functions and HTTPS endpoints, for parallel processing, and Amazon Kinesis Data Firehose.
You can subscribe one or more Amazon SQS queues to an Amazon SNS topic from a list of topics available for the selected queue.
Amazon SQS manages the subscription and any necessary permissions.
When you publish a message to a topic, Amazon SNS sends the message to every subscribed queue.
Fanout is supported for application-to-application (A2A) messaging:
- Fanout to Kinesis Data Firehose delivery streams.
- Fanout to Lambda functions.
- Fanout to Amazon SQS queues.
- Fanout to HTTP/S endpoints.
- Fanout to AWS Event Fork Pipelines.
Amazon Simple Queue Service (SQS) is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component.
Amazon SQS enables you to send, store, and receive messages between software components.
An Amazon SQS queue is a temporary repository for messages that are awaiting processing.
The SQS queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing.
The SQS queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network.
This is known as decoupling / loose coupling and helps to enable elasticity for your application.
Amazon SQS is pull-based, not push-based (like Amazon SNS).
Messages are up to 256KB in size.
Messages can be kept in the queue from 1 minute to 14 days.
The default retention period is 4 days.
SQS guarantees that your messages will be processed at least once.
Queue names must be unique within a region.
There are two types of queue – standard queues and FIFO queues.
Default queue type.
Nearly unlimited transactions per second.
Guarantee that a message is delivered at least once.
Occasionally more than one copy of a message might be delivered out of order.
Provides best-effort ordering which ensures that messages are generally delivered in the same order as they are sent.
First in First Out (FIFO)
Delivers exactly-once processing.
The order in which messages are sent and received is strictly preserved and a message is delivered once and remains available until a consumer processes and deletes it.
Duplicates are not introduced into the queue.
FIFO queues also support message groups that allow multiple ordered message groups within a single queue.
Limited to 300 transactions per second (TPS) but have all the capabilities of standard queues.
Deduplication with FIFO queues:
- Provide a MessageDeduplicationId with the message.
- The de-duplication interval is 5 minutes.
- Content based duplication – the MessageDeduplicationId is generated as the SHA-256 with the message body.
Sequencing with FIFO queues:
- To ensure strict ordering between messages, specify a MessageGroupId.
- Messages with a different Group ID may be received out of order.
- Messages with the same Group ID are delivered to one consumer at a time.
FIFO queues require the Message Group ID and Message Deduplication ID parameters to be added to messages.
Message Group ID:
- The tag that specifies that a message belongs to a specific message group. Messages that belong to the same message group are guaranteed to be processed in a FIFO manner.
Message Deduplication ID:
- The token used for deduplication of messages within the deduplication interval.
The following table provides a side-by-side comparison of standard and FIFO queues:
|Standard Queue||FIFO Queue|
|Unlimited Throughput: Standard queues support a nearly unlimited number of transactions per second (TPS) per API action.||High Throughput: FIFO queues support up to 300 messages per second (300 send, receive, or delete operations per second). When you batch 10 messages per operation (maximum), FIFO queues can support up to 3,000 messages per second|
|Best-Effort Ordering: Occasionally, messages might be delivered in an order different from which they were sent||First-ln-First-out Delivery: The order in which messages are sent and received is strictly preserved|
|At-Least-Once Delivery: A message is delivered at least once, but occasionally more than one copy of a message is delivered||Exactly-Once Processing: A message is delivered once and remains available until a consumer processes and deletes it. Duplicates are not introduced into the queue|
Scalability and Durability
You can have multiple queues with different priorities.
Scaling is performed by creating more queues.
SQS stores all message queues and messages within a single, highly available AWS region with multiple redundant AZs.
The amount of time a message is invisible in the queue after a reader picks it up.
Provided the job is processed before the visibility timeout expires, the message will then be deleted from the queue.
If the job is not processed within the visibility timeout, the message will become visible again and another reader will process it.
This could result in the same message being delivered twice.
The default visibility timeout is 30 seconds.
Increase it if your task takes >30 seconds.
The maximum is 12 hours.
SQS uses short polling and long polling.
Amazon SQS long polling is a way to retrieve messages from SQS queues.
Short polling returns immediately (even if the message queue is empty).
Long polling doesn’t return a response until a message arrives in the message queue or the long poll times out.
Long polling can be enabled at the queue level or at the API level using WaitTimeSeconds.
Long polling is in effect when the Receive Message Wait Time is a value greater than 0 seconds and up to 20 seconds.
- Does not wait for messages to appear in the queue.
- It queries only a subset of the available servers for messages (based on weighted random execution).
- Short polling is the default.
- ReceiveMessageWaitTime is set to 0.
- More requests are used, which implies higher cost.
- Uses fewer requests and reduces cost.
- Eliminates false empty responses by querying all servers.
- SQS waits until a message is available in the queue before sending a response.
- Requests contain at least one of the available messages up to the maximum number of messages specified in the ReceiveMessage action.
- Shouldn’t be used if your application expects an immediate response to receive message calls.
- ReceiveMessageWaitTime is set to a non-zero value (up to 20 seconds).
- Same charge per million requests as short polling.
The main task of a dead-letter queue is handling message failure.
A dead-letter queue lets you set aside and isolate messages that can’t be processed correctly to determine why their processing didn’t succeed.
It is not a queue type; it is a standard or FIFO queue that has been specified as a dead-letter queue in the configuration of another standard or FIFO queue.
Messages are moved to the dead-letter queue when the ReceiveCount for a message exceeds the maxReceiveCount for a queue.
Dead-letter queues should not be used with standard queues when your application will keep retrying transmission.
Dead-letter queues will break the order of messages in FIFO queues.
SQS Delay Queues
A delay queue postpones delivery of new messages to a queue for a number of seconds.
Messages sent to the Delay Queue remain invisible to consumers for the duration of the delay period.
Default delay is 0 seconds, maximum is 900 seconds (15 minutes).
For standard SQS queues, changing this setting doesn’t affect the delay of messages already in the queue, only new messages.
For FIFO queues, this affects the delay of messages already in the queue.
When to use a delay queue:
- Large distributed applications which may need to introduce a delay in processing.
- You need to apply a delay to an entire queue of messages.
- For example, adding a delay of a few seconds to allow updates to sales or stock control databases before sending a notification to a customer confirming an online transaction.
DLQs must be the same type as the source.
Lambda with Amazon SQS
SQS can be configured as an event source for AWS Lambda functions.
Lambda with SQS:
- You can use an AWS Lambda function to process messages in an Amazon Simple Queue Service (Amazon SQS) queue.
- Lambda event source mappings support standard queues and first-in, first-out (FIFO) queues.
- With Amazon SQS, you can offload tasks from one component of your application by sending them to a queue and processing them asynchronously.
- Lambda polls the queue and invokes your Lambda function synchronously with an event that contains queue messages.
- Lambda reads messages in batches and invokes your function once for each batch.
- When your function successfully processes a batch, Lambda deletes its messages from the queue.
SQS Extended Client
The maximum message size in SQS is 256 KB.
You can use Amazon S3 and the Amazon SQS Extended Client Library for Java to manage Amazon SQS messages.
Useful for storing and consuming messages up to 2 GB in size.
You can use the Amazon SQS Extended Client Library for Java library to do the following:
- Specify whether messages are always stored in Amazon S3 or only when the size of a message exceeds 256 KB.
- Send a message that references a single message object stored in an Amazon S3 bucket.
- Get the corresponding message object from an Amazon S3 bucket.
- Delete the corresponding message object from an Amazon S3 bucket.
CloudWatch is integrated with SQS and you can view and monitor queue metrics.
CloudWatch metrics are automatically collected every 5 minutes.
CloudWatch considers a queue to be active for up to 6 hours if it contains any messages or if any API action accesses it.
No charge for CloudWatch (no detailed monitoring).
CloudTrail captures API calls from SQS and logs to a specified S3 bucket.
You can use IAM policies to control who can read/write messages.
Authentication can be used to secure messages within queues (who can send and receive).
SQS is PCI DSS level 1 compliant and HIPAA eligible.
In-flight security is provided using HTTPS.
Can enable server-side encryption (SSE) using KMS.
- Can set the CMK you want to use.
- Can set the data key reuse period.
- SSE only encrypts the message body not the message attributes.
IAM policy must allow usage of SQS.
You can also specify permissions in an SQS queue access policy:
- Providers finer grained control.
- Control over the requests that come in.
Amazon SQS API’s
APIs you should know for the exam:
CreateQueue (aws sqs create-queue):
- Creates a new standard or FIFO queue. You can pass one or more attributes in the request.
DeleteQueue (aws sqs delete-queue):
- Deletes the queue specified by the QueueUrl , regardless of the queue’s contents.
- If the specified queue doesn’t exist, Amazon SQS returns a successful response.
PurgeQueue (aws sqs purge-queue):
- Deletes the messages in a queue specified by the QueueURL parameter.
SendMessage (aws sqs send-message):
- Delivers a message to the specified queue.
ReceiveMessage (aws sqs receive-messsage):
- Retrieves one or more messages (up to 10), from the specified queue.
- Using the WaitTimeSeconds parameter enables long-poll support.
DeleteMessage (aws sqs delete-message):
- Deletes the specified message from the specified queue.
- To select the message to delete, use the ReceiptHandle of the message (not the MessageId which you receive when you send the message).
ChangeMessageVisibility (aws sqs change-message-visibility):
- Changes the visibility timeout of a specified message in a queue to a new value.
- The default visibility timeout for a message is 30 seconds.
- The minimum is 0 seconds.
- The maximum is 12 hours.
To reduce costs or manipulate up to 10 messages with a single action, you can use the following actions:
Amazon Simple Workflow Service (SWF) is a web service that makes it easy to coordinate work across distributed application components.
Amazon SWF is used for processing background jobs that have parallel or sequential steps.
You can think of Amazon SWF as a fully managed state tracker and task coordinator.
Use Amazon SWF if your app’s steps take more than 500 milliseconds to complete, you need to track the state of processing, or you need to recover or retry if a task fails.
With SWF you can create distributed asynchronous systems as workflows.
Tracks the state of your workflow which you interact and update via API.
Best suited for human-enabled workflows like an order fulfilment system or for procedural requests.
AWS recommends that for new applications customers consider AWS Step Functions instead of SWF.
SWF enables applications for a range of use cases, including media processing, web application back-ends, business process workflows, and analytics pipelines.
Registration is a one-time step that you perform for each different type of workflow and activity.
SWF has a completion time of up to 1 year for workflow executions.
SWF uses a task-oriented API.
SWF ensures a task is assigned once and never duplicated.
SWF keeps track of all the tasks and events in an application.
A domain is a logical container for application resources such as workflows, activities, and executions.
Workers are programs that interact with Amazon SWF to get tasks, process received tasks, and return the results.
The decider is a program that controls the coordination of tasks, i.e. their ordering, concurrency, and scheduling according to the application logic.
SWF applications include the following logical components:
- Task Lists.
- Workflow Execution.
Amazon MQ is a managed message broker service for ActiveMQ..
Amazon MQ supports industry-standard APIs and protocols so you can migrate messaging and applications without rewriting code.
Amazon MQ provides cost-efficient and flexible messaging capacity – you pay for broker instance and storage usage as you go.
Amazon MQ manages the administration and maintenance of ActiveMQ brokers and automatically provisions infrastructure for high availability.
With Amazon MQ, you can use the AWS Management Console, AWS CloudFormation, the Command Line Interface (CLI), or simple API calls to launch a production-ready message broker in minutes.
Amazon MQ is fully managed and highly available within a region.
Amazon MQ stores your messages redundantly across multiple Availability Zones (AZs).
Active/standby brokers are designed for high availability.
In the event of a failure of the broker, or even a full AZ outage, Amazon MQ automatically fails over to the standby broker so you can continue sending and receiving messages.
ActiveMQ API and support for JMS, NMS, MQTT, and WebSockets.
It is designed as a drop-in replacement for on-premises message brokers.
Use SQS if you’re creating a new application from scratch.
Use MQ if you want an easy low-hassle path to migrate from existing message brokers to AWS.
Amazon MQ provides encryption of your messages at rest and in transit.
It’s easy to ensure that your messages are securely stored in an encrypted format. Connections to the broker use SSL, and access can be restricted to a private endpoint within your Amazon VPC, which allows you to isolate your broker in your own virtual network.
You can configure security groups to control network access to your broker.
Amazon MQ is integrated with Amazon CloudWatch and AWS CloudTrail. With CloudWatch you can monitor metrics on your brokers, queues, and topics.
AWS Step Functions
AWS Step Functions can be used to coordinate the components of distributed applications as a series of steps in a visual workflow.
You can quickly build and run state machines to execute the steps of your application in a reliable and scalable fashion.
How it works:
- Define the steps of your workflow in the JSON-based Amazon States Language. The visual console automatically graphs each step in the order of execution.
- Start an execution to visualize and verify the steps of your application are operating as intended. The console highlights the real-time status of each step and provides a detailed history of every execution.
- AWS Step Functions operates and scales the steps of your application and underlying compute for you to help ensure your application executes reliably under increasing demand.
It is a managed workflow and orchestration platform.
It is scalable and highly available.
You define your app as a state machine.
Create tasks, sequential steps, parallel steps, branching paths or timers.
Uses Amazon State Language declarative JSON.
Apps can interact and update the stream via Step Function API.
Provides a visual interface which describes flow and real-time status.
Provides detailed logs of each step execution.
AWS Step Functions features:
- Built-in error handling – AWS Step Functions tracks the state of each step, so you can automatically retry failed or timed-out tasks, catch specific errors, and recover gracefully, whether the task takes seconds or months to complete.
- Automatic Scaling – AWS Step Functions automatically scales the operations and underlying compute to run the steps of your application for you in response to changing workloads. Step Functions scales automatically to help ensure the performance of your application workflow remains consistently high as the frequency of requests increases.
- Pay per use – With AWS Step Functions, you pay only for the transition from one step of your application workflow to the next, called a state transition. Billing is metered by state transition, regardless of how long each state persists (up to one year).
- Execution event history – AWS Step Functions creates a detailed event log for every execution, so when things do go wrong, you can quickly identify not only where, but why. All of the execution history is available visually and programmatically to quickly troubleshoot and remediate failures.
- High availability – AWS Step Functions has built-in fault tolerance. Step Functions maintains service capacity across multiple Availability Zones in each region to help protect application workflows against individual machine or data center facility failures. There are no maintenance windows or scheduled downtimes.
- Administrative security – AWS Step Functions is integrated with AWS Identity and Access Management (IAM). IAM policies can be used to control access to the Step Functions APIs.