AWS Certified Data Engineer Associate Cheat Sheet

Preparing for the AWS Certified Data Engineer Associate (DEA-C01) exam requires a deep understanding of AWS services and data engineering principles. The coverage of AWS services is both broad and, in some cases, deep. You’ll need solid data engineering experience to pass this challenging exam.

This AWS cheat sheet for the AWS Certified Data Engineer Associate exam consolidates the core facts you need to know to pass the exam for each AWS service. Coupled with our practice tests this knowledge will give you the edge on exam day.

Compute

In the Compute category of our AWS Certified Data Engineer Associate (DEA-C01) exam cheat sheet, we delve into the essential AWS compute services that are integral to the exam.

This section provides key insights and facts about services including Amazon EC2, and Amazon ECS/EKS, which are fundamental in data engineering on AWS.

Understanding these compute services is vital for tackling the DEA-C01 exam, as they form the backbone of many data processing and analytics solutions in the AWS ecosystem. 

Amazon EC2 (Elastic Compute Cloud):

• EC2 Instances: Amazon EC2 provides resizable compute capacity in the cloud, allowing you to launch virtual servers (instances) as needed.
• Instance Types: EC2 offers a variety of instance types optimized for different use cases, such as compute-optimized, memory-optimized, and storage-optimized instances.
• Elastic Block Store (EBS): EC2 instances use EBS volumes for persistent storage. EBS volumes offer different types like General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic.
• Security Groups: These act as virtual firewalls for EC2 instances, controlling inbound and outbound traffic at the instance level.
• Elastic IP Addresses: These are static IP addresses designed for dynamic cloud computing, allowing you to allocate and assign a fixed IP address to an EC2 instance.
• Key Pairs: EC2 uses public-key cryptography to encrypt and decrypt login information. To log into your instances, you must create a key pair.
• Amazon Machine Images (AMIs): AMIs are templates that contain the software configuration (operating system, application server, applications) required to launch an EC2 instance.
• Instance Store Volumes: These provide temporary block-level storage for some EC2 instances. Data on instance store volumes is lost if the instance is stopped or terminated.
• Auto Scaling: This feature allows you to automatically scale your EC2 capacity up or down according to conditions you define.
• Elastic Load Balancing (ELB): ELB automatically distributes incoming application traffic across multiple EC2 instances, improving application scalability and reliability.
• Pricing Models: EC2 offers several pricing options, including On-Demand, Reserved Instances, and Spot Instances, each catering to different business needs and cost optimization strategies.
• VPC Integration: EC2 instances are launched in a Virtual Private Cloud (VPC) to provide network isolation and connection to your own network.
• Monitoring and Logging: Integration with Amazon CloudWatch allows for monitoring the performance of EC2 instances, providing metrics like CPU utilization, disk I/O, and network usage.
• EC2 Instance Lifecycle: Understanding the lifecycle phases of an EC2 instance, including launching, starting, stopping, rebooting, and terminating.
• AMI Customization: Ability to create custom AMIs from existing instances, which can be used to launch new instances with pre-configured settings.

Amazon Elastic Kubernetes Service (Amazon EKS):

• EKS Overview: Amazon EKS is a managed service that makes it easy to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane or nodes.
• Kubernetes Clusters: EKS runs Kubernetes control plane instances across multiple Availability Zones to ensure high availability. It automatically detects and replaces unhealthy control plane instances, and it provides automated version upgrades.
• Integration with AWS Services: EKS integrates with AWS services like Elastic Load Balancing for distributing traffic, IAM for authentication, and Amazon VPC for networking.
• Worker Nodes Management: While EKS manages the Kubernetes control plane, the responsibility of managing worker nodes that run the applications lies with the user. These can be EC2 instances or AWS Fargate.
• EKS with Fargate: AWS Fargate is a serverless compute engine for containers that works with EKS. Fargate removes the need to provision and manage servers, and you only pay for the resources required to run your containers.
• Networking in EKS: EKS can be integrated with Amazon VPC, allowing you to isolate your cluster within your own network and connect to your existing services or resources.
• Load Balancing: EKS supports Elastic Load Balancing (ELB), which automatically distributes incoming application traffic across multiple targets, such as EC2 instances.
• Security: EKS integrates with AWS IAM, providing granular control over AWS resources. Security groups can be used to control the traffic allowed to and from worker node instances.
• Logging and Monitoring: EKS integrates with Amazon CloudWatch and AWS CloudTrail for logging and monitoring. CloudWatch collects and tracks metrics, collects and monitors log files, and sets alarms.
• Persistent Storage: EKS supports Amazon EBS and Amazon EFS for persistent storage of Kubernetes pods.
• Scalability: EKS supports horizontal pod autoscaling and cluster autoscaling. Horizontal Pod Autoscaler automatically scales the number of pods, and Cluster Autoscaler adjusts the number of nodes.
• Kubernetes API Compatibility: EKS provides a fully managed Kubernetes API server that you can interact with using your existing tools and workflows.
• EKS Console: AWS provides a management console for EKS, simplifying the process of creating, updating, and deleting clusters.
• EKS Pricing: EKS pricing is based on the number of hours that your Kubernetes control plane runs, with no minimum fees or upfront commitments.

Amazon Elastic Container Service (Amazon ECS):

• ECS Overview: Amazon ECS is a fully managed container orchestration service that makes it easy to deploy, manage, and scale containerized applications using Docker.
• Container Definition and Task Definitions: In ECS, a container definition is part of a task definition. It specifies how to run a Docker container, including CPU and memory allocations, environment variables, and networking settings.
• ECS Tasks and Services: A task is the instantiation of a task definition within a cluster. An ECS service allows you to run and maintain a specified number of instances of a task definition simultaneously.
• Cluster Management: ECS clusters are logical groupings of tasks or services. You can run ECS on a serverless infrastructure that’s managed by AWS Fargate or on a cluster of EC2 instances that you manage.
• Integration with AWS Fargate: AWS Fargate is a serverless compute engine for containers that removes the need to provision and manage servers. With Fargate, you specify and pay for resources per application.
• Networking and Load Balancing: ECS can be integrated with Amazon VPC to provide isolation for the containerized applications. It also supports Elastic Load Balancing (ELB) for distributing incoming traffic.
• Storage with EBS and EFS: ECS tasks can use EBS volumes for persistent storage, which persists beyond the life of a single task. ECS can also integrate with EFS for shared storage between multiple tasks.
• IAM Roles for Tasks: ECS tasks can have IAM roles associated with them, allowing each task to have specific permissions.
• Logging and Monitoring: ECS integrates with Amazon CloudWatch for logging and monitoring. CloudWatch Logs can collect and store container logs, and CloudWatch metrics can monitor resource utilization.
• ECS Scheduling and Orchestration: ECS includes built-in schedulers for running containers based on resource needs and other requirements. It also supports integration with third-party schedulers.
• Service Discovery: ECS supports service discovery, which makes it easy for your containerized services to discover and connect with each other.
• ECS Security: Security in ECS involves securing the container instances, managing access to the ECS resources through IAM, and network traffic control with security groups and network ACLs.
• ECS Pricing: Pricing for ECS is based on the resources you use, such as EC2 instances or Fargate compute resources. There is no additional charge for ECS itself.
• Container Agent: ECS uses a container agent running on each container instance in an ECS cluster. The agent sends information about the resource’s current running tasks and resource utilization to ECS.

Storage

In the Storage section of our AWS Certified Data Engineer Associate (DEA-C01) exam cheat sheet, we focus on Amazon S3, S3 Select, Glacier, and EBS – key AWS storage services essential for data engineering.

This section provides detailed insights into Amazon S3 for object storage, S3 Select for efficient data querying, Glacier for long-term archival, and EBS for block-level storage.

Understanding the functionalities, use cases, and best practices of these services is crucial for the DEA-C01 exam, as they are fundamental in designing and implementing effective, scalable, and cost-efficient storage solutions in AWS.

Amazon S3 (Simple Storage Service):

• S3 Overview: Amazon S3 (Simple Storage Service) is an object storage service offering scalability, data availability, security, and performance.
• Buckets and Objects: S3 stores data as objects within buckets. A bucket is a container for objects stored in Amazon S3.
• S3 Data Consistency Model: Amazon S3 offers strong read-after-write consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES.
• Storage Classes: S3 offers a range of storage classes designed for different use cases, including S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA (Infrequent Access), S3 One Zone-IA, and S3 Glacier.
• S3 Glacier: S3 Glacier is a secure, durable, and low-cost storage class for data archiving. Retrieval times can range from minutes to hours.
• S3 Select: This feature allows retrieval of only a subset of data from an object, using simple SQL expressions. S3 Select improves the performance of applications by retrieving only the needed data from an S3 object.
• Versioning: S3 supports versioning, enabling multiple versions of an object to be stored in the same bucket.
• Lifecycle Policies: Lifecycle policies automate moving your objects between different storage tiers and can be used to expire objects at the end of their lifecycles.
• Security and Encryption: S3 offers various encryption options for data at rest and in transit. It also integrates with AWS Identity and Access Management (IAM) for secure access control.
• Performance Optimization: Techniques like multipart uploads, S3 Transfer Acceleration, and using byte-range fetches can optimize the performance of S3.
• Data Replication: S3 offers cross-region replication (CRR) and same-region replication (SRR) for replicating objects across buckets.
• Event Notifications: S3 can send notifications when specified events happen in a bucket, which can trigger workflows, alerts, or other processing.
• Access Management: S3 provides various mechanisms for managing access, including bucket policies, ACLs (Access Control Lists), and Query String Authentication.
• S3 Analytics and Monitoring: Integration with Amazon CloudWatch and S3 Storage Class Analysis tools help monitor and analyze storage usage.
• S3 Pricing: Costs are based on storage used, number of requests, data transfer, and additional features like S3 Select and Glacier retrieval.

Amazon EBS (Elastic Block Store):

• EBS Overview: Amazon EBS provides block-level storage volumes for use with Amazon EC2 instances. EBS volumes are highly available and reliable storage volumes that can be attached to any running instance in the same Availability Zone.
• Volume Types: EBS offers different types of volumes for different needs, such as General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. Each type has distinct performance characteristics and cost implications.
• Data Durability and Availability: EBS volumes are designed for high durability, protecting against failures by replicating within the same Availability Zone.
• Snapshots: EBS allows you to create snapshots (backups) of volumes, which are stored in Amazon S3. Snapshots can be used for data recovery and creating new volumes.
• Encryption: EBS provides the ability to encrypt volumes and snapshots with AWS Key Management Service (KMS), ensuring data security.
• Performance Metrics: Understanding EBS performance metrics like IOPS (Input/Output Operations Per Second) and throughput is crucial for optimizing storage performance.
• Scalability and Flexibility: EBS volumes can be easily resized, and their performance can be changed depending on the workload requirements.
• EBS-Optimized Instances: Certain EC2 instances are EBS-optimized, offering dedicated bandwidth for EBS volumes, which is essential for high-performance workloads.
• Lifecycle Management: Knowledge of EBS volume lifecycle, from creation to deletion, and how it impacts EC2 instances is important.
• Cost Management: Understanding the pricing model of EBS, including volume types and snapshot storage costs, is crucial for cost-effective solutions.
• Integration with EC2: EBS is tightly integrated with EC2, and knowledge of how they work together is essential for effective data engineering on AWS.
• Use Cases: EBS is commonly used for databases, file systems, and any applications that require a file system or direct block-level access to storage.
• Data Transfer: Data transfer between EBS and EC2 is a key concept, especially regarding performance and costs.
• EBS and High Availability: Strategies for using EBS in high availability configurations, such as with EC2 Auto Scaling and across multiple Availability Zones.
• Disaster Recovery: Using EBS snapshots for disaster recovery and understanding the process of restoring data from snapshots.

Networking

In the Networking section of our AWS Certified Data Engineer Associate (DEA-C01) exam cheat sheet, we delve into the intricacies of Amazon Virtual Private Cloud (VPC), AWS Direct Connect, and AWS Transit Gateway.

This segment is tailored to enhance your understanding of AWS’s networking services, which are pivotal in establishing secure, scalable, and efficient network architectures. Mastery of VPC for isolated cloud resources, Direct Connect for dedicated network connections, and Transit Gateway for network scaling and connectivity is essential for the DEA-C01 exam.

These services form the backbone of network management and optimization in AWS, crucial for any data engineering solution.

Amazon VPC (Virtual Private Cloud):

• VPC Overview: Amazon VPC allows you to provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define.
• Subnets: A VPC can be segmented into subnets, which are subsets of the VPC’s IP address range. Subnets can be public (with direct access to the internet) or private (without direct access).
• Internet Gateways (IGW): To enable access to or from the internet in a VPC, you must attach an Internet Gateway.
• Route Tables: These define rules, known as routes, which determine where network traffic from your subnet or gateway is directed.
• Network Access Control Lists (NACLs): Stateless firewalls for controlling traffic at the subnet level, allowing, or denying traffic based on IP protocol, port number, and source/destination IP.
• Security Groups: Act as virtual firewalls for EC2 instances to control inbound and outbound traffic at the instance level.
• VPC Peering: Allows you to connect one VPC with another via a direct network route using private IP addresses.
• NAT Devices: Network Address Translation (NAT) devices enable instances in a private subnet to connect to the internet or other AWS services but prevent the internet from initiating connections with the instances.
• Elastic IP Addresses: These are static IPv4 addresses designed for dynamic cloud computing, which can be associated with instances or network interfaces in a VPC.
• Virtual Private Gateway (VPG) and VPN Connection: A VPG is the VPN concentrator on the Amazon side of a VPN connection, and the VPN connection links your VPC to your own network.
• Direct Connect: AWS Direct Connect bypasses the internet and provides a direct connection from your network to AWS, which can be used to create a more consistent network experience.
• Endpoint Services: VPC Endpoint Services allow you to expose your own services within your VPC to other AWS accounts.
• Flow Logs: Capture information about the IP traffic going to and from network interfaces in your VPC, which can be used for network monitoring, forensics, and security.
• VPC Pricing: There is no additional charge for creating and using the VPC itself. Charges are for the AWS resources you create in your VPC and for data transfer.

AWS Direct Connect:

• Direct Connect Overview: AWS Direct Connect is a cloud service solution that makes it easy to establish a dedicated network connection from your premises to AWS.
• Private Connectivity: It provides a private, dedicated network connection between your data center and AWS, bypassing the public internet.
• Reduced Bandwidth Costs: By using AWS Direct Connect, you can reduce network costs, increase bandwidth throughput, and provide a more consistent network experience than internet-based connections.
• Connection Options: AWS Direct Connect offers different connection speeds, starting from 50 Mbps up to 100 Gbps.
• Virtual Interfaces (VIFs): You can create virtual interfaces directly to public AWS services (Public VIF) or to resources in your VPC (Private VIF).
• Data Transfer: AWS Direct Connect changes the data transfer rate for AWS services, often reducing the cost of data transfer compared to internet rates.
• Direct Connect Gateway: Allows you to connect to multiple VPCs in different AWS regions with the same AWS Direct Connect connection.
• Partner Network: AWS Direct Connect can be set up through AWS partners who can help in establishing the physical connection between your network and AWS.
• Hybrid Environments: Ideal for hybrid cloud architectures, providing a secure and reliable connection to AWS for workloads that require higher bandwidth or lower latency.
• Consistent Performance: Offers more consistent network performance and lower latency compared to the internet.
• Data Privacy: Since traffic is not traversing the public internet, it provides a higher level of security and privacy for your data.
• Resilience and Redundancy: For enhanced resilience, you can set up multiple Direct Connect connections for redundancy.
• Use Cases: Commonly used for high-volume data transfers, such as large-scale migrations, real-time data feeds, and hybrid cloud architectures.
• Pricing Model: Pricing is based on the port speed and data transfer rates. There are no minimum commitments or long-term contracts required.

AWS Transit Gateway:

• Transit Gateway Overview: AWS Transit Gateway acts as a network transit hub, enabling you to connect your VPCs and on-premises networks through a central point of management.
• Simplified Network Architecture: It simplifies the network architecture by reducing the number of required peering connections and managing them centrally.
• Inter-Region Peering: Transit Gateway supports peering connections across different AWS Regions, facilitating a global network architecture.
• Integration with Direct Connect: It can be integrated with AWS Direct Connect to create a unified network interface for both cloud and on-premises environments.
• Routing and Segmentation: Offers advanced routing features for network segmentation and traffic management, including support for both static and dynamic routing.
• Centralized Management: Provides a single gateway for all network traffic, simplifying management and monitoring of inter-VPC and VPC-to-on-premises connectivity.
• Scalability: AWS Transit Gateway is designed to scale horizontally, allowing it to handle a growing amount of network traffic as your AWS environment expands.
• Security and Compliance: Enhances network security and compliance by providing a single point to enforce security policies and network segmentation.
• Cost-Effective: Reduces the overall operational cost by minimizing the complexity of network topology and reducing the number of peering connections.
• VPN Support: Supports VPN connections, enabling secure connectivity between your on-premises networks and the AWS cloud.
• Multicast Support: Transit Gateway supports multicast routing, which is useful for applications that need to send the same content to multiple destinations simultaneously.
• High Availability and Resilience: Designed for high availability and resilience, Transit Gateway automatically scales with the increase in the volume of network traffic.
• Flow Logs: Supports VPC Flow Logs for Transit Gateway, allowing you to capture information about the IP traffic going to and from network interfaces in your Transit Gateway.
• Pricing Model: Pricing is based on the amount of data processed through the Transit Gateway and the number of connections.

Serverless

In the Networking section of our AWS Certified Data Engineer Associate (DEA-C01) exam cheat sheet, we delve into the intricacies of Amazon Virtual Private Cloud (VPC), AWS Direct Connect, and AWS Transit Gateway.

This segment is tailored to enhance your understanding of AWS’s networking services, which are pivotal in establishing secure, scalable, and efficient network architectures.

Mastery of VPC for isolated cloud resources, Direct Connect for dedicated network connections, and Transit Gateway for network scaling and connectivity is essential for the DEA-C01 exam.

These services form the backbone of network management and optimization in AWS, crucial for any data engineering solution. 

AWS Lambda:

• Lambda Overview: AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes.
• Event-Driven Architecture: Lambda functions are designed to be triggered by AWS services like S3, DynamoDB, Kinesis, SNS, and SQS, making it a key component in event-driven architectures.
• Scaling: AWS Lambda automatically scales your application by running code in response to each trigger. Your code runs in parallel and processes each trigger individually, scaling precisely with the size of the workload.
• Stateless: Lambda functions are stateless, meaning they do not retain any state between invocations. For state management, you need to use external services like S3 or DynamoDB.
• Supported Languages: Lambda supports multiple programming languages, including Node.js, Python, Ruby, Java, Go, .NET Core, and custom runtimes.
• Time Limits: Lambda functions have a maximum execution time limit, which as of the latest update, is 15 minutes.
• Resource Allocation: You can allocate CPU power to Lambda functions in increments of 64MB up to 10GB of memory.
• Pricing: Lambda charges are based on the number of requests for your functions and the time your code executes.
• Integration with AWS Services: Lambda can be integrated with various AWS services for logging (CloudWatch), monitoring (X-Ray), and security (IAM, VPC).
• Deployment Packages: Lambda code can be deployed as a ZIP file or a container image.
• Versioning and Aliases: Lambda supports versioning of functions. You can use aliases to route traffic between different versions.
• Environment Variables: Lambda allows you to set environment variables for your functions, which can be used to store configuration settings and secrets.
• Cold Starts: Understanding the concept of cold starts – when a new instance of a function is created in response to an event – and strategies to mitigate them.
• Concurrency and Throttling: AWS Lambda has a concurrency limit, which can be managed and configured. Throttling may occur when these limits are reached.
• Security: Lambda functions run within a VPC by default, but you can configure them to access resources within your own VPC.

AWS Step Functions:

• Step Functions Overview: AWS Step Functions is a serverless orchestration service that lets you combine AWS Lambda functions and other AWS services to build business-critical applications through visual workflows.
• State Machine: Step Functions are based on the concept of a state machine, where each state represents a step in the workflow and can perform different functions like calculations, data retrieval, or decision-making.
• Types of States:
• Task State: Represents a single unit of work performed by a workflow. It can invoke Lambda functions, run ECS tasks, or interact with other supported AWS services.
• Choice State: Adds branching logic to the workflow, allowing for decisions to be made based on the input.
• Wait State: Delays the state machine from transitioning to the next state for a specified time.
• Succeed and Fail States: Indicate the successful or unsuccessful termination of the state machine.
• Parallel State: Allows for the concurrent execution of multiple branches of a workflow.
• Map State: Processes multiple input elements dynamically, iterating through a set of steps for each element of an array.
• Integration with AWS Services: Step Functions can integrate with various AWS services, enabling complex workflows that include functions like data transformation, batch processing, and report generation.
• Error Handling: Provides robust error handling mechanisms, allowing you to catch errors and implement retry logic or fallback states.
• Execution History: Keeps a detailed history of each execution, which is useful for debugging and auditing purposes.
• Visual Interface: Offers a graphical console to visualize the components of your workflow and their real-time status.
• Scalability and Reliability: Automatically scales the execution of workflows and ensures the reliable execution of each step.
• Pricing: Charges are based on the number of state transitions in your workflows, making it cost-effective for a wide range of use cases.
• Use Cases: Commonly used for data processing, task coordination, microservices orchestration, and automated IT and business processes.
• IAM Integration: Uses AWS Identity and Access Management (IAM) to control access to resources and services used in workflows.
• API Support: Provides APIs for managing and executing workflows programmatically.

Amazon Managed Streaming for Apache Kafka (Amazon MSK):

• MSK Overview: Amazon MSK is a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data.
• Apache Kafka Integration: MSK is fully compatible with Apache Kafka, allowing you to use Kafka APIs for creating, configuring, and managing your Kafka clusters.
• Cluster Management: MSK handles the provisioning, configuration, and maintenance of Kafka clusters, including tasks like patching and updates.
• Scalability: MSK can scale out to handle high throughput and large numbers of topics and partitions, making it suitable for big data streaming applications.
• High Availability: MSK is designed for high availability with replication across multiple AWS Availability Zones.
• Security: Supports encryption at rest and in transit, VPC integration, IAM for access control, and private connectivity to ensure secure data handling.
• Monitoring and Logging: Integrates with Amazon CloudWatch for metrics and logging, allowing you to monitor the health and performance of your Kafka clusters.
• Data Retention: MSK allows you to configure the data retention period, enabling you to store data for a specified duration.
• Consumer Lag Metrics: Provides consumer lag metrics, which are critical for monitoring the health of streaming applications.
• Automatic Scaling: Supports automatic scaling of the storage associated with your MSK clusters.
• Kafka Connect and Kafka Streams: Compatible with Kafka Connect for data integration and Kafka Streams for stream processing.
• Pricing: Pricing is based on the resources consumed, including the number of broker nodes, storage, and data transfer.
• Use Cases: Commonly used for real-time analytics, log aggregation, message brokering, and event-driven architectures.
• Broker Node Configuration: Allows you to select the type and number of broker nodes, providing flexibility based on your workload requirements.

Database

In the Database section of our AWS Certified Data Engineer Associate (DEA-C01) exam cheat sheet, we focus on a range of pivotal AWS database services including Amazon RDS, Aurora, DynamoDB, Redshift, and AWS Data Pipeline.

This segment is crafted to equip you with a thorough understanding of these database technologies, each playing a significant role in AWS data engineering. From the managed relational database capabilities of RDS and Aurora to the NoSQL solutions offered by DynamoDB, the powerful data warehousing features of Redshift, and the data orchestration provided by AWS Data Pipeline, mastering these services is essential for the DEA-C01 exam.

This section aims to provide the knowledge needed to design, implement, and manage robust, scalable, and efficient database solutions in the AWS ecosystem. 

Amazon RDS (Relational Database Service):

• Database Engines Supported: Amazon RDS supports several database engines, including MySQL, PostgreSQL, MariaDB, Oracle, and SQL Server.
• Automated Backups: RDS automatically performs a daily backup of your database (during a specified backup window) and captures the entire DB instance and its data.
• DB Snapshots: RDS allows you to create manual backups of your database, known as DB Snapshots, which are user-initiated and retained until explicitly deleted.
• Multi-AZ Deployments: RDS offers Multi-AZ deployments for high availability. In a Multi-AZ deployment, RDS automatically provisions and maintains a synchronous standby replica in a different Availability Zone.
• Read Replicas: RDS supports read replicas to increase read scaling. Changes to the primary DB instance are asynchronously copied to the read replica.
• Storage Types: RDS offers three types of storage: General Purpose SSD (gp2), Provisioned IOPS SSD (io1), and Magnetic. The choice depends on the type of workload.
• Scaling: RDS allows vertical scaling (changing the instance type) and storage scaling. Storage scaling is online and does not require downtime.
• Security: RDS integrates with AWS Identity and Access Management (IAM) and offers encryption at rest using AWS Key Management Service (KMS). It also supports encryption in transit using SSL.
• Monitoring and Metrics: RDS integrates with Amazon CloudWatch for monitoring the performance and health of databases. Key metrics include CPU utilization, read/write throughput, and database connections.
• Parameter Groups: RDS uses DB Parameter Groups to manage the configuration and tuning of the database engine.
• Subnet Groups: DB Subnet Groups in RDS define which subnets and IP ranges the database can use in a VPC, allowing for network isolation.
• Maintenance and Updates: RDS provides a maintenance window for updates to the database engine, which can be specified by the user.
• Endpoint Types: RDS instances have endpoints, and each type (primary, read replica, custom) serves different purposes in database connectivity.
• Pricing Model: RDS pricing is based on the resources consumed, such as DB instance hours, provisioned storage, provisioned IOPS, and data transfer.
• Failover Process: In Multi-AZ deployments, RDS automatically performs failover to the standby in case of an issue with the primary instance.

Amazon Aurora:

• Aurora Overview: Amazon Aurora is a MySQL and PostgreSQL-compatible relational database built for the cloud, providing the performance and availability of high-end commercial databases at a fraction of the cost.
• High Performance and Scalability: Aurora provides up to five times the throughput of standard MySQL and three times the throughput of standard PostgreSQL. It’s designed to scale storage automatically, growing in 10GB increments up to 64TB.
• Aurora Replicas: Supports up to 15 low latency read replicas across three Availability Zones to increase read scalability and fault tolerance.
• Aurora Serverless: Aurora Serverless is an on-demand, auto-scaling configuration for Aurora where the database will automatically start-up, shut down, and scale capacity up or down based on your application’s needs.
• Storage and Replication: Aurora replicates data across multiple Availability Zones for improved availability and reliability. It uses a distributed, fault-tolerant, self-healing storage system.
• Backup and Recovery: Continuous backup to Amazon S3 and point-in-time recovery are supported. Snapshots can be shared with other AWS accounts.
• Security: Offers encryption at rest using AWS Key Management Service (KMS) and encryption in transit with SSL. Also integrates with AWS Identity and Access Management (IAM).
• Aurora Global Database: Designed for globally distributed applications, allowing a single Aurora database to span multiple AWS regions with fast replication.
• Database Cloning: Supports fast database cloning, which is useful for development and testing.
• Compatibility: Offers full compatibility with existing MySQL and PostgreSQL open-source databases.
• Monitoring and Maintenance: Integrates with Amazon CloudWatch for monitoring. Aurora automates time-consuming tasks like patching and backups.
• Custom Endpoints: Aurora allows you to create custom endpoints that can direct read/write operations to specific instances.
• Pricing: Aurora pricing is based on instance hours, storage consumed, and I/O operations. Aurora Serverless charges for actual consumption by the second.
• Failover: Automatic failover to a replica in the case of a failure, improving the database’s availability.
• Aurora Parallel Query: Enhances performance by pushing query processing down to the Aurora storage layer, speeding up analytical queries.

Amazon DynamoDB:

• DynamoDB Overview: Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability.
• Data Model: DynamoDB is a key-value and document database. It supports JSON-like documents and simple key-value pairs.
• Primary Key Types: DynamoDB supports two types of primary keys:
  • Partition Key: A simple primary key, composed of one attribute.
  • Composite Key: Consists of a partition key and a sort key.
• Read/Write Capacity Modes: Offers two read/write capacity modes:
  • Provisioned Throughput Mode: Pre-allocate capacity units.
  • On-Demand Mode: Automatically scales to accommodate workload demands.
• Secondary Indexes: Supports two types of secondary indexes for more complex queries:
  • Global Secondary Indexes (GSI): Index with a partition key and sort key that can be different from those on the table.
  • Local Secondary Indexes (LSI): Index with the same partition key as the table but a different sort key.
• Consistency Models: Offers both strongly consistent and eventually consistent read options.
• DynamoDB Streams: Captures a time-ordered sequence of item-level modifications in any DynamoDB table and stores this information in a log for up to 24 hours.
• Auto Scaling: Automatically adjusts read and write throughput capacity, in response to dynamically changing request volumes.
• DynamoDB Accelerator (DAX): In-memory caching service for DynamoDB, delivering fast read performance.
• Data Backup and Restore: Supports on-demand and continuous backups, point-in-time recovery, and restoration of table data.
• Security: Integrates with AWS Identity and Access Management (IAM) for authentication and authorization. Supports encryption at rest.
• Integration with AWS Lambda: Enables the triggering of AWS Lambda functions directly from DynamoDB Streams.
• Global Tables: Provides fully replicated, multi-region, multi-master tables for high availability and global data access.
• Pricing: Based on provisioned throughput and stored data. Additional charges for optional features like DAX, Streams, and backups.
• Use Cases: Ideal for web-scale applications, gaming, mobile apps, IoT, and many other applications requiring low-latency data access.

Amazon Redshift:

• Redshift Overview: Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud, allowing users to analyze data using standard SQL and existing Business Intelligence (BI) tools.
• Columnar Storage: Redshift uses columnar storage, which is optimized for data warehousing and analytics, leading to faster query performance and efficient storage.
• Node Types: Redshift offers two types of nodes – dense compute (DC) and dense storage (DS) – which are chosen based on the amount of data and the computational power required.
• Data Distribution Styles:
  • Even Distribution: Distributes table rows evenly across all slices and nodes.
  • Key Distribution: Distributes rows based on the values of the specified column.
  • All Distribution: Copies the entire table to every node, beneficial for smaller dimension tables.
• Sort Keys: Sort keys determine the order of data within each block and can significantly impact query performance. Redshift supports both compound and interleaved sort keys.
• Redshift Spectrum: Allows querying data directly in Amazon S3, enabling a data lake architecture. It’s used for running queries on large datasets in S3 without loading them into Redshift.
• Concurrency Scaling: Automatically adds additional cluster capacity to handle an increase in concurrent read queries.
• Workload Management (WLM): Redshift WLM allows users to define multiple queues and assign memory and concurrency limits to manage query performance.
• Elastic Resize: Quickly adds or removes nodes to match workload demands, enabling fast scaling of the cluster’s compute resources.
• VACUUM Command: Used to reclaim space and resort rows in tables where data has been updated or deleted, optimizing storage efficiency and query performance.
• Redshift Data API: Enables running SQL queries on data in Redshift asynchronously and retrieving the results through a simple API call, useful for integrating with web services and AWS Lambda.
• Encryption and Security: Supports encryption at rest and in transit, along with VPC integration and IAM for access control.
• Backup and Restore: Automated and manual snapshots for data backup and point-in-time recovery.
• Query Optimization: Redshift’s query optimizer uses cost-based algorithms and machine learning to deliver fast query performance.
• Pricing Model: Based on the type and number of nodes in the cluster, with additional costs for features like Redshift Spectrum and data transfer.
• Use Cases: Ideal for complex querying and analysis of large datasets, business intelligence applications, and data warehousing.

AWS Data Pipeline:

• Data Pipeline Overview: AWS Data Pipeline is a web service for processing and moving data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals.
• Data Movement and Transformation: It can be used to regularly transfer and transform data between AWS services like Amazon S3, RDS, DynamoDB, and EMR.
• Workflow Management: Data Pipeline allows you to create complex data processing workloads that are fault-tolerant, repeatable, and highly available.
• Scheduling: You can schedule regular data movement and data processing activities. The service ensures that these tasks are carried out at defined intervals.
• Prebuilt Templates: AWS Data Pipeline provides prebuilt templates for common scenarios like copying data between Amazon S3 and RDS or running queries on a schedule.
• Custom Scripts: Supports custom scripts written in SQL, Python, and other scripting languages for data transformation tasks.
• Error Handling: Provides options to retry failed tasks and to notify users of success or failure through Amazon SNS.
• Resource Management: Manages the underlying resources needed to perform data movements and transformations, automatically spinning up EC2 instances or EMR clusters as needed.
• Integration with AWS IAM: Uses AWS Identity and Access Management (IAM) for security and access control to resources and pipeline activities.
• Visual Interface: Offers a drag-and-drop web interface to create and manage data processing workflows.
• Pipeline Definition: Pipelines are defined in JSON format, specifying the data sources, destinations, activities, and scheduling information.
• Logging and Monitoring: Integrates with Amazon CloudWatch for monitoring pipeline performance and logs activities for auditing and troubleshooting.
• Pricing: Charges are based on the number of preconditions and activities used in your pipeline and the compute resources consumed.
• Use Cases: Commonly used for regular data extraction, transformation, and loading (ETL) tasks, data backup, and log processing.

Analytics

In the Analytics section of our AWS Certified Data Engineer Associate (DEA-C01) exam cheat sheet, we delve into the core AWS analytics services, including AWS Glue, Amazon Athena, Amazon EMR, Amazon Kinesis, AWS Lake Formation, and Amazon QuickSight.

This part of the guide is essential for understanding how to leverage these services to analyze and process large datasets effectively. 

AWS Glue:

• AWS Glue Overview: AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy to prepare and load data for analytics.
• Glue Data Catalog: Acts as a centralized metadata repository for all your data assets, regardless of where they are stored. It integrates with Amazon Athena, Amazon Redshift Spectrum, and AWS Lake Formation.
• AWS Glue Crawlers: Automatically discover and profile your data. Crawlers scan various data stores to infer schemas and populate the Glue Data Catalog with table definitions and other metadata.
• ETL Jobs in Glue: Allows you to author and orchestrate ETL jobs. These jobs can be triggered on a schedule or in response to an event.
• AWS Glue Studio: A visual interface to create, run, and monitor ETL jobs. It simplifies the process of writing ETL scripts with a drag-and-drop editor.
• AWS Glue DataBrew: A visual data preparation tool that enables data analysts and data scientists to clean and normalize data without writing code.
• Glue Schema Registry: Manages schema versioning and validation. It’s used to track different versions of data schemas and validate data formats to ensure data quality.
• Script Generation: Glue automatically generates ETL scripts in PySpark or Scala that can be customized as needed.
• Serverless Architecture: AWS Glue is serverless, so there is no infrastructure to manage. It automatically provisions the resources required to run your ETL jobs.
• Data Sources and Targets: Supports various data sources and targets, including Amazon S3, RDS, Redshift, and third-party databases.
• Built-in Transforms: Provides a library of predefined transforms to perform operations like joining, filtering, and sorting data.
• Security: Integrates with AWS IAM for access control and supports encryption of data in transit and at rest.
• Monitoring and Logging: Integrates with Amazon CloudWatch for monitoring ETL job execution and logs.
• Pricing: Based on the resources consumed by the ETL jobs and the number of DataBrew interactive sessions.
• Use Cases: Commonly used for data integration, data cleansing, data normalization, and building data lakes.

Amazon Athena:

• Athena Overview: Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.
• Serverless: Athena is serverless, so there is no infrastructure to manage. You pay only for the queries you run.
• S3 Integration: Directly works with data stored in S3. It’s commonly used for querying log files, clickstream data, and other unstructured/semi-structured data.
• SQL Compatibility: Supports most of the standard SQL functions, including joins, window functions, and arrays.
• Data Formats: Works with multiple data formats such as JSON, CSV, Parquet, ORC, and Avro.
• Schema Definition: Uses the AWS Glue Data Catalog for schema management, which stores metadata and table definitions.
• Partitioning: Supports partitioning of data, which improves query performance and reduces costs by scanning only relevant data.
• Query Results: Athena stores query results in S3, and you can specify the output location.
• Security: Integrates with AWS IAM for access control. Supports encryption at rest for query results in S3.
• Performance Optimization: Query performance can be optimized by using columnar formats like Parquet or ORC, compressing data, and partitioning datasets.
• Cost Optimization: Athena charges are based on the amount of data scanned per query. Costs can be optimized by compressing data, partitioning, and using columnar data formats.
• Use Cases: Ideal for ad-hoc querying, data analysis, and business intelligence applications.
• Integration with Other AWS Services: Integrates with AWS Glue for ETL, AWS QuickSight for visualization, and AWS Lambda for advanced processing.
• Federated Query: Athena supports federated queries, allowing you to run SQL queries across data stored in relational, non-relational, object, and custom data sources.
• Saved Queries and History: Athena allows saving queries and maintains a history of executed queries for auditing and review purposes.

Amazon EMR (Elastic MapReduce):

• EMR Overview: Amazon EMR is a cloud-native big data platform, allowing processing of vast amounts of data quickly and cost-effectively across resizable clusters of Amazon EC2 instances.
• Hadoop Ecosystem: EMR supports a broad array of big data frameworks, including Apache Hadoop, Spark, HBase, Presto, and Flink, making it suitable for a variety of processing tasks like batch processing, streaming, machine learning, and interactive analytics.
• Cluster Management: EMR simplifies the setup, management, and scaling of big data processing clusters. It offers options for auto-scaling the cluster size based on workload.
• Data Storage: EMR can process data from Amazon S3, DynamoDB, Amazon RDS, and Amazon Redshift. It also supports HDFS (Hadoop Distributed File System) and EMR File System (EMRFS).
• EMRFS (EMR File System): An implementation of HDFS that allows EMR clusters to store data directly in Amazon S3, providing durability and cost savings on storage.
• Pricing Model: Offers a pay-as-you-go pricing model. You pay for the EC2 instances and other AWS resources (like Amazon S3) used while your cluster is running.
• Security: Integrates with AWS IAM for authentication and authorization. Supports encryption in transit and at rest, and can be configured to launch in a VPC.
• Spot Instances: Supports the use of EC2 Spot Instances to optimize the cost of processing large datasets.
• Customization and Flexibility: Allows customization of clusters with bootstrap actions and supports multiple instance types and configurations.
• Data Processing Optimization: Offers optimizations for processing with Spark and Hadoop, including optimized versions of these frameworks.
• Monitoring and Logging: Integrates with Amazon CloudWatch for monitoring the performance of the cluster. Also supports logging to Amazon S3 for audit and troubleshooting purposes.
• Notebook Integration: Supports Jupyter and Zeppelin notebooks for interactive data exploration and visualization.
• EMR Studio: An integrated development environment (IDE) for developing, visualizing, and debugging data engineering and data science applications written in R, Python, Scala, and PySpark.
• Use Cases: Commonly used for log analysis, real-time analytics, web indexing, data transformations (ETL), machine learning, scientific simulation, and bioinformatics.
• EMR Managed Scaling: Automatically resizes clusters for optimal performance and cost with EMR managed scaling.

Amazon Kinesis (including Kinesis Data Streams, Data Firehose, and Data Analytics):

• Amazon Kinesis Overview: Amazon Kinesis is a platform for streaming data on AWS, offering powerful services to load and analyze streaming data, and also providing the ability to build custom streaming data applications.
• Kinesis Data Streams (KDS):
  • Purpose: Enables real-time processing of streaming data at a massive scale.
  • Key Features: Allows you to continuously collect and store terabytes of data per hour from hundreds of thousands of sources.
  • Consumers: Data can be processed with custom applications using Kinesis Client Library (KCL) or other AWS services like Kinesis Data Analytics, Kinesis Data Firehose, and AWS Lambda.
• Kinesis Data Firehose:
  • Purpose: Automatically loads streaming data into AWS data stores and analytics tools.
  • Key Features: Supports near-real-time loading of data into Amazon S3, Redshift, Elasticsearch Service, and Splunk.
  • Transformation and Conversion: Offers capabilities to transform and convert incoming streaming data before loading it to destinations.
• Kinesis Data Analytics:
  • Purpose: Enables you to analyze streaming data with SQL or Apache Flink without having to learn new programming languages or processing frameworks.
  • Key Features: Provides built-in functions to filter, aggregate, and transform streaming data for advanced analytics.
  • Integration: Seamlessly integrates with Kinesis Data Streams and Kinesis Data Firehose for sourcing data.
• Shards in Kinesis Data Streams:
  • Functionality: A stream is composed of one or more shards, each of which provides a fixed unit of capacity.
  • Scaling: The total capacity of a stream is the sum of the capacities of its shards.
• Data Retention: Kinesis Data Streams stores data for 24 hours by default, which can be extended up to 7 days.
• Real-Time Processing: Kinesis is designed for real-time processing of data as it arrives, unlike batch processing.
• Security: Supports encryption at rest and in transit, IAM for access control, and VPC endpoints for private network access.
• Monitoring and Logging: Integrates with Amazon CloudWatch for monitoring the performance of streams and firehoses.
• Use Cases: Ideal for real-time analytics, log and event data collection, real-time metrics, and reporting, and IoT data processing.
• Pricing: Based on the volume of data processed and the number of shards used in Kinesis Data Streams, and the amount of data ingested and transformed in Kinesis Data Firehose.

AWS Lake Formation:

• Lake Formation Overview: AWS Lake Formation simplifies the process of setting up a secure and well-architected data lake. It automates the provisioning and configuration of the underlying resources needed for a data lake on AWS.
• Data Lake Creation and Management: Lake Formation assists in collecting, cleaning, and cataloging data from various sources. It organizes data into a central repository in Amazon S3.
• Integration with AWS Services: Works seamlessly with other AWS services like Amazon Redshift, Amazon Athena, and AWS Glue. It uses the AWS Glue Data Catalog as a central metadata repository.
• Security and Access Control: Provides granular access control to data stored in the data lake. It integrates with AWS Identity and Access Management (IAM) to manage permissions and access.
• Data Cataloging: Automatically crawls data sources to identify and catalog data, making it searchable and queryable.
• Data Cleaning and Transformation: Offers tools to clean and transform data using AWS Glue, making it ready for analysis.
• Blueprints: Lake Formation provides blueprints for common data ingestion patterns, such as database replication or log processing, simplifying the process of data loading.
• Machine Learning Integration: Facilitates the use of machine learning with data in the data lake using services like Amazon SageMaker.
• Audit and Monitoring: Integrates with AWS CloudTrail and Amazon CloudWatch for auditing and monitoring data lake activities.
• Self-service Data Access: Enables end-users to access and analyze data with their choice of analytics and machine learning services.
• Cross-Account Data Sharing: Supports sharing data across different AWS accounts, enhancing collaboration while maintaining security and governance.
• Data Lake Optimization: Provides recommendations for optimizing data storage and access, improving performance, and reducing costs.
• Use Cases: Ideal for organizations looking to set up a secure data lake quickly, enabling various analytics and machine learning applications.

Amazon QuickSight:

• QuickSight Overview: Amazon QuickSight is a scalable, serverless, embeddable, machine learning-powered business intelligence (BI) service built for the cloud.
• Data Sources: QuickSight can connect to a wide array of data sources within AWS, such as Amazon RDS, Redshift, S3, Athena, and more, as well as external databases and flat files.
• SPICE Engine: QuickSight uses the Super-fast, Parallel, In-memory Calculation Engine (SPICE) to perform advanced calculations and render visualizations quickly.
• Visualizations: Offers a variety of visualization types, including graphs, charts, tables, and more, which can be used to create interactive dashboards.
• Dashboards and Stories: Users can create and publish interactive dashboards, and share insights with others through stories within QuickSight.
• Machine Learning Insights: Integrates machine learning capabilities to provide insights, forecast trends, and highlight patterns in data.
• Security and Access Control: Integrates with AWS IAM for managing access and uses row-level security to control access to data based on user roles.
• Embedding and API: Supports embedding analytics into applications and provides an API for interaction with other services and applications.
• Mobile Access: Offers mobile applications for iOS and Android, allowing access to dashboards and insights on the go.
• Scalability: As a serverless service, QuickSight scales automatically to accommodate the number of users and the volume of data.
• Pay-per-Session Pricing: Offers a unique pay-per-session pricing model, making it cost-effective for wide deployment across many users.
• Themes and Customization: Supports custom themes and layouts for dashboards, enabling alignment with company branding.
• Collaboration and Sharing: Facilitates sharing of dashboards and analyses within and outside the organization, with fine-grained control over permissions.
• Data Preparation: Includes data preparation tools for cleaning and transforming data before analysis.
• Use Cases: Ideal for building interactive BI dashboards, performing ad-hoc analysis, and embedding analytics in applications.

Deployment and Management

In the Deployment and Management section of our AWS Certified Data Engineer Associate (DEA-C01) exam cheat sheet, we concentrate on pivotal AWS services like AWS CloudFormation, Amazon CloudWatch, Amazon AppFlow, and Amazon Managed Workflows for Apache Airflow (MWAA).

This segment is tailored to provide a deep dive into the tools and services essential for efficiently deploying, monitoring, and managing data engineering workflows and resources in AWS.

AWS CloudFormation:

• CloudFormation Overview: AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications.
• Infrastructure as Code: CloudFormation allows you to use programming languages or a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts.
• Templates: Resources are defined in CloudFormation templates, which are JSON or YAML files describing the AWS resources and their dependencies so you can launch and configure them together as a stack.
• Stacks: A stack is a collection of AWS resources that you can manage as a single unit. You can create, update, or delete a collection of resources by managing the stack.
• Change Sets: Before making changes to your stack, Change Sets allow you to see how those changes might impact your existing resources.
• Resource Management: CloudFormation manages the complete lifecycle of resources: creation, updating, and deletion.
• Custom Resources: Enables the creation of custom resources when existing resources do not meet all your needs.
• Nested Stacks: Allows organizing stacks in a hierarchical manner by creating a parent stack and including other stacks as child stacks.
• Rollback Capabilities: In case of errors during deployment, CloudFormation automatically rolls back to the previous state, ensuring resource integrity.
• Integration with AWS Services: Works with a wide range of AWS services, enabling comprehensive management of an application’s resources.
• Declarative Programming: You declare the desired state of your AWS resources, and CloudFormation takes care of the provisioning and configuration.
• Drift Detection: CloudFormation can detect if the configuration of a resource has drifted from its expected state.
• Security: Integrates with AWS Identity and Access Management (IAM) for secure access to resources and supports encryption for sensitive data.
• Cross-Account and Cross-Region Management: Supports managing resources across different AWS accounts and regions.
• Use Cases: Commonly used for repeatable and consistent deployment of applications, infrastructure automation, and managing multi-tier complex architectures.

Amazon CloudWatch:

• CloudWatch Overview: Amazon CloudWatch is a monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers.
• Metrics: CloudWatch provides data and actionable insights to monitor applications, understand and respond to system-wide performance changes, optimize resource utilization, and get a unified view of operational health.
• Custom Metrics: You can publish your own metrics to CloudWatch using the AWS CLI or API.
• Alarms: CloudWatch Alarms allow you to watch a single CloudWatch metric or the result of a math expression based on CloudWatch metrics. You can set alarms to notify you when a threshold is breached.
• Logs: CloudWatch Logs can be used to monitor, store, and access log files from AWS EC2 instances, AWS CloudTrail, Route 53, and other sources.
• Events/EventBridge: CloudWatch Events/EventBridge delivers a near real-time stream of system events that describe changes in AWS resources. It can trigger AWS Lambda functions, create SNS topics, or perform other actions.
• Dashboards: CloudWatch Dashboards are customizable home pages in the CloudWatch console that you can use to monitor your resources in a single view, even those spread across different regions.
• High-Resolution Metrics: Supports high-resolution metrics (down to one-second granularity).
• Integration with AWS Services: Integrates with various AWS services for monitoring, logging, and events, providing a comprehensive view of AWS resources and applications.
• Real-time Monitoring: Offers real-time monitoring of AWS resources and applications, with metrics updated continuously.
• Automated Actions: Can automatically respond to changes in your AWS resources.
• CloudWatch Logs Insights: Provides an interactive interface to search and analyze your log data in CloudWatch Logs.
• CloudWatch Synthetics: Allows you to create canaries to monitor your endpoints and APIs from the outside-in.
• Pricing: Offers a basic level of monitoring and logging at no cost, with additional charges for extended metric retention, additional dashboards, and logs data ingestion and storage.
• Use Cases: Commonly used for performance monitoring, operational troubleshooting, application monitoring, and ensuring the security and compliance of AWS environments.

Amazon AppFlow:

• AppFlow Overview: Amazon AppFlow is a fully managed integration service that enables you to securely transfer data between AWS services and SaaS applications like Salesforce, ServiceNow, Slack, and Google Analytics.
• Data Transfer and Integration: AppFlow allows you to automate the flow of data between AWS services and SaaS applications without writing custom integration code.
• Secure Data Movement: Ensures secure and private data transfer with encryption at rest and in transit.
• Data Transformation Capabilities: Offers data transformation features such as mapping, merging, masking, filtering, and validation to prepare data for analysis.
• No Code Required: Provides a simple, no-code interface to create and execute data flows.
• Event-Driven Flows: Supports triggering data flows based on events in SaaS applications, enabling real-time data integration.
• Batch and Scheduled Data Transfers: Allows for both batch and scheduled data transfers, giving flexibility in how and when data is moved.
• Error Handling: Includes robust error handling capabilities, ensuring reliable data transfer even in case of intermittent connectivity issues.
• Integration with AWS Analytics Services: Seamlessly integrates with AWS analytics services like Amazon Redshift, Amazon S3, and AWS Lambda for advanced data processing and analytics.
• Use Cases: Commonly used for CRM data integration, marketing analytics, operational reporting, and data backup and archival.
• Scalability: Scales automatically to meet the data transfer demands of the applications.
• Monitoring and Logging: Integrates with Amazon CloudWatch for monitoring the performance and logging the activities of data flows.
• Pricing: Pay-as-you-go pricing model based on the number of flows run and the volume of data processed.
• Connectors: Provides a range of pre-built connectors for popular SaaS applications, making it easy to set up data flows.
• Data Governance and Compliance: Adheres to AWS’s high standards for data governance and compliance, ensuring data is handled securely.

Amazon Managed Workflows for Apache Airflow (Amazon MWAA):

• Amazon MWAA Overview: Amazon Managed Workflows for Apache Airflow (MWAA) is a managed service that makes it easier to set up and operate end-to-end data pipelines in the cloud with Apache Airflow.
• Apache Airflow Integration: Amazon MWAA is built on Apache Airflow, an open-source platform used for orchestrating complex computational workflows and data processing pipelines.
• Managed Service: AWS manages the underlying infrastructure for Apache Airflow, including the setup, maintenance, scaling, and patching, reducing the operational overhead for users.
• Workflow Automation: Enables the creation of workflows using directed acyclic graphs (DAGs) in Python, which specify the tasks to be executed, their dependencies, and the order in which they should run.
• Scalability: Automatically scales workflow execution capacity to match the workload.
• Integration with AWS Services: Seamlessly integrates with various AWS services like Amazon S3, Amazon Redshift, AWS Lambda, and AWS Step Functions, facilitating the creation of diverse data pipelines.
• Monitoring and Logging: Integrates with Amazon CloudWatch for monitoring and logging, providing insights into workflow performance and execution.
• Security: Offers built-in security features, including encryption in transit and at rest, IAM roles for execution, and VPC support for network isolation.
• Customization: Supports custom plugins and configurations, allowing users to tailor the environment to their specific workflow requirements.
• High Availability: Designed for high availability, with workflows running in a highly available manner across multiple Availability Zones.
• Cost-Effective: Offers a pay-as-you-go pricing model, charging based on the number of vCPU and GB of memory used per hour.
• DAG Scheduling and Triggering: Supports complex scheduling and triggering mechanisms for DAGs, enabling sophisticated workflow orchestration.
• User Interface: Provides a web interface for managing and monitoring Airflow DAGs, making it easy to visualize pipelines and their execution status.
• Use Cases: Ideal for data engineering tasks, ETL processing, machine learning model training pipelines, and any scenario requiring complex data workflow orchestration.
• Version Support: Regularly updated to support the latest versions of Apache Airflow, ensuring access to new features and improvements.

Security, Identity, and Compliance

In the Security, Identity, and Compliance section of our AWS Certified Data Engineer Associate (DEA-C01) exam cheat sheet, we delve into critical AWS services such as AWS Identity and Access Management (IAM), AWS Secrets Manager, Amazon EventBridge, and AWS CloudTrail. This part of the guide is designed to enhance your understanding of the security and compliance aspects within AWS, which are fundamental to any data engineering role.

You’ll gain insights into IAM for managing access to AWS resources, Secrets Manager for securing sensitive information, EventBridge for event-driven security monitoring, and CloudTrail for logging and tracking user activity. Mastery of these services is essential for the DEA-C01 exam, as they play a crucial role in ensuring the security and compliance of data engineering solutions in the AWS cloud.

AWS Identity and Access Management (IAM):

• IAM Overview: AWS Identity and Access Management (IAM) is a web service that helps securely control access to AWS resources. It allows you to manage users, security credentials (like access keys), and permissions that control which AWS resources users and applications can access.
• Users, Groups, and Roles:
  • Users: IAM identities that represent a person or service.
  • Groups: Collections of IAM users, managed as a unit with shared permissions.
  • Roles: IAM identities with specific permissions that can be assumed by users, applications, or AWS services.
• Policies and Permissions:
  • Policies are objects in IAM that define permissions and can be attached to users, groups, and roles.
  • Supports JSON policy language to specify permissions and resources.
• Access Management:
  • Provides tools to set up authentication and authorization for AWS resources.
  • Supports Multi-Factor Authentication (MFA) for enhanced security.
• IAM Best Practices:
  • Principle of least privilege: Granting only the permissions required to perform a task.
  • Regularly rotate security credentials.
  • Use IAM roles for applications running on EC2 instances.
• Integration with AWS Services:
  • Integrates with almost all AWS services, enabling fine-grained access control to AWS resources.
• Identity Federation:
  • Supports identity federation to allow users to authenticate with external identity providers and then access AWS resources without needing to create an IAM user.
• IAM Access Analyzer:
  • Helps identify the resources in your organization and accounts that are shared with an external entity.
• Security Auditing:
  • Integrates with AWS CloudTrail for auditing IAM activity.
  • Enables tracking of changes in permissions and resource policies.
• Cross-Account Access:
  • Allows users from one AWS account to access resources in another AWS account.
• Conditional Access Control:
  • Supports the use of conditions in IAM policies for finer control, such as allowing access only from specific IP ranges or at certain times.
• IAM Roles for EC2:
  • Allows EC2 instances to securely make API requests using temporary credentials.
• Service-Linked Roles:
  • Predefined roles that provide permissions for AWS services to access other AWS services on your behalf.
• Tagging IAM Entities:
  • Supports tagging of IAM users and roles for easier management and cost allocation.
• Use Cases:
  • Essential for managing security and access in AWS environments, including scenarios like multi-user AWS accounts, cross-account access, and automated access by AWS services.

AWS Secrets Manager:

• Secrets Manager Overview: AWS Secrets Manager is a service for managing, retrieving, and rotating database credentials, API keys, and other secrets throughout their lifecycle.
• Secret Rotation: Secrets Manager can automatically rotate secrets on a scheduled basis without user intervention. It supports AWS databases like RDS, DocumentDB, and Redshift, as well as third-party services.
• Secure Storage of Secrets: Secrets are encrypted using encryption keys that you create using AWS Key Management Service (KMS). This ensures that the secrets are stored securely.
• Integration with AWS Services: Seamlessly integrates with other AWS services, allowing you to retrieve secrets from within AWS Lambda functions, EC2 instances, RDS databases, and more.
• Centralized Management: Provides a centralized interface to manage secrets across various AWS services and applications.
• Versioning of Secrets: Supports versioning of secrets, allowing you to retrieve previous versions of a secret if needed.
• Fine-Grained Access Control: Integrates with AWS IAM, allowing you to control which users or services have access to specific secrets.
• Audit and Monitoring: Integrates with AWS CloudTrail for auditing secret access and changes, providing a record of who accessed what secret and when.
• Cross-Account Access: Allows sharing of secrets across different AWS accounts, facilitating secure access in multi-account environments.
• API and CLI Access: Secrets can be managed and retrieved using the AWS Management Console, AWS CLI, or Secrets Manager APIs.
• Secrets Retrieval: Applications can retrieve secrets with a simple API call, which makes it easier to manage credentials for databases and other services.
• Disaster Recovery: Secrets Manager is designed for high availability and durability, storing secrets across multiple Availability Zones.
• Custom Rotation Logic: Supports the use of custom AWS Lambda functions for defining custom secret rotation logic for non-AWS databases and other types of secrets.
• Pricing: Charged based on the number of secrets managed and the number of secret versions accessed each month.
• Use Cases: Commonly used for managing database credentials, API keys, and other sensitive information, especially in automated and scalable environments.

Amazon EventBridge:

• EventBridge Overview: Amazon EventBridge is a serverless event bus service that enables you to connect applications using events. It facilitates building event-driven architectures by routing events between AWS services, integrated SaaS applications, and custom applications.
• Event Sources: EventBridge can receive events from AWS services, SaaS applications, and custom applications. It supports a wide range of event sources, making it versatile for various use cases.
• Event Rules: You can create rules that define how to process and route events. These rules can filter events or transform their content before routing.
• Targets: Events can be routed to multiple AWS service targets for processing. Common targets include AWS Lambda functions, Amazon SNS topics, Amazon SQS queues, and more.
• Schema Registry: EventBridge includes a schema registry that defines the structure of event data. It helps in understanding the format of incoming events and simplifies the process of writing code to handle those events.
• Integration with SaaS Applications: EventBridge has built-in integrations with various SaaS applications, enabling you to easily route events from these applications to AWS services.
• Custom Event Buses: Supports creating custom event buses in addition to the default event bus. Custom event buses can be used for routing events from your own applications or third-party SaaS applications.
• Scalability and Reliability: As a serverless service, EventBridge scales automatically to handle a high number of events and offers high availability and reliability.
• Event Pattern Matching: Event rules use event patterns for filtering events. These patterns can match event attributes, enabling precise control over which events trigger actions.
• Security and Access Control: Integrates with AWS IAM for access control, ensuring secure handling of events.
• Cross-Account Event Delivery: EventBridge supports sending events to different AWS accounts, facilitating cross-account communication, and decoupling of services.
• Real-Time Data Flow: Enables real-time data flow between services, making it suitable for applications that require immediate response to changes.
• Monitoring and Logging: Integrates with Amazon CloudWatch for monitoring and logging, providing insights into event patterns and rule invocations.
• API Destinations: Allows you to route events to HTTP APIs, expanding the range of possible integrations and actions.
• Use Cases: Commonly used for building loosely coupled, scalable, and reliable event-driven architectures in the cloud.

AWS CloudTrail:

• CloudTrail Overview: AWS CloudTrail is a service that provides a record of actions taken by a user, role, or an AWS service in AWS, enabling governance, compliance, operational auditing, and risk auditing of your AWS account.
• Activity Logging: CloudTrail tracks user activity and API usage, recording AWS Management Console actions and API calls, including who made the call, from what IP address, and when.
• Event History: CloudTrail Event history allows you to view, search, and download the recent AWS account activity.
• Management and Data Events: CloudTrail provides two types of events:
  • Management events: Operations that are performed on resources in your AWS account.
  • Data events: Operations performed on or within a resource.
• Multiple Trails: You can create multiple trails, each of which can be configured to capture different types of events or to log events in different S3 buckets.
• Integration with Amazon S3: CloudTrail logs can be delivered to an Amazon S3 bucket for storage and analysis. You can set up S3 lifecycle policies to archive or delete logs after a specified period.
• Log File Integrity Validation: CloudTrail provides log file integrity validation, ensuring that your log files have not been tampered with after CloudTrail has delivered them to your S3 bucket.
• Encryption: Log files are encrypted using Amazon S3 server-side encryption (SSE).
• Real-Time Monitoring: Integrates with Amazon CloudWatch Logs and Amazon CloudWatch Events for real-time monitoring and alerting of specific API activity or error rates.
• Global Service Events: CloudTrail can be configured to log API calls and activities from AWS global services such as IAM and AWS STS.
• Lookup API: Provides the Lookup API to programmatically access and search CloudTrail event history for specific activities.
• Cross-Account Access: Supports logging of events in multi-account AWS environments, allowing centralized logging and analysis.
• Compliance and Auditing: CloudTrail logs are crucial for compliance and auditing processes, providing evidence of who did what in the AWS environment.
• AWS Organizations Integration: CloudTrail supports AWS Organizations, enabling you to set up a single trail to log events for all AWS accounts in an organization.
• Use Cases: Commonly used for security analysis, resource change tracking, troubleshooting, and ensuring compliance with internal policies and regulatory standards.

Migration

In the Migration section of our AWS Certified Data Engineer Associate (DEA-C01) exam cheat sheet, we focus on AWS DataSync and AWS Database Migration Service (DMS), two key services for data migration in the AWS ecosystem.

This segment is specifically designed to provide you with essential knowledge and practical insights into these services, crucial for any data engineering professional. Understanding the functionalities and best practices of DataSync for efficient data transfer and synchronization, along with DMS for seamless database migration, is vital for excelling in the DEA-C01 exam.

These services are instrumental in facilitating smooth and secure migration of data to AWS, making them indispensable tools in your data engineering toolkit.

AWS DataSync:

1. DataSync Overview: AWS DataSync is a data transfer service that simplifies, automates, and accelerates moving data between on-premises storage systems and AWS storage services, as well as between AWS storage services.
2. High-Speed Data Transfer: DataSync uses a purpose-built network protocol and parallel transfer to achieve high-speed data transfer, significantly faster than traditional transfer protocols like FTP and HTTP.
3. Automated Data Synchronization: It automates the replication of data between NFS or SMB file systems, Amazon S3 buckets, and Amazon EFS file systems.
4. Data Transfer Management: DataSync handles tasks like scheduling, monitoring, and validating data transfers, reducing the need for manual intervention and scripting.
5. Integration with AWS Storage Services: Works seamlessly with AWS storage services like Amazon S3, Amazon EFS, and Amazon FSx for Windows File Server.
6. Data Encryption and Integrity Checks: Encrypts data in transit and performs data integrity checks both during and after the transfer to ensure data is securely and accurately transferred.
7. On-Premises to AWS Transfer: Ideal for moving large volumes of data from on-premises storage into AWS for processing, backup, or archiving.
8. AWS to AWS Transfer: Supports transferring data between AWS storage services across different regions, useful for data migration, replication for disaster recovery, and data distribution.
9. Bandwidth Throttling: Offers bandwidth throttling to manage network bandwidth usage during data transfers.
10. Agent Deployment: Requires the deployment of a DataSync agent in the on-premises environment for communication with AWS storage services.
11. Scheduled Transfers: Allows scheduling of data transfers, enabling regular, automated synchronization of data.
12. Monitoring and Logging: Integrates with Amazon CloudWatch for monitoring and AWS CloudTrail for logging, providing visibility into data transfer operations.
13. Pricing: Charges based on the amount of data transferred, with no minimum fees or setup costs.
14. Use Cases: Commonly used for data migration, online data transfer for analytics and processing, and disaster recovery.
15. Simple Setup and Configuration: Offers a simple interface for setting up and configuring data transfer tasks, reducing the complexity of data migration and synchronization.

AWS Database Migration Service (DMS):

• DMS Overview: AWS Database Migration Service (DMS) is a service that enables easy and secure migration of databases to AWS, between on-premises instances, or between different AWS cloud services.
• Support for Various Database Types: DMS supports a wide range of database platforms, including relational databases, NoSQL databases, and data warehouses.
• Minimal Downtime: DMS is designed to ensure minimal downtime during database migration, making it suitable for migrating production databases with minimal impact on operations.
• Data Replication: Apart from migration, DMS can also be used for continuous data replication with high availability.
• Schema Conversion: Works in conjunction with the AWS Schema Conversion Tool (SCT) to convert the source database schema and code to a format compatible with the target database.
• Homogeneous and Heterogeneous Migrations: Supports both homogeneous migrations (like Oracle to Oracle) and heterogeneous migrations (like Oracle to Amazon Aurora).
• Incremental Data Sync: Capable of syncing only the data that has changed, which is useful for keeping the source and target databases in sync during the migration process.
• Secure Data Transfer: Ensures data security during migration by encrypting data in transit.
• Monitoring and Logging: Integrates with Amazon CloudWatch and AWS CloudTrail for monitoring the performance and auditing the migration process.
• Easy to Set Up and Use: Provides a simple-to-use interface for setting up and managing database migrations.
• Resilience and Scalability: Automatically manages the replication and network resources required for migration, scaling resources as needed to match the volume of data.
• Change Data Capture (CDC): Supports CDC, capturing and replicating ongoing changes to the database.
• Pricing: Charges based on the compute resources used during the migration process and the amount of data transferred.
• Use Cases: Commonly used for database migration projects, including migrating from on-premises databases to AWS, consolidating databases onto AWS, and migrating between different AWS database services.
• Endpoint Compatibility: Supports various source and target endpoints, including Amazon RDS, Amazon Redshift, Amazon DynamoDB, and other non-AWS database services.

Test your Knowledge with Free Practice Questions

Get access to FREE practice questions and check out the difficulty of AWS Certified Data Engineer Associate (DEA-C01) exam questions for yourself:
https://learn.digitalcloud.training/course/free-aws-certified-data-engineer-associate-practice-exam