Well-defined priorities will maximize the benefits of your efforts.

Review your priorities regularly so that they can be updated as your organization's needs change.

Involve key stakeholders, including business, development, and operations teams, to determine where to focus efforts on external customer needs.

This will ensure that you have a thorough understanding of the operations support that is required to achieve your desired business outcomes.

Customers whose needs are satisfied are much more likely to remain customers.

Evaluating and understanding external customer needs will inform how you prioritize your efforts to deliver business value.

Involve key stakeholders when determining where to focus efforts on internal customer needs.

Ensure you understand the operations support that is required to achieve business outcomes.

Use your established priorities to focus your improvement efforts where they will have the greatest impact (for example, developing team skills, improving workload performance, reducing costs, automating runbooks, or enhancing monitoring).

Update your priorities as needs change.

Ensure that you are aware of guidelines or obligations defined by your organization that may mandate or emphasize specific focus.

Evaluate internal factors, such as organization policy, standards, and requirements.

Validate that you have mechanisms to identify changes to governance. If no governance requirements are identified, ensure that you have applied due diligence to this determination.

Evaluating and understanding the governance requirements that your organization applies to your workload will inform how you prioritize your efforts to deliver business value.

Evaluate external factors, such as regulatory compliance requirements and industry standards, to ensure that you are aware of guidelines or obligations that might mandate or emphasize specific focus.

If no compliance requirements are identified, ensure that you apply due diligence to this determination.

Evaluating and understanding the compliance requirements that apply to your workload will inform how you prioritize your efforts to deliver business value.

Evaluate threats to the business (for example, competition, business risk and liabilities, operational risks, and information security threats) and maintain current information in a risk registry.

Include the impact of risks when determining where to focus efforts.

Evaluate the impact of tradeoffs between competing interests or alternative approaches, to help make informed decisions when determining where to focus efforts or choosing a course of action.

For example, accelerating speed to market for new features may be emphasized over cost optimization, or you may choose a relational database for non-relational data to simplify the effort to migrate a system, rather than migrating to a database optimized for your data type and updating your application.

Manage benefits and risks to make informed decisions when determining where to focus efforts.

For example, it may be beneficial to deploy a workload with unresolved issues so that significant new features can be made available to customers.

It may be possible to mitigate associated risks, or it may become unacceptable to allow a risk to remain, in which case you will take action to address the risk.

You might find that you want to emphasize a small subset of your priorities at some point in time.

Use a balanced approach over the long term to ensure the development of needed capabilities and management of risk. Update your priorities as needs change.

Identifying the available benefits of your choices, and being aware of the risks to your organization, enables you to make informed decisions.

Teams need to understand their roles in the success of other teams, the role of other teams in their success, and have shared goals.

Understanding responsibility, ownership, how decisions are made, and who has authority to make decisions will help focus efforts and maximize the benefits from your teams.

The needs of a team will be shaped by their industry, their organization, the makeup of the team, and the characteristics of their workload.

It is unreasonable to expect a single operating model to be able to support all teams and their workloads.

Understand who has ownership of each application, workload, platform, and infrastructure component, what business value is provided by that component, and why that ownership exists.

Understanding the business value of these individual components and how they support business outcomes informs the processes and procedures applied against them.

Understanding ownership identifies whom can approve improvements, implement those improvements, or both.

Understand who has ownership of the definition of individual processes and procedures, why those specific process and procedures are used, and why that ownership exists.

Understanding the reasons that specific processes and procedures are used enables identification of improvement opportunities.

Benefits of establishing this best practice: Understanding ownership identifies who can approve improvements, implement those improvements, or both.

Understand who has responsibility to perform specific activities on defined workloads and why that responsibility exists.

Understanding who has responsibility to perform activities informs who will conduct the activity, validate the result, and provide feedback to the owner of the activity.

Benefits of establishing this best practice: Understanding who is responsible to perform an activity informs whom to notify when action is needed and who will perform the action, validate the result, and provide feedback to the owner of the activity.

Understanding the responsibilities of your role and how you contribute to business outcomes informs the prioritization of your tasks and why your role is important.

This enables team members to recognize needs and respond appropriately.

Benefits of establishing this best practice: Understanding your responsibilities informs the decisions you make, the actions you take, and your hand off activities to their proper owners.

Where no individual or team is identified, there are defined escalation paths to someone with the authority to assign ownership or plan for that need to be addressed.

Benefits of establishing this best practice: Understanding who has responsbility or ownership allows you to reach out to the proper team or team member to make a request or transition a task.

Having an identified person who has the authority to assign responsbility or ownership or plan to address needs reduces the risk of inaction and needs not being addressed.

You are able to make requests to owners of processes, procedures, and resources.

Make informed decisions to approve requests where viable and determined to be appropriate after an evaluation of benefits and risks.

Benefits of establishing this best practice: It's critical that mechanisms exist to request additions, changes, and exceptions in support of teams' activities. Without this option, current state become a constraint on innovation.

Have defined or negotiated agreements between teams describing how they work with and support each other (for example, response times, service level objectives, or service level agreements).

Understanding the impact of the teams work on business outcomes, and the outcomes of other teams and organizations, informs the prioritization of their tasks and enables them to respond appropriately.

When responsibility and ownership are undefined or unknown, you are at risk of both not addressing necessary activities in a timely fashion and of redundant and potentially conflicting efforts emerging to address those needs.

Benefits of establishing this best practice: Establishing the responsibilities between teams, the objectives, and the methods for communicating needs, eases the flow of requests and helps ensures the necessary information is provided.

This reduces the delay introduced by transition tasks between teams and help support the achievement of business outcomes.

Senior leadership clearly sets expectations for the organization and evaluates success.

Senior leadership is the sponsor, advocate, and driver for the adoption of best practices and evolution of the organization

Benefits of establishing this best practice: Engaged leadership, clearly communicated expectations, and shared goals ensures that team members know what is expected of them.

Evaluating success enables identification of barriers to success so that they can be addressed through intervention by the sponsor advocate or their delegates.

The workload owner has defined guidance and scope empowering team members to respond when outcomes are at risk.

Escalation mechanisms are used to get direction when events are outside of the defined scope.

By testing and validating changes early, you are able to address issues with minimized costs and limit the impact on your customers.

By testing prior to deployment you minimize the introduction of errors.

Team members have mechanisms and are encouraged to escalate concerns to decision makers and stakeholders if they believe outcomes are at risk.

Escalation should be performed early and often so that risks can be identified, and prevented from causing incidents.

Mechanisms exist and are used to provide timely notice to team members of known risks and planned events. Necessary context, details, and time (when possible) are provided to support determining if action is necessary, what action is required, and to take action in a timely manner.

For example, providing notice of software vulnerabilities so that patching can be expedited, or providing notice of planned sales promotions so that a change freeze can be implemented to avoid the risk of service disruption.

Planned events can be recorded in a change calendar or maintenance schedule so that team members can identify what activities are pending.

On AWS, AWS Systems Manager Change Calendar can be used to record these details. It supports programmatic checks of calendar status to determine if the calendar is open or closed to activity at a particular point of time.

Operations activities can be planned around specific approved windows of time that are reserved for potentially disruptive activities. AWS Systems Manager Maintenance Windows allows you to schedule activities against instances and other supported resources to automate the activities and make those activities discoverable.

Experimentation accelerates learning and keeps team members interested and engaged.

An undesired result is a successful experiment that has identified a path that will not lead to success.

Team members are not punished for successful experiments with undesired results. Experimentation is required for innovation to happen and turn ideas into outcomes.

Teams must grow their skill sets to adopt new technologies, and to support changes in demand and responsibilities in support of your workloads.

Growth of skills in new technologies is frequently a source of team member satisfaction and supports innovation.

Support your team members pursuit and maintenance of industry certifications that validate and acknowledge their growing skills.

Cross train to promote knowledge transfer and reduce the risk of significant impact when you lose skilled and experienced team members with institutional knowledge.

Provide dedicated structured time for learning.

Maintain team member capacity, and provide tools and resources to support your workload needs.

Overtasking team members increases the risk of incidents resulting from human error.

nvestments in tools and resources (for example, providing automation for frequently performed activities) can scale the effectiveness of your team, enabling them to support additional activities.

Leverage cross-organizational diversity to seek multiple unique perspectives.

Use this perspective to increase innovation, challenge your assumptions, and reduce the risk of confirmation bias.

Grow inclusion, diversity, and accessibility within your teams to gain beneficial perspectives.

Organizational culture has a direct impact on team member job satisfaction and retention. Enable the engagement and capabilities of your team members to enable the success of your business.

Iterate to develop the telemetry necessary to monitor the health of your workload, identify when outcomes are at risk, and enable effective responses. In AWS, you can emit and collect logs, metrics, and events from your applications and workloads components to enable you to understand their internal state and health. You can integrate distributed tracing to track requests as they travel through your workload. Use this data to understand how your application and underlying components interact and to analyze issues and performance.

Application telemetry is the foundation for observability of your workload.

Your application should emit telemetry that provides insight into the state of the application and the achievement of business outcomes.

From troubleshooting to measuring the impact of a new feature, application telemetry informs the way you build, operate, and evolve your workload.

Collecting metrics over time can be used to develop baselines and detect anomalies.

Design and configure your workload to emit information about its internal state and current status, for example, API call volume, HTTP status codes, and scaling events.

Use this information to help determine when a response is required.

Benefits of establishing this best practice: Understanding what is going on inside your workload enables you to respond if necessary.

Instrument your application code to emit information about user activity, for example, click streams, or started, abandoned, and completed transactions.

Use this information to help understand how the application is used, patterns of usage, and to determine when a response is required.

Design and configure your workload to emit information about the status (for example, reachability or response time) of resources it depends on.

Examples of external dependencies can include, external databases, DNS, and network connectivity.

Use this information to determine when a response is required.

Implement your application code and configure your workload components to emit information about the flow of transactions across the workload.

Use this information to determine when a response is required and to assist you in identifying the factors contributing to an issue.

These accelerate beneficial changes entering production, limit issues deployed, and enable rapid identification and remediation of issues introduced through deployment activities.

In AWS, you can view your entire workload (applications, infrastructure, policy, governance, and operations) as code. It can all be defined in and updated using code. This means you can apply the same engineering discipline that you use for application code to every element of your stack.

Use version control to enable tracking of changes and releases.

Test and validate changes to help limit and detect errors. Automate testing to reduce errors caused by manual processes, and reduce the level of effort to test.

Use configuration management systems to make and track configuration changes.

These systems reduce errors caused by manual processes and reduce the level of effort to deploy changes.

Static configuration management sets values when initializing a resource that are expected to remain consistent throughout the resource’s lifetime.

Use build and deployment management systems. These systems reduce errors caused by manual processes and reduce the level of effort to deploy changes.

Perform patch management to gain features, address issues, and remain compliant with governance.

Automate patch management to reduce errors caused by manual processes, and reduce the level of effort to patch.

Patch and vulnerability management are part of your benefit and risk management activities. d.

Share best practices across teams to increase awareness and maximize the benefits of development efforts.

On AWS, application, compute, infrastructure, and operations can be defined and managed using code methodologies. This allows for easy release, sharing, and adoption.

Use this information to determine when a response is required.

Implement practices to improve code quality and minimize defects. Some examples include test-driven development, code reviews, and standards adoption.

Use multiple environments to experiment, develop, and test your workload.

Use increasing levels of controls as environments approach production to gain confidence your workload will operate as intended when deployed.

Frequent, small, and reversible changes reduce the scope and impact of a change.

This eases troubleshooting, enables faster remediation, and provides the option to roll back a change.

Automate build, deployment, and testing of the workload.

This reduces errors caused by manual processes and reduces the effort to deploy changes.

Apply metadata using Resource Tags and AWS Resource Groups following a consistent tagging strategy to enable identification of your resources

Tag your resources for organization, cost accounting, access controls, and targeting the execution of automated operations activities.

Using these practices mitigates the impact of issues introduced through the deployment of changes.

Plan to revert to a known good state, or remediate in the production environment if a change does not have the desired outcome. This preparation reduces recovery time through faster responses.

Test changes and validate the results at all lifecycle stages to confirm new features and minimize the risk and impact of failed deployments.

On AWS, you can create temporary parallel environments to lower the risk, effort, and cost of experimentation and testing. Automate the deployment of these environments using AWS CloudFormation to ensure consistent implementations of your temporary environments.

Use deployment management systems to track and implement change. This reduces errors caused by manual processes and reduces the effort to deploy changes.

Build Continuous Integration/Continuous Deployment (CI/CD) pipelines

Test with limited deployments alongside existing systems to confirm desired outcomes prior to full scale deployment. For example, use deployment canary testing or one-box deployments.

Implement changes onto parallel environments, and then transition over to the new environment. Maintain the prior environment until there is confirmation of successful deployment.

Doing so minimizes recovery time by enabling rollback to the previous environment.

Use frequent, small, and reversible changes to reduce the scope of a change. This results in easier troubleshooting and faster remediation with the option to roll back a change.

Automate build, deployment, and testing of the workload. This reduces errors cause by manual processes and reduces the effort to deploy changes.

Automate testing of deployed environments to confirm desired outcomes. Automate rollback to a previous known good state when outcomes are not achieved to minimize recovery time and reduce errors caused by manual processes.

Manage the flow of change into your environments. You should use a consistent process (including manual or automated checklists) to know when you are ready to go live with your workload or a change. This will also enable you to find any areas that you need to make plans to address. You will have runbooks that document your routine activities and playbooks that guide your processes for issue resolution. Use a mechanism to manage changes that supports the delivery of business value and help mitigate risks associated to change.

Have a mechanism to validate that you have the appropriate number of trained personnel to provide support for operational needs. .

Train personnel and adjust personnel capacity as necessary to maintain effective support.

You will need to have enough team members to cover all activities (including on-call). Ensure that your teams have the necessary skills to be successful with training on your workload, your operations tools, and AWS.

Use Operational Readiness Reviews (ORRs) to validate that you can operate your workload.

ORR is a mechanism developed at Amazon to validate that teams can safely operate their workloads.

An ORR is a review and inspection process using a checklist of requirements.

A runbook is a documented process to achieve a specific outcome.

Runbooks consist of a series of steps that someone follows to get something done.

Runbooks are an essential part of operating your workload. From onboarding a new team member to deploying a major release, runbooks are the codified processes that provide consistent outcomes no matter who uses them.

Playbooks are step-by-step guides used to investigate an incident. When incidents happen, playbooks are used to investigate, scope impact, and identify a root cause.

Playbooks are used for a variety of scenarios, from failed deployments to security incidents.

In many cases, playbooks identify the root cause that a runbook is used to mitigate. Playbooks are an essential component of your organization's incident response plans.

Evaluate the capabilities of the team to support the workload and the workload's compliance with governance.

Your team should be able to understand the health of your workload easily. You will want to use metrics based on workload outcomes to gain useful insights. You should use these metrics to implement dashboards with business and technical viewpoints that will help team members make informed decisions.

Have a mechanism to validate that you have the appropriate number of trained personnel to provide support for operational needs. .

Train personnel and adjust personnel capacity as necessary to maintain effective support.

You will need to have enough team members to cover all activities (including on-call). Ensure that your teams have the necessary skills to be successful with training on your workload, your operations tools, and AWS.

Have a mechanism to validate that you have the appropriate number of trained personnel to provide support for operational needs. .

Train personnel and adjust personnel capacity as necessary to maintain effective support.

You will need to have enough team members to cover all activities (including on-call). Ensure that your teams have the necessary skills to be successful with training on your workload, your operations tools, and AWS.

Perform regular proactive reviews of metrics to identify trends and determine where appropriate responses are needed.

You should aggregate log data from your application, workload components, services, and API calls to a service such as CloudWatch Logs.

Establish baselines for metrics to provide expected values as the basis for comparison and identification of under- and over-performing components. Identify thresholds for improvement, investigation, and intervention.

Establish patterns of workload activity to identify anomalous behavior so that you can respond appropriately if required.

CloudWatch through the CloudWatch Anomaly Detection feature applies statistical and machine learning algorithms to generate a range of expected values that represent normal metric behavior.

Raise an alert when workload outcomes are at risk so that you can respond appropriately if necessary.

Ideally, you have previously identified a metric threshold that you are able to alarm upon or an event that you can use to trigger an automated response.

Raise an alert when workload anomalies are detected so that you can respond appropriately if necessary.

Your analysis of your workload metrics over time may establish patterns of behavior that you can quantify sufficiently to define an event or raise an alarm in response.

Create a business-level view of your workload operations to help you determine if you are satisfying needs and to identify areas that need improvement to reach business goals.

Validate the effectiveness of KPIs and metrics and revise them if necessary.

Your team should be able to understand the health of your operations easily. You will want to use metrics based on operations outcomes to gain useful insights. You should use these metrics to implement dashboards with business and technical viewpoints that will help team members make informed decisions.

Identify key performance indicators (KPIs) based on desired business outcomes (for example, new features delivered) and customer outcomes (for example, customer support cases).

Define operations metrics to measure the achievement of KPIs (for example, successful deployments, and failed deployments).

Perform regular, proactive reviews of metrics to identify trends and determine where appropriate responses are needed.

You should aggregate log data from the execution of your operations activities and operations API calls, into a service such as CloudWatch Logs. G

Establish baselines for metrics to provide expected values as the basis for comparison and identification of under and over performing operations activities.

Establish patterns of operations activities to identify anomalous activity so that you can respond appropriately if necessary.

Whenever operations outcomes are at risk, an alert must be raised and acted upon. Operations outcomes are any activity that supports a workload in production.

This includes everything from deploying new versions of applications to recovering from an outage

Raise an alert when operations anomalies are detected so that you can respond appropriately if necessary.

Your analysis of your operations metrics over time may established patterns of behavior that you can quantify sufficiently to define an event or raise an alarm in response.

Create a business-level view of your operations activities to help you determine if you are satisfying needs and to identify areas that need improvement to reach business goals.

You should use your existing runbooks and playbooks to deliver consistent results when you respond to alerts. Defined alerts should be owned by a role or a team that is accountable for the response and escalations. You will also want to know the business impact of your system components and use this to target efforts when needed. You should perform a root cause analysis (RCA) after events, and then prevent recurrence of failures or document workarounds.

Your organization has processes to handle events, incidents, and problems.

Events are things that occur in your workload but may not need intervention.

Incidents are events that require intervention.

Problems are recurring events that require intervention or cannot be resolved.

You need processes to mitigate the impact of these events on your business and make sure that you respond appropriately

Have a well-defined response (runbook or playbook), with a specifically identified owner, for any event for which you raise an alert.

Ensure that when multiple events require intervention, those that are most significant to the business are addressed first.

Impacts can include loss of life or injury, financial loss, or damage to reputation or trust.

Define escalation paths in your runbooks and playbooks, including what triggers escalation, and procedures for escalation.

Specifically identify owners for each action to ensure effective and prompt responses to operations events.

Communicate directly with your users (for example, with email or SMS) when the services they use are impacted, and again when the services return to normal operating conditions, to enable users to take appropriate action.

Provide dashboards tailored to their target audiences (for example, internal technical teams, leadership, and customers) to communicate the current operating status of the business and provide metrics of interest.

Automate responses to events to reduce errors caused by manual processes, and to ensure prompt and consistent responses.

When things fail, you will want to ensure that your team, as well as your larger engineering community, learns from those failures. You should analyze failures to identify lessons learned and plan improvements. You will want to regularly review your lessons learned with other teams to validate your insights.

Regularly evaluate and prioritize opportunities for improvement to focus efforts where they can provide the greatest benefits.

Review customer-impacting events, and identify the contributing factors and preventative actions.

Use this information to develop mitigations to limit or prevent recurrence. Develop procedures for prompt and effective responses

Feedback loops provide actionable insights that drive decision making. Build feedback loops into your procedures and workloads.

This helps you identify issues and areas that need improvement. They also validate investments made in improvements.

Mechanisms exist for your team members to discover the information that they are looking for in a timely manner, access it, and identify that it’s current and complete.

Mechanisms are present to identify needed content, content in need of refresh, and content that should be archived so that it’s no longer referenced.

Identify drivers for improvement to help you evaluate and prioritize opportunities.

On AWS, you can aggregate the logs of all your operations activities, workloads, and infrastructure to create a detailed activity history.

You can then use AWS tools to analyze your operations and workload health over time.

Review your analysis results and responses with cross-functional teams and business owners. Use these reviews to establish common understanding, identify additional impacts, and determine courses of action. Adjust responses as appropriate.

Regularly perform retrospective analysis of operations metrics with cross-team participants from different areas of the business.

Use these reviews to identify opportunities for improvement, potential courses of action, and to share lessons learned.

Document and share lessons learned from the operations activities so that you can use them internally and across teams.

You should share what your teams learn to increase the benefit across your organization.

Dedicate time and resources within your processes to make continuous incremental improvements possible.

On AWS, you can create temporary duplicates of environments, lowering the risk, effort, and cost of experimentation and testing.

In AWS, accounts are a hard boundary. For example, account-level separation is strongly recommended for isolating production workloads from development and test workloads.

Manage accounts centrally: AWS Organizations automates AWS account creation and management, and control of those accounts after they are created.

Set controls centrally: Control what your AWS accounts can do by only allowing specific services, Regions, and service actions at the appropriate level.

Configure services and resources centrally: AWS Organizations helps you configure AWS services that apply to all of your accounts.

Start with security and infrastructure in mind to enable your organization to set common guardrails as your workloads grow. This approach provides boundaries and controls between workloads.

Account level separation is strongly recommended for isolating production environments from development and test environments, or providing a strong logical boundary between workloads that process data of different sensitivity levels, as defined by external compliance requirements (such as PCI-DSS or HIPAA), and workloads that don’t.

There are a number of aspects to securing your AWS accounts, including the securing of, and not using the root user, and keeping your contact information up-to-date.

You can use AWS Organizations to centrally manage and govern your accounts as you grow and scale your workloads in AWS.

AWS Organizations helps you manage accounts, set controls, and configure services across your accounts.

One of the ways to improve your ability to operate securely in the cloud is by taking an organizational approach to governance

Governance is the way that decisions are guided consistently without depending solely on the good judgment of the people involved.

Based on your compliance requirements and risks identified from your threat model, derive and validate the control objectives and controls that you need to apply to your workload.

Ongoing validation of control objectives and controls help you measure the effectiveness of risk mitigation.

To help you define and implement appropriate controls, recognize attack vectors by staying up to date with the latest security threats.

Consume AWS Managed Services to make it easier to receive notification of unexpected or unusual behavior in your AWS accounts. Investigate using AWS Partner tools or thirdparty threat information feeds as part of your security information flow.

Stay up-to-date with both AWS and industry security recommendations to evolve the security posture of your workload.

AWS Security Bulletins contain important information about security and privacy notifications.

Establish secure baselines and templates for security mechanisms that are tested and validated as part of your build, pipelines, and processes.

Use tools and automation to test and validate all security controls continuously.

Use a threat model to identify and maintain an up-to-date register of potential threats. Prioritize your threats and adapt your security controls to prevent, detect, and respond.

Revisit and maintain this in the context of the evolving security landscape.

Evaluate and implement security services and features from AWS and AWS Partners that allow you to evolve the security posture of your workload.

The AWS Security Blog highlights new AWS services and features, implementation guides, and general security guidance.

What's New with AWS? is a great way to stay up to date with all new AWS features, services, and announcements.

Human identities: The administrators, developers, operators, and consumers of your applications require an identity to access your AWS environments and applications.

Machine identities: Your workload applications, operational tools, and components require an identity to make requests to AWS services, for example, to read data. These identities include machines running in your AWS environment, such as Amazon EC2 instances or AWS Lambda functions.

Enforce minimum password length, and educate your users to avoid common or reused passwords.

Enforce multi-factor authentication (MFA) with software or hardware mechanisms to provide an additional layer of verification

For human identities using the AWS Management Console, require users to acquire temporary credentials and federate into AWS. You can do this using the AWS IAM Identity Center user portal.

For users requiring CLI access, ensure that they use AWS CLI v2, which supports direct integration with IAM Identity Center.

For machine identities, you should rely on IAM roles to grant access to AWS.

For workforce and machine identities that require secrets such as passwords to third-party applications, store them with automatic rotation.

Secrets Manager makes it easy to manage, rotate, and securely store encrypted secrets using supported services.

For workforce identities, rely on an identity provider that enables you to manage identities in a centralized place.

This makes it easier to manage access across multiple applications and services,because you are creating, managing, and revoking access from a single location.

When you cannot rely on temporary credentials and require long-term credentials, audit credentials to ensure that the defined controls for example, multi-factor authentication (MFA), are enforced, rotated regularly, and have the appropriate access level.

As the number of users you manage grows, you will need to determine ways to organize them so that you can manage them at scale.

Place users with common security requirements in groups defined by your identity provider, and put mechanisms in place to ensure that user attributes that may be used for access control (for example, department or location) are correct and updated.

Use these groups and attributes to control access, rather than individual users.

Permissions control who can access what, and under what conditions. Set permissions to specific human and machine identities to grant access to specific service actions on specific resources.

There are a number of ways to grant access to different types of resources. One way is by using different policy types.

Identity-based policies in IAM attach to IAM identities, including users, groups, or roles. These policies let you specify what that identity can do (its permissions).

Each component or resource of your workload needs to be accessed by administrators, end users, or other components.

Have a clear definition of who or what should have access to each component, choose the appropriate identity type and method of authentication and authorization.

Grant only the access that identities require by allowing access to specific actions on specific AWS resources under specific conditions.

Rely on groups and identity attributes to dynamically set permissions at scale, rather than defining permissions for individual users.

A process that allows emergency access to your workload in the unlikely event of an automated process or pipeline issue.

This will help you rely on least privilege access, but ensure users can obtain the right level of access when they require it.

As teams and workloads determine what access they need, remove permissions they no longer use and establish review processes to achieve least privilege permissions.

Continuously monitor and reduce unused identities and permissions.

Establish common controls that restrict access to all identities in your organization

For example, you can restrict access to specific AWS Regions, or prevent your operators from deleting common resources, such as an IAM role used for your central security team.

Integrate access controls with operator and application lifecycle and your centralized federation provider.

For example, remove a user’s access when they leave the organization or change roles.

Continuously monitor findings that highlight public and cross-account access.

Reduce public access and cross-account access to only resources that require this type of access.

Govern the consumption of shared resources across accounts or within your AWS Organizations.

Monitor shared resources and review shared resource access.

Detection enables you to identify a potential security misconfiguration, threat, or unexpected behavior.

It’s an essential part of the security lifecycle and can be used to support a quality process, a legal or compliance obligation, and for threat identification and response efforts.

Configure logging throughout the workload, including application logs, resource logs, and AWS service logs.

A foundational practice is to establish a set of detection mechanisms at the account level. This base set of mechanisms is aimed at recording and detecting a wide range of actions on all resources in your account.

Security operations teams rely on the collection of logs and the use of search tools to discover potential events of interest, which might indicate unauthorized activity or unintentional change.

A best practice for building a mature security operations team is to deeply integrate the flow of security events and findings into a notification and workflow system such as a ticketing system, a bug or issue system, or other security information and event management (SIEM) system.

Using automation to investigate and remediate events reduces human effort and error, and enables you to scale investigation capabilities.

In AWS, investigating events of interest and information on potentially unexpected changes into an automated workflow can be achieved using Amazon EventBridge.

Detecting change and routing this information to the correct workflow can also be accomplished using AWS Config Rules and Conformance Packs.

Create alerts that are sent to and can be actioned by your team. Ensure that alerts include relevant information for the team to take action.

For each detective mechanism you have, you should also have a process, in the form of a runbook or playbook, to investigate

When you follow the principle of applying security at all layers, you employ a Zero Trust approach.

Zero Trust security is a model where application components or microservices are considered discrete from each other and no component or microservice trusts any other.

Group components that share reachability requirements into layers.

For example, a database cluster in a virtual private cloud (VPC) with no need for internet access should be placed in subnets with no route to or from the internet.

For network connectivity that can include thousands of VPCs, AWS accounts, and on-premises networks, you should use AWS Transit Gateway. It acts as a hub that controls how traffic is routed among all the connected networks, which act like spokes.

When architecting your network topology, you should examine the connectivity requirements of each component.

For example, if a component requires internet accessibility (inbound and outbound), connectivity to VPCs, edge services, and external data centers.

Automate protection mechanisms to provide a self-defending network based on threat intelligence and anomaly detection.

For example, intrusion detection and prevention tools that can adapt to current threats and reduce their impact.

A web application firewall is an example of where you can automate network protection, to automatically block requests originating from IP addresses associated with known threat actors

Inspect and filter your traffic at each layer. You can inspect your VPC configurations for potential unintended access using VPC Network Access Analyzer.

Inspect and filter your traffic at each layer. You can inspect your VPC configurations for potential unintended access using VPC Network Access Analyzer.

For components transacting over HTTP-based protocols, a web application firewall can help protect from common attacks.

Each of these compute resource types require different approaches to secure them.

However, they do share common strategies that you need to consider: defense in depth, vulnerability management, reduction in attack surface, automation of configuration and operation, and performing actions at a distance.

Frequently scan and patch for vulnerabilities in your code, dependencies, and in your infrastructure to help protect against new threats.

You are responsible for patch management for your AWS resources, including Amazon Elastic Compute Cloud(Amazon EC2) instances, Amazon Machine Images (AMIs), and many other compute resources.

Reduce your exposure to unintended access by hardening operating systems and minimizing the components, libraries, and externally consumable services in use.

Start by reducing unused components.

You can find many hardening and security configuration guides for common operating systems and server software. For example, you can start with the Center for Internet Security and iterate.

Implement services that manage resources, such as Amazon Relational Database Service (Amazon RDS), AWS Lambda, and Amazon Elastic Container Service (Amazon ECS), to reduce your security maintenance tasks as part of the shared responsibility model.

For example, Amazon RDS helps you set up, operate, and scale a relational database, automates administration tasks such as hardware provisioning, database setup, patching, and backups.

Automate your protective compute mechanisms including vulnerability management, reduction in attack surface, and management of resources.

The automation will help you invest time in securing other aspects of your workload, and reduce the risk of human error.

Removing the ability for interactive access reduces the risk of human error, and the potential for manual configuration or management.

For example, use a change management workflow to deploy Amazon Elastic Compute Cloud (Amazon EC2) instances using infrastructure-as-code, then manage Amazon EC2 instances using tools such as AWS Systems Manager instead of allowing direct access or through a bastion host.

AWS Systems Manager can automate a variety of maintenance and deployment tasks, using features including automation workflows, documents (playbooks), and the run command.

Implement mechanisms (for example, code signing) to validate that the software, code and libraries used in the workload are from trusted sources and have not been tampered with.

For example, you should verify the code signing certificate of binaries and scripts to confirm the author, and ensure it has not been tampered with since created by the author

You need to understand the type and classification of data your workload is processing, the associated business processes, data owner, applicable legal and compliance requirements, where it’s stored, and the resulting controls that are needed to be enforced.

This may include classifications to indicate if the data is intended to be publicly available, if the data is internal use only such as customer personally identifiable information (PII), or if the data is for more restricted access such as intellectual property, legally privileged or marked sensitive, and more.

By carefully managing an appropriate data classification system, along with each workload’s level of protection requirements, you can map the controls and level of access or protection appropriate for the data.

Protect data according to its classification level. For example, secure data classified as public by using relevant recommendations while protecting sensitive data with additional controls.

By using resource tags, separate AWS accounts per sensitivity (and potentially also for each caveat, enclave, or community of interest), IAM policies, AWS Organizations SCPs, AWS Key Management Service (AWS KMS), and AWS CloudHSM, you can define and implement your policies for data classification and protection with encryption.

Automating the identification and classification of data can help you implement the correct controls.

Using automation for this instead of direct access from a person reduces the risk of human error and exposure.

You should evaluate using a tool, such as Amazon Macie, that uses machine learning to automatically discover, classify, and protect sensitive data in AWS.

Your defined lifecycle strategy should be based on sensitivity level as well as legal and organization requirements.

Aspects including the duration for which you retain data, data destruction processes, data access management, data transformation, and data sharing should be considered.

When choosing a data classification methodology, balance usability versus access. You should also accommodate the multiple levels of access and nuances for implementing a secure, but still usable, approach for each level.

This includes block storage, object storage, databases, archives, IoT devices, and any other storage medium on which data is persisted.

Protecting your data at rest reduces the risk of unauthorized access, when encryption and appropriate access controls are implemented.

By defining an encryption approach that includes the storage, rotation, and access control of keys, you can help provide protection for your content against unauthorized users and against unnecessary exposure to authorized users.

AWS Key Management Service (AWS KMS) helps you manage encryption keys and integrates with many AWS services. This service provides durable, secure, and redundant storage for your AWS KMS keys.

You should ensure that the only way to store data is by using encryption.

AWS Key Management Service (AWS KMS) integrates seamlessly with many AWS services to make it easier for you to encrypt all your data at rest.

Use automated tools to validate and enforce data at rest controls continuously, for example, verify that there are only encrypted storage resources.

You can automate validation that all EBS volumes are encrypted using AWS Config Rules.

AWS Security Hub can also verify several different controls through automated checks against security standards. Additionally, your AWS Config Rules can automatically remediate noncompliant resources.

Enforce access control with least privileges and mechanisms, including backups, isolation, and versioning, to help protect your data at rest. Prevent operators from granting public access to your data.

Different controls including access (using least privilege), backups (see Reliability whitepaper), isolation, and versioning can all help protect your data at rest.

Keep all users away from directly accessing sensitive data and systems under normal operational circumstances.

For example, use a change management workflow to manage Amazon Elastic Compute Cloud (Amazon EC2) instances using tools instead of allowing direct access or a bastion host. This can be achieved using AWS Systems Manager Automation.

By providing the appropriate level of protection for your data in transit, you protect the confidentiality and integrity of your workload’s data.

Store encryption keys and certificates securely and rotate them at appropriate time intervals with strict access control.

The best way to accomplish this is to use a managed service, such as AWS Certificate Manager (ACM).

Enforce your defined encryption requirements based on appropriate standards and recommendations to help you meet your organizational, legal, and compliance requirements.

AWS services provide HTTPS endpoints using TLS for communication, thus providing encryption in transit when communicating with the AWS APIs.

Use tools such as Amazon GuardDuty to automatically detect suspicious activity or attempts to move data outside of defined boundaries.

Verify the identity of communications by using protocols that support authentication, such as Transport Layer Security (TLS) or IPsec.

Using network protocols that support authentication, allows for trust to be established between the parties. This adds to the encryption used in the protocol to reduce the risk of communications being altered or intercepted.

Make sure that your teams have appropriate pre-provisioned access to perform their duties before an event occurs. All tools, access, and plans should be documented and tested before an event occurs to make sure that they can provide a timely response.

Identify internal and external personnel, resources, and legal obligations that would help your organization respond to an incident.

When you define your approach to incident response in the cloud, in unison with other teams (such as your legal counsel, leadership, business stakeholders, AWS Support Services, and others), you must identify key personnel, stakeholders, and relevant contacts.

Create plans to help you respond to, communicate during, and recover from an incident.

For example, you can start an incident response plan with the most likely scenarios for your workload and organization.

It’s important for your incident responders to understand when and how the forensic investigation fits into your response plan.

Your organization should define what evidence is collected and what tools are used in the process. Identify and prepare forensic investigation capabilities that are suitable, including external specialists, tools, and automation.

Verify that incident responders have the correct access pre-provisioned in AWS to reduce the time needed for investigation through to recovery.

Ensure that security personnel have the right tools pre-deployed into AWS to reduce the time for investigation through to recovery.

To automate security engineering and operations functions, you can use a comprehensive set of APIs and tools from AWS. You can fully automate identity management, network security, data protection, and monitoring capabilities and deliver them using popular software development methods that you already have in place.

The value derived from participating in a simulation activity increases an organization's effectiveness during stressful events.

Game days, also known as simulations or exercises, are internal events that provide a structured opportunity to practice your incident management plans and procedures during a realistic scenario

These events should exercise responders using the same tools and techniques that would be used in a real-world scenario - even mimicking real-world environments. Game days are fundamentally about being prepared and iteratively improving your response capabilities.

Once you create and practice the processes and tools from your playbooks, you can deconstruct the logic into a code-based solution, which can be used as a tool by many responders to automate the response and remove variance or guess-work by your responders.

This can speed up the lifecycle of a response.

These quotas exist to prevent accidentally provisioning more resources than you need and to limit request rates on API operations so as to protect services from abuse.

There are also resource constraints, for example, the rate that you can push bits down a fiber-optic cable, or the amount of storage on a physical disk.

You are aware of your default quotas and quota increase requests for your workload architecture.

You additionally know which resource constraints, such as disk or network, are potentially impactful.

Service Quotas is an AWS service that helps you manage your quotas for over 100 AWS services from one location.

If you are using multiple AWS accounts or AWS Regions, ensure that you request the appropriate quotas in all environments in which your production workloads run.

Service quotas are tracked per account. Unless otherwise noted, each quota is AWS Region-specific.

In addition to the production environments, also manage quotas in all applicable non-production environments, so that testing and development are not hindered.

Be aware of unchangeable service quotas and physical resources, and architect to prevent these from impacting reliability.

Examples include network bandwidth, AWS Lambda payload size, throttle burst rate for API Gateway, and concurrent user connections to an Amazon Redshift cluster.

Evaluate your potential usage and increase your quotas appropriately, allowing for planned growth in usage.

For supported services, you can manage your quotas by configuring CloudWatch alarms to monitor usage and alert you to approaching quotas.

These alarms can be triggered from Service Quotas or from Trusted Advisor.

You can also use metric filters on CloudWatch Logs to search and extract patterns in logs to determine if usage is approaching quota thresholds.

Implement tools to alert you when thresholds are being approached.

You can automate quota increase requests by using AWS Service Quotas APIs.

If you integrate your Configuration Management Database (CMDB) or ticketing system with Service Quotas, you can automate the tracking of quota increase requests and current quotas.

When a resource fails, it might still be counted against quotas until it’s successfully terminated.

Ensure that your quotas cover the overlap of all failed resources with replacements before the failed resources are terminated. You should consider an Availability Zone failure when calculating this gap.

Plans must include network considerations, such as intrasystem and intersystem connectivity, public IP address management, private IP address management, and domain name resolution.

When architecting systems using IP address-based networks, you must plan network topology and addressing in anticipation of possible failures, and to accommodate future growth and integration with other systems and their networks.

Amazon Virtual Private Cloud (Amazon VPC) lets you provision a private, isolated section of the AWS Cloud where you can launch AWS resources in a virtual network.

These endpoints and the routing to them must be highly available.

To achieve this, use highly available DNS, content delivery networks (CDNs), API Gateway, load balancing, or reverse proxies.

Amazon Route 53, AWS Global Accelerator, Amazon CloudFront, Amazon API Gateway, and Elastic Load Balancing (ELB) all provide highly available public endpoints.

You might also choose to evaluate AWS Marketplace software appliances for load balancing and proxying.

Use multiple AWS Direct Connect connections or VPN tunnels between separately deployed private networks.

Use multiple Direct Connect locations for high availability. If using multiple AWS Regions, ensure redundancy in at least two of them.

Amazon VPC IP address ranges must be large enough to accommodate workload requirements, including factoring in future expansion and allocation of IP addresses to subnets across Availability Zones.

This includes load balancers, EC2 instances, and container-based applications.

If more than two network address spaces (for example, VPCs and on-premises networks) are connected via VPC peering, AWS Direct Connect, or VPN, then use a hub-and-spoke model, like that provided by AWS Transit Gateway.

The IP address ranges of each of your VPCs must not overlap when peered or connected via VPN.

You must similarly avoid IP address conflicts between a VPC and on-premises environments or with other cloud providers that you use.

You must also have a way to allocate private IP address ranges when needed.

Service-oriented architecture (SOA) is the practice of making software components reusable via service interfaces.

Microservices architecture goes further to make components smaller and simpler.

Workload segmentation is important when determining the resilience requirements of your application. Monolithic architecture should be avoided whenever possible.

Instead, carefully consider which application components can be broken out into microservices. Depending on your application requirements, this may end up being a combination of a service-oriented architecture (SOA) with microservices where possible.

Workloads that are capable of statelessness are more capable of being deployed as microservices.

Service-oriented architecture (SOA) builds services with well-delineated functions defined by business needs.

Microservices use domain models and bounded context to limit this further so that each service does just one thing. Focusing on specific functionality enables you to differentiate the reliability requirements of different services, and target investments more specifically.

Service contracts are documented agreements between teams on service integration and include a machine-readable API definition, rate limits, and performance expectations.

A versioning strategy allows your clients to continue using the existing API and migrate their applications to the newer API when they are ready.

Components of the distributed system must operate in a way that does not negatively impact other components or the workload.

These best practices prevent failures and improve mean time between failures (MTBF).

Best Practices

Hard real-time distributed systems require responses to be given synchronously and rapidly, while soft real-time systems have a more generous time window of minutes or more for response.

Offline systems handle responses through batch or asynchronous processing.

Hard real-time distributed systems have the most stringent reliability requirements.

Dependencies such as queuing systems, streaming systems, workflows, and load balancers are loosely coupled.

Loose coupling helps isolate behavior of a component from other components that depend on it, increasing resiliency and agility.

Systems can fail when there are large, rapid changes in load. For example, if your workload is doing a health check that monitors the health of thousands of servers, it should send the same size payload (a full snapshot of the current state) each time.

Whether no servers are failing, or all of them, the health check system is doing constant work with no large, rapid changes.

An idempotent service promises that each request is completed exactly once, such that making multiple identical requests has the same effect as making a single request.

An idempotent service makes it easier for a client to implement retries without fear that a request will be erroneously processed multiple times.

Your workload must operate reliably despite data loss or latency over these networks. Components of the distributed system must operate in a way that does not negatively impact other components or the workload.

These best practices enable workloads to withstand stresses or failures, more quickly recover from them, and mitigate the impact of such impairments. The result is improved mean time to recovery (MTTR)

When a component's dependencies are unhealthy, the component itself can still function, although in a degraded manner.

For example, when a dependency call fails, failover to a predetermined static response.

Throttling requests is a mitigation pattern to respond to an unexpected increase in demand.

Some requests are honored but those over a defined limit are rejected and return a message indicating they have been throttled. The expectation on clients is that they will back off and abandon the request or try again at a slower rate.

Use exponential backoff to retry after progressively longer intervals. Introduce jitter to randomize those retry intervals, and limit the maximum number of retries.

If the workload is unable to respond successfully to a request, then fail fast.

This allows the releasing of resources associated with a request, and permits the service to recover if it’s running out of resources.

Set timeouts appropriately, verify them systematically, and do not rely on default values as they are generally set too high.

This best practice applies to the client-side, or sender, of the request.

Services should either not require state, or should offload state such that between different client requests, there is no dependence on locally stored data on disk and in memory.

This enables servers to be replaced at will without causing an availability impact. Amazon ElastiCache or Amazon DynamoDB are good destinations for offloaded state.

Emergency levers are rapid processes that can mitigate availability impact on your workload.

You can configure your workload to monitor logs and metrics and send notifications when thresholds are crossed or significant events occur

Monitoring enables your workload to recognize when low-performance thresholds are crossed or failures occur, so it can recover automatically in response.

All components of your workload should be monitored, including the front-end, business logic, and storage tiers.

Monitor the components of the workload with Amazon CloudWatch or third-party tools. Monitor AWS services with AWS Health Dashboard.

Store log data and apply filters where necessary to calculate metrics, such as counts of a specific log event, or latency calculated from log event timestamps.

Organizations that need to know, receive notifications when significant events occur.

Alerts can be sent to Amazon Simple Notification Service (Amazon SNS) topics, and then pushed to any number of subscribers.

For example, Amazon SNS can forward alerts to an email alias so that technical staff can respond.

Use automation to take action when an event is detected, for example, to replace failed components.

Collect log files and metrics histories and analyze these for broader trends and workload insights.

Frequently review how workload monitoring is implemented and update it based on significant events and changes.

Effective monitoring is driven by key business metrics.

Ensure these metrics are accommodated in your workload as business priorities change.

Use AWS X-Ray or third-party tools so that developers can more easily analyze and debug distributed systems to understand how their applications and its underlying services are performing.

When replacing impaired resources or scaling your workload, automate the process by using managed AWS services, such as Amazon S3 and AWS Auto Scaling.

You can also use third-party tools and AWS SDKs to automate scaling.

Scale resources reactively when necessary if availability is impacted, to restore workload availability.

You first must configure health checks and the criteria on these checks to indicate when availability is impacted by lack of resources.

Then either notify the appropriate personnel to manually scale the resource, or trigger automation to automatically scale it.

Scale resources proactively to meet demand and avoid availability impact.

Adopt a load testing methodology to measure if scaling activity meets workload requirements.

It’s important to perform sustained load testing.

Load tests should discover the breaking point and test the performance of your workload.

If these changes are uncontrolled, then it makes it difficult to predict the effect of these changes, or to address issues that arise because of them.

Runbooks are the predefined procedures to achieve specific outcomes.

Use runbooks to perform standard activities, whether done manually or automatically.

Examples include deploying a workload, patching a workload, or making DNS modifications.

Functional tests are run as part of automated deployment. If success criteria are not met, the pipeline is halted or rolled back.

These tests are run in a pre-production environment, which is staged prior to production in the pipeline.

Resiliency tests (using the principles of chaos engineering) are run as part of the automated deployment pipeline in a pre-production environment.

These tests are staged and run in the pipeline in a pre-production environment.

They should also be run in production as part of game days.

Immutable infrastructure is a model that mandates that no updates, security patches, or configuration changes happen in-place on production workloads.

When a change is needed, the architecture is built onto new infrastructure and deployed into production.

Deployments and patching are automated to eliminate negative impact.

All AWS data stores offer backup capabilities.

Services such as Amazon RDS and Amazon DynamoDB additionally support automated backup that enables point-in-time recovery (PITR), which allows you to restore a backup to any time up to five minutes or less before the current time.

Many AWS services offer the ability to copy backups to another AWS Region. AWS Backup is a tool that gives you the ability to centralize and automate data protection across AWS services.

Control and detect access to backups using authentication and authorization, such as AWS IAM.

Prevent and detect if data integrity of backups is compromised using encryption.

Configure backups to be taken automatically based on a periodic schedule informed by the Recovery Point Objective (RPO), or by changes in the dataset.

Critical datasets with low data loss requirements need to be backed up automatically on a frequent basis, whereas less critical data where some loss is acceptable can be backed up less frequently.

Validate that your backup process implementation meets your recovery time objectives (RTO) and recovery point objectives (RPO) by performing a recovery test.

Components outside of the boundary are unaffected by the failure. Using multiple fault isolated boundaries, you can limit the impact on your workload.

Distribute workload data and resources across multiple Availability Zones or, where necessary, across AWS Regions.

These locations can be as diverse as required.

For high availability, always (when possible) deploy your workload components to multiple Availability Zones (AZs).

For workloads with extreme resilience requirements, carefully evaluate the options for a multi-Region architecture.

If components of the workload can only run in a single Availability Zone or in an on-premises data center, you must implement the capability to do a complete rebuild of the workload within your defined recovery objectives.

Like the bulkheads on a ship, this pattern ensures that a failure is contained to a small subset of requests or clients so that the number of impaired requests is limited, and most can continue without error.

Bulkheads for data are often called partitions, while bulkheads for services are known as cells.

Continuously monitor the health of your workload so that you and your automated systems are aware of degradation or failure as soon as they occur.

Ensure that if a resource failure occurs, that healthy resources can continue to serve requests.

For location failures (such as Availability Zone or AWS Region) ensure that you have systems in place to fail over to healthy resources in unimpaired locations.

Upon detection of a failure, use automated capabilities to perform actions to remediate.

The control plane is used to configure resources, and the data plane delivers services.

Data planes typically have higher availability design goals than control planes and are usually less complex.

When implementing recovery or mitigation responses to potentially resiliency-impacting events, using control plane operations can lower the overall resiliency of your architecture.

Bimodal behavior is when your workload exhibits different behavior under normal and failure modes, for example, relying on launching new instances if an Availability Zone fails.

You should instead build workloads that are statically stable and operate in only one mode.

Notifications are sent upon the detection of significant events, even if the issue caused by the event was automatically resolved.

Test to validate that your workload meets functional and non-functional requirements, because bugs or performance bottlenecks can impact the reliability of your workload. Test the resiliency of your workload to help you find latent bugs that only surface in production. Exercise these tests regularly.

Enable consistent and prompt responses to failure scenarios that are not well understood, by documenting the investigation process in playbooks.

Playbooks are the predefined steps performed to identify the factors contributing to a failure scenario.

The results from any process step are used to determine the next steps to take until the issue is identified or escalated.

Review customer-impacting events, and identify the contributing factors and preventative action items.

Use this information to develop mitigations to limit or prevent recurrence.

Develop procedures for prompt and effective responses.

Use techniques such as unit tests and integration tests that validate required functionality.

Use techniques such as load testing to validate that the workload meets scaling and performance requirements.

Run chaos experiments regularly in environments that are in or as close to production as possible to understand how your system responds to adverse conditions.

Use game days to regularly exercise your procedures for responding to events and failures as close to production as possible (including in production environments) with the people who will be involved in actual failure scenarios.

Game days enforce measures to ensure that production events do not impact users.

Set these based on business needs. Implement a strategy to meet these objectives, considering locations and function of workload resources and data

Both Availability and Disaster Recovery rely on the same best practices such as monitoring for failures, deploying to multiple locations, and automatic failover. However Availability focuses on components of the workload, while Disaster Recovery focuses on discrete copies of the entire workload. Disaster Recovery has different objectives from Availability, focusing on time to recovery after a disaster

The workload has a recovery time objective (RTO) and recovery point objective (RPO).

Recovery Time Objective (RTO) is the maximum acceptable delay between the interruption of service and restoration of service.

Recovery Point Objective (RPO) is the maximum acceptable amount of time since the last data recovery point.

Define a disaster recovery (DR) strategy that meets your workload's recovery objectives.

Choose a strategy such as: backup and restore; standby (active/passive); or active/active.

Regularly test failover to your recovery site to ensure proper operation, and that RTO and RPO are met.

Ensure that the infrastructure, data, and configuration are as needed at the DR site or Region.

For example, check that AMIs and service quotas are up to date.

Use AWS or third-party tools to automate system recovery and route traffic to the DR site or Region.

AWS Solutions Architects, AWS Reference Architectures, and AWS Partners can help you select an architecture based on industry knowledge, but data obtained through benchmarking or load testing will be required to optimize your architecture.

Your architecture will likely combine a number of different architectural approaches (for example, eventdriven, ETL, or pipeline). The implementation of your architecture will use the AWS services that are specific to the optimization of your architecture's performance. In the following sections we discuss the four main resource types to consider (compute, storage, database, and network).

Learn about and understand the wide range of services and resources available in the cloud.

Identify the relevant services and configuration options for your workload, and understand how to achieve optimal performance.

Use internal experience and knowledge of the cloud, or external resources such as published use cases, relevant documentation, or whitepapers, to define a process to choose resources and services. Y

You should define a process that encourages experimentation and benchmarking with the services that could be used in your workload

Workloads often have cost requirements for operation. Use internal cost controls to select resource types and sizes based on predicted resource need.

Determine which workload components could be replaced with fully managed services, such as managed databases, in-memory caches, and ETL services. Reducing your operational workload allows you to focus resources on business outcomes.

Maximize performance and efficiency by evaluating internal policies and existing reference architectures and using your analysis to select services and configurations for your workload.

Use cloud company resources, such as solutions architects, professional services, or an appropriate partner to guide your decisions. These resources can help review and improve your architecture for optimal performance.

Reach out to AWS for assistance when you need additional guidance or product information. AWS Solutions Architects and AWS Professional Services provide guidance for solution implementation. AWS Partners provide AWS expertise to help you unlock agility and innovation for your business.

Benchmark the performance of an existing workload to understand how it performs on the cloud. Use the data collected from benchmarks to drive architectural decisions.

Use benchmarking with synthetic tests and real-user monitoring to generate data about how your workload’s components perform. Benchmarking is generally quicker to set up than load testing and is used to evaluate the technology for a particular component. Benchmarking is often used at the start of a new project, when you lack a full solution to load test.

Deploy your latest workload architecture on the cloud using different resource types and sizes. Monitor the deployment to capture performance metrics that identify bottlenecks or excess capacity.

Use this performance information to design or improve your architecture and resource selection.

Architectures may use different compute choices for various components and enable different features to improve performance.

Selecting the wrong compute choice for an architecture can lead to lower performance efficiency

Understand how your workload can benefit from the use of different compute options, such as instances, containers and functions.

Each compute solution has options and configurations available to you to support your workload characteristics.

Learn how various options complement your workload, and which configuration options are best for your application.

To understand how your compute resources are performing, you must record and track the utilization of various systems.

This data can be used to make more accurate determinations about resource requirements.

Analyze the various performance characteristics of your workload and how these characteristics relate to memory, network, and CPU usage.

Use this data to choose resources that best match your workload's profile.

The cloud provides the flexibility to expand or reduce your resources dynamically through a variety of mechanisms to meet changes in demand.

Combined with compute-related metrics, a workload can automatically respond to changes and use the optimal set of resources to achieve its goal.

Use system-level metrics to identify the behavior and requirements of your workload over time.

Evaluate your workload's needs by comparing the available resources with these requirements and make changes to your compute environment to best match your workload's profile

Well-architected systems use multiple storage solutions and enable different features to improve performance.

In AWS, storage is virtualized and is available in a number of different types. This makes it easier to match your storage methods with your needs, and offers storage options that are not easily achievable with on-premises infrastructure.

Identify and document the workload storage needs and define the storage characteristics of each location.

Examples of storage characteristics include: shareable access, file size, growth rate, throughput, IOPS, latency, access patterns, and persistence of data. Use these characteristics to evaluate if block, file, object, or instance storage services are the most efficient solution for your storage needs.

Evaluate the various characteristics and configuration options and how they relate to storage.

Understand where and how to use provisioned IOPS, SSDs, magnetic storage, object storage, archival storage, or ephemeral storage to optimize storage space and performance for your workload.

Choose storage systems based on your workload's access patterns and configure them by determining how the workload accesses data.

Increase storage efficiency by choosing object storage over block storage. Configure the storage options you choose to match your data access patterns.

Many systems use different database solutions for various sub-systems and enable different features to improve performance.

Selecting the wrong database solution and features for a system can lead to lower performance efficiency.

Choose your data management solutions to optimally match the characteristics, access patterns, and requirements of your workload datasets.

When selecting and implementing a data management solution, you must ensure that the querying, scaling, and storage characteristics support the workload data requirements.

Learn how various database options match your data models, and which configuration options are best for your use-case

Understand the available database options and how it can optimize your performance before you select your data management solution.

Understand the available database options and how it can optimize your performance before you select your data management solution. Use load testing to identify database metrics that matter for your workload.

To understand how your data management systems are performing, it is important to track relevant metrics. These metrics will help you to optimize your data management resources, to ensure that your workload requirements are met, and that you have a clear overview on how the workload performs.

Use tools, libraries, and systems that record performance measurements related to database performance.

Use the access patterns of the workload to decide which services and technologies to use. In addition to non-functional requirements such as performance and scale, access patterns heavily influence the choice of the database and storage solutions.

The first dimension is the need for transactions, ACID compliance, and consistent reads. Not every database supports these and most of the NoSQL databases provide an eventual consistency model. The second important dimension would be the distribution of write and reads over time and space

Use performance characteristics and access patterns that optimize how data is stored or queried to achieve the best possible performance.

Measure how optimizations such as indexing, key distribution, data warehouse design, or caching strategies impact system performance or overall efficiency.

On AWS, networking is virtualized and is available in a number of different types and configurations. This makes it easier to match your networking methods with your needs.

AWS offers product features (for example, Enhanced Networking, Amazon EC2 networking optimized instances, Amazon S3 transfer acceleration, and dynamic Amazon CloudFront) to optimize network traffic.

AWS also offers networking features (for example, Amazon Route 53 latency routing, Amazon VPC endpoints, AWS Direct Connect, and AWS Global Accelerator) to reduce network distance or jitter.

Analyze and understand how network-related decisions impact workload performance. The network is responsible for the connectivity between application components, cloud services, edge networks and on-premises data and therefor it can highly impact workload performance.

In addition to workload performance, user experience is also impacted by network latency, bandwidth, protocols, location, network congestion, jitter, throughput, and routing rules.

Evaluate networking features in the cloud that may increase performance. Measure the impact of these features through testing, metrics, and analysis.

For example, take advantage of network-level features that are available to reduce latency, packet loss, or jitter.

When a common network is required to connect on-premises and cloud resources in AWS, ensure that you have adequate bandwidth to meet your performance requirements.

Estimate the bandwidth and latency requirements for your hybrid workload. These numbers will drive the sizing requirements for AWS Direct Connect or your VPN endpoints.

Distribute traffic across multiple resources or services to allow your workload to take advantage of the elasticity that the cloud provides.

You can also use load balancing for offloading encryption termination to improve performance and to manage and route traffic effectively

Make decisions about protocols for communication between systems and networks based on the impact to the workload’s performance.

There is a relationship between latency and bandwidth to achieve throughput. If your file transfer is using TCP, higher latencies will reduce overall throughput. There are approaches to fix this with TCP tuning and optimized transfer protocols, some approaches use UDP.

Use the cloud location options available to reduce network latency or improve throughput.

Use AWS Regions, Availability Zones, placement groups, and edge locations such as AWS Outposts, AWS Local Zones, and AWS Wavelength, to reduce network latency or improve throughput.

Use collected and analyzed data to make informed decisions about optimizing your network configuration. Measure the impact of those changes and use the impact measurements to make future decisions.

Enable VPC Flow Logs for all VPC networks that are used by your workload. VPC Flow Logs are a feature that allows you to capture information about the IP traffic going to and from network interfaces in your VPC.

Evaluate ways to improve performance as new services, design patterns, and product offerings become available.

Determine which of these could improve performance or increase the efficiency of the workload through evaluation, internal discussion, or external analysis.

Define a process to evaluate new services, design patterns, resource types, and configurations as they become available.

For example, run existing performance tests on new instance offerings to determine their potential to improve your workload.

As an organization, use the information gathered through the evaluation process to actively drive adoption of new services or resources when they become available.

Use the information you gather when evaluating new services or technologies to drive change. As your business or workload changes, performance needs also change.

Use a monitoring and observability service to record performance-related metrics.

Examples of metrics include record database transactions, slow queries, I/O latency, HTTP request throughput, service latency, or other key data.

In response to (or during) an event or incident, use monitoring dashboards or reports to understand and diagnose the impact. These views provide insight into which portions of the workload are not performing as expected.

Identify the KPIs that quantitatively and qualitatively measures workload performance. KPIs help to measure the health of a workload as it relates to a business goal.

KPIs allow business and engineering teams to align on the measurement of goals and strategies and how this combines to produce business outcomes.

KPIs should be revisited when business goals, strategies, or end-user requirements change.

Using the performance-related key performance indicators (KPIs) that you defined, use a monitoring system that generates alarms automatically when these measurements are outside expected boundaries.

Amazon CloudWatch can collect metrics across the resources in your architecture. You can also collect and publish custom metrics to surface business or derived metrics.

As routine maintenance, or in response to events or incidents, review which metrics are collected.

Use these reviews to identify which metrics were essential in addressing issues and which additional metrics, if they were being tracked, would help to identify, address, or prevent issues.

Use key performance indicators (KPIs), combined with monitoring and alerting systems, to proactively address performance-related issues.

Use alarms to trigger automated actions to remediate issues where possible. Escalate the alarm to those able to respond if automated response is not possible

Often you can improve performance by trading consistency, durability, and space for time and latency. Trade-offs can increase the complexity of your architecture and require load testing to ensure that a measurable benefit is obtained.

Understand and identify areas where increasing the performance of your workload will have a positive impact on efficiency or customer experience.

For example, a website that has a large amount of customer interaction can benefit from using edge services to move content delivery closer to customers.

Research and understand the various design patterns and services that help improve workload performance. As part of the analysis, identify what you could trade to achieve higher performance.

For example, using a cache service can help to reduce the load placed on database systems. However, caching can introduce eventual consistency and requires engineering effort to implement within business requirements and customer expectations.

When evaluating performance-related improvements, determine which choices will impact your customers and workload efficiency.

For example, if using a key-value data store increases system performance, it is important to evaluate how the eventually consistent nature of it will impact customers.

As changes are made to improve performance, evaluate the collected metrics and data. Use this information to determine impact that the performance improvement had on the workload, the workload’s components, and your customers.

Where applicable, use multiple strategies to improve performance.

For example, using strategies like caching data to prevent excessive network or database calls, using read-replicas for database engines to improve read rates, sharding or compressing data where possible to reduce data volumes, and buffering and streaming of results as they are available to avoid blocking.

Create a team (Cloud Business Office or Cloud Center of Excellence) that is responsible for establishing and maintaining cost awareness across your organization. The team requires people from finance, technology, and business roles across the organization.

Involve finance and technology teams in cost and usage discussions at all stages of your cloud journey. Teams regularly meet and discuss topics such as organizational goals and targets, current state of cost and usage, and financial and accounting practices.

Adjust existing organizational budgeting and forecasting processes to be compatible with the highly variable nature of cloud costs and usage. Processes must be dynamic using trend-based or business driver-based algorithms, or a combination of both.

Implement cost awareness, create transparency, and accountability of costs into new or existing processes that impact usage, and leverage existing processes for cost awareness. Implement cost awareness into employee training.

Configure AWS Budgets and AWS Cost Anomaly Detection to provide notifications on cost and usage against targets. Have regular meetings to analyze your workload's cost efficiency and to promote cost- aware culture.

Implement tooling and dashboards to monitor cost proactively for the workload. Regularly review the costs with configured tools or out of the box tools, do not just look at costs and categories when you receive notifications. Monitoring and analyzing costs proactively helps to identify positive trends and allows you to promote them throughout your organization.

Consult regularly with experts or AWS Partners to consider which services and features provide lower cost. Review AWS blogs and other information sources.

Implement changes or programs across your organization to create a cost-aware culture. It is recommended to start small, then as your capabilities increase and your organization’s use of the cloud increases, implement large and wide ranging programs.

Quantifying business value from cost optimization allows you to understand the entire set of benefits to your organization. Because cost optimization is a necessary investment, quantifying business value allows you to explain the return on investment to stakeholders. Quantifying business value can help you gain more buy-in from stakeholders on future cost optimization investments, and provides a framework to measure the outcomes for your organization’s cost optimization activities.

Develop policies that define how resources are managed by your organization. Policies should cover cost aspects of resources and workloads, including creation, modification and decommission over the resource lifetime.

Implement both cost and usage goals for your workload. Goals provide direction to your organization on cost and usage, and targets provide measurable outcomes for your workloads.

Implement a structure of accounts that maps to your organization. This assists in allocating and managing costs throughout your organization.

Implement groups and roles that align to your policies and control who can create, modify, or decommission instances and resources in each group. For example, implement development, test, and production groups. This applies to AWS services and third-party solutions.

Implement controls based on organization policies and defined groups and roles. These certify that costs are only incurred as defined by organization requirements: for example, control access to regions or resource types with AWS Identity and Access Management (IAM) policies.

Track, measure, and audit the lifecycle of projects, teams, and environments to avoid using and paying for unnecessary resources.

Cost optimization begins with a granular understanding of the breakdown in cost and usage, the ability to model and forecast future spend, usage, and features, and the implementation of sufficient mechanisms to align cost and usage to your organization’s objectives.

Configure the AWS Cost and Usage Report, and Cost Explorer hourly granularity, to provide detailed cost and usage information. Configure your workload to have log entries for every delivered business outcome. Tag resources.

Identify organization categories that could be used to allocate cost within your organization.

Establish the organization metrics that are required for this workload. Example metrics of a workload are customer reports produced, or web pages served to customers.

Configure AWS Cost Explorer and AWS Budgets inline with your organization policies.

Define a tagging schema based on organization, and workload attributes, and cost allocation categories. Implement tagging across all resources. Use Cost Categories to group costs and usage according to organization attributes.

Allocate the workload's costs by metrics or business outcomes to measure workload cost efficiency. Implement a process to analyze the AWS Cost and Usage Report with Amazon Athena, which can provide insight and charge back capability.

Define and implement a method to track resources and their associations with systems over their lifetime. You can use tagging to identify the workload or function of the resource.

Implement a process to identify and decommission orphaned resources.

Decommission resources triggered by events such as periodic audits, or changes in usage. Decommissioning is typically performed periodically, and is manual or automated.

Design your workload to gracefully handle resource termination as you identify and decommission non- critical resources, resources that are not required, or resources with low utilization.

Work with team members to define the balance between cost optimization and other pillars, such as performance and reliability, for this workload.

Verify every workload component is analyzed, regardless of current size or current costs. The review effort should reflect the potential benefit, such as current and projected costs.

Look at overall cost to the organization of each component. Look at total cost of ownership by factoring in cost of operations and management, especially when using managed services. The review effort should reflect potential benefit, for example, time spent analyzing is proportional to component cost.

Open-source software eliminates software licensing costs, which can contribute significant costs to workloads. Where licensed software is required, avoid licenses bound to arbitrary attributes such as CPUs, look for licenses that are bound to output or outcomes. The cost of these licenses scales more closely to the benefit they provide.

Factor in cost when selecting all components. This includes using application level and managed services, such as Amazon Relational Database Service (Amazon RDS), Amazon DynamoDB, Amazon Simple Notification Service (Amazon SNS), and Amazon Simple Email Service (Amazon SES) to reduce overall organization cost. Use serverless and containers for compute, such as AWS Lambda, Amazon Simple Storage Service (Amazon S3)for static websites, and Amazon Elastic Container Service (Amazon ECS). Minimize license costs by using open source software, or software that does not have license fees: for example, Amazon Linux for compute workloads or migrate databases to Amazon Aurora.

Workloads can change over time. Some services or features are more cost effective at different usage levels. By performing the analysis on each component over time and at projected usage, the workload remains cost-effective over its lifetime.

Right-sizing activities takes into account all of the resources of a workload, all of the attributes of each individual resource, and the effort involved in the right-sizing operation. Right-sizing can be an iterative process, triggered by changes in usage patterns and external factors, such as AWS price drops or new AWS resource types. Right-sizing can also be one-off if the cost of the effort to right-size, outweighs the potential savings over the life of the workload.

Identify organization requirements and perform cost modeling of the workload and each of its components. Perform benchmark activities for the workload under different predicted loads and compare the costs. The modeling effort should reflect the potential benefit. For example, time spent is proportional to component cost.

Select resource size or type based on data about the workload and resource characteristics. For example, compute, memory, throughput, or write intensive. This selection is typically made using a previous (on- premises) version of the workload, using documentation, or using other sources of information about the workload.

Use metrics from the currently running workload to select the right size and type to optimize for cost. Appropriately provision throughput, sizing, and storage for services such as Amazon Elastic Compute Cloud (Amazon EC2), Amazon DynamoDB, Amazon Elastic Block Store (Amazon EBS) (PIOPS), Amazon Relational Database Service (Amazon RDS), Amazon EMR, and networking. This can be done with a feedback loop such as automatic scaling or by custom code in the workload.

Perform regular account level analysis: Performing regular cost modeling ensures that opportunities to optimize across multiple workloads can be implemented.

• On-Demand Instances • Spot Instances • Commitment discounts - Savings Plans • Commitment discounts - Reserved Instances/Capacity • Geographic selection • Third-party agreements and pricing

Analyze each component of the workload. Determine if the component and resources will be running for extended periods (for commitment discounts), or dynamic and short-running (for Spot or On-Demand Instances). Perform an analysis on the workload using the Recommendations feature in AWS Cost Explorer.

Resource pricing can be different in each Region. Factoring in Region cost helps ensure that you pay the lowest overall price for this workload.

Cost efficient agreements and terms ensure the cost of these services scales with the benefits they provide. Select agreements and pricing that scale when they provide additional benefits to your organization.

Permanently running resources should utilize reserved capacity such as Savings Plans or Reserved Instances. Short-term capacity is configured to use Spot Instances, or Spot Fleet. On-Demand Instances are only used for short-term workloads that cannot be interrupted and do not run long enough for reserved capacity, between 25% to 75% of the period, depending on the resource type.

Use Cost Explorer Savings Plans and Reserved Instance recommendations to perform regular analysis at the management account level for commitment discounts.

Gather organization requirements and perform data transfer modeling of the workload and each of its components. This identifies the lowest cost point for its current data transfer requirements.

All components are selected, and architecture is designed to reduce data transfer costs. This includes using components such as wide-area-network (WAN) optimization and Multi-Availability Zone (AZ) configurations

Implement services to reduce data transfer. For example, using a content delivery network (CDN) such as Amazon CloudFront to deliver content to end users, caching layers using Amazon ElastiCache, or using AWS Direct Connect instead of VPN for connectivity to AWS.

Align service levels to customer needs.

Position resources to limit the network required for users to consume them.

Remove existing, unused assets.

Identify created assets that are unused and stop generating them. Provide your team members with devices that support their needs with minimized sustainability impact.

Identify periods of low or no utilization and scale down resources to eliminate excess capacity and improve efficiency.

Define and update Service Level Agreements (SLAs) such as availability or data retention periods to minimize the number of resources required to support your workload while continuing to meet business requirements.

Analyze application assets (such as pre-compiled reports, datasets, and static images) and asset access patterns to identify redundancy, underutilization, and potential decommission targets.

Consolidate generated assets with redundant content (for example, monthly reports with overlapping or common datasets and outputs) to remove the resources consumed when duplicating outputs

Decommission unused assets (for example, images of products that are no longer sold) to free consumed resources and reduce the number of resources used to support the workload.

Analyze network access patterns to identify where your customers are connecting from geographically. Select Regions and services that reduce the distance network traffic must travel to decrease the total network resources required to support your workload.

Optimize resources provided to team members to minimize the sustainability impact while supporting their needs. For example, perform complex operations, such as rendering and compilation, on highly utilized shared cloud desktops instead of on underutilized high-powered single-user systems.

Revise patterns and architecture to consolidate under-utilized components to increase overall utilization.

Retire components that are no longer required.

patterns to minimize the need for device upgrades.