AWS re:Invent 2024 - Don't get stuck: How connected telemetry keeps you moving forward (COP322)

The speaker wakes up to an alert about a production site problem, leading to a troubleshooting scenario with teammates coming online to assist but facing challenges in identifying the root cause.

Explains the purpose of troubleshooting, which is to find actionable solutions to resolve issues quickly, focusing on resolving immediate problems and making follow-ups for long-term improvements.

Discusses the five common triggers of outages: changes in the system, reaching limits, dependencies failing, instances failing, and workload changes, emphasizing the need to address these triggers to stop the problem.

Illustrates how to investigate issues faster by identifying and addressing specific triggers systematically, emphasizing the importance of quick identification and mitigation strategies.

Explains the benefits of application instrumentation, including faster navigation during troubleshooting, enhanced visibility into system performance, and the importance of dimensionality in measurements.

Emphasizes the significance of navigation and connectivity in troubleshooting to avoid getting stuck, navigate efficiently between components, and utilize manual instrumentation for effective tracking and resolution.

Demonstrates how to leverage CloudWatch logs and traces for detailed analysis, investigating faults, and tracing the flow of requests to identify and resolve issues effectively.

Introduces new capabilities in CloudWatch for faster and more efficient troubleshooting, such as filtering noise, summarizing results, and identifying issues through log analysis and query functionalities.

Using an AI agent to conduct investigations and sift through telemetry and topology to identify issues.

Starting the investigation process from the entry signals, focusing on the bot service and gateway, and identifying configuration API issues.

Exploring bot schedule service as a dependency of bot service and following causal pathways to uncover errors.

Detecting access denied messages in logs related to specific microservices like DynamoDB and tracing errors back to resource policy changes.

The AI agent generates hypotheses to explain issues, providing insights into the root cause and recommending actions like reviewing recent changes and implementing change control policies.

Exploring five troubleshooting steps to identify root causes, gather information, and investigate using a systematic approach.