In modern cloud-native applications, monitoring and troubleshooting are critical for ensuring performance, reliability, and security. Amazon CloudWatch an AWS monitoring service, provides a powerful feature known as CloudWatch Logs Insights.

This feature has allowed our teams and developers to quickly and easily search log data, helping them get to the root cause and find actionable insights quickly and efficiently.

What is Amazon CloudWatch Logs Insights?

Amazon CloudWatch Logs Insights is a fully managed service that enables users to explore, query, and filter logs stored in CloudWatch.

It offers an intuitive query language similar to SQL and an interactive interface to simplify log analysis. Perfect for filtering larger logs when you’re debugging an application, troubleshooting an infrastructure issue, or performing security audits. Since our team started using CloudWatch Logs Insights we have been able to make the fault resolution process faster and more precise.

Why use Amazon CloudWatch Logs Insights?

Powerful Query Language

Logs Insights provides a robust, SQL-like query language. With commands like fields, filter, sort, and stats, users can slice and dice logs to uncover patterns, identify anomalies, or pinpoint specific events.

Seamless Integration

Logs Insights integrates seamlessly with other AWS services, including AWS Lambda, Amazon EC2, and Amazon ECS. This makes it easy to analyse logs across a wide range of AWS resources.

On-Demand Scalability

The service is designed to handle log volumes of any size, scaling automatically to meet demand.

Cost-Efficient

CloudWatch Logs Insights is pay-as-you-go. You only pay for the queries you run, based on the amount of data scanned.

Ready to give Logs Insights a go?

Step 1 - Enter the CloudWatch Dashboard

Using an AWS account with sufficient access privilege to access your CloudWatch services log into your AWS Console and search for CloudWatch and click to arrive at the CloudWatch Services console

Step 2 - Select Logs Insights

Under the Logs dropdown on the left hand pane, expand the Logs element and select Logs Insights

Step 3 - Choose your Log Group

Log groups in CloudWatch represent collections of log streams, typically from a single resource type e.g a Lambda function or an EC2 instance

Step 4 - Write and run your Query

In the query editor window, you can write queries to extract specific information from the logs, such as most recent entries, customer data, product id's, error codes etc.

In our example below we are writing a query and selecting the fields we are interested in (timestamp, message etc) and filtering the message field to show all logs with a string match of "level:ERROR" within the last 3 hours and limiting the response to 1000 records.

fields @timestamp, @message, @logStream, @log
| filter @message like /level":"ERROR"/
| sort @timestamp desc
| limit 1000

Once happy with your query, simply click the Run Query button and wait for the results.

Step 5 - Analyse Results

If your search query finds any matches, all results will be returned via a timestamped histogram graph and list of matching CloudWatch logs.

From here you can visualise patterns, such as when and how often errors occurred and quickly home in on the exact CloudWatch log to take further action.

Step 6 - Save and Share Queries

Once you have crafted the perfect query, why not save it for later or share it with your team! Once you have saved a query it is available for everyone with console access to use.

Some useful queries

List the most recent logs

fields @timestamp, @message, @logStream, @log
| sort @timestamp desc
| limit 20

Filter logs for a specific string

fields @timestamp, @message, @logStream, @log
| filter @message like /[email protected]/
| sort @timestamp desc
| limit 1000

Filter and count logs of matching strings (error levels etc)

fields @timestamp, @message, @logStream, @log
| filter @message like /ERROR/
| stats count() by level

Best Practices for CloudWatch Logs Insights

Structure Your Logs Use structured logging (e.g., JSON) for easier parsing and querying.

Use Filters Efficiently Reduce data scanned by filtering logs at the source level. This speeds up queries and reduces costs.

Monitor Costs Regularly monitor the amount of data scanned by queries to optimise usage and manage expenses.

Automate with APIs Use AWS SDKs or the CLI to automate log analysis and integrate Logs Insights into your CI/CD pipelines.

Combine with CloudWatch Alarms Use Logs Insights in conjunction with CloudWatch Alarms to proactively monitor for specific log patterns or thresholds.

Conclusion

Congratulations on getting started with Amazon CloudWatch Logs Insights. In this article we covered how to access Logs Insights within the console, create our first query and analyse the results.

I hope you and your teams can make use of Logs Insights effectively and start to enhance your monitoring capabilities, resolve issues faster, and make data-driven decisions to optimise your applications and infrastructure.

CloudWatch Logs Insights Official Documentation

CloudWatch Logs Insights Samples