Making the Most of CloudWatch Log Insights: 7 Best Practices

What is CloudWatch Logs Insights?

Amazon CloudWatch provides Log Insights, a feature that can help you:

• Search and analyze log data located in Amazon CloudWatch Logs
• Perform queries required for operational issue response
• Identify root causes and validate fixes

CloudWatch Log Insights uses a proprietary query language with several basic commands. It provides sample queries for common AWS service log types, as well as query autocompletion. Learn more about CloudWatch Log Insights capabilities and how to use them.

Users can also leverage log field discovery, which automatically locates fields in JSON-based AWS service logs, including Amazon Route 53, AWS Lambda, AWS CloudTrail, and Amazon VPC. 

How Amazon CloudWatch Works

Amazon CloudWatch is a metrics repository that collects data from all Amazon services. Infrastructure as a Service (IaaS) offerings like Amazon EC2 or S3 place metrics in the repository, and retrieve statistics based on these metrics. You can also place custom metrics in the repository and retrieve statistics.

In addition, CloudWatch provides:

• CloudWatch Console—lets you visualize metrics and statistics in dashboards
• Alarms—you can set up alarm actions to stop, start, or terminate Amazon EC2 instances when certain conditions are met
• Integration with Auto Scaling and SNS—CloudWatch alarms can trigger Amazon EC2 Auto Scaling events and notifications by Amazon Simple Notification Service (SNS)

The AWS global infrastructure is divided into several regions. CloudWatch metrics are stored in each Amazon region separately, but you can use CloudWatch's cross-region feature to summarize statistics from multiple regions.

CloudWatch Log Insights Best Practices

Visualizing Log Data in Graphs

CloudWatch Log Insights generates bar charts, line charts, and stacked area charts using the stat function and aggregation functions.

Visualizing time series data

Time series visualizations are possible for queries that contain one or more aggregation functions and use the bin() function to group data by a single field.

Visualizing log data grouped by fields

Bar charts can be generated by running a query using the Visualization tab. Clicking on the arrow next to Line reveals the Bar option. Charts are limited to 100 bars.

Filter and Pattern Syntax

You can use metric filters to find and organize terms, phrases, or values in log events. For example, you can create a metric filter to detect and count occurrences of the word WARNING in log events for a specific Amazon service.

Metric filters can be used in several ways:

• Counter—when the "metric" filter detects a matching term, phrase, or value in a log event, it can increment the metric.
• Numeric value—metric filters can extract numbers from log events, such as the latency of a web request. In this example, the indicator value can reflect the actual number extracted from the log.
• Conditional operators—you can also use condition operators (such as AND, OR) and wildcards for more precise matches.

Before creating a metric filter, you can test your search pattern in the CloudWatch console.

Saving and Re-running CloudWatch Logs Insights Queries

CloudWatch Logs Insights provides a query language, allowing you to perform structured queries on log data. After creating a query, you can save it to run it again later. This can save time and help you build a library of routine analysis patterns. Saved queries are stored in a folder structure to keep them organized. Each account can store up to 1000 CloudWatch Logs Insights queries per region.

Note that to save queries or view saved queries, you must have the appropriate permissions in your Identity and Access Management (IAM) role.

View Currently Running and Recent Queries

You can view the query currently running, and the history of recent queries. It can be useful to see which analyses your colleagues recently performed, and reuse them instead of running new queries. Also, each account can run up to 10 CloudWatch Logs Insights queries simultaneously, including queries added to the dashboard.

So it is important to see how many queries are available before running new ones. To view current and historical queries, open the CloudWatch Console, select Insights from the navigation pane, then select History.

Real-time Processing of Log Data with Subscriptions

Subscriptions provide access to a real-time feed of CloudWatch Log events. This feed can be sent to other Amazon streams, such as AWS Lambda, Amazon Kinesis, or Amazon Kinesis Data Firehose for custom processing and analysis. It can also be delivered to other systems, encoded in Base64, and compressed as a gzip file or stream.

Before subscribing to log events, you must create a receiving resource to which events will be delivered—for example, a Kinesis stream. Events and their destination can be filtered using up to two subscription filters. Cloudwatch Logs also generates metrics describing the forwarding of events to subscriptions.

Analyze Patterns with Contributor Insights

Contributor Insights identifies common system behavior patterns by analyzing log events. It also identifies top influencers, such as specific host IPs with higher than normal calls to an application. By specifying particular values for specific fields, users can filter log entries for further analysis.

They can then compare them to other CloudWatch metrics and add the results to a CloudWatch dashboard, for further troubleshooting and optimization. To do this, select one or more CloudWatch Log groups, then specify the fields to be examined, and select filtering conditions. You can also set conditions for taking specific actions, such as triggering an alarm.

Check End-User Performance with Synthetic Tests

CloudWatch Synthetics enables the configuration of canaries for triggering and monitoring common interactions on a web application by evaluating data and status codes returned by APIs. If needed, these canaries can also trigger a CloudWatch alarm based on response.

Synthetic logs let you evaluate a realistic end-user experience. They can replicate real-life interactions and evaluate responses, using flexible Node.js scripts.

Recommended reading:Getting started with AWS CloudWatch 

Conclusion

In this article, we've covered the basics of Amazon CloudWatch Log Insights, a tool that can help you search and analyze logs from across your AWS deployment. In addition, we’ve also covered seven best practices that can help you do more with CloudWatch Log Insights:

1. Visualizing log data in graphs to identify patterns and trends in system logs
2. Use filters and pattern syntax to search through large volumes of log data
3. Save and re-run common queries to save time and create a library of analysis patterns
4. View currently running and recent queries to manage concurrent queries
5. Use subscriptions to process log data in real-time
6. Use Contributor Insights to identify common system behavior
7. Use synthetic tests to measure performance from an end user's performance

We hope this will be helpful as you deepen your use of Amazon CloudWatch and learn to extract more valuable insights from your cloud log data.