Retrieving Activity Log Data from Azure Log Analytics – Part 1

Overview

Ever wonder how to go about retrieving Activity Log Data from Azure Log Analytics? Perhaps you want to create a shared dashboard that shows you Virtual Machine creations from the Activity Log. Or perhaps instead of a shared dashboard you want to ingest this data into a custom PowerShell Object that can then be exported to a CSV, an HTML file, etc…? This article will explain how to do both.

Note: I previously blogged about ingesting Activity Log Data into Azure Log Analytics using Log Analytics Data Sources which you will see as Option #1 below. You can see that article here. This is a new article series as there is a new option to ingest Activity Log Data into Log Analytics. This new method provides some benefits but also modifies column behavior. This article series provides comprehensive insight in showing the new method and old/current method of Activity Log data ingestion into a Log Analytics Workspace and how to leverage Kusto Queries to retrieve information from both methods of Activity Log data ingestion.

This will be a multiple-part series.

Part 1

• What is Log Analytics?
• What is the Activity Log?
• Two methods for ingesting Activity Log Data into Log Analytics
  • Option #1 – Old/Current Method Being Deprecated where you go into your Log Analytics Workspace and hook the Activity Log directly into the workspace
  • Option #2 – New Method leveraging Activity Log Diagnostic Settings

Part 2

• Leverage Kusto Query Language (KQL) to build a custom shared dashboard for Virtual Machines that have been created

Part 3

• Leverage the KQL Query we build within PowerShell to pull data into a variable which will then be exported to CSV

What is Log Analytics?

Log Analytics is comprised of one or more Log Analytics Workspaces. This workspace, or workspaces, allows data to be ingested from Platform Logs. This data can then be queried using log queries which use the Kusto Query Language (KQL). An Administrator can specify what Platform Logs are ingested into an Azure Log Analytics Workspace. For example, you can go onto the Diagnostic Settings for most resources and tell that resource to dump its logs into a specific Azure Log Analytics Workspace. You can then use KQL to retrieve those logs and even set up alerts based on conditions on the data that is retrieved from those KQL Queries.

Virtual Machines is a resource that can dump a wealth of information into a Log Analytics Workspace. Data that can be collected about a Virtual Machine includes, but not limited to:

• Virtual Machine Writes (e.g. creations) via Activity Log ingestion into a Log Analytics Workspace
• Event Log Data by connecting your Virtual Machine into a Log Analytics Workspace which installs the Microsoft Monitoring Agent which is referred to in documentation as the Log Analytics Agent/Extension.
• Performance Counters by connecting your Virtual Machine into a Log Analytics Workspace which installs the Microsoft Monitoring Agent which is referred to in documentation as the Log Analytics Agent/Extension.

Going deep into Log Analytics is outside of the scope of this article. I wanted to provide a rough overview of Log Analytics and how Virtual Machine Operations and Logs are ingested.

What is in scope for this article is the first bullet – Virtual Machine Writes (e.g. creations) via Activity Log ingestion into a Log Analytics Workspace. Therefore, let’s look at taking an existing Log Analytics Workspace and ingesting Activity Log Data into it. Then we’ll talk about creating KQL Queries for discovering Virtual Machines that were created.

Something to note is that by default, a Log Analytics Workspace will provide up to 31 days of data retention included in a Per GB pricing plan. The default retention setting is set to 30 days. All Activity Log Data is kept for 90 days. When ingesting Activity Log Data into Log Analytics, even if your Log Analytics Workspace is set to 30 days, it will keep the 90 days of Activity Log Data for free. However, if your Log Analytics Workspace is 180 days, your Activity Log Data will be kept for 180 days. More on configuring Log Analytics Data Retention is provided here.

What is the Activity Log?

The Azure Activity Log provides insight into the operations on each Azure resource in the subscription from the outside (the management plane) in addition to updates on Service Health events. Use the Activity Log, to determine the what, who, and when for any write operations (PUT, POST, DELETE) taken on the resources in your subscription. You can also understand the status of the operation and other relevant properties. There is a single Activity log for each Azure subscription.

Ingesting Activity Log Data into Log Analytics

There are two ways to ingest Activity Log data into Log Analytics:

• Option #1 (Old/Current Method Being Deprecated) – Go directly into the Log Analytics Workspace and hook the Activity Log into your Log Analytics Workspace. The Activity Log data using this method will be ingested into a Log Analytics Workspace in about 15 minutes and will store all Activity Log Data for the subscription selected.
• Option #2 (New Preferred Method) – Go into the Activity Log’s Diagnostic Setting and specify what pieces of data from the Activity Log should be stored into the Log Analytics Workspace. The Activity Log data using this method will be ingested into a Log Analytics Workspace in about 1 minute and provides flexibility into what specific Azure Activity Log data you want to ingest.

Microsoft provides documentation here for discussing the new method (Option #2) here and advantages and considerations for leveraging this new option. For completeness, we will look at both options.

Something of importance to note, is pay attention to that Azure Doc as it mentions that some of the columns change. One of the things I discover is that the Resource column changes as well which is not documented. With the new capability, Option #2, Resource column is completely missing. When the KQL Query is built in Part 2, we take a look at pulling data from both Option #1 and Option #2, for those of you that use Option #1 and are moving to Option #2, and build our KQL to return results from both data sets.

Option #1 (Deprecated)

In the Azure Portal, go to All Services, and click on Log Analytics Workspaces.

We see we have an existing Log Analytics Workspace named ShudLogAnalytics. Let’s go into this workspace.

Click on the line where it shows your Subscription and the Log Analytics Connection being Not connected.

Simply click Connect and your Activity Log data will start being ingested into your Log Analytics Workspace. The data will take approximately 15 minutes to begin being ingested.

Option #2 (Preferred)

In the Azure Portal, go to All Services, and click on Activity Log.

Click on Diagnostic settings.

Click on Add diagnostic setting.

Select the log options you want to collect from the Activity Log. Click Send to Log Analytics and select your Subscription and the Log Analytics workspace you want to leverage.

In Part 2, we’ll take a look at using KQL to pull Activity Log Data from our Log Analytics Workspace. We’ll then use the KQL queries to build custom shared dashboards.