• Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • Disclaimer & Policy

Elan Shudnow's Blog

MVP Logo
  • Azure
  • Exchange
  • Lync

Retrieving Activity Log Data from Azure Log Analytics – Part 1

March 5, 2020 by Elan Shudnow Leave a Comment

Overview

Ever wonder how to go about retrieving Activity Log Data from Azure Log Analytics? Perhaps you want to create a shared dashboard that shows you Virtual Machine creations from the Activity Log. Or perhaps instead of a shared dashboard you want to ingest this data into a custom PowerShell Object that can then be exported to a CSV, an HTML file, etc…? This article will explain how to do both.

Note: I previously blogged about ingesting Activity Log Data into Azure Log Analytics using Log Analytics Data Sources which you will see as Option #1 below. You can see that article here. This is a new article series as there is a new option to ingest Activity Log Data into Log Analytics. This new method provides some benefits but also modifies column behavior. This article series provides comprehensive insight in showing the new method and old/current method of Activity Log data ingestion into a Log Analytics Workspace and how to leverage Kusto Queries to retrieve information from both methods of Activity Log data ingestion.

This will be a multiple-part series.

Part 1

  • What is Log Analytics?
  • What is the Activity Log?
  • Two methods for ingesting Activity Log Data into Log Analytics
    • Option #1 – Old/Current Method Being Deprecated where you go into your Log Analytics Workspace and hook the Activity Log directly into the workspace
    • Option #2 – New Method leveraging Activity Log Diagnostic Settings

Part 2

  • Leverage Kusto Query Language (KQL) to build a custom shared dashboard for Virtual Machines that have been created

Part 3

  • Leverage the KQL Query we build within PowerShell to pull data into a variable which will then be exported to CSV

What is Log Analytics?

Log Analytics is comprised of one or more Log Analytics Workspaces. This workspace, or workspaces, allows data to be ingested from Platform Logs. This data can then be queried using log queries which use the Kusto Query Language (KQL). An Administrator can specify what Platform Logs are ingested into an Azure Log Analytics Workspace. For example, you can go onto the Diagnostic Settings for most resources and tell that resource to dump its logs into a specific Azure Log Analytics Workspace. You can then use KQL to retrieve those logs and even set up alerts based on conditions on the data that is retrieved from those KQL Queries.

Virtual Machines is a resource that can dump a wealth of information into a Log Analytics Workspace. Data that can be collected about a Virtual Machine includes, but not limited to:

  • Virtual Machine Writes (e.g. creations) via Activity Log ingestion into a Log Analytics Workspace
  • Event Log Data by connecting your Virtual Machine into a Log Analytics Workspace which installs the Microsoft Monitoring Agent which is referred to in documentation as the Log Analytics Agent/Extension.
  • Performance Counters by connecting your Virtual Machine into a Log Analytics Workspace which installs the Microsoft Monitoring Agent which is referred to in documentation as the Log Analytics Agent/Extension.

Going deep into Log Analytics is outside of the scope of this article. I wanted to provide a rough overview of Log Analytics and how Virtual Machine Operations and Logs are ingested.

What is in scope for this article is the first bullet – Virtual Machine Writes (e.g. creations) via Activity Log ingestion into a Log Analytics Workspace. Therefore, let’s look at taking an existing Log Analytics Workspace and ingesting Activity Log Data into it. Then we’ll talk about creating KQL Queries for discovering Virtual Machines that were created.

Something to note is that by default, a Log Analytics Workspace will provide up to 31 days of data retention included in a Per GB pricing plan. The default retention setting is set to 30 days. All Activity Log Data is kept for 90 days. When ingesting Activity Log Data into Log Analytics, even if your Log Analytics Workspace is set to 30 days, it will keep the 90 days of Activity Log Data for free. However, if your Log Analytics Workspace is 180 days, your Activity Log Data will be kept for 180 days. More on configuring Log Analytics Data Retention is provided here.

What is the Activity Log?

The Azure Activity Log provides insight into the operations on each Azure resource in the subscription from the outside (the management plane) in addition to updates on Service Health events. Use the Activity Log, to determine the what, who, and when for any write operations (PUT, POST, DELETE) taken on the resources in your subscription. You can also understand the status of the operation and other relevant properties. There is a single Activity log for each Azure subscription.

Ingesting Activity Log Data into Log Analytics

There are two ways to ingest Activity Log data into Log Analytics:

  • Option #1 (Old/Current Method Being Deprecated) – Go directly into the Log Analytics Workspace and hook the Activity Log into your Log Analytics Workspace. The Activity Log data using this method will be ingested into a Log Analytics Workspace in about 15 minutes and will store all Activity Log Data for the subscription selected.
  • Option #2 (New Preferred Method) – Go into the Activity Log’s Diagnostic Setting and specify what pieces of data from the Activity Log should be stored into the Log Analytics Workspace. The Activity Log data using this method will be ingested into a Log Analytics Workspace in about 1 minute and provides flexibility into what specific Azure Activity Log data you want to ingest.

Microsoft provides documentation here for discussing the new method (Option #2) here and advantages and considerations for leveraging this new option. For completeness, we will look at both options.

Something of importance to note, is pay attention to that Azure Doc as it mentions that some of the columns change. One of the things I discover is that the Resource column changes as well which is not documented. With the new capability, Option #2, Resource column is completely missing. When the KQL Query is built in Part 2, we take a look at pulling data from both Option #1 and Option #2, for those of you that use Option #1 and are moving to Option #2, and build our KQL to return results from both data sets.

Option #1 (Deprecated)

In the Azure Portal, go to All Services, and click on Log Analytics Workspaces.

Navigate to your existing Log Analytics Workspace

We see we have an existing Log Analytics Workspace named ShudLogAnalytics. Let’s go into this workspace.

Enabling Activity Log data ingesting into a Log Analytics Workspace using the old/current method

Click on the line where it shows your Subscription and the Log Analytics Connection being Not connected.

Connecting Activity Log into your Log Analytics Workspace if not already done using the old/current method.

Simply click Connect and your Activity Log data will start being ingested into your Log Analytics Workspace. The data will take approximately 15 minutes to begin being ingested.

Option #2 (Preferred)

In the Azure Portal, go to All Services, and click on Activity Log.

Enabling Activity Log data ingesting into a Log Analytics Workspace using the new preferred method

Click on Diagnostic settings.

Enabling Activity Log data ingesting into a Log Analytics Workspace using the new preferred method

Click on Add diagnostic setting.

Enabling Activity Log data ingesting into a Log Analytics Workspace using the new preferred method

Select the log options you want to collect from the Activity Log. Click Send to Log Analytics and select your Subscription and the Log Analytics workspace you want to leverage.

In Part 2, we’ll take a look at using KQL to pull Activity Log Data from our Log Analytics Workspace. We’ll then use the KQL queries to build custom shared dashboards.

Share this:

  • Twitter
  • LinkedIn
  • Reddit

Filed Under: Azure Tagged With: Azure, Log Analytics, Virtual Machines

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • YouTube

More to See

Azure Runbooks Connecting to Exchange Online and Microsoft Graph

July 22, 2022 By Elan Shudnow

Using Python 3.8.0 Azure Runbooks with Python Packages

July 11, 2022 By Elan Shudnow

Preserving UNC Path after Azure Files Migration using DFS-N

April 10, 2022 By Elan Shudnow

Pin Azure VM Availability Sets into an Availability Zone

April 9, 2022 By Elan Shudnow

Tags

ACR Always Encrypted Ansible Automation Availability Sets Availability Zones Azure Azure Active Directory Azure Application Gateway Azure Files Azure Firewall Azure Key Vault Azure Load Balancer Azure Migrate Azure Monitor Azure Web App Backup Exec CDN Cluster DevOps DFS Docker DPM Event Grid Exchange Exchange 2010 Function App ISA iSCSI Log Analytics Logic App Lync Microsoft Graph OCS Office Personal PowerShell Proximity Placement Groups Runbook SCOM SQL Storage Accounts Virtual Machines Windows Server 2008 Windows Server 2008 R2

Footer

About Me

Microsoft Cloud Solution Architect focused on Azure IaaS, PaaS, DevOps, Ansible, Terraform, ARM and PowerShell.

Previously a 6x Microsoft MVP in Exchange Server and Lync Server.

My hobbies include watching sports (Baseball, Football and Hockey) as well as Aviation.

Recent

  • Pre-creating Azure AD App for Azure Migrate
  • Azure Runbooks Connecting to Exchange Online and Microsoft Graph
  • Using Python 3.8.0 Azure Runbooks with Python Packages
  • Preserving UNC Path after Azure Files Migration using DFS-N
  • Pin Azure VM Availability Sets into an Availability Zone

Search

Tags

ACR Always Encrypted Ansible Automation Availability Sets Availability Zones Azure Azure Active Directory Azure Application Gateway Azure Files Azure Firewall Azure Key Vault Azure Load Balancer Azure Migrate Azure Monitor Azure Web App Backup Exec CDN Cluster DevOps DFS Docker DPM Event Grid Exchange Exchange 2010 Function App ISA iSCSI Log Analytics Logic App Lync Microsoft Graph OCS Office Personal PowerShell Proximity Placement Groups Runbook SCOM SQL Storage Accounts Virtual Machines Windows Server 2008 Windows Server 2008 R2

Copyright © 2023 · Magazine Pro on Genesis Framework · WordPress · Log in