Increasing Azure Log Analytics Retention Per Data Type

On October 8th, 2019, Microsoft announced the support of increasing retention independently per data type. What this means, is that if you want to collect 6 months worth of log data for the SecurityEvent Log, you can do so independently of the retention period for other data types. The official documentation on that is here: https://docs.microsoft.com/en-us/azure/azure-monitor/platform/manage-cost-storage#retention-by-data-type. At the time of writing this document, increasing the retention per data type requires leveraging the Rest API to make these changes.

Note: The Usage and AzureActivity data types cannot be set with custom retention.  They are retained for 90 days by default, and there is no charge for for this 90 day retention. These data types are also free from data ingestion charges. They will take on the maximum of the default workspace retention or 90 days.

Let’s have a look and how this works, first by looking at how increasing retention works by utilizing the portal using the old fashioned way and then how we can configure this per data type leveraging the Rest API.

While logged into the Azure Portal, go into your Log Analytics Workspace. In order to configure your Data Retention, you will want to navigate to “Usage and estimated costs” and then “Data Retention.”

This will bring up the following selection:

As can be seen, you can increase the slider. Anything above 31 days or lower are included with your pricing plan. So let’s say we want to capture 6 months worth of SecurityEvent Log data but everything else we want to leave Data Retention at 30 days. Prior to October 8th, 2019, this was impossible. If it was imperative that we collect SecurityEvent Log data for 6 months, we would have two options:

• Increase the Data Retention to 6 months and have all data in that Log Analytics Workspace be retained for 6 months. The downside to this is the increased costs for all data retaining it for 6 months.
• Create another Log Analytics Workspace just for Azure SecurityEvent Log Data and set the Data Retention for 6 months while your other Log Analytics Workspace that collects data for all other data types remains at 30 days.

But since this new feature was introduced on October 8th, 2019, there is no longer a need to be limited to the above two options.

Installing the software necessary to make the Rest API calls

We will use a tool called ARMclient to make the Rest API calls necessary to increase the Azure SecurityEvent Logs for 6 months while leaving all other data types at the default retention. We will install ARMclient using Chocolatey. You can install Chocolatey by following the instructions available here.

To install ARMClient, you can run the following command:

choco install armclient --source=https://chocolatey.org/api/v2/  

After installation, you can verify the installation was successful by typing in ARMClient into the Command Prompt or PowerShell Prompt.

Leveraging ARMClient to configure retention per data type

Let’s log in to our Azure Subscription within PowerShell using the following command (this command will bring up a Modern Authentication form asking you to enter your Organization ID Username and Password):

ARMClient Login

I do like using PowerShell for ARMClient as I like to set variables for certain values such as Subscription that gets passed in the Rest API URL. You will see this in action in a little bit.

Note: It is important to note that you may want to use a Service Principal, a specific Azure AD Tenant, or a specific Azure environment when logging in with ARMClient. All these login parameters are documented here. To use a specific subscription, it is only required to connect to the Azure AD Tenant the subscription belongs to. You specify the subscription on the Rest API URL you pass.

We will need to collect a few pieces of information:

• Subscription ID
• Resource Group containing our Log Analytics Workspace
• Name of the Log Analytics Workspace

To get the Subscription ID, in the Azure Portal, navigate to your Subscriptions and copy down the Subscription ID.

In preparation of running the Rest API Commands, I have the three variables configured and ready to store in PowerShell:

The Rest API command to look at the existing retention by datatype in a Log Analytics Workspace is:

GET /subscriptions/00000000-0000-0000-0000-00000000000/resourceGroups/MyResourceGroupName/providers/Microsoft.OperationalInsights/workspaces/MyWorkspaceName/Tables/SecurityEvent?api-version=2017-04-26-preview

Because we have a variable for Subscription, the Log Analytics Workspace Name, and the Resource Group, we will update the Rest API command as such:

GET /subscriptions/$Subscription/resourceGroups/$ResourceGroup/providers/Microsoft.OperationalInsights/workspaces/$LogAnalyticsName/Tables/SecurityEvent?api-version=2017-04-26-preview

Let’s set those variables and use ARMClient to call the Rest API command. I have already previously increased retentionInDays for Security event to 90. Therefore, you will see the retentionInDays property being set to 90.

Our goal now is to change the SecurityEvent RetentionInDays to 6 months (180 days). We do this by running the following Rest API command:

PUT /subscriptions/00000000-0000-0000-0000-00000000000/resourceGroups/MyResourceGroupName/providers/Microsoft.OperationalInsights/workspaces/MyWorkspaceName/Tables/SecurityEvent?api-version=2017-04-26-preview "{properties: {retentionInDays: 180}}"

But because we have our three variables defined, we will update the Rest API command as such:

PUT /subscriptions/$Subscription/resourceGroups/$ResourceGroup/providers/Microsoft.OperationalInsights/workspaces/$LogAnalyticsName/Tables/SecurityEvent?api-version=2017-04-26-preview "{properties: {retentionInDays: 180}}"

Note: Be sure to update the Table used to SecurityEvent. I have highlighted it in the command above in bold.

Now let’s go ahead and run the Rest API Command using ARMClient.

Now the data collected for SecurityEvent within your Log Analytics Workspace will be retained for 180 days while everything that does not have a specified retentionInDays property will be retained for the 30 days we have configured in our portal.