This topic is to address some incorrect information out there regarding leveraging Azure Private Endpoints and Storage Account Firewalls. The recommendations I have seen are to create a Private Endpoint on an Azure Storage Account and restrict subnet access to a specific subnet.
Let’s walk through an example of an incorrect statement I have seen out there. Then I’ll walk through how the Storage Account Firewall works with and without a Private Endpoint and walk-through various scenarios so you gain a complete understanding.
An example I have seen is when doing Azure Virtual Desktop. Azure Virtual Desktop, depending on the Architecture, may call for the use of storing User Profiles in an Azure Storage Account. For example, using FSLogix to store user profiles. In AVD, the VDI Servers that a user accesses are known as a Session Host. I typically architect AVD so that Session Hosts are deployed in their own Subnet. The recommendation I have seen out there is to secure the Azure Storage Account by deploying a Private Endpoint for Azure Files and then restricting access to only the AVD Subnet.
Seems simple and straightforward, right? Well, it’s actually not that straightforward and doesn’t quite work that way. It works quite differently than Service Endpoints. Therefore I am writing this blog post to help you understand the difference between leveraging Service Endpoints and Private Endpoints in regards to Storage Account Firewalls.
Service Endpoints and the Azure Storage Account Firewall
Most Azure PaaS Services work based on having a Public Endpoint with no Private Connectivity. I say most because not all PaaS Services have a Public Endpoint such as Azure NetApp Files. An Azure Storage Account, by default, only has a Public Endpoint meaning that it is accessible only via a Public IP Address out of the box. What this means, any connectivity from on-premises (if not using an ExpressRoute Microsoft Peering connection configured and also configured for Storage Account BGP Communities) or from an Azure Virtual Network will route out to the internet and to the Azure Storage Account.
Service Endpoints are a feature that are enabled on a given subnet within an Azure Virtual Network. Service Endpoints create a custom route on your subnet that directs Virtual Machines to route to the corresponding PaaS Service Public IP Range within the Azure Backbone directly to the PaaS Service. This traffic never goes out to the internet. What this also allows is the Storage Account to see the Virtual Machine’s Private IP Address rather than the Virtual Machine’s Public IP Address.
In this example, I have a Storage Account configured with all default options (for example, no Private Endpoint).
Go into the Storage Account > Networking, choose Selected Networks. Then click Add existing virtual network.
When we are prompted to add our subnets, we see in parenthesis that a Service endpoint is required. This is because in order for the Storage Account Firewall to allow specific Virtual Networks, we must enable the Service Endpoint on that subnet so any Virtual Machine’s connecting from that subnet will leverage the Azure Backbone for the Virtual Machine to Storage Account connection so the Storage Account can see the Private IP Address of the Virtual Machine and allow or restrict traffic.
I went ahead and selected subnet1 and clicked Enable. This created the Service Endpoint for us.
We can verify this by looking at the Selected Networks. Our subnet is now defined as a Selected Network in the Storage Account Firewall.
If we go to our Virtual Network, open subnet1, we can see the Service Endpoint has been enabled for Microsoft.Storage. Therefore, anytime a Virtual Machine in this subnet needs to route to a Storage Account, they will stay on the Azure Backbone.
There is a limitation with Service Endpoints in general which, by default, only allow Service Endpoints to work with the corresponding PaaS Service only in the same region in which the Virtual Network exists. However, work is being down to allow Service Endpoints to also work inter-region. For example, at the time of this writing, there is a Public Preview to enable this functionality for Storage Accounts: Configure Azure Storage firewalls and virtual networks | Microsoft Docs
By default, service endpoints work between virtual networks and service instances in the same Azure region. When using service endpoints with Azure Storage, service endpoints also work between virtual networks and service instances in a paired region. If you want to use a service endpoint to grant access to virtual networks in other regions, you must register the
AllowGlobalTagsForStoragefeature. This capability is currently in public preview.
As we have enabled Microsoft.Storage Service Endpoint on subnet1, when a Virtual Machine on this subnet connects to the Storage Account, that Storage Account will see the Private IP of the Azure VM. Because our Azure Storage Account Firewall permits Subnet 1, the traffic will be allowed.
I have two Virtual Machines created:
- subnet1VM
- subnet2VM
Let’s look at subnet1VM Effective Routes on its’ Network Interface Card. We will see we have two VirtualNetworkServiceEndpoint routes. Now anytime a Virtual Machine in subnet1 for which we have enabled Service Endpoint for Microsoft.Storage attempts to connect to a Storage Account, it routes to the VirtualNetworkServiceEndpoint next hop via the Azure Backbone rather than routing out to the internet.
subnet2VM does not have a Service Endpoint enabled on it for Microsoft.Storage and it is not allowed through the Azure Storage Account Firewall and therefore, access will be denied to any Virtual Machine on subnet 2.
Let’s test this out. I created a File Share on this Storage Account and configured the Storage Account Firewall to allow access to my home machine’s Public IP so I can interact with the File Share. I uploaded two files to this file share, file1.txt and file2.txt.
Click Connect and obtain the PowerShell Script for mounting the File Share. We’ll hop on both subnet1VM and subnet2VM and try mounting the File Share.
As we can see, the File Share mounts on subnet1VM
If I try the same on subnet2VM, I get Access Denied.
Private Endpoints and the Azure Storage Account Firewall
A lack of capability of Service Endpoints is that they only allowed Private Connectivity from Azure Virtual Networks to PaaS Services. You could not leverage Service Endpoints for On-Premises to PaaS private connectivity. What if you didn’t have ExpressRoute Microsoft Peering and had ExpressRoute Private Peering and/or a Site-to-Site VPN Tunnel into your Azure Virtual Network. Private Endpoints allow you to have a Network Interface Card representing a PaaS Service on your Virtual Network. When an on-premises VM or an Azure VM needs to access a PaaS Service such as an Azure Storage Account, you can leverage the Azure Private Endpoint as the Connection Endpoint which then connects to the PaaS Service. It acts as a Reverse Proxy in a sense.
More on how Private Endpoints work in Microsoft Docs: What is a private endpoint? | Microsoft Docs
I’m not going to go deep into how DNS needs to be configured with Private Endpoints as that is covered in depth in the Microsoft Docs as well as John Savill’s video’s on Private Endpoints: (1) Microsoft Azure Private Link Deep Dive – YouTube and (1) Private Endpoint DNS Exploration – YouTube. Please watch those videos prior to reading on if you’re not familiar with Private Endpoints and DNS configurations.
As mentioned at the beginning of this blog post, the misconception out there is that you get the same flexibility that you have with Service Endpoints with Storage Account Firewalls. And that simply is not true.
The jist of it is as follows: Any subnets in a VNET linked to the Private DNS zone or a DNS Server that has the Private Endpoint CNAME added will bypass the Storage Account Firewall. When leveraging a Private Endpoint, if you’re in a VNET that is linked to the Private DNS Zone that houses the DNS Configuration for your Storage Account’s Private Endpoint, the Storage Account Firewall does not honor the VM’s Private IP Address because the Private Endpoint itself is enabled on the Storage Account and the connection is coming from the Private Endpoint.
To demonstrate this, let’s enable a Private Endpoint on the same Storage Account we set up with Service Endpoints. We’ll hop back on subnet2VM which previously failed, verify we see the Private Endpoint IP come back when doing an NSLookup for the Storage Account FQDN, and we’ll see the mount successfully works even though subnet2 is not allowed on the Storage Account Firewall.
When creating the private endpoint, be sure to choose Target sub-resource as we’re testing the mounting of File Shares.
We’ll put the Private Endpoint in our virtual network and choose to integrate it with a Private DNS zone. I already have this zone created from previous Private Endpoint for integrations with Azure Files. If you do not see the same option, you can elect to create the Private DNS zone. If you choose no, you will instead leverage your own DNS Servers for the Private Endpoint DNS records. If you choose to leverage the Private DNS zones, you will want to understand Private Endpoint DNS: Azure Private Endpoint DNS configuration | Microsoft Docs as well as John Savill’s Private Endpoint videos linked above. Again, I am not here to teach how Private Endpoints work and their DNS gets configured, but rather to demonstrate a very specific scenario.
In the Virtual Network tab when creating our Private Endpoint, I first created a third subnet specifically to contain the private endpoint rather than storing it in the subnet1 or subnet2 subnets that contain the virtual machines (subnet1vm and subnet2vm).
Our Private Endpoint was successfully created.
The privatelink.file.core.windows.net private DNS zone also has a record for our new private endpoint.
Let’s go ahead and link our Private DNS zone to our VNET that was created as part of this demonstration. That way, our VMs on the new VNET will be able to see the new private endpoint and leverage it rather than going out through the internet to reach the Storage Account via its Public IP Address; for which we would need to allow through our Storage Account Firewall.
Let’s hop on our subnetvm2 that previously received an Access Denied and do an nslookup for our Storage Account (after doing an ipconfig /flushdns). We successfully verify that doing a DNS Query on our Storage Account for a VM that is linked to the private dns zone for privatelink.file.core.windows.net successfully returns the private endpoint private IP address.
Now let’s try running the script again to mount the Azure File Share on subnetvm2 which previously failed. The only difference now is that we are connecting through the Private Endpoint. No changes have been made at all to the Storage Account Firewall. It’s still configured to only allow connectivity from Subnet1, not Subnet2.
PaaS Public Endpoints when using Private Endpoints
There is another common misconception out there that when you enable Private Endpoints for a PaaS Service, that the Public Endpoint goes away. This is true for some PaaS Services but it works differently for each PaaS Service. For Storage Accounts, the Public Endpoint is not disabled. But after you create the Private Endpoint for your Storage Account, the Storage Account Firewall will begin blocking access to the Public Endpoint. You do this by adding in a Private IP Address Range as depicted in the following screenshot:
Most PaaS Services do not block the Public Endpoint after enabling the Private Endpoint. PaaS Services that have a Firewall built into them will deny Public Endpoint access after enabling Private Endpoint and will require you to go into the PaaS Firewall as displayed in the above screenshot and add IP Address Ranges to allow access to the Public Endpoint. A couple examples of this are Azure Storage Accounts and Azure Web Apps.
Conclusion
And there you have it. Any VMs connecting through a Private Endpoint will bypass the Azure Storage Account Firewall. Administrators that want more control over this will be able to leverage Public preview of Private Link Network Security Group Support | Azure updates | Microsoft Azure and/or Public preview of Private Link UDR Support | Azure updates | Microsoft Azure to control access. NSGs will allow you to create 5-Tuple (source IP, source port, protocol, destination IP, destination port) denies or allows to Private Endpoints. UDRs will allow you to direct traffic destined to a Private Endpoint to first flow through a Network Virtual Appliance (NVA) to restrict traffic.
Leave a Reply