OnCallReady

Lesson 23.10 · Azure II: Networking & AKS · 16 min read

Private endpoints and the DNS that makes them work

In plain words

Imagine a big department store on the high street. Normally you go through the main entrance. The store can also build a private door from its warehouse straight into your office building, so you never walk on the street. But your office's reception desk has to know about that door: when someone asks "how do I get to the store?", reception must point them to the private door, not the high street. If reception doesn't know, people walk out to the main entrance, and when it's closed, they're turned away.

A private endpoint is that private door: a NIC with a private IP like 10.20.3.4 in your subnet for one PaaS resource. Reception is DNS: a private zone like privatelink.blob.core.windows.net, linked to your VNet, must answer the normal name with the private IP.

Your Key Vault and storage account (Ch 22) have public addresses: anyone on the internet can reach them and is stopped only by authentication. Security wants them reachable only from your VNet. A private endpoint gives the service an address inside your subnet so you can switch the public one off - and then DNS decides whether anything still works.

What you need to know already: 23.1 (subnets, NICs), 8.16-8.22 (DNS: A and CNAME records, resolvers, split horizon), 22.23 (Key Vault), 22.29 (storage accounts and blob), 9.15 (TLS and certificate names).

What a private endpoint is

A private endpoint (PE) is a NIC in your subnet with a private IP that maps to one specific PaaS resource (platform as a service: a service Azure runs for you, like one vault or one storage account's blob service). Traffic to that IP stays on the Microsoft backbone (Microsoft's own network between its data centres) and never touches the resource's public endpoint - which you can then switch off. The feature behind it is called Private Link.

az network private-endpoint show ... --query below prints the endpoint's IP, the name it serves, and whether the connection was approved by the resource.

# pe-blob and its DNS zone come from the next mission
az network private-endpoint show -g rg-oncall-lab -n pe-blob --query "{ip:customDnsConfigs[0].ipAddresses[0], fqdn:customDnsConfigs[0].fqdn, state:privateLinkServiceConnections[0].privateLinkServiceConnectionState.status}"
{
  "fqdn": "stsysoplab.blob.core.windows.net",
  "ip": "10.20.3.4",
  "state": "Approved"
}

The part everyone gets wrong: DNS

The application still connects to stsysoplab.blob.core.windows.net (TLS certificates and SDKs require the real name). So that name has to resolve to 10.20.3.4 from inside your VNet, and to the public IP from everywhere else. This is split horizon DNS (8.22): the same name, different answers inside and outside. Azure does it with a CNAME chain:

stsysoplab.blob.core.windows.net
   CNAME  stsysoplab.privatelink.blob.core.windows.net      (added when a PE exists)
   CNAME  blob.ams07prdstr02a.store.core.windows.net        (public answer)

A private DNS zone is a DNS zone (a set of records for one domain) that only the VNets you link to it can see. If the VNet can see a private DNS zone called privatelink.blob.core.windows.net with an A record stsysoplab -> 10.20.3.4, the chain stops there with the private IP. If it cannot, resolution continues to the public IP - and the connection either goes over the internet or, once public access is disabled, fails with a 403 that looks like an auth problem.

The zone names are fixed per service:

privatelink.blob.core.windows.net      blob
privatelink.file.core.windows.net      file
privatelink.vaultcore.azure.net        Key Vault
privatelink.azurecr.io                 Container Registry (ACR, Azure's image registry, Ch 10)
privatelink.westeurope.azmk8s.io       AKS private cluster API (23.20)
privatelink.database.windows.net       Azure SQL

Three pieces, all required

  1. the zone - az network private-dns zone create -n <zone name>;
  2. a VNet link - az network private-dns link vnet create: lets one VNet see the zone (--registration-enabled false: do not auto-add VM names);
  3. the A record - best made by a zone group (az network private-endpoint dns-zone-group create), which ties the record to the endpoint.
# 1. the zone (usually once, in a central "hub" subscription)
az network private-dns zone create -g rg-oncall-lab -n privatelink.blob.core.windows.net

# 2. link it to every VNet whose clients must resolve privately
az network private-dns link vnet create -g rg-oncall-lab --zone-name privatelink.blob.core.windows.net \
  -n link-vnet-sysop --virtual-network vnet-sysop --registration-enabled false

# 3. the A record - let a zone group maintain it for you
az network private-endpoint dns-zone-group create -g rg-oncall-lab --endpoint-name pe-blob \
  -n default --private-dns-zone privatelink.blob.core.windows.net --zone-name blob
# after the private-endpoint mission
az network private-dns record-set a list -g rg-oncall-lab -z privatelink.blob.core.windows.net -o table
Name        ResourceGroup    Ttl    Type    AutoRegistered    Metadata
----------  ---------------  -----  ------  ----------------  ----------
stsysoplab  rg-oncall-lab     10     A       False

A zone group creates and deletes the A record together with the endpoint; a hand-made record goes stale the day someone recreates the endpoint and it gets a different IP.

Failure modes, in order of frequency:

  1. zone exists, no VNet link -> public IP from inside the VNet
  2. linked to the app VNet but the app resolves through a custom DNS server (on-prem, a DNS VM) that does not forward (pass on) privatelink.* questions to Azure DNS (168.63.129.16) -> public IP
  3. record points at an old IP (hand-made records)
  4. the zone is created per team in every subscription -> split-brain (two copies that disagree); one central zone per service, linked everywhere, is the pattern

Then turn the public door off

az storage account update -n stsysoplab --public-network-access Disabled
az keyvault update -n kv-shop-prod --public-network-access Disabled

Only private endpoints work afterwards. Your laptop, the portal's data browser and any automation running on machines outside the VNet lose access too - plan how they get in (automation machines inside the VNet, a jump host, Bastion).

Service endpoints: the older, weaker option

A service endpoint (Microsoft.Storage enabled on a subnet) lets that subnet reach the service's public endpoint over the backbone, and the service's firewall can allow "subnet X". No private IP, no DNS change, free - but the service stays public, it works per service type rather than per resource (data exfiltration - someone copying your data out - to any storage account is possible), and it does not extend to on-prem. New designs use private endpoints.

How to prove the resolution from inside

From a pod or VM in the VNet:

# from a VM inside the linked VNet
nslookup stsysoplab.blob.core.windows.net
Non-authoritative answer:
stsysoplab.blob.core.windows.net       canonical name = stsysoplab.privatelink.blob.core.windows.net.
Name:   stsysoplab.privatelink.blob.core.windows.net
Address: 10.20.2.4

From outside (your laptop, or oncall-lab here), the same name answers with the public IP after the privatelink CNAME - the CNAME is public, the A record behind it only exists in your private zone.

If the answer inside the VNet is public:

# the checklist, once the private endpoint exists
az network private-dns link vnet list -g rg-oncall-lab -z privatelink.blob.core.windows.net -o table
az network private-dns record-set a list -g rg-oncall-lab -z privatelink.blob.core.windows.net -o table
az network private-endpoint show -g rg-oncall-lab -n pe-blob-app --query "customDnsConfigs[0].ipAddresses[0]" -o tsv

Link missing, record missing, or record IP different from the endpoint IP - those three commands cover almost every case. The drill after the Key Vault mission trains exactly this triage.

One endpoint per sub-resource

A storage account has separate endpoints per service: blob, file, queue, table, dfs, web. A private endpoint with --group-id blob does nothing for Azure Files - an AKS cluster mounting Azure Files privately needs a second endpoint with --group-id file and the privatelink.file.core.windows.net zone. (--group-id names the sub-resource the endpoint is for.)

What you can now do

Why it helps

Private endpoints are the standard way to reach Key Vault, storage, container registries, databases and private AKS APIs at a bank, and "it works from my laptop but not from the pod" or "it broke when we disabled public access" is almost always their DNS. The failure is nasty because it looks like an auth problem: public access disabled, the name resolves to the public IP, and the service answers 403.

Knowing the three pieces, the endpoint, the zone and the VNet link, plus the zone group that keeps the record correct and the custom DNS forwarding to 168.63.129.16, lets you diagnose it with three commands. You'll also make the design choices: one central zone per service in the hub, not a zone per team, and separate endpoints for blob and file.

FAQ

Why does the app still use the public name?

Because TLS certificates and SDKs are issued for and configured with the real name, like stsysoplab.blob.core.windows.net, not an IP or a privatelink name. Azure publishes a CNAME from the real name to stsysoplab.privatelink.blob.core.windows.net, and a private DNS zone with that name, linked to your VNet, answers it with the private IP. Outside your network the same chain continues to the public IP.

What is a DNS zone group?

A setting on the private endpoint that links it to a private DNS zone, so Azure creates the A record when the endpoint is created and removes it when it's deleted. Without one, someone adds the record by hand, and the day the endpoint is recreated with a different IP, the record is stale and clients connect to nothing. Always use zone groups, in Terraform through private_dns_zone_group on the endpoint.

What goes wrong with custom DNS servers?

If the VNet uses a custom DNS server, on-prem DNS, a DNS VM or a resolver, clients ask it instead of Azure DNS. It must forward privatelink.* zones, or the service zones, to Azure DNS at 168.63.129.16, usually through Azure DNS Private Resolver in the hub, or it resolves the public IP. Linking the private zone to the VNet isn't enough when nobody in that VNet asks Azure DNS directly.

What is the difference between a private endpoint and a service endpoint?

A service endpoint lets a subnet reach a service's public endpoint over the Azure backbone, and the service's firewall can allow that subnet. No private IP, no DNS change, free, but the service stays public, it applies to a service type rather than one resource, and it doesn't extend to on-prem. A private endpoint gives one specific resource a private IP in your subnet, so you can disable public access entirely. New designs use private endpoints.

Why does blob work privately but Azure Files doesn't?

Each storage sub-resource, blob, file, queue, table, dfs and web, has its own endpoint and its own private DNS zone. A private endpoint with --group-id blob only covers blob traffic. An AKS cluster mounting Azure Files privately needs a second endpoint with --group-id file and the privatelink.file.core.windows.net zone linked to its VNet. Same pattern for other services with multiple sub-resources.

In an interview Mid

After disabling public network access on a storage account or Key Vault, apps inside the VNet get 403. How do you diagnose it?

Almost always DNS. A private endpoint gives the service a private IP in your subnet, but the app still uses the public name (stsysoplab.blob.core.windows.net), which must resolve to that IP from inside the VNet. Azure does it with a CNAME to *.privatelink.blob.core.windows.net, answered privately only where a matching private DNS zone is visible. Otherwise the name resolves to the public IP - which is now closed, hence a 403 that looks like auth.

  1. From inside the VNet: nslookup <name> - a public IP means the chain did not stop at the private zone.
  2. az network private-dns link vnet list ... - is the zone linked to this VNet?
  3. az network private-dns record-set a list ... - does the A record exist, and does it match the endpoint's IP? (A zone group keeps it in sync; hand-made records go stale.)
  4. Custom DNS servers must forward privatelink.* to Azure DNS 168.63.129.16.
  5. One endpoint per sub-resource: a blob endpoint does nothing for file.

Also asked: What is an Azure private endpoint? · What is the difference between a private endpoint and a service endpoint? · How would you organise private DNS zones across many subscriptions?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.