Lab Guide: DNS and VNet Resolution Failure¶

This lab reproduces a private DNS resolution failure on an Azure Functions Flex Consumption (FC1) app with private endpoints. By removing the blob storage Private DNS zone VNet link, the function app loses private DNS resolution for blob storage, causing storage operations to route to the public endpoint and fail with AuthorizationFailure. Restoring the link restores private resolution and storage access.

Lab Metadata¶

1) Background¶

Azure Functions on Flex Consumption with VNet integration and private endpoints depends on Private DNS zones for name resolution. Each storage service (blob, queue, table, file) has its own Private DNS zone (privatelink.{service}.core.windows.net) linked to the VNet. If a zone's VNet link is removed, the function app resolves that service's hostname to the public IP instead of the private endpoint IP.

1.1 Private endpoint DNS architecture¶

When allowSharedKeyAccess: false is set on the storage account (common in FC1 identity-based deployments), the managed identity token works over the private endpoint path but fails when traffic routes to the public endpoint — because the storage account's network rules block public access.

flowchart TD
    A[Function host starts] --> B[Resolve storage FQDN]
    B --> C[DNS resolver checks linked Private DNS zone]
    C --> D{VNet link present?}
    D -->|Yes| E[Resolve to private IP via PE]
    E --> F[Storage call via private endpoint]
    F --> G[Auth succeeds with MI token]

    D -->|No| H[Resolve to public IP]
    H --> I[Storage call via public endpoint]
    I --> J[Auth fails: AuthorizationFailure]
    J --> K[502 error returned]

1.2 Why individual service DNS links matter¶

Each storage service has an independent Private DNS zone and VNet link:

Removing one zone's VNet link only affects that service. The other three continue resolving to private IPs. This selective breakage is a key diagnostic signal — it rules out VNet integration failure (which would affect all services) and points directly to DNS zone configuration.

1.3 FC1 behavior and why this scenario appears suddenly¶

Flex Consumption scales quickly and can instantiate new workers frequently. Each new worker performs fresh DNS resolution:

• A DNS drift may remain unnoticed while cached resolution is still valid.
• New worker startup forces fresh DNS queries.
• The first query after a VNet link removal resolves to the public IP.

1.4 Failure progression model¶

sequenceDiagram
    participant App as Function App (FC1)
    participant DNS as DNS Resolver
    participant BlobZone as privatelink.blob DNS Zone
    participant PE as Blob Private Endpoint (10.0.2.7)
    participant Public as Blob Public Endpoint

    Note over BlobZone: VNet link present
    App->>DNS: Resolve labdnsfc1storage.blob.core.windows.net
    DNS->>BlobZone: Query A record
    BlobZone-->>DNS: 10.0.2.7
    DNS-->>App: Private IP returned
    App->>PE: Storage call via PE
    PE-->>App: 200 OK

    Note over BlobZone: VNet link REMOVED
    App->>DNS: Resolve labdnsfc1storage.blob.core.windows.net
    DNS->>BlobZone: Query A record
    BlobZone--xDNS: No VNet link — zone not visible
    DNS-->>App: Public IP 20.150.4.36
    App->>Public: Storage call via public endpoint
    Public-->>App: 403 AuthorizationFailure

1.5 Signal map: normal vs incident¶

2) Hypothesis¶

2.1 Formal hypothesis statement¶

If the blob Private DNS zone VNet link is removed from an FC1 function app with private endpoints and allowSharedKeyAccess: false, then blob DNS resolves to the public IP instead of the private endpoint IP, and storage operations fail with AuthorizationFailure because the public endpoint rejects managed identity tokens when public network access is restricted. Restoring the VNet link restores private resolution and storage access without code changes.

2.2 Causal chain¶

flowchart LR
    A[Blob DNS VNet link removed] --> B[Blob FQDN resolves to public IP]
    B --> C[Storage call routes to public endpoint]
    C --> D[MI token rejected: AuthorizationFailure]
    D --> E[Storage probe returns 502]
    E --> F["Queue/table/file unaffected — private IPs intact"]

    G[Blob DNS VNet link restored] --> H[Blob FQDN resolves to private IP]
    H --> I[Storage call routes via PE]
    I --> J[MI token accepted]
    J --> K[Storage probe returns 200]

2.3 Proof criteria¶

2.4 Disproof criteria¶

2.5 Competing hypotheses¶

3) Runbook¶

Prerequisites¶

3.1 Variables¶

RG="rg-lab-fc1-dns"
LOCATION="koreacentral"
BASE_NAME="labdnsfc1"
APP_NAME="labdnsfc1-func"
STORAGE_NAME="labdnsfc1storage"
VNET_NAME="labdnsfc1-vnet"
AI_NAME="labdnsfc1-insights"

3.2 Deploy infrastructure¶

Deploy using the Flex Consumption Bicep template with VNet and private endpoints:

az group create \
  --name "$RG" \
  --location "$LOCATION"

az deployment group create \
  --resource-group "$RG" \
  --template-file "infra/flex-consumption/main.bicep" \
  --parameters baseName="$BASE_NAME"

Deploy the diagnostics app:

az functionapp deployment source config-zip \
  --name "$APP_NAME" \
  --resource-group "$RG" \
  --src "./apps/python/app.zip"

Verify all 4 Private DNS zone VNet links exist:

for zone in blob queue table file; do
  echo "=== privatelink.${zone}.core.windows.net ==="
  az network private-dns link vnet list \
    --zone-name "privatelink.${zone}.core.windows.net" \
    --resource-group "$RG" \
    --output table
done

3.3 Capture baseline evidence¶

Baseline DNS resolution (all 4 endpoints)¶

for svc in blob queue table file; do
  curl --silent \
    "https://${APP_NAME}.azurewebsites.net/api/diagnostics/dns-resolve?host=${STORAGE_NAME}.${svc}.core.windows.net"
  echo ""
done

Baseline output:

{"status": "resolved", "host": "labdnsfc1storage.blob.core.windows.net", "addresses": ["10.0.2.7"], "elapsedMs": 2}
{"status": "resolved", "host": "labdnsfc1storage.queue.core.windows.net", "addresses": ["10.0.2.5"], "elapsedMs": 1}
{"status": "resolved", "host": "labdnsfc1storage.table.core.windows.net", "addresses": ["10.0.2.6"], "elapsedMs": 3}
{"status": "resolved", "host": "labdnsfc1storage.file.core.windows.net", "addresses": ["10.0.2.4"], "elapsedMs": 1}

Baseline storage probe¶

curl --silent \
  "https://${APP_NAME}.azurewebsites.net/api/diagnostics/storage-probe"

Output:

{"status": "success", "account": "labdnsfc1storage", "containers": ["azure-webjobs-hosts", "azure-webjobs-secrets", "deployment-packages"], "elapsedMs": 370}

Baseline health and identity¶

curl --silent "https://${APP_NAME}.azurewebsites.net/api/health"
curl --silent "https://${APP_NAME}.azurewebsites.net/api/diagnostics/identity-probe"

Output:

{"status": "healthy", "timestamp": "2026-04-07T09:25:05+00:00", "version": "1.0.0"}
{"status": "success", "identityType": "user-assigned", "clientId": "<managed-identity-client-id>", "tokenAcquired": true, "tokenExpiresOn": "2026-04-08T09:25:07Z", "elapsedMs": 44}

3.4 Trigger the failure¶

Remove the blob Private DNS zone VNet link:

az network private-dns link vnet delete \
  --resource-group "$RG" \
  --zone-name "privatelink.blob.core.windows.net" \
  --name "${BASE_NAME}-blob-dns-link" \
  --yes

Verify the link is gone:

az network private-dns link vnet list \
  --zone-name "privatelink.blob.core.windows.net" \
  --resource-group "$RG" \
  --output table

Expected: empty table (no VNet links for blob zone).

CLI output confirming blob VNet link is absent:

$ az network private-dns link vnet list \
    --zone-name "privatelink.blob.core.windows.net" \
    --resource-group "$RG" \
    --output table

Result
--------
(empty — no VNet links found for blob zone)

Verify other zones are unaffected:

for zone in queue table file; do
  echo "=== privatelink.${zone}.core.windows.net ==="
  az network private-dns link vnet list \
    --zone-name "privatelink.${zone}.core.windows.net" \
    --resource-group "$RG" \
    --output table
done

Expected: queue, table, and file VNet links still present.

3.5 Collect incident evidence¶

Incident DNS resolution — the key diagnostic¶

for svc in blob queue table file; do
  curl --silent \
    "https://${APP_NAME}.azurewebsites.net/api/diagnostics/dns-resolve?host=${STORAGE_NAME}.${svc}.core.windows.net"
  echo ""
done

Incident output:

{"status": "resolved", "host": "labdnsfc1storage.blob.core.windows.net", "addresses": ["20.150.4.36"], "elapsedMs": 87}
{"status": "resolved", "host": "labdnsfc1storage.queue.core.windows.net", "addresses": ["10.0.2.5"], "elapsedMs": 1}
{"status": "resolved", "host": "labdnsfc1storage.table.core.windows.net", "addresses": ["10.0.2.6"], "elapsedMs": 3}
{"status": "resolved", "host": "labdnsfc1storage.file.core.windows.net", "addresses": ["10.0.2.4"], "elapsedMs": 1}

Incident storage probe¶

curl --silent --write-out "\nHTTP Status: %{http_code}\n" \
  "https://${APP_NAME}.azurewebsites.net/api/diagnostics/storage-probe"

Incident output:

{"status": "error", "error": "AuthorizationFailure: This request is not authorized to perform this operation.\nRequestId: a1b2c3d4-e5f6-7890-abcd-ef1234567890\nTime: 2026-04-07T09:28:08Z\nErrorCode: AuthorizationFailure"}
HTTP Status: 502

Incident health and identity (confirming app and MI are fine)¶

curl --silent "https://${APP_NAME}.azurewebsites.net/api/health"
curl --silent "https://${APP_NAME}.azurewebsites.net/api/diagnostics/identity-probe"

Output:

{"status": "healthy", "timestamp": "2026-04-07T09:28:24+00:00", "version": "1.0.0"}
{"status": "success", "identityType": "user-assigned", "clientId": "<managed-identity-client-id>", "tokenAcquired": true, "tokenExpiresOn": "2026-04-08T09:28:26Z", "elapsedMs": 47}

Incident persistence after restart¶

az functionapp restart \
  --name "$APP_NAME" \
  --resource-group "$RG"

# Wait 30 seconds for restart
sleep 30

curl --silent \
  "https://${APP_NAME}.azurewebsites.net/api/diagnostics/dns-resolve?host=${STORAGE_NAME}.blob.core.windows.net"
curl --silent --write-out "\nHTTP Status: %{http_code}\n" \
  "https://${APP_NAME}.azurewebsites.net/api/diagnostics/storage-probe"

Post-restart output:

{"status": "resolved", "host": "labdnsfc1storage.blob.core.windows.net", "addresses": ["20.150.4.36"], "elapsedMs": 87}

{"status": "error", "error": "AuthorizationFailure: ..."}
HTTP Status: 502

3.6 Investigation summary¶

3.7 Triage decision flow¶

flowchart TD
    A[Storage probe returns 502 AuthorizationFailure] --> B{DNS resolves to private IP?}
    B -->|Yes| C[Investigate RBAC, storage networking rules, MI config]
    B -->|No - resolves to public IP| D{All storage services affected?}
    D -->|Yes| E[Check VNet integration, NSG, route table]
    D -->|No - only one service| F{Private DNS zone VNet link present?}
    F -->|No| G[Restore VNet link for affected service]
    F -->|Yes| H[Check DNS zone A record, PE NIC IP]
    G --> I[Verify DNS resolves to private IP]
    I --> J[Verify storage probe succeeds]

3.8 Recover from the induced failure¶

Restore the blob Private DNS zone VNet link:

az network private-dns link vnet create \
  --name "${BASE_NAME}-blob-dns-link" \
  --zone-name "privatelink.blob.core.windows.net" \
  --resource-group "$RG" \
  --virtual-network "$VNET_NAME" \
  --registration-enabled false

Restart the app:

az functionapp restart \
  --name "$APP_NAME" \
  --resource-group "$RG"

Wait 30 seconds for DNS propagation.

3.9 Verify recovery¶

Recovery DNS resolution¶

for svc in blob queue table file; do
  curl --silent \
    "https://${APP_NAME}.azurewebsites.net/api/diagnostics/dns-resolve?host=${STORAGE_NAME}.${svc}.core.windows.net"
  echo ""
done

Recovery output:

{"status": "resolved", "host": "labdnsfc1storage.blob.core.windows.net", "addresses": ["10.0.2.7"], "elapsedMs": 12}
{"status": "resolved", "host": "labdnsfc1storage.queue.core.windows.net", "addresses": ["10.0.2.5"], "elapsedMs": 4}
{"status": "resolved", "host": "labdnsfc1storage.table.core.windows.net", "addresses": ["10.0.2.6"], "elapsedMs": 28}
{"status": "resolved", "host": "labdnsfc1storage.file.core.windows.net", "addresses": ["10.0.2.4"], "elapsedMs": 19}

Recovery storage probe¶

curl --silent \
  "https://${APP_NAME}.azurewebsites.net/api/diagnostics/storage-probe"

Output:

{"status": "success", "account": "labdnsfc1storage", "containers": ["azure-webjobs-hosts", "azure-webjobs-secrets", "deployment-packages"], "elapsedMs": 307}

Recovery health and identity¶

curl --silent "https://${APP_NAME}.azurewebsites.net/api/health"
curl --silent "https://${APP_NAME}.azurewebsites.net/api/diagnostics/identity-probe"

Output:

{"status": "healthy", "timestamp": "2026-04-07T09:38:11+00:00", "version": "1.0.0"}
{"status": "success", "identityType": "user-assigned", "clientId": "<managed-identity-client-id>", "tokenAcquired": true, "tokenExpiresOn": "2026-04-08T09:38:36Z", "elapsedMs": 52}

4) Experiment Log¶

Artifact inventory¶

Evidence timeline¶

Representative log patterns¶

Baseline endpoint responses¶

[Baseline 09:25:07Z]
dns-resolve blob: {"status":"resolved","addresses":["10.0.2.7"],"elapsedMs":2}
dns-resolve queue: {"status":"resolved","addresses":["10.0.2.5"],"elapsedMs":1}
storage-probe: {"status":"success","containers":["azure-webjobs-hosts","azure-webjobs-secrets","deployment-packages"],"elapsedMs":370}
identity-probe: {"status":"success","tokenAcquired":true,"elapsedMs":44}

Incident endpoint responses¶

[Incident 09:27:56Z — after blob VNet link removal]
dns-resolve blob: {"status":"resolved","addresses":["20.150.4.36"],"elapsedMs":87}
dns-resolve queue: {"status":"resolved","addresses":["10.0.2.5"],"elapsedMs":1}  (unchanged)
storage-probe: {"status":"error","error":"AuthorizationFailure: This request is not authorized..."}  HTTP 502
health: {"status":"healthy"}  HTTP 200  (app itself is fine)
identity-probe: {"status":"success","tokenAcquired":true}  (MI works fine)

Recovery endpoint responses¶

[Recovery 09:38:46Z — after blob VNet link restored]
dns-resolve blob: {"status":"resolved","addresses":["10.0.2.7"],"elapsedMs":12}
storage-probe: {"status":"success","containers":[...],"elapsedMs":307}
health: {"status":"healthy"}  HTTP 200
identity-probe: {"status":"success","tokenAcquired":true,"elapsedMs":52}

Core finding¶

Hypothesis verdict table¶

Final verdict: Hypothesis supported.

Practical implications for production¶

1. Monitor each DNS zone VNet link independently — a missing link for one service does not affect others, making it easy to miss in aggregate health checks.
2. DNS resolution check is the fastest diagnostic — resolving the storage FQDN from within the function app immediately reveals public vs private IP routing.
3. Restart does not fix DNS configuration — unlike transient DNS cache issues, a missing VNet link persists across restarts.
4. allowSharedKeyAccess: false amplifies DNS failures — with shared key access disabled, any path change from private to public immediately causes auth failures.
5. Automate VNet link validation in deployment pipelines — include link existence checks in IaC validation and post-deployment smoke tests.

Expected Evidence¶

Before Trigger (Baseline)¶

During Incident¶

After Recovery¶

Evidence Timeline¶

gantt
    title DNS / VNet Resolution Failure Timeline (FC1 Lab)
    dateFormat  HH:mm
    axisFormat  %H:%M
    section Baseline
    Baseline capture (all endpoints private)    :done, b1, 09:25, 2m
    section Trigger
    Remove blob DNS VNet link                   :crit, t1, 09:27, 1m
    section Incident
    Blob resolves to public IP + storage 502    :crit, i1, 09:28, 2m
    Confirm restart does not fix                :crit, i2, 09:29, 1m
    section Recovery
    Restore blob DNS VNet link                  :done, r1, 09:36, 2m
    App restart + DNS propagation               :done, r2, 09:37, 1m
    Verify recovery (private IP + storage 200)  :done, r3, 09:38, 2m

Evidence Chain: Why This Proves the Hypothesis¶

Clean Up¶

az group delete \
  --name "$RG" \
  --yes \
  --no-wait

Related Playbook¶

• Functions Failing with Errors

See Also¶

• Troubleshooting Lab Guides
• First 10 Minutes Triage
• Troubleshooting Methodology
• KQL Reference for Troubleshooting
• Evidence Map

Sources¶

• Azure Functions networking options
• Azure private endpoint DNS configuration
• Azure Functions storage considerations
• Azure Private DNS overview
• Monitor Azure Functions
• Azure DNS private zones CLI reference