Managed Identity and RBAC Authentication Failure¶

1. Summary¶

This playbook is for incidents where Azure Functions cannot authenticate to downstream services by using managed identity, and requests fail with 401, 403, or host listener startup errors. It covers both system-assigned and user-assigned managed identities, role assignment drift, stale token cache windows after RBAC changes, Key Vault authorization mode mismatch, and token audience or scope mistakes when DefaultAzureCredential is used.

The goal is to restore trigger execution and outbound dependency access quickly, while separating identity-plane problems from application code regressions. Focus on proving or disproving six competing hypotheses in a bounded time window, then apply the smallest safe mitigation.

Decision Flow¶

flowchart TD
    A[Auth failures or listener startup errors] --> B{Managed identity enabled?}
    B -->|No| C[Enable identity and restart host]
    B -->|Yes| D{Correct principal assigned to app?}
    D -->|No| E[Attach expected user-assigned identity]
    D -->|Yes| F{Required RBAC role present?}
    F -->|No| G[Create missing role assignment]
    F -->|Yes| H{Role assignment recently changed?}
    H -->|Yes| I[Wait 5-10 min and refresh token path]
    H -->|No| J{Target is Key Vault?}
    J -->|Yes| K{Vault auth mode aligned?}
    K -->|No| L[Align RBAC vs access policy model]
    K -->|Yes| M{Audience and scope correct?}
    J -->|No| M
    M -->|No| N[Fix token scope and SDK settings]
    M -->|Yes| O[Check network and service-side deny]
    C --> P[Verify dependency success recovery]
    E --> P
    G --> P
    I --> P
    L --> P
    N --> P
    O --> P

Severity guidance¶

Signal snapshot¶

Failure progression model¶

sequenceDiagram
    participant H as Functions Host
    participant M as Managed Identity Endpoint
    participant A as Azure AD Token Service
    participant R as Target Resource
    H->>M: Request token for resource scope
    M->>A: Exchange identity assertion
    A-->>M: Access token
    M-->>H: Access token returned
    H->>R: Call dependency with bearer token
    R-->>H: 403 AuthorizationPermissionMismatch
    H->>H: Retry policy executes
    H-->>H: Listener startup may fail if trigger auth blocked

Identity authorization map¶

flowchart LR
    F[Function App] --> I[(System-assigned MI)]
    F --> U[(User-assigned MI)]
    I --> RBAC1[Role Assignment: Storage Blob Data Contributor]
    U --> RBAC2[Role Assignment: Azure Service Bus Data Receiver]
    I --> KV[Key Vault]
    KV --> MODE{Permission model}
    MODE -->|RBAC| R1[Key Vault Secrets User or Reader]
    MODE -->|Access Policy| P1["Get/List secret policy"]
    F --> SDK[DefaultAzureCredential]
    SDK --> AUD[Resource audience or scope]
    AUD --> TARGET["Storage / Service Bus / Key Vault"]

2. Common Misreadings¶

3. Competing Hypotheses¶

4. What to Check First¶

1. Confirm which identity is active (system-assigned, user-assigned, or both) and capture principal IDs.
2. Validate that each impacted resource has the exact required role assignment at the proper scope.
3. Compare incident start time to recent RBAC or identity changes to detect propagation windows.
4. If Key Vault is involved, verify vault permission model and secret reference behavior.

Quick portal checks¶

• Function App -> Identity: status is On and expected user-assigned identity appears.
• Target resource -> Access control (IAM): principal has least-privilege role at effective scope.
• Application Insights -> Logs: ManagedIdentityCredential and 401/403 burst aligns with outage window.

Quick CLI checks¶

az functionapp identity show --name "$APP_NAME" --resource-group "$RG" --output json
az role assignment list --assignee "$PRINCIPAL_ID" --scope "$RESOURCE_ID" --output table
az keyvault show --name "$KV_NAME" --resource-group "$RG" --query "{name:name,enableRbacAuthorization:properties.enableRbacAuthorization}" --output table

Example output¶

{
  "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "tenantId": "<tenant-id>",
  "type": "SystemAssigned, UserAssigned",
  "userAssignedIdentities": {
    "/subscriptions/<subscription-id>/resourceGroups/rg-functions-prod/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mi-func-prod": {
      "clientId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    }
  }
}

Role                              Scope
--------------------------------  --------------------------------------------------------------------------------
Storage Blob Data Contributor     /subscriptions/<subscription-id>/resourceGroups/rg-functions-prod/providers/Microsoft.Storage/storageAccounts/stfuncprod
Azure Service Bus Data Receiver   /subscriptions/<subscription-id>/resourceGroups/rg-functions-prod/providers/Microsoft.ServiceBus/namespaces/sb-func-prod

Name            EnableRbacAuthorization
--------------  -----------------------
kv-func-prod    True

5. Evidence to Collect¶

6. Validation and Disproof by Hypothesis¶

H1: Managed identity missing or wrong identity selected¶

Confirming KQL¶

let appName = "$APP_NAME";
traces
| where timestamp > ago(2h)
| where cloud_RoleName =~ appName
| where message has_any ("ManagedIdentityCredential authentication unavailable", "No managed identity endpoint found", "User-assigned identity not found")
| project timestamp, severityLevel, operation_Id, message
| order by timestamp desc

Expected output¶

timestamp                    severityLevel  operation_Id                           message
---------------------------  -------------  ------------------------------------   --------------------------------------------------------------------------------------
2026-04-05T01:14:52.000000Z  3              xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   ManagedIdentityCredential authentication unavailable. No managed identity endpoint found.
2026-04-05T01:14:51.000000Z  3              xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   User-assigned identity not found for clientId xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
2026-04-05T01:14:49.000000Z  2              xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   DefaultAzureCredential failed to retrieve a token from the included credentials.

Disproving check¶

If traces show successful token acquisition for the intended principal and there are no identity-unavailable messages in the same window, H1 is weakened. Confirm with az functionapp identity show that principalId and expected user-assigned identity IDs match deployment intent.

H2: Missing or incorrect RBAC role assignment¶

Confirming KQL¶

let appName = "$APP_NAME";
dependencies
| where timestamp > ago(2h)
| where cloud_RoleName =~ appName
| where resultCode in ("401", "403")
| summarize Failures=count(), Samples=make_set(tostring(target), 5) by DependencyType=type, Name=name, Result=resultCode
| order by Failures desc

Expected output¶

DependencyType  Name                                 Result  Failures  Samples
--------------  -----------------------------------  ------  --------  --------------------------------------------------------------------
Azure blob      BlobContainerClient.GetProperties    403     182       ["stfuncprod.blob.core.windows.net"]
Azure ServiceBus ServiceBusReceiver.ReceiveMessages  403     146       ["sb-func-prod.servicebus.windows.net"]
Azure Key Vault SecretClient.GetSecret               403     91        ["kv-func-prod.vault.azure.net"]

Disproving check¶

If required roles are present at the exact data-plane scope and 403 events disappear without role changes, H2 is unlikely. Validate with az role assignment list --assignee "$PRINCIPAL_ID" --scope "$RESOURCE_ID" --output table for each affected resource.

H3: Propagation delay or stale token cache after RBAC update¶

Confirming KQL¶

let appName = "$APP_NAME";
dependencies
| where timestamp > ago(4h)
| where cloud_RoleName =~ appName
| where resultCode in ("401", "403")
| summarize AuthFailures=count() by bin(timestamp, 5m)
| join kind=leftouter (
    traces
    | where timestamp > ago(4h)
    | where cloud_RoleName =~ appName
    | where message has_any ("Role assignment", "ManagedIdentityCredential", "token")
    | summarize TokenLogs=count() by bin(timestamp, 5m)
) on timestamp
| project timestamp, AuthFailures, TokenLogs
| order by timestamp asc

Expected output¶

timestamp                    AuthFailures  TokenLogs
---------------------------  ------------  ---------
2026-04-05T01:00:00.000000Z  88            14
2026-04-05T01:05:00.000000Z  95            16
2026-04-05T01:10:00.000000Z  61            12
2026-04-05T01:15:00.000000Z  19            8
2026-04-05T01:20:00.000000Z  2             5
2026-04-05T01:25:00.000000Z  0             4

Disproving check¶

If failures continue beyond 30 minutes after confirmed RBAC updates and host recycle, propagation delay alone does not explain the incident. Continue with H2, H4, and H5 checks.

H4: Key Vault permission model mismatch¶

Confirming KQL¶

let appName = "$APP_NAME";
dependencies
| where timestamp > ago(2h)
| where cloud_RoleName =~ appName
| where target has "vault.azure.net"
| where resultCode in ("401", "403")
| project timestamp, operation_Id, name, resultCode, target, data
| order by timestamp desc

Expected output¶

timestamp                    operation_Id                           name                  resultCode  target                     data
---------------------------  ------------------------------------   --------------------  ----------  -------------------------  -----------------------------------------------------------
2026-04-05T01:16:22.000000Z  xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   SecretClient.GetSecret 403         kv-func-prod.vault.azure.net ForbiddenByPolicy: Caller is not authorized by access policy.
2026-04-05T01:16:21.000000Z  xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   SecretClient.GetSecret 403         kv-func-prod.vault.azure.net ForbiddenByRbac: Principal does not have secrets/get permission.

Disproving check¶

When vault permission mode is known and matching permissions are granted (enableRbacAuthorization=true with RBAC roles, or false with access policies), and secret reads succeed in test calls, H4 is disconfirmed.

H5: Audience or scope mismatch in token acquisition¶

Confirming KQL¶

let appName = "$APP_NAME";
traces
| where timestamp > ago(2h)
| where cloud_RoleName =~ appName
| where message has_any ("invalid audience", "scope", "AADSTS500011", "The audience is invalid")
| project timestamp, severityLevel, operation_Id, message
| order by timestamp desc

Expected output¶

timestamp                    severityLevel  operation_Id                           message
---------------------------  -------------  ------------------------------------   ----------------------------------------------------------------------------------------------
2026-04-05T01:18:02.000000Z  3              xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   AADSTS500011: The resource principal named https://storage.azure.com was not found.
2026-04-05T01:18:01.000000Z  3              xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   The audience is invalid for the requested token. Scope used: api://wrong-resource/.default

Disproving check¶

If dependency calls succeed after forcing the documented service scope (for example, https://storage.azure.com/.default or https://servicebus.azure.net/.default) and no audience errors remain, H5 was causal and now resolved.

H6: Non-auth outage masquerading as auth failure¶

Confirming KQL¶

let appName = "$APP_NAME";
dependencies
| where timestamp > ago(2h)
| where cloud_RoleName =~ appName
| summarize
    Auth=countif(resultCode in ("401", "403")),
    Timeout=countif(resultCode == "0" or tostring(data) has_any ("timeout", "Name or service not known", "No such host")),
    Server=countif(resultCode startswith "5")
  by Target=target
| order by Timeout desc, Auth desc

Expected output¶

Target                               Auth  Timeout  Server
-----------------------------------  ----  -------  ------
sb-func-prod.servicebus.windows.net  6     188      0
stfuncprod.blob.core.windows.net     4     172      0
kv-func-prod.vault.azure.net         3     97       0

Disproving check¶

If timeout and DNS errors dominate while 401/403 are sparse, treat auth findings as secondary and execute network outage workflow first. H6 remains strongest until connectivity normalizes.

Host startup correlation¶

timeline
    title Managed identity auth incident timeline
    01h00 : RBAC assignment removed accidentally
    01h03 : First 403 in dependencies
    01h05 : Listener startup warnings appear
    01h12 : Incident declared SEV-1
    01h14 : Role assignment restored
    01h20 : Auth failures decline after propagation
    01h25 : Invocations recover
    01h40 : Monitoring confirms steady state

Normal vs abnormal comparison¶

Evidence interpretation checklist¶

1. Verify the same principal ID appears across identity metadata, role assignments, and dependency call context.
2. Confirm the failure signature is authorization (401/403) rather than connectivity timeout (resultCode 0).
3. Align incident start minute with IAM change history before declaring propagation delay.
4. For Key Vault, verify exactly one permission model expectation is used in operational runbooks.
5. Capture at least one post-mitigation query showing AuthFailures=0 before closure.

7. Likely Root Cause Patterns¶

8. Immediate Mitigations¶

1. Re-enable or attach the expected managed identity.

   az functionapp identity assign --name "$APP_NAME" --resource-group "$RG" --output json
   
   # For user-assigned identity:
   az functionapp identity assign --resource-group "$RG" --name "$APP_NAME" \
     --identities "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/$IDENTITY_NAME"
   

2. Grant missing blob data-plane permissions for blob-trigger workloads.

   az role assignment create --assignee-object-id "$PRINCIPAL_ID" --assignee-principal-type ServicePrincipal --role "Storage Blob Data Contributor" --scope "$STORAGE_RESOURCE_ID" --output json
   

3. Grant missing Service Bus data-plane permissions for Service Bus trigger workloads.

   az role assignment create --assignee-object-id "$PRINCIPAL_ID" --assignee-principal-type ServicePrincipal --role "Azure Service Bus Data Receiver" --scope "$SERVICEBUS_RESOURCE_ID" --output json
   

4. Align Key Vault permission mode and grants. !!! warning "Verify Before Switching" Switching from access-policy to RBAC authorization may break existing access-policy-based consumers. Verify all Key Vault consumers support RBAC before running this command.

az keyvault update --name "$KV_NAME" --resource-group "$RG" --enable-rbac-authorization true --output json

az functionapp restart --name "$APP_NAME" --resource-group "$RG"

az monitor log-analytics query --workspace "$WORKSPACE_ID" --analytics-query "dependencies | where timestamp > ago(15m) | summarize AuthFailures=countif(resultCode in ('401','403'))" --output table

9. Prevention¶

1. Standardize managed identity and RBAC assignments in infrastructure as code with review gates.
2. Add deployment checks that assert required roles per trigger type before slot swap.
3. Track IAM and identity changes in change management with explicit blast-radius notes.
4. Add alert rules on dependencies 401/403 rate and ManagedIdentityCredential trace spikes.
5. Document canonical token scopes for each dependency and enforce through shared SDK wrappers.
6. Run quarterly access reviews to remove stale assignments without breaking trigger identities.

See Also¶

• Troubleshooting Architecture
• Troubleshooting Methodology
• KQL Query Guide
• Troubleshooting Lab Guides
• Managed Identity Authentication Lab
• App Settings Misconfiguration Playbook

Sources¶

• Azure Functions identity-based connections
• Use managed identities for App Service and Azure Functions
• Assign Azure roles using Azure CLI
• Authorize access to Key Vault secrets
• DefaultAzureCredential authentication flow