Lab Guide: Managed Identity Authentication Failure¶

This Level 3 lab guide reproduces a managed identity authorization failure in Azure Functions and demonstrates a complete, evidence-driven chain from RBAC drift to host storage unhealthy state, listener startup failure, and trigger interruption. The scenario uses Python v2 on Flex Consumption (FC1) with a system-assigned identity that reads and writes blob containers.

Lab Metadata¶

1) Background¶

Managed identity simplifies secretless authentication, but it does not remove authorization design and operations risk.

In this scenario, Azure Functions uses a system-assigned identity to access Azure Blob containers for app workflows and host-level storage probes.

When Storage Blob Data Contributor is removed from the storage scope, token acquisition still succeeds, but storage authorization fails with 403 AuthorizationPermissionMismatch.

1.1 Managed identity authentication flow¶

sequenceDiagram
    autonumber
    participant Func as Azure Function App
    participant IMDS as Managed Identity Endpoint (IMDS)
    participant AAD as Microsoft Entra ID
    participant Blob as Azure Storage Blob API
    participant AuthZ as RBAC Authorization Engine

    Func->>IMDS: Request token for https://storage.azure.com/
    IMDS->>AAD: Resolve managed identity and audience
    AAD-->>IMDS: Access token (JWT)
    IMDS-->>Func: Return access token
    Func->>Blob: Blob request with Bearer token
    Blob->>AuthZ: Evaluate role assignments at scope
    AuthZ-->>Blob: Allow or deny operation
    Blob-->>Func: 200/201 on success, 403 on missing role

1.2 System-assigned vs user-assigned identity¶

1.3 RBAC propagation timeline¶

gantt
    title RBAC Change to Effective Authorization Timeline
    dateFormat  YYYY-MM-DD HH:mm:ss
    axisFormat  %H:%M:%S

    section Change Operations
    Role assignment deleted                 :done, a1, 2026-04-04 12:14:58, 1m
    Function app restart initiated          :done, a2, 2026-04-04 12:15:12, 1m

    section Propagation Window
    RBAC propagation in control plane       :active, b1, 2026-04-04 12:15:00, 8m
    Token/cache refresh across components   :active, b2, 2026-04-04 12:15:00, 10m

    section Runtime Effects
    Authorization failures observed         :crit, c1, 2026-04-04 12:17:54, 7m
    Listener startup failures               :crit, c2, 2026-04-04 12:15:42, 9m
    Host unhealthy storage signal           :crit, c3, 2026-04-04 12:22:42, 3m

    section Recovery
    Role assignment restored                :done, d1, 2026-04-04 12:23:20, 1m
    Access becomes effective                :done, d2, 2026-04-04 12:24:31, 1m

1.4 Common identity failure modes¶

2) Hypothesis¶

2.1 Formal statement¶

If the Function App system-assigned managed identity loses Storage Blob Data Contributor on the target storage scope, then Azure Functions blob calls will return AuthorizationPermissionMismatch (403), the storage subsystem can become unhealthy, and listener startup for Functions.scheduled_cleanup can fail until RBAC is restored and propagation completes.

2.2 Causal chain¶

flowchart LR
    A[Role assignment removed] --> B[Blob data-plane authorization denied]
    B --> C[403 AuthorizationPermissionMismatch]
    C --> D[Host storage probe failures]
    D --> E[Listener startup failure]
    E --> F[scheduled_cleanup not executing]
    F --> G[Operational symptom: maintenance jobs stop]

    H[Role assignment restored] --> I[Propagation delay window]
    I --> J[Storage probe succeeded 70ms 4 containers]
    J --> K[Listener startup succeeds]
    K --> L[Timer + HTTP blob flows recover]

2.3 Proof criteria¶

2.4 Disproof criteria¶

Hypothesis status is valid only if the evidence chain is temporally coherent and reproducible across at least one restart cycle.

3) Runbook¶

Prerequisites¶

Use canonical variables in command examples:

RG="rg-lab-managed-identity"
LOCATION="koreacentral"
APP_NAME="func-lab-mi-auth"
STORAGE_NAME="stlabmiauth"
PLAN_NAME="fc1-lab-mi"
SUBSCRIPTION_ID="<subscription-id>"

3.1 Deploy infrastructure¶

Deploy the lab resources (Function App, Storage Account, identity-enabled configuration, and baseline RBAC):

az group create \
  --name "$RG" \
  --location "$LOCATION"

az deployment group create \
  --resource-group "$RG" \
  --template-file "labs/managed-identity-auth/main.bicep" \
  --parameters \
    "appName=$APP_NAME" \
    "storageName=$STORAGE_NAME" \
    "location=$LOCATION" \
    "planName=$PLAN_NAME"

Confirm Function App identity state:

az functionapp identity show \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --output json

Example output (sanitized):

{
  "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "tenantId": "<tenant-id>",
  "type": "SystemAssigned",
  "userAssignedIdentities": null
}

Capture storage resource ID and identity principal for later steps:

STORAGE_ID=$(az storage account show \
  --resource-group "$RG" \
  --name "$STORAGE_NAME" \
  --query "id" \
  --output tsv)

APP_PRINCIPAL_ID=$(az functionapp identity show \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --query "principalId" \
  --output tsv)

Validate baseline RBAC:

az role assignment list \
  --assignee "$APP_PRINCIPAL_ID" \
  --scope "$STORAGE_ID" \
  --output table

Example output (sanitized):

Principal                             Role                          Scope
------------------------------------  ----------------------------  -------------------------------------------------------------------------------
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  Storage Blob Data Contributor /subscriptions/<subscription-id>/resourceGroups/rg-lab-managed-identity/providers/Microsoft.Storage/storageAccounts/stlabmiauth

3.2 Collect baseline¶

Wait until the app is stable (2-5 minutes after deployment), then run baseline queries.

3.2.1 Baseline dependency health¶

let appName = "func-lab-mi-auth";
dependencies
| where timestamp > ago(30m)
| where cloud_RoleName =~ appName
| where target has "blob.core.windows.net"
| summarize
    Calls=count(),
    Failed=countif(success == false),
    FailureRatePercent=round(100.0 * countif(success == false) / count(), 2),
    P50Ms=percentile(duration, 50),
    P95Ms=percentile(duration, 95)
  by target, operation_Name, resultCode
| order by Calls desc

Example output:

3.2.2 Baseline request success¶

let appName = "func-lab-mi-auth";
requests
| where timestamp > ago(30m)
| where cloud_RoleName =~ appName
| where operation_Name in ("Functions.http_blob_read", "Functions.http_blob_write", "Functions.scheduled_cleanup")
| summarize
    Invocations=count(),
    Failures=countif(success == false),
    FailureRatePercent=round(100.0 * countif(success == false) / count(), 2),
    P95Ms=percentile(duration, 95)
  by operation_Name
| order by operation_Name asc

Example output:

3.3 Trigger the failure¶

Remove Storage Blob Data Contributor role from the Function App identity at storage scope.

az role assignment delete \
  --assignee-object-id "$APP_PRINCIPAL_ID" \
  --role "Storage Blob Data Contributor" \
  --scope "$STORAGE_ID"

Example output:

Role assignment deleted successfully.

Restart the Function App to force a fresh startup/storage probe path:

az functionapp restart \
  --resource-group "$RG" \
  --name "$APP_NAME"

Timestamp this trigger point in your logbook.

2026-04-04T12:15:12Z  Failure trigger applied: removed Storage Blob Data Contributor and restarted app

3.4 Observe failure¶

During this phase, collect high-signal evidence and preserve exact timestamps.

3.4.1 Auth failure detection query¶

let appName = "func-lab-mi-auth";
dependencies
| where timestamp > ago(45m)
| where cloud_RoleName =~ appName
| where target has "blob.core.windows.net"
| summarize
    Calls=count(),
    Failed=countif(success == false),
    FailureRatePercent=round(100.0 * countif(success == false) / count(), 2),
    P95Ms=percentile(duration, 95)
  by target, resultCode, operation_Name
| order by Failed desc, Calls desc

Example output:

3.4.2 Trace signatures for listener and host health¶

let appName = "func-lab-mi-auth";
traces
| where timestamp > ago(45m)
| where cloud_RoleName =~ appName
| where message has_any ("AuthorizationPermissionMismatch", "listener for function 'Functions.scheduled_cleanup'", "azure.functions.webjobs.storage")
| project timestamp, severityLevel, message
| order by timestamp asc

Example output:

3.4.3 Exception trend during incident¶

let appName = "func-lab-mi-auth";
exceptions
| where timestamp between (ago(2h) .. now())
| where cloud_RoleName =~ appName
| summarize Count=count() by bin(timestamp, 5m), type
| order by timestamp asc

Example output:

3.4.4 Correlated request failures¶

let appName = "func-lab-mi-auth";
requests
| where timestamp > ago(45m)
| where cloud_RoleName =~ appName
| where operation_Name in ("Functions.http_blob_read", "Functions.http_blob_write", "Functions.scheduled_cleanup")
| summarize
    Invocations=count(),
    Failures=countif(success == false),
    FailureRatePercent=round(100.0 * countif(success == false) / count(), 2),
    P95Ms=percentile(duration, 95)
  by operation_Name, resultCode
| order by operation_Name asc, resultCode asc

Example output:

Reference failure log anchors used in this lab:

[2026-04-04T12:17:54Z] AuthorizationPermissionMismatch: This request is not authorized to perform this operation using this permission.
[2026-04-04T12:15:42Z] The listener for function 'Functions.scheduled_cleanup' was unable to start.
[2026-04-04T12:22:42Z] Process reporting unhealthy: azure.functions.webjobs.storage: Unhealthy (AuthorizationPermissionMismatch)

3.5 Investigate¶

Use CLI to prove identity state, role assignments, service principal mapping, and change events.

3.5.1 Confirm managed identity details¶

az functionapp identity show \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --output json

Example output:

{
  "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "tenantId": "<tenant-id>",
  "type": "SystemAssigned",
  "userAssignedIdentities": null
}

3.5.2 Inspect role assignments at storage scope¶

az role assignment list \
  --assignee "$APP_PRINCIPAL_ID" \
  --scope "$STORAGE_ID" \
  --output table

Example output during incident:

Principal                             Role                          Scope
------------------------------------  ----------------------------  -------------------------------------------------------------------------------
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  Reader                        /subscriptions/<subscription-id>/resourceGroups/rg-lab-managed-identity/providers/Microsoft.Storage/storageAccounts/stlabmiauth

Interpretation: identity exists, but required Blob Data Contributor role is absent.

3.5.3 Verify service principal object¶

az ad sp show \
  --id "$APP_PRINCIPAL_ID" \
  --output json

Example output:

{
  "id": "<object-id>",
  "appId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "displayName": "func-lab-mi-auth",
  "servicePrincipalType": "ManagedIdentity",
  "accountEnabled": true,
  "appOwnerOrganizationId": "<tenant-id>"
}

3.5.4 Trace RBAC modifications in Activity Log¶

az monitor activity-log list \
  --resource-group "$RG" \
  --offset 2h \
  --status Succeeded \
  --output table

Example output:

EventTimestamp          OperationNameValue                                   ResourceGroup             Caller                         Status
----------------------  ---------------------------------------------------  ------------------------  -----------------------------  ---------
2026-04-04T12:14:58Z    Microsoft.Authorization/roleAssignments/delete       rg-lab-managed-identity   <user-email>                  Succeeded
2026-04-04T12:15:12Z    Microsoft.Web/sites/restart/action                   rg-lab-managed-identity   <user-email>                  Succeeded
2026-04-04T12:23:20Z    Microsoft.Authorization/roleAssignments/write        rg-lab-managed-identity   <user-email>                  Succeeded

3.6 Fix and verify¶

Restore the missing role assignment:

az role assignment create \
  --assignee-object-id "$APP_PRINCIPAL_ID" \
  --role "Storage Blob Data Contributor" \
  --scope "$STORAGE_ID"

Example output:

{
  "id": "/subscriptions/<subscription-id>/resourceGroups/rg-lab-managed-identity/providers/Microsoft.Storage/storageAccounts/stlabmiauth/providers/Microsoft.Authorization/roleAssignments/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "name": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "roleDefinitionName": "Storage Blob Data Contributor",
  "scope": "/subscriptions/<subscription-id>/resourceGroups/rg-lab-managed-identity/providers/Microsoft.Storage/storageAccounts/stlabmiauth"
}

Wait for propagation (typically 2-10 minutes), then restart app:

az functionapp restart \
  --resource-group "$RG" \
  --name "$APP_NAME"

Run recovery verification query:

let appName = "func-lab-mi-auth";
requests
| where timestamp > ago(30m)
| where cloud_RoleName =~ appName
| where operation_Name in ("Functions.http_blob_read", "Functions.http_blob_write", "Functions.scheduled_cleanup")
| summarize
    Invocations=count(),
    Failures=countif(success == false),
    FailureRatePercent=round(100.0 * countif(success == false) / count(), 2),
    SuccessRatePercent=round(100.0 * countif(success == true) / count(), 2),
    P95Ms=percentile(duration, 95)
  by operation_Name
| order by operation_Name asc

Example output:

Confirm recovery signatures in traces:

[2026-04-04T12:24:31Z] Storage probe succeeded (70ms, 4 containers)
[2026-04-04T12:23:57Z] Executed 'Functions.scheduled_cleanup' (Succeeded, Duration=34ms)

3.7 Triage decision flow¶

flowchart TD
    A[Symptom: blob operations failing or timer not running] --> B{403 AuthorizationPermissionMismatch present?}
    B -->|No| C[Investigate non-auth causes: DNS, private endpoint, app bugs]
    B -->|Yes| D{Function App identity enabled?}
    D -->|No| E[Enable system-assigned identity and redeploy]
    D -->|Yes| F{Required role present on correct scope?}
    F -->|No| G[Create role assignment Storage Blob Data Contributor]
    F -->|Yes| H{Propagation window elapsed 10+ minutes?}
    H -->|No| I[Wait and re-test]
    H -->|Yes| J{Host unhealthy storage or listener errors persist?}
    J -->|No| K[Issue resolved]
    J -->|Yes| L["Correlate with network/DNS/access restrictions"]

3.8 Clean up¶

az group delete \
  --name "$RG" \
  --yes \
  --no-wait

4) Experiment Log¶

This section records representative artifacts and timestamped findings from one complete run.

Artifact inventory¶

Baseline evidence¶

Incident observations¶

Core finding¶

The observed incident chain supports a strict authorization-failure model: the identity object remained present and valid, but the missing blob data-plane role caused deterministic 403 dependency failures. Those failures aligned with listener initialization errors and host unhealthy storage signals. Restoring the exact missing role and waiting for RBAC propagation resolved all symptoms without code changes.

Verdict¶

Expected Evidence¶

Before Trigger¶

During Incident¶

After Recovery¶

Evidence Timeline¶

gantt
    title Managed Identity Auth Failure Evidence Timeline
    dateFormat  YYYY-MM-DD HH:mm:ss
    axisFormat  %H:%M:%S

    section Baseline
    Host healthy + blob success               :done, b1, 2026-04-04 12:12:00, 3m

    section Trigger
    Remove Blob Data Contributor role         :done, t1, 2026-04-04 12:14:58, 1m
    Restart Function App                      :done, t2, 2026-04-04 12:15:12, 1m

    section Incident
    Listener unable to start                  :crit, i1, 2026-04-04 12:15:42, 7m
    AuthorizationPermissionMismatch spikes    :crit, i2, 2026-04-04 12:17:54, 6m
    Storage subsystem unhealthy               :crit, i3, 2026-04-04 12:22:42, 2m

    section Recovery
    Restore Blob Data Contributor role        :done, r1, 2026-04-04 12:23:20, 1m
    Storage probe success + timer resume      :done, r2, 2026-04-04 12:24:31, 2m

Evidence Chain: Why This Proves the Hypothesis¶

The hypothesis requires more than one isolated signal. It requires a coherent chain: 1. A concrete authorization change event (role deletion) is visible in Activity Log. 2. Soon after restart, blob dependencies fail with 403 and AuthorizationPermissionMismatch. 3. In the same window, listener startup for Functions.scheduled_cleanup fails. 4. Host emits explicit unhealthy storage state. 5. Role restoration event appears, followed by successful probe and resumed timer execution.

This sequence satisfies both causality and reversibility. Causality is established by temporal ordering from change to failure. Reversibility is established by targeted fix (role restore) followed by recovery without unrelated modifications.

If any middle link is missing (for example 403 dependency evidence), the confidence level drops and competing hypotheses (network path failure, runtime crash, extension issue) must be tested.

If recovery occurs before role restoration, the hypothesis is falsified for that run and the lab should be rerun with stricter change control.¶

Related Playbook¶

• Functions Failing with Errors

See Also¶

• Functions Failing with Errors
• First 10 Minutes
• KQL Query Library
• Lab Guides Index

Sources¶

• Use managed identities in Azure Functions
• Configure identity-based connections in Azure Functions
• Assign Azure roles using Azure CLI
• Azure Blob Storage authorization with Microsoft Entra ID
• Monitor Azure Functions
• Azure Monitor activity log