Lab: Intermittent 5xx Under Load¶

This lab reproduces intermittent request failures caused by sync worker starvation on Azure App Service Linux.

The app intentionally mixes slow and fast endpoints:

• /slow sleeps 5-15 seconds.
• /fast returns immediately.

When a burst of concurrent slow requests occupies the limited Gunicorn sync workers, fast requests queue behind them and start timing out.

graph TD
    A[Deploy B1 Linux app with 2 sync workers] --> B[Launch 20 concurrent /slow requests]
    B --> C[Immediately call /fast 10 times]
    C --> D[Observe queued fast requests]
    D --> E[Collect trigger CSV + /diag/stats]
    E --> F[Query AppServiceHTTPLogs for 200 vs 499 pattern]
    F --> G[Confirm worker starvation causal chain]

This guide is designed to diagnose intermittent 5xx-like customer symptoms where the root cause is worker pool saturation and request queueing rather than random platform instability, using a B1 Linux Python 3.11 app with Gunicorn sync, --workers 2, --timeout 30, a trigger of 20 concurrent /slow requests followed by 10 /fast requests, and sanitized artifacts from a real run.

Lab Metadata¶

1) Background¶

1.1 Mechanism overview¶

This failure mode is a queueing problem:

1. A small sync worker pool accepts work.
2. Long-running requests occupy workers.
3. New requests cannot execute until a worker is free.
4. Waiting clients hit timeout thresholds.
5. Logs show a mixed pattern of successes and timeouts.

1.2 App and process model in this lab¶

From baseline/app-config.json:

With sync workers, a worker handles one request at a time. If two workers are blocked on /slow, /fast waits in queue.

1.3 Request starvation diagram¶

flowchart TD
    A[20 concurrent /slow requests] --> B[Worker 1 busy]
    A --> C[Worker 2 busy]
    D[/fast requests arrive/] --> E[Queue behind /slow]
    E --> F{Client timeout reached?}
    F -->|Yes| G[Client abort / 499 in logs]
    F -->|No| H[Eventually 200]

    style B fill:#ef6c00,color:#fff
    style C fill:#ef6c00,color:#fff
    style G fill:#c62828,color:#fff
    style H fill:#2e7d32,color:#fff

1.4 Endpoint behavior from app code¶

Because /slow blocks worker threads in sync mode, it is an ideal lab endpoint to reproduce starvation.

1.5 Why this resembles intermittent 5xx incidents¶

In real systems, users often report:

• Some requests succeed.
• Some requests fail with gateway/service-unavailable messages.
• Failures are clustered during brief load bursts.

That pattern matches this lab's mixed-success output under constrained workers.

1.6 Internal queueing timeline¶

sequenceDiagram
    participant ClientA as Slow Clients (x20)
    participant ClientB as Fast Clients (x10)
    participant FE as App Service Front End
    participant Gunicorn as Gunicorn sync workers=2
    participant App as Flask app

    ClientA->>FE: Burst /slow
    FE->>Gunicorn: Dispatch first two /slow
    Gunicorn->>App: worker 1 executes /slow
    Gunicorn->>App: worker 2 executes /slow
    ClientB->>FE: /fast requests
    FE->>Gunicorn: Queue /fast (no free worker)
    Note over Gunicorn: queue wait grows
    alt Client timeout reached
        FE-->>ClientB: timeout/aborted (logged as 499)
    else Worker frees up in time
        Gunicorn->>App: execute /fast
        FE-->>ClientB: 200
    end

1.7 Signal map for this failure mode¶

1.8 Distinguishing from other incident classes¶

1.9 Practical troubleshooting takeaway¶

If fast endpoints fail only when slow endpoints are concurrent, investigate worker model first:

1. Worker class (sync vs async/threaded)
2. Worker count
3. Timeout alignment between client and server
4. Concurrency profile of requests

2) Hypothesis¶

2.1 Falsifiable hypothesis statement¶

If we run 20 concurrent /slow requests against a Gunicorn app configured with only 2 sync workers and then send 10 /fast requests, then:

• a significant share of /fast will time out,
• /slow will also partially time out,
• HTTP logs will show mixed 200/499 pattern,

which confirms worker starvation and queueing.

2.2 Causal chain¶

1. SLOW_CONCURRENCY=20 saturates the two sync workers.
2. /fast requests queue behind /slow.
3. Client-side timeout thresholds are reached (15s for fast, 45s for slow in trigger artifacts).
4. Timed-out requests surface as status 000 in trigger output and 499 in App Service HTTP logs.

flowchart TD
    A[Limited sync workers=2] --> B[Concurrent /slow saturates workers]
    B --> C[/fast cannot start promptly]
    C --> D[Queue wait increases]
    D --> E[Client timeout threshold exceeded]
    E --> F[499 in HTTP logs and timeout at caller]

2.3 Proof criteria¶

2.4 Disproof criteria¶

Any of these weaken/disprove starvation hypothesis:

• Fast requests remain consistently low-latency under slow concurrency burst.
• Slow requests do not occupy workers long enough to cause queueing.
• Timeouts occur without corresponding slow burst.
• Failures align with platform restarts instead of queueing behavior.

2.5 Variables¶

Independent variables¶

Dependent variables¶

Controlled conditions¶

2.6 Causal validation matrix¶

2.7 Confounders and boundaries¶

• Trigger script captures one burst window; repeated runs can vary slightly.
• App Service HTTP status 499 indicates client abort/timeouts; external monitoring may categorize this as availability impact akin to intermittent 5xx symptoms.
• Console timeout logs may be absent for short windows even when queueing is evident.

3) Runbook¶

3.1 Prerequisites¶

az version
az bicep version
az account show --output table

3.2 Set standard variables¶

export RG="rg-lab-5xx"
export LOCATION="koreacentral"
export BASE_NAME="lab5xx"
export APP_PACKAGE_PATH="/tmp/intermittent-5xx-app.zip"

3.3 Create resource group¶

az group create --name "$RG" --location "$LOCATION"

3.4 Deploy Bicep (actual lab template path)¶

az deployment group create \
  --resource-group "$RG" \
  --template-file "labs/intermittent-5xx/main.bicep" \
  --parameters baseName="$BASE_NAME" location="$LOCATION"

Extract deployment outputs:

export APP_NAME=$(az deployment group show \
  --resource-group "$RG" \
  --name "main" \
  --query "properties.outputs.appName.value" \
  --output tsv)

export APP_HOSTNAME=$(az deployment group show \
  --resource-group "$RG" \
  --name "main" \
  --query "properties.outputs.defaultHostName.value" \
  --output tsv)

export APP_URL="https://${APP_HOSTNAME}"

3.5 Deploy application package¶

cd "labs/intermittent-5xx/app"
zip --recurse-paths "$APP_PACKAGE_PATH" .

az webapp deploy \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --src-path "$APP_PACKAGE_PATH" \
  --type zip

Restart app for clean test state:

az webapp restart --resource-group "$RG" --name "$APP_NAME"

3.6 Baseline health checks¶

curl --silent "$APP_URL/"
curl --silent "$APP_URL/health"
curl --silent "$APP_URL/fast"
curl --silent "$APP_URL/diag/stats"

Baseline artifact values from this run:

3.7 Trigger starvation workload (actual trigger script)¶

bash "labs/intermittent-5xx/trigger.sh" "$APP_URL"

Script behavior:

1. Launches 20 concurrent /slow requests.
2. Immediately sends 10 /fast requests.
3. Prints per-request status and elapsed time.

3.8 Capture post-trigger diagnostics¶

curl --silent "$APP_URL/diag/stats" > /tmp/intermittent-5xx-diag-stats-after.json
curl --silent "$APP_URL/diag/env" > /tmp/intermittent-5xx-diag-env-after.json

Portal view: Application Insights overview (post-trigger triage anchor)¶

The Application Insights overview is the highest-signal Portal view to open immediately after the trigger script in section 3.7 finishes. The Failed requests tile (10 here) is the primary failure-burst indicator, Server response time separates fast-path latency from slow-endpoint queueing, and Server requests lets you visually correlate the failure burst with the request volume burst from /slow. The Show data for last 1 hour tab keeps the window tight enough that the starvation pattern is not smoothed away by older traffic - widen it to 24 hours only after the immediate-window signal is validated. From here, click the Failed requests tile to drill into the App Insights requests failure telemetry for the same failure window the platform AppServiceHTTPLogs queries in section 3.10 aggregate from a different angle.

3.9 Query Log Analytics¶

Resolve workspace identifiers:

export LOG_WORKSPACE_NAME=$(az deployment group show \
  --resource-group "$RG" \
  --name "main" \
  --query "properties.outputs.logAnalyticsWorkspaceName.value" \
  --output tsv)

export LOG_WORKSPACE_ID=$(az monitor log-analytics workspace show \
  --resource-group "$RG" \
  --workspace-name "$LOG_WORKSPACE_NAME" \
  --query "customerId" \
  --output tsv)

HTTP detail query¶

az monitor log-analytics query \
  --workspace "$LOG_WORKSPACE_ID" \
  --analytics-query "AppServiceHTTPLogs | where TimeGenerated > ago(2h) | project TimeGenerated, CsUriStem, ScStatus, TimeTaken, CsHost | order by TimeGenerated desc" \
  --output json

Status by endpoint query¶

az monitor log-analytics query \
  --workspace "$LOG_WORKSPACE_ID" \
  --analytics-query "AppServiceHTTPLogs | where TimeGenerated > ago(2h) | summarize total=count(), s200=countif(ScStatus==200), s499=countif(ScStatus==499), s5xx=countif(ScStatus>=500) by CsUriStem | order by total desc" \
  --output json

Endpoint latency profile query¶

az monitor log-analytics query \
  --workspace "$LOG_WORKSPACE_ID" \
  --analytics-query "AppServiceHTTPLogs | where TimeGenerated > ago(2h) | summarize avgMs=avg(TimeTaken), p95Ms=percentile(TimeTaken,95), maxMs=max(TimeTaken) by CsUriStem | order by p95Ms desc" \
  --output json

Console and platform query¶

az monitor log-analytics query \
  --workspace "$LOG_WORKSPACE_ID" \
  --analytics-query "AppServiceConsoleLogs | where TimeGenerated > ago(2h) | project TimeGenerated, ResultDescription | order by TimeGenerated desc" \
  --output json

az monitor log-analytics query \
  --workspace "$LOG_WORKSPACE_ID" \
  --analytics-query "AppServicePlatformLogs | where TimeGenerated > ago(2h) | project TimeGenerated, Level, Message | order by TimeGenerated desc" \
  --output json

3.10 KQL snippets for portal troubleshooting¶

Portal view: Logs blade (Log Analytics query editor)¶

The Logs blade is where you paste the KQL snippets below - this capture shows the Application Insights Logs experience (ai-test-20251107), but the workspace-based Log Analytics blade has the same query editor and toolbar. Use the New Query 1 tab, keep Time range: Last 24 hours to cover the intermittent failure window (the lab's 5xx pattern usually spans hours, not minutes), and leave Show: 1000 results so summarized status distributions are not truncated. The Queries hub button in the top-right gives you a saved-query library; once you have run one of the snippets below, save it there so the next on-call engineer can re-run the same triage in one click. The empty Query history pane confirms this is a fresh session - paste the first KQL block to populate it.

AppServiceHTTPLogs
| where TimeGenerated > ago(2h)
| summarize total=count(), s200=countif(ScStatus==200), s499=countif(ScStatus==499), s5xx=countif(ScStatus>=500) by CsUriStem
| order by total desc

AppServiceHTTPLogs
| where TimeGenerated > ago(2h)
| summarize avgMs=avg(TimeTaken), p95Ms=percentile(TimeTaken, 95), maxMs=max(TimeTaken) by CsUriStem, ScStatus
| order by p95Ms desc

AppServiceHTTPLogs
| where TimeGenerated > ago(2h)
| where CsUriStem in ("/slow", "/fast")
| project TimeGenerated, CsUriStem, ScStatus, TimeTaken
| order by TimeGenerated asc

AppServicePlatformLogs
| where TimeGenerated > ago(2h)
| project TimeGenerated, Level, Message
| order by TimeGenerated desc

3.11 Verification checklist¶

• [ ] Trigger executed with 20 /slow and 10 /fast.
• [ ] Slow and fast CSV artifacts captured.
• [ ] /fast timeout ratio increased during /slow burst.
• [ ] KQL HTTP logs show mixed status pattern.
• [ ] /diag/stats shows endpoint counters after run.
• [ ] Console/platform exports checked.

3.12 Common pitfalls¶

3.13 Decision tree for incident triage¶

flowchart TD
    A[Observe intermittent failures] --> B{Are slow endpoints active concurrently?}
    B -->|No| C[Investigate dependency or platform issues]
    B -->|Yes| D{Fast endpoints degrade simultaneously?}
    D -->|No| E[Investigate endpoint-specific logic]
    D -->|Yes| F[Check worker count and worker class]
    F --> G[Validate queueing pattern in TimeTaken and status mix]
    G --> H[Apply concurrency/worker tuning]

4) Experiment Log¶

4.1 Artifact inventory used¶

All values below come from:

labs/intermittent-5xx/artifacts-sanitized/

4.2 Baseline state¶

From baseline/diag-stats.json:

From baseline/app-config.json:

4.3 Trigger CSV evidence: slow endpoint¶

From slow-responses-20260404T053453Z.csv (20 requests):

Representative rows:

4.4 Trigger CSV evidence: fast endpoint¶

From fast-responses-20260404T053453Z.csv (10 requests):

Per-request rows (exact):

4.5 Post-trigger app state¶

From diag-stats-after-20260404T053453Z.json:

Interpretation:

• Endpoint counters reflect requests that reached the app handler.
• Trigger CSV includes client-observed timeouts before some requests completed server-side.

4.6 KQL HTTP aggregate summary¶

From kql-http-20260404T060610Z.json:

Status by endpoint:

4.7 KQL latency profile by endpoint¶

From KQL HTTP export (TimeTaken in milliseconds):

Key signature:

• /fast shows both very low and very high durations in the same run, which is expected when queueing behind occupied sync workers.

4.8 Raw KQL sample rows (sanitized)¶

Representative rows from kql-http-20260404T060610Z.json:

4.9 Console and platform exports¶

Console log export¶

From kql-console-20260404T060610Z.json:

• rows: []

No console entries were captured in this window.

Platform log export¶

From kql-platform-20260404T060610Z.json:

• Lifecycle informational events (site start, warmup probe success, container running).
• No explicit failure/restart entry in sampled period.

Sample rows:

4.10 Hypothesis verdict¶

Result: Supported¶

Why supported:

1. Under a 20-request slow burst, slow endpoint timeout ratio reached 55%.
2. Fast endpoint timeout ratio reached 60% despite fast handler logic.
3. KQL HTTP logs show mixed 200/499 pattern with high durations for /fast and /slow.
4. Pattern is strongly consistent with sync worker starvation and queueing.

Not required for support in this run:

• Console timeout text was absent.
• Platform restart event was absent.

These are optional corroborators, not mandatory if timing/status pattern already proves queueing.

4.11 Recommendations¶

1. Increase Gunicorn worker capacity or move to async/threaded worker model for mixed latency workloads.
2. Isolate slow operations from front-door request path (queue, background job, or dedicated endpoint plan).
3. Align client timeout and server timeout intentionally (avoid accidental mismatch where client cancels first).
4. Monitor endpoint-specific latency and status distribution, not only aggregate app success rate.
5. Use synthetic mixed-load probes (/slow + /fast) in pre-production to detect starvation early.

4.12 Suggested follow-up experiments¶

Expected Evidence¶

This section defines what you SHOULD observe at each phase of the lab. Use it to validate your investigation is on track.

Before Trigger (Baseline)¶

During Incident¶

After Recovery¶

Evidence Timeline¶

graph TD
    A[Baseline Capture] --> B[Trigger Fault]
    B --> C[During: Collect Evidence]
    C --> D[After: Compare to Baseline]
    D --> E[Verdict: Confirmed/Falsified]

Evidence Chain: Why This Proves the Hypothesis¶

Clean Up¶

az group delete --name "$RG" --yes --no-wait

Related Playbook¶

• Intermittent 5xx Under Load

See Also¶

• Memory Pressure and Worker Degradation Lab
• First 10 Minutes: App Service Troubleshooting

Sources¶

• Troubleshoot HTTP 502 and 503 in Azure App Service
• Monitor Azure App Service
• Enable diagnostics logging in Azure App Service
• Configure Python apps in Azure App Service
• Azure App Service plan overview
• Quickstart: Build Bicep templates