Lab: Memory Pressure and Worker Degradation¶

This lab reproduces memory pressure behavior on Azure App Service (Linux, B1) using a Python Flask app that intentionally grows heap allocations (/leak) and triggers CPU/memory-intensive work (/heavy).

The goal is not only to "cause load," but to understand the full chain from user traffic to Gunicorn process behavior to Linux memory reclaim and App Service platform signals.

graph TD
    A[Deploy B1 Linux App Service] --> B[Deploy Flask memory-lab app]
    B --> C[Trigger /leak 100x]
    C --> D[Trigger /heavy burst]
    D --> E[Capture /diag/stats and /diag/proc]
    E --> F[Query AppServiceHTTPLogs AppServiceConsoleLogs AppServicePlatformLogs]
    F --> G[Compare against falsifiable hypothesis]

Lab Metadata¶

1) Background¶

1.1 Why this failure mode happens¶

Memory-pressure incidents in App Service are usually multi-layer issues:

1. Application layer allocates memory over time (intentional leak in this lab).
2. Worker process layer (Gunicorn) competes for RSS across multiple workers.
3. Linux kernel layer starts reclaim/scanning under pressure.
4. Platform layer may still report mostly healthy requests until pressure crosses a tipping point.

In short: rising memory pressure can be present long before obvious 5xx appears.

1.2 App Service Linux execution model in this lab¶

From the deployed app configuration artifact:

Because B1 is constrained and this app uses --workers=4, memory fragmentation and worker-level RSS contention become visible faster under leak growth.

1.3 Failure progression model¶

flowchart TD
    A[/leak requests append large lists/] --> B[Per-worker RSS rises]
    B --> C[MemAvailable drops]
    C --> D[Kernel reclaim intensifies]
    D --> E[pgscan and allocstall counters rise]
    E --> F[Latency volatility increases]
    F --> G{Further pressure?}
    G -->|Yes| H[Worker timeout or recycle risk]
    G -->|No| I[Temporary steady state]

    style A fill:#1565c0,color:#fff
    style C fill:#ef6c00,color:#fff
    style D fill:#c62828,color:#fff
    style H fill:#b71c1c,color:#fff
    style I fill:#2e7d32,color:#fff

1.4 Request path and where memory accumulates¶

sequenceDiagram
    participant Client
    participant FrontEnd as App Service Front End
    participant Worker as Linux Worker
    participant Gunicorn as Gunicorn Master/Workers
    participant Flask as Flask App
    participant Kernel as Linux Kernel

    Client->>FrontEnd: HTTPS request /leak
    FrontEnd->>Worker: Route request
    Worker->>Gunicorn: Dispatch to sync worker
    Gunicorn->>Flask: Execute leak() (allocate list)
    Flask-->>Gunicorn: JSON 200 with block count
    Gunicorn-->>Worker: Response
    Worker-->>FrontEnd: Response
    FrontEnd-->>Client: 200 OK
    Note over Kernel: MemAvailable drops, reclaim activity rises

1.5 Why /leak + /heavy is a useful pair¶

• /leak stresses persistent memory growth (retained list in LEAK_BUCKET).
• /heavy stresses CPU + transient allocations (500k list creation + sort).
• Combined workload reveals whether the system is:
  • still handling requests,
  • but already paying increasing reclaim cost.

1.6 Signals used in this lab¶

1.7 Linux counters that matter most here¶

From the app code (/diag/proc) and artifacts:

1.8 Architectural context diagram¶

graph TD
    subgraph Azure
        FE[App Service Front End]
        W[Linux Worker VM]
        LA[Log Analytics Workspace]
    end

    subgraph SiteContainer[Web App Site Container]
        G[Gunicorn workers=4 timeout=120]
        F[Flask routes /leak /heavy /diag/*]
    end

    FE --> W
    W --> SiteContainer
    F --> G
    SiteContainer --> LA

1.9 Practical troubleshooting interpretation¶

Memory-pressure troubleshooting should not depend on a single symptom (for example, only 5xx rate).

Use a stacked interpretation:

1. Pressure trend (MemAvailable, SwapFree, reclaim counters)
2. Latency trend (TimeTaken, trigger timings)
3. Error trend (5xx, 499)
4. Lifecycle trend (restart/recycle events)

This lab gives a full chain for (1) and (2), with a "no 5xx yet" outcome for (3).

2) Hypothesis¶

2.1 Falsifiable hypothesis statement¶

If repeated /leak requests are used to consume memory on a B1 Linux App Service with 4 Gunicorn workers, then:

• MemAvailable will drop materially,
• reclaim/swap counters will increase,
• latency variance will rise,

even if HTTP 5xx does not immediately appear.

2.2 Causal chain¶

1. /leak appends large lists to LEAK_BUCKET.
2. Process RSS grows and available memory shrinks.
3. Kernel reclaim mechanisms intensify (pgscan_*, allocstall, swap churn).
4. /heavy runs under less headroom and produces longer tail latency.
5. Platform may still return mostly HTTP 200 during this intermediate state.

flowchart TD
    A[/leak accumulation/] --> B[Low headroom]
    B --> C[Reclaim and swap activity]
    C --> D[Request latency tail growth]
    D --> E{Observed window}
    E -->|Early| F[No 5xx, risk building]
    E -->|Late| G[Possible timeout/recycle/5xx]

2.3 Proof criteria¶

All criteria below should be met to support the hypothesis:

2.4 Disproof criteria¶

Any one of the following would weaken/disprove this hypothesis:

• MemAvailable stays near baseline despite /leak volume.
• Reclaim counters stay flat while leak count grows.
• Swap does not change at all and no reclaim counters move.
• No measurable latency impact in trigger/KQL while pressure counters remain flat.

2.5 Variables¶

Independent variables (controlled)¶

Dependent variables (measured)¶

Controlled conditions¶

2.6 Confounders and caveats¶

• KQL export windows may include extra baseline requests.
• Concurrent requests can be load-balanced across Gunicorn workers; one worker's leak count does not represent all workers.
• The heavy-responses artifact format is concatenated in one line; status extraction is still possible, and KQL cross-check is used for latency analysis.

3) Runbook¶

3.1 Prerequisite checks¶

Use these commands before deployment.

az version
az bicep version
az account show --output table

Expected checks:

• Azure CLI installed and authenticated
• Bicep available via Azure CLI
• Correct subscription context selected

3.2 Set standard variables¶

Use repository variable conventions.

export RG="rg-lab-memory"
export LOCATION="koreacentral"
export BASE_NAME="labmem"
export APP_PACKAGE_PATH="/tmp/memory-pressure-app.zip"

3.3 Create resource group¶

az group create --name "$RG" --location "$LOCATION"

Example output:

{
  "location": "koreacentral",
  "name": "rg-lab-memory",
  "properties": {
    "provisioningState": "Succeeded"
  }
}

3.4 Deploy the lab infrastructure (actual Bicep path)¶

az deployment group create \
  --resource-group "$RG" \
  --template-file "labs/memory-pressure/main.bicep" \
  --parameters baseName="$BASE_NAME" location="$LOCATION"

Capture outputs:

export APP_NAME=$(az deployment group show \
  --resource-group "$RG" \
  --name "main" \
  --query "properties.outputs.webAppName.value" \
  --output tsv)

export APP_HOSTNAME=$(az deployment group show \
  --resource-group "$RG" \
  --name "main" \
  --query "properties.outputs.webAppDefaultHostName.value" \
  --output tsv)

export APP_URL="https://${APP_HOSTNAME}"

3.5 Package and deploy the lab app code¶

cd "labs/memory-pressure/app"
zip --recurse-paths "$APP_PACKAGE_PATH" .

az webapp deploy \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --src-path "$APP_PACKAGE_PATH" \
  --type zip

Restart app after deployment:

az webapp restart --resource-group "$RG" --name "$APP_NAME"

3.6 Verify baseline endpoints¶

curl --silent "$APP_URL/"
curl --silent "$APP_URL/health"
curl --silent "$APP_URL/diag/stats"
curl --silent "$APP_URL/diag/proc"

Baseline artifact snapshot from this run:

3.7 Confirm App Service runtime configuration¶

az webapp config show \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --output json

Look specifically for:

• linuxFxVersion
• appCommandLine
• alwaysOn
• numberOfWorkers

3.8 Trigger memory pressure (actual trigger script)¶

bash "labs/memory-pressure/trigger.sh" "$APP_URL"

Script behavior:

1. Sends 100 sequential /leak requests.
2. Sends 50 concurrent /heavy requests with max 10 concurrent curl jobs.
3. Prints 5xx failure counts for each phase.

3.9 Mid-run diagnostic capture¶

During the leak phase, capture diagnostics:

curl --silent "$APP_URL/diag/stats" > /tmp/memory-mid-diag-stats.json
curl --silent "$APP_URL/diag/proc" > /tmp/memory-mid-diag-proc.json

After heavy phase:

curl --silent "$APP_URL/diag/stats" > /tmp/memory-post-diag-stats.json
curl --silent "$APP_URL/diag/proc" > /tmp/memory-post-diag-proc.json

3.10 Query Log Analytics workspace¶

Resolve workspace name from deployment output or resource query:

export LOG_WORKSPACE_NAME=$(az deployment group show \
  --resource-group "$RG" \
  --name "main" \
  --query "properties.outputs.logAnalyticsWorkspaceName.value" \
  --output tsv)

export LOG_WORKSPACE_ID=$(az monitor log-analytics workspace show \
  --resource-group "$RG" \
  --workspace-name "$LOG_WORKSPACE_NAME" \
  --query "customerId" \
  --output tsv)

HTTP logs query¶

az monitor log-analytics query \
  --workspace "$LOG_WORKSPACE_ID" \
  --analytics-query "AppServiceHTTPLogs | where TimeGenerated > ago(2h) | where CsHost has 'app-' | project TimeGenerated, CsUriStem, ScStatus, TimeTaken, CsHost | order by TimeGenerated desc" \
  --output json

Console logs query¶

az monitor log-analytics query \
  --workspace "$LOG_WORKSPACE_ID" \
  --analytics-query "AppServiceConsoleLogs | where TimeGenerated > ago(2h) | where ResultDescription has_any ('OutOfMemory','OOM','Killed','WORKER TIMEOUT','memory') | project TimeGenerated, ResultDescription | order by TimeGenerated desc" \
  --output json

Platform logs query¶

az monitor log-analytics query \
  --workspace "$LOG_WORKSPACE_ID" \
  --analytics-query "AppServicePlatformLogs | where TimeGenerated > ago(2h) | project TimeGenerated, Level, Message | order by TimeGenerated desc" \
  --output json

3.11 KQL query snippets for portal use¶

Portal view: Diagnose and solve problems blade (memory triage entry point)¶

The Diagnose and solve problems blade is the Portal entry point that surfaces App Service's pre-built performance detectors before you reach the metric-by-metric exploration in the Metrics blade embed shown next - the Web App Slow link under Popular troubleshooting tools is a direct quick-access path to a curated detector that App Service correlates from platform-side signals (response times, worker recycles, memory percentages) without requiring you to write the KQL queries shown in section 3.10. The Risk alerts card at the top is the fastest signal that App Service has already flagged your app: in this capture, Availability 2 Critical means the platform's continuous diagnostic engine has detected availability degradation, which often correlates with the worker recycle risk path shown in section 1.3's failure progression diagram. For deeper categorical drill-downs, the Availability and Performance tile leads to detectors organized by symptom (Application Logs, App Down Workflow, Web App Down), and the Diagnostic Tools tile exposes Auto-Heal where you can verify whether memory-based recycle rules are active and matching the worker-recycle predictions in section 1.3. This blade sits one layer above the Metrics blade shown next: use this hub first to confirm the platform agrees a problem exists, then drill into specific MemAvailable and worker-RSS metrics via the chart configuration in the next H4.

Portal view: Metrics blade (memory pressure anchor)¶

The Metrics blade is the Portal entry point for the memory-pressure KQL queries below. Click Add metric in the chart command bar and choose a memory or response-time signal from the App Service standard metric namespace shown in the configuration row - this surfaces the same dimensions the KQL snippets aggregate from AppServiceHTTPLogs. The Drill into Logs button on the same toolbar pivots from a metric spike directly into a Log Analytics query, which is faster than typing the queries below from scratch. The chart in this capture is empty (no metric selected yet), and the Sample data help cards Filter + Split, Plot multiple metrics, and Build custom dashboards walk you through the standard setup for ongoing memory-pressure monitoring.

AppServiceHTTPLogs
| where TimeGenerated > ago(2h)
| summarize total=count(), errors5xx=countif(ScStatus >= 500) by CsUriStem
| order by total desc

AppServiceHTTPLogs
| where TimeGenerated > ago(2h)
| summarize avgMs=avg(TimeTaken), p95Ms=percentile(TimeTaken, 95), maxMs=max(TimeTaken) by CsUriStem, ScStatus
| order by p95Ms desc

AppServiceConsoleLogs
| where TimeGenerated > ago(2h)
| where ResultDescription has_any ("WORKER TIMEOUT", "OutOfMemory", "OOM", "Killed")
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc

AppServicePlatformLogs
| where TimeGenerated > ago(2h)
| project TimeGenerated, Level, Message
| order by TimeGenerated desc

3.12 Verification checklist¶

Use this checklist to determine whether the lab produced useful memory-pressure evidence:

• [ ] Leak endpoint count increased significantly.
• [ ] MemAvailable dropped materially from baseline.
• [ ] pgscan_kswapd and/or pgscan_direct increased.
• [ ] Swap counters changed (SwapFree, pswp*).
• [ ] KQL HTTP data shows endpoint timing under pressure.
• [ ] Console/platform logs captured (even if no errors).

3.13 Common execution pitfalls¶

3.14 Runbook decision tree¶

flowchart TD
    A[Run trigger.sh] --> B{Did /leak and /heavy return 200 mostly?}
    B -->|No| C[Investigate immediate app/runtime issue]
    B -->|Yes| D[Collect /diag/proc snapshots]
    D --> E{Did MemAvailable drop and vmstat rise?}
    E -->|No| F[Pressure not reproduced, increase trigger or validate deploy]
    E -->|Yes| G[Query HTTP/Console/Platform logs]
    G --> H[Record experiment log and verdict]

4) Experiment Log¶

4.1 Artifact inventory used¶

All values below are taken directly from sanitized artifacts:

labs/memory-pressure/artifacts-sanitized/

4.2 Baseline measurements¶

From baseline/diag-proc.json and baseline/diag-stats.json:

4.3 Mid-leak snapshot¶

From trigger/diag-proc-midleak-20260404T053438Z.json and trigger/diag-stats-midleak-20260404T053438Z.json:

4.4 Post-heavy snapshot¶

From trigger/diag-proc-postheavy-20260404T053438Z.json and trigger/diag-stats-postheavy-20260404T053438Z.json:

4.5 Delta analysis: baseline → mid-leak¶

4.6 Delta analysis: baseline → post-heavy¶

4.7 Trigger response evidence¶

Leak phase CSV (100 rows)¶

From leak-responses-20260404T053438Z.csv:

Sample rows:

Heavy phase evidence¶

The artifact heavy-responses-20260404T053438Z.csv is captured as a concatenated one-line stream. Status entries remain 200 in parsed segments, and KQL HTTP export is used as primary timing source.

From KQL HTTP export (/heavy endpoint rows):

4.8 KQL HTTP summary (real export)¶

From kql-http-20260404T060610Z.json:

Total rows: 314, all status 200.

4.9 KQL console and platform evidence¶

Console logs¶

From kql-console-20260404T060610Z.json:

• rows: []
• No WORKER TIMEOUT, OOM, or kill signature during this window.

Platform logs¶

From kql-platform-20260404T060610Z.json:

• Site/container startup lifecycle entries present.
• No restart/failure event in observed window.

Representative platform rows:

4.10 Raw KQL output sample (sanitized)¶

Representative rows from HTTP export:

4.11 Hypothesis verdict¶

Result: Supported (pre-failure stage)¶

Evidence that supports hypothesis:

1. MemAvailable dropped from 523896 kB to 44760 kB mid-leak.
2. pgscan_kswapd and pgscan_direct rose substantially.
3. SwapFree dropped by ~1.4 GB from baseline to post-heavy.
4. PSI memory pressure moved from near-zero avg10 to very high values.
5. /heavy latency tail in KQL reached 15+ seconds while status remained 200.

Evidence against immediate outage:

• No 5xx in trigger/KQL window.
• No console timeout/OOM records.
• No platform restart event in sampled interval.

Interpretation:

This run captured a high-risk intermediate state where memory pressure and reclaim were demonstrably active, but failure thresholds were not yet crossed.

4.12 Recommendations¶

1. Treat low MemAvailable + rising pgscan_direct as an early warning signal.
2. Reduce worker count or cap per-request memory growth before user-visible errors appear.
3. Add periodic /diag/proc telemetry in non-production labs for trend baselining.
4. Correlate HTTP latency tail with reclaim counters, not only with 5xx rate.
5. If this pattern appears in production, scale up plan memory or split workloads.

4.13 Reproducibility notes¶

• Trigger volume and timing are deterministic in script structure but still subject to worker scheduling variance.
• HTTP log windows can include extra baseline requests; always correlate by endpoint and timestamp.
• For strict phase accounting, export logs immediately after each phase boundary.

Expected Evidence¶

This section defines what you SHOULD observe at each phase of the lab. Use it to validate your investigation is on track.

Before Trigger (Baseline)¶

During Incident¶

After Recovery¶

Evidence Timeline¶

graph TD
    A[Baseline Capture] --> B[Trigger Fault]
    B --> C[During: Collect Evidence]
    C --> D[After: Compare to Baseline]
    D --> E[Verdict: Confirmed/Falsified]

Evidence Chain: Why This Proves the Hypothesis¶

Clean Up¶

az group delete --name "$RG" --yes --no-wait

Related Playbook¶

• Memory Pressure and Worker Degradation

See Also¶

• Intermittent 5xx Under Load Lab
• First 10 Minutes: App Service Troubleshooting

Sources¶

• Monitor Azure App Service
• Azure App Service diagnostics logs
• Troubleshoot high memory for an app in Azure App Service
• Overview of Azure App Service plan
• Configure Python on Azure App Service
• Quickstart: Create Bicep files with Visual Studio Code