Lab Guide: Slot Swap Config Drift (Sticky vs Non-Sticky Settings)¶

This Level 3 lab guide reproduces a slot swap on Azure App Service Linux and proves, using real artifacts, how configuration behaves across swap boundaries. The lab shows why teams often perceive "drift" after a healthy swap.

Lab Metadata¶

1) Background¶

Slot swap incidents are often not platform failures. They are frequently interpretation failures.

To troubleshoot accurately, you must separate:

1. Slot routing behavior.
2. Sticky/non-sticky config behavior.
3. Process restart and warm-up lifecycle behavior.

1.1 Deployment slots are separate runtime surfaces¶

Each slot is a separate runtime endpoint with its own hostname and runtime lifecycle.

flowchart TD
    subgraph AppService[Single App Service App]
        P[Production Slot\nHostname: app.azurewebsites.net]
        S[Staging Slot\nHostname: app-staging.azurewebsites.net]
    end
    U[Users] --> P
    O[Operators/Testers] --> S

Even when slots share plan resources, they maintain independent runtime state and per-slot app setting context.

1.2 Swap behavior model¶

Swap can be thought of as role exchange with warm-up safeguards, not just a file copy.

sequenceDiagram
    participant Ops as Operator
    participant Platform as App Service Platform
    participant Stage as Staging Slot
    participant Prod as Production Slot

    Ops->>Platform: Start swap (staging -> production)
    Platform->>Stage: Validate and warm source runtime
    Stage-->>Platform: Warm-up probe succeeded
    Platform->>Prod: Shift production role
    Platform->>Stage: Shift staging role
    Platform-->>Ops: Swap completed

1.3 Sticky and non-sticky settings¶

The critical rule set:

This lab intentionally mixes one of each to produce a clear differential.

1.4 Why this creates perceived "drift"¶

After swap, operators may expect all values to follow code.

If that mental model is wrong:

• Non-sticky setting movement appears surprising.
• Sticky setting stability appears inconsistent.

In reality, both are correct outcomes.

1.5 Template-level implementation in this lab¶

From main.bicep:

• Production and staging are both configured with Python 3.11 runtime.
• FEATURE_FLAG differs by slot (v1 vs v2) and is non-sticky.
• DB_CONNECTION_STRING differs by slot and is marked sticky via slotConfigNames.

slotConfigNames excerpt:

resource slotConfigNames 'Microsoft.Web/sites/config@2023-12-01' = {
  name: '${webApp.name}/slotConfigNames'
  properties: {
    appSettingNames: [
      'DB_CONNECTION_STRING'
    ]
    connectionStringNames: []
  }
}

1.6 Expected before/after state map¶

flowchart TB
    subgraph BeforeSwap[Before Swap]
        P1[Production\nFEATURE_FLAG=v1\nDB_CONNECTION_STRING=prod-server]
        S1[Staging\nFEATURE_FLAG=v2\nDB_CONNECTION_STRING=staging-server]
    end

    subgraph AfterSwap[After Swap]
        P2[Production\nFEATURE_FLAG=v2\nDB_CONNECTION_STRING=prod-server]
        S2[Staging\nFEATURE_FLAG=v1\nDB_CONNECTION_STRING=staging-server]
    end

    P1 --> S2
    S1 --> P2

1.7 Warm-up lifecycle context¶

Swap operations are tightly coupled with startup/warm-up behavior:

• Container and app startup events occur around swap windows.
• Probe success transitions appear in AppServicePlatformLogs.
• Gunicorn startup appears in AppServiceConsoleLogs.

This guide includes both config-state proof and lifecycle proof.

1.8 Why this lab matters¶

For production release safety, teams must be able to answer:

1. Did the swap perform correctly?
2. Did sticky settings stay where intended?
3. Did non-sticky settings move as expected?
4. Were restarts/warm-up transitions healthy?

This lab provides an evidence-driven method to answer all four.

2) Hypothesis¶

2.1 Formal hypothesis statement¶

When performing a slot swap, non-sticky app settings move with the swapped app context while slot settings (deployment slot settings) remain with the destination slot, creating apparent config drift only for operators who expect all settings to swap uniformly.

2.2 Causal chain¶

flowchart TD
    A[Pre-swap config divergence by slot] --> B[Swap staging to production]
    B --> C[Non-sticky values exchange]
    B --> D[Sticky values remain slot-bound]
    C --> E[Production FEATURE_FLAG changes]
    D --> F[Production DB setting remains]
    E --> G[Perceived drift if sticky model unknown]
    F --> G

2.3 Proof criteria¶

The hypothesis is supported only if all conditions are met:

1. Pre-swap production and staging values are different for both targeted settings.
2. Post-swap FEATURE_FLAG values are exchanged across slots.
3. Post-swap DB_CONNECTION_STRING values are unchanged per physical slot.
4. App settings metadata confirms DB_CONNECTION_STRING is sticky (slotSetting=true).
5. Lifecycle logs show restart/warm-up progression around swap time.

2.4 Disproof criteria¶

Any of these disproves the hypothesis:

• FEATURE_FLAG does not exchange across slots after confirmed swap.
• DB_CONNECTION_STRING exchanges despite sticky metadata.
• Sticky metadata absent or inconsistent in pre-swap app settings.
• No lifecycle evidence near swap operation window.

2.5 Controlled and dependent variables¶

2.6 Anticipated data pattern¶

3) Runbook¶

This runbook is designed for deterministic replay. Commands use long flags only.

3.1 Prerequisites¶

3.2 Environment setup¶

RG="rg-lab-slotswap"
LOCATION="koreacentral"
BASE_NAME="labswap"

Convention note:

• In command usage, this guide references $RG, $APP_NAME, etc.
• In declarations, variable names are written without $ (shell syntax).

3.3 Deploy lab infrastructure¶

az group create \
  --name "$RG" \
  --location "$LOCATION"

az deployment group create \
  --resource-group "$RG" \
  --template-file "labs/slot-swap-config-drift/main.bicep" \
  --parameters "baseName=$BASE_NAME"

3.4 Resolve app and slot URLs¶

APP_NAME=$(az webapp list \
  --resource-group "$RG" \
  --query "[0].name" \
  --output tsv)

PRODUCTION_HOST=$(az webapp show \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --query "defaultHostName" \
  --output tsv)

STAGING_HOST="$APP_NAME-staging.azurewebsites.net"
PRODUCTION_URL="https://$PRODUCTION_HOST"
STAGING_URL="https://$STAGING_HOST"

3.5 Validate slot settings metadata¶

Inspect production settings:

az webapp config appsettings list \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --query "[?name=='SCM_DO_BUILD_DURING_DEPLOYMENT' || name=='DB_CONNECTION_STRING' || name=='FEATURE_FLAG'].{name:name,value:value,slotSetting:slotSetting}"

Inspect staging settings:

az webapp config appsettings list \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --slot "staging" \
  --query "[?name=='SCM_DO_BUILD_DURING_DEPLOYMENT' || name=='DB_CONNECTION_STRING' || name=='FEATURE_FLAG'].{name:name,value:value,slotSetting:slotSetting}"

Expected pre-swap model:

Portal view: Configuration blade (slot-aware settings entry point)¶

The Configuration blade is the Portal control plane for the same settings the CLI commands above expose. The Application settings sub-tab (not shown in this capture, which focuses on General settings) is where the per-row Deployment slot setting checkbox appears - that checkbox is the visual representation of the slotSetting:true flag in the expected pre-swap table above (DB_CONNECTION_STRING row). General settings (active here) governs platform behavior like Always on and Session affinity and follows different swap rules than App Settings; consult section 1.3 before changing anything here mid-incident. Use this blade to spot-check the sticky/non-sticky shape of your settings without parsing CLI JSON, then cross-reference the post-swap behavior with the KQL evidence in section 3.10.

3.6 Capture pre-swap runtime evidence¶

curl --silent --show-error "$PRODUCTION_URL/config"
curl --silent --show-error "$STAGING_URL/config"

curl --silent --show-error "$PRODUCTION_URL/diag/slots"
curl --silent --show-error "$STAGING_URL/diag/slots"

curl --silent --show-error "$PRODUCTION_URL/diag/stats"
curl --silent --show-error "$PRODUCTION_URL/diag/env"

Representative pre-swap output:

{"DB_CONNECTION_STRING":"prod-server.database.windows.net","FEATURE_FLAG":"v1","PROCESS_START_UTC":"2026-04-04T05:13:59.196136+00:00","WEBSITE_SLOT_NAME":"Production"}

{"DB_CONNECTION_STRING":"staging-server.database.windows.net","FEATURE_FLAG":"v2","PROCESS_START_UTC":"2026-04-04T05:34:02.102318+00:00","WEBSITE_SLOT_NAME":"Production"}

3.7 Execute swap¶

az webapp deployment slot swap \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --slot "staging" \
  --target-slot "production"

Wait for immediate post-swap stabilization:

sleep 20

3.8 Capture post-swap runtime evidence¶

curl --silent --show-error "$PRODUCTION_URL/config"
curl --silent --show-error "$STAGING_URL/config"

curl --silent --show-error "$PRODUCTION_URL/diag/slots"
curl --silent --show-error "$PRODUCTION_URL/diag/stats"

Representative post-swap output:

{"DB_CONNECTION_STRING":"prod-server.database.windows.net","FEATURE_FLAG":"v2","PROCESS_START_UTC":"2026-04-04T05:46:56.764695+00:00","WEBSITE_SLOT_NAME":"Production"}

{"DB_CONNECTION_STRING":"staging-server.database.windows.net","FEATURE_FLAG":"v1","PROCESS_START_UTC":"2026-04-04T05:49:05.541050+00:00","WEBSITE_SLOT_NAME":"Production"}

3.9 Run full trigger script (optional)¶

bash "labs/slot-swap-config-drift/trigger.sh" "$RG" "$APP_NAME"

The script does:

1. Package deployment to both slots.
2. Pre-swap config capture.
3. Swap execution.
4. Post-swap capture.
5. Behavior summary booleans:
   • feature swapped
   • sticky DB remained
   • process restart observed

3.10 KQL runbook queries¶

Portal view: Activity log (swap audit trail)¶

The Activity log blade is the Portal anchor for swap-drift investigations - it gives you the audit trail of every ARM operation against the app, with timestamps, initiator, and status, while the KQL queries below give you the application-side evidence. This capture already shows the right scoping posture: the Timespan: Last 6 hours and Resource: app-test-20251107 chips are applied, narrowing the table to 11 items. of Succeeded operations - the same scope you want for a swap-window audit. The Event initiated by column (here user@example.com) tells you which principal triggered each operation, which is useful when correlating against the sticky-vs-non-sticky settings table in section 2. Use the KQL block below to verify the application-side restart and request behavior aligns with the audited operations.

3.10.1 HTTP request evidence¶

AppServiceHTTPLogs
| where TimeGenerated > ago(2h)
| where CsHost has "app-labswap"
| project TimeGenerated, CsUriStem, ScStatus, TimeTaken, CsHost
| order by TimeGenerated desc

3.10.2 Console startup evidence¶

AppServiceConsoleLogs
| where TimeGenerated > ago(2h)
| where ResultDescription has_any ("Starting gunicorn", "Booting worker", "App path is set", "Oryx", "Listening at")
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc

3.10.3 Platform lifecycle evidence¶

AppServicePlatformLogs
| where TimeGenerated > ago(2h)
| where Message has_any ("WarmUpProbeSucceeded", "Site startup probe succeeded", "Site started", "StoppingSite", "CreatingContainer", "PullingImage")
| project TimeGenerated, Level, Message
| order by TimeGenerated desc

3.11 Real output snippets from this lab¶

Platform warm-up progression:

2026-04-04T05:49:05.8804667Z  State: Starting, Action: WarmUpProbeSucceeded ... Site startup probe succeeded after 55.2013831 seconds.
2026-04-04T05:49:06.6769441Z  Site started.

Console startup progression:

[2026-04-04 05:49:04 +0000] [1895] [INFO] Starting gunicorn 25.3.0
[2026-04-04 05:49:04 +0000] [1896] [INFO] Booting worker with pid: 1896
[2026-04-04 05:49:04 +0000] [1897] [INFO] Booting worker with pid: 1897

HTTP verification calls:

2026-04-04T05:51:21.610129Z  /config      200   47
2026-04-04T05:51:22.477189Z  /config      200   14
2026-04-04T05:51:24.143418Z  /diag/slots  200   11

3.12 Operator checklist¶

3.13 Decision tree for incident responders¶

flowchart TD
    A[Post-swap config surprise] --> B{Is setting sticky?}
    B -->|Yes| C[Expected to remain with slot]
    B -->|No| D[Expected to move with swapped context]
    C --> E{Observed unchanged?}
    D --> F{Observed exchanged?}
    E -->|Yes| G[Behavior correct]
    E -->|No| H[Investigate setting metadata]
    F -->|Yes| G
    F -->|No| H
    G --> I[Check warm-up and HTTP health]
    H --> J[Review slotConfigNames and deployment pipeline]

4) Experiment Log¶

All data below is sourced from:

labs/slot-swap-config-drift/artifacts-sanitized/

4.1 Artifact inventory¶

4.2 Baseline evidence¶

4.2.1 Baseline config snapshots¶

Production baseline:

{"DB_CONNECTION_STRING":"prod-server.database.windows.net","FEATURE_FLAG":"v1","PROCESS_START_UTC":"2026-04-04T05:13:59.196136+00:00","WEBSITE_SLOT_NAME":"Production"}

Staging baseline:

{"DB_CONNECTION_STRING":"staging-server.database.windows.net","FEATURE_FLAG":"v2","PROCESS_START_UTC":"2026-04-04T05:34:01.911025+00:00","WEBSITE_SLOT_NAME":"Production"}

4.2.2 Baseline app setting metadata¶

Production (baseline/app-settings-prod.json):

Staging (baseline/app-settings-staging.json):

4.2.3 Baseline diagnostics¶

baseline/diag-stats-prod.json:

{"endpoint_counters":{"<unknown>":1,"diag_env":1,"diag_slots":1,"diag_stats":1,"index":1},"last_config_snapshot":null,"pid":1898,"process_start_time":"2026-04-04T05:13:59.135627+00:00","request_count":6,"uptime_seconds":1212.651}

baseline/diag-env-prod.json highlights:

4.3 Trigger pre-swap captures¶

From trigger artifacts:

4.4 Trigger post-swap captures¶

From trigger artifacts:

4.5 Assertion table: expected vs observed¶

4.6 Programmatic summary checks (artifact-derived)¶

These checks produce a deterministic proof of sticky/non-sticky mechanics.

4.7 Process restart evidence¶

Using timestamp differences from before/after JSON:

This indicates new process generations became active during swap lifecycle.

4.8 /diag/slots snapshots and config hash evolution¶

Pre-swap production (baseline/diag-slots-prod.json):

{"config_hash":"027d574d88e7b2722dc7da98bc5d1bb337a75f1331bb795d2a83aba6ce291fd7","current_config":{"DB_CONNECTION_STRING":"prod-server.database.windows.net","FEATURE_FLAG":"v1","PROCESS_START_UTC":"2026-04-04T05:13:59.196136+00:00","WEBSITE_SLOT_NAME":"Production"}}

Pre-swap staging (baseline/diag-slots-staging.json):

{"config_hash":"7f5ed03ca7b0fdd301f27ed036155631ca62ffce8a18b4fe5bce0235f02216bc","current_config":{"DB_CONNECTION_STRING":"staging-server.database.windows.net","FEATURE_FLAG":"v2","PROCESS_START_UTC":"2026-04-04T05:34:02.102318+00:00","WEBSITE_SLOT_NAME":"Production"}}

Post-swap sample (trigger/diag-slots-postswap-20260404T055128Z.json):

{"config_hash":"4069415d1d95ca3d38f723e476f8956726634a0b7fddf338499168a7d7198b3d","current_config":{"DB_CONNECTION_STRING":"prod-server.database.windows.net","FEATURE_FLAG":"v2","PROCESS_START_UTC":"2026-04-04T05:46:57.025552+00:00","WEBSITE_SLOT_NAME":"Production"}}

4.9 KQL export volume summary¶

4.10 HTTP evidence details¶

Representative rows from HTTP export:

Observation: both slots remained reachable and healthy during/after swap evidence window.

4.11 Console evidence details¶

Representative console lines:

Observation: process lifecycle aligns with swap-time startup sequence.

4.12 Platform lifecycle evidence details¶

Representative platform lines:

These lines validate that the swap involved observable start/stop/warm-up transitions.

4.13 Hypothesis evaluation matrix¶

Final hypothesis verdict: Supported.

4.14 Key finding (required conclusion)¶

4.15 Practical lessons for release engineering¶

1. Treat sticky/non-sticky classification as release-critical metadata.
2. Include pre/post slot snapshots in every production swap runbook.
3. Align incident response language with slot semantics ("expected movement" vs "drift").
4. Use KQL lifecycle evidence to distinguish swap mechanics from app defects.

4.16 Recommended production controls¶

4.17 Pre-swap checklist template¶

4.18 Post-swap checklist template¶

4.19 Reproducibility and data integrity notes¶

• Every numeric and string value in this section is sourced from checked-in sanitized artifacts.
• Hostnames and subscription identifiers are redacted where required.
• No placeholders were used for experimental result tables.

4.20 Limitations and scope¶

This lab validates swap/config semantics for this specific setup:

• Linux App Service, Python runtime.
• S1 plan, single app with one staging slot.
• Two controlled app settings.

It does not attempt to benchmark multi-slot, multi-instance, or multi-region swap complexity.

Expected Evidence¶

This section defines what you SHOULD observe at each phase of the lab. Use it to validate your investigation is on track.

Before Trigger (Baseline)¶

During Incident¶

After Recovery¶

Evidence Timeline¶

graph TD
    A[Baseline Capture] --> B[Trigger Fault]
    B --> C[During: Collect Evidence]
    C --> D[After: Compare to Baseline]
    D --> E[Verdict: Confirmed/Falsified]

Evidence Chain: Why This Proves the Hypothesis¶

Clean Up¶

az group delete --name "$RG" --yes --no-wait

Related Playbook¶

• Slot Swap Restart / Config Drift / Warm-up Race

See Also¶

• Playbook: Slot Swap Restart / Config Drift / Warm-up Race
• Playbook: Slot Swap Failed During Warm-up
• Playbook: Warm-up vs Health Check
• KQL: Restart Timing Correlation
• Troubleshooting Method

Sources¶

• Set up staging environments in Azure App Service
• Configure an App Service app in Azure App Service
• Enable diagnostic logging for apps in Azure App Service
• Monitor Azure App Service
• App Service plan overview