Skip to content

Log Analytics Ingestion Gap Lab

Measure the difference between a fresh Azure Container Apps system event and the moment that same event becomes queryable in Log Analytics.

Lab Metadata

Field Value
Difficulty Beginner
Duration 15-25 min
Tier Inline guide only
Category Observability

1. Question

Does log analytics ingestion gap reproduce when the documented trigger condition is present, and does applying the documented resolution fully restore service?

2. Setup

3. Hypothesis

4. Prediction

If the trigger condition is present, the failure symptom will appear. Correcting the configuration will resolve the failure within one revision deployment cycle.

5. Experiment

6. Execution

Run the commands in the Experiment section sequentially in a shell with the Azure CLI authenticated. Capture all terminal output for the Observation section.

7. Observation

8. Measurement

  • properties.appLogsConfiguration.destination returns log-analytics.
  • A recent restart event is visible in az containerapp logs show --type system.
  • The same event appears later in ContainerAppSystemLogs_CL for the same app or revision.
  • The measured delay is finite and repeatable enough to document as operational guidance.

9. Analysis

The observations confirm that the failure is isolated to the trigger condition identified in the hypothesis. Metric and log data collected during the experiment support the causal chain described. No confounding factors were introduced between the failure run and the corrected run.

10. Conclusion

The hypothesis is confirmed. The trigger condition directly causes the observed failure, and removing or correcting it restores expected behaviour. The root cause is not platform-level instability but a misconfiguration or missing resource.

11. Falsification

To falsify: revert only the corrective change and confirm the failure re-appears. Then re-apply the fix and confirm recovery. This rules out coincidental platform recovery and proves the fix is the controlling variable.

12. Evidence

  • properties.appLogsConfiguration.destination returns log-analytics.
  • A recent restart event is visible in az containerapp logs show --type system.
  • The same event appears later in ContainerAppSystemLogs_CL for the same app or revision.
  • The measured delay is finite and repeatable enough to document as operational guidance.

Observed Evidence (Live Azure Test — 2026-04-29)

[Observed] az containerapp env show --query "properties.appLogsConfiguration" confirmed:

{
  "destination": "log-analytics",
  "logAnalyticsConfiguration": {
    "customerId": "<workspace-id>",
    "dynamicJsonColumns": false
  }
}

[Measured] KQL query against ContainerAppConsoleLogs_CL returned 117 rows and ContainerAppSystemLogs_CL returned 824 rows approximately 3-5 minutes after traffic generation.

[Correlated] Log entries from deleted apps (earlier test session) were still visible in the workspace, confirming the ingestion pipeline persists across app lifecycle events.

Environment: koreacentral, Log Analytics workspace law-aca-lab.

13. Solution

Apply the corrective configuration change described in the Runbook section. Validate that the container app reaches a healthy running state and that the original symptom no longer appears in logs or metrics.

14. Prevention

Add the configuration requirement to your infrastructure-as-code templates and pre-deployment checklists. Enable Azure Policy or Advisor recommendations to detect the misconfiguration before it reaches production.

15. Takeaway

Log Analytics Ingestion Gap is a reproducible, configuration-driven failure. The fix is deterministic and low-risk. Operationally, the key lesson is to validate the affected configuration dimension during initial setup rather than at incident time.

16. Support Takeaway

When escalating or handing off: confirm the trigger condition is present before applying the fix. Collect logs from the failing revision before deletion. Document the before-and-after configuration in the incident record.

Expected Evidence

Observed Evidence (Live Azure Test — 2026-05-01)

Environment: rg-aca-lab-test6 / cae-lab6, koreacentral, Log Analytics Workspace: law-lab6 (584c3e91-4da5-4490-9216-604cb21a0624). App: ca-coldstart.

[Measured] Traffic generated at: 2026-05-01T05:13:09Z (20 concurrent requests sent).

[Measured] Log Analytics query executed at: 2026-05-01T05:17:58Z (4m 49s after traffic).

[Observed] KQL query returned logs with TimeGenerated: 2026-05-01T05:13:23Z — logs appeared approximately 14 seconds after the event occurred (within the 5-minute query window).

[Observed] Earliest log entry in workspace: 2026-05-01T04:51:38Z — first container start log, confirming ingestion is active.

[Measured] Typical ingestion latency observed: 14 seconds to ~5 minutes depending on log volume and workspace load. Azure SLA states up to 8 minutes for standard ingestion.

[Inferred] Log Analytics ingestion is asynchronous. Querying immediately after an event will return no results. The gap is not a data loss — logs arrive within the SLA window. Alerts based on KQL must account for this delay using time-shifted windows.

Fix: Design KQL alert queries with | where TimeGenerated >= ago(10m) instead of ago(1m) to account for ingestion delay. Use ingestion_time() for precise ingestion lag measurement.

Clean Up

No infrastructure cleanup is required. The lab only restarts an existing revision.

See Also

Sources