Reliability Best Practices for Azure Functions¶

Reliable Azure Functions systems are built around the runtime execution model: most event-driven triggers (queues, Event Hubs, Service Bus, blob) deliver at-least-once, retries are normal, host instances can restart at any time, and dependencies can throttle. HTTP triggers behave differently — they are synchronous request/response and do not retry automatically. This guide focuses on safe execution patterns that keep behavior correct during duplicates, retries, scale-out, and partial outages.

Why This Matters¶

Function instances are ephemeral. The runtime can:

• start multiple instances during scale-out,
• recycle an instance during host upgrade or health event,
• reassign future invocations to different workers.

Any in-memory state can disappear or become inconsistent across instances.

What breaks when a function is stateful¶

• In-memory dedup caches miss duplicates after restart.
• Per-instance counters diverge during parallel scale-out.
• "Last processed timestamp" stored in memory causes skipped or duplicated work.
• Temporary local files are lost across cold starts and host moves.

Externalize state safely¶

Use external stores based on state type:

• Dedup/index state: Table Storage, Cosmos DB, SQL unique constraints.
• Workflow state: Durable Functions instance state.
• Checkpoint state: trigger-native checkpointing where provided.
• Long-running orchestration progress: Durable orchestrator history.

Recommended Practices¶

Idempotency is mandatory for every trigger¶

At-least-once delivery means a message or event can be processed more than once. Build every handler so duplicate invocation is safe.

Trigger-specific deduplication patterns¶

Retry-safe processing design¶

Treat each invocation as potentially repeated.

1. Validate input and compute deterministic operation key.
2. Check whether operation already completed.
3. Execute side effect with timeout budget.
4. Persist completion marker atomically.
5. Ack/complete message only after durable success.

flowchart TD
    S[Invocation starts] --> K[Compute deterministic operation key]
    K --> D{Already completed?}
    D -->|Yes| R1[Return cached result or ack]
    D -->|No| E[Execute side effect]
    E --> T{Failure type}
    T -->|Transient| RT[Retry with backoff]
    RT --> S
    T -->|Terminal| DL[Send to poison queue or DLQ with reason]
    E --> OK{Succeeded?}
    OK -->|Yes| M[Persist completion marker atomically]
    M --> R2[Ack or complete message]

Queue trigger safety (Storage Queue)¶

• visibilityTimeout controls message reappearance window.
• dequeueCount increments on each receive.
• maxDequeueCount moves failed messages to poison queue.

Operational guidance:

• Set visibilityTimeout longer than normal processing time plus downstream jitter.
• Keep processing idempotent because visibility expiration can cause concurrent duplicate processing.
• Alert on poison queue growth rate, not just absolute count.

Service Bus trigger safety¶

• Delivery lock can expire if processing exceeds lock duration.
• Retries can redeliver same message.
• Max delivery count routes to dead-letter queue.

Operational guidance:

• Keep handler duration inside lock window or enable lock renewal where supported.
• Move terminal business-rule failures to dead-letter with explicit reason metadata.
• Build replay tooling that can requeue selectively after remediation.

HTTP trigger safety with client retries¶

Clients, proxies, and API gateways can retry when they see 408/429/5xx or connection resets.

Design pattern:

• Require Idempotency-Key for state-changing operations.
• Store key with operation status (started, completed, failed-retryable, failed-terminal).
• Return existing result when key already completed.
• Expire keys with retention aligned to client retry behavior.

Timer trigger safety¶

Timer schedules can be delayed by host restarts and scale events.

Design pattern:

• Read schedule metadata and detect missed occurrences.
• Process windows by deterministic range (from, to) instead of "now only" logic.
• Use singleton/lease patterns for non-partitionable maintenance jobs.

Storage dependency resilience (AzureWebJobsStorage)¶

AzureWebJobsStorage is a runtime dependency for core host behavior and many triggers.

Failure modes and impact:

• Unavailable storage: triggers may stop polling or fail initialization.
• Throttled storage: increased latency, repeated retries, backlog growth.
• Misconfigured identity or connection: startup failure, binding errors, trigger silence.

Mitigations:

• Use supported identity-based configuration where available.
• Monitor host startup failures and storage-related exceptions.
• Separate application data storage from host storage when isolation is needed.
• Test failure behavior in staging by simulating storage deny/throttle conditions.

Durable Functions: when to use orchestrations¶

Use plain functions when work is short, stateless, and independently retryable. Use Durable Functions when you need deterministic workflow state across retries and restarts.

Choose Durable when you need:

• fan-out/fan-in with durable checkpoints,
• long-running human or external-system waits,
• compensation workflows,
• deterministic replayed orchestration logic.

Keep plain functions when you need:

• single-step event handling,
• low-latency stateless transformations,
• simple queue-to-API processing.

Timeout and cancellation handling¶

Plan-specific execution limits must drive handler design.

• Break long tasks into chunked operations.
• Use cancellation tokens or equivalent cancellation signals.
• Stop creating new outbound calls once cancellation is requested.
• Persist safe resume checkpoints before exiting.

Example cancellation-aware strategy:

1. Read cancellation signal at loop boundaries.
2. Complete current atomic unit.
3. Persist checkpoint state.
4. Exit gracefully so retry resumes from checkpoint.

Common Mistakes / Anti-Patterns¶

Do not treat poison/dead-letter paths as final discard queues.

Minimum production pattern:

• capture payload, trigger metadata, and failure reason,
• classify transient versus terminal failures,
• create replay process with dedup protection,
• track mean time to remediate poison items.

Trigger-specific poison handling¶

• Storage Queue: monitor <queue>-poison; create poison processor.
• Service Bus: monitor DLQ counts and age; preserve dead-letter reason.
• Event-based systems: implement side-channel failed-event store if no native poison queue exists.

Message processing lifecycle (retry and poison path)¶

flowchart TD
    T[Trigger event arrives] --> H[Function handler starts]
    H --> D{Duplicate already processed?}
    D -->|Yes| A1[Ack or return cached result]
    D -->|No| X[Execute business side effect]
    X --> S{Success?}
    S -->|Yes| C[Persist completion marker]
    C --> A2[Ack or complete message]
    S -->|No transient| R[Runtime retry path]
    R --> H
    S -->|No terminal| P[Poison queue or DLQ]
    P --> O[Ops triage and replay]
    O --> H

Validation Checklist¶

• [ ] Every trigger handler has explicit idempotency key logic.
• [ ] Side effects are guarded by dedup store or domain unique constraint.
• [ ] Queue visibilityTimeout and maxDequeueCount are tuned and documented.
• [ ] Service Bus lock duration/renewal and DLQ strategy are documented.
• [ ] HTTP write operations require idempotency key and replay-safe status model.
• [ ] Timer jobs handle missed schedules and avoid overlap issues.
• [ ] AzureWebJobsStorage identity/permissions/config are validated in pre-production.
• [ ] Timeout budget exists for each dependency call path.
• [ ] Poison and dead-letter queues have alerts, dashboard, and replay runbook.
• [ ] Durable Functions is used for multi-step workflows that need durable state.

See Also¶

• Platform Reliability
• Retries and Poison Handling
• Recovery
• Security Best Practices

Sources¶

• Designing Azure Functions for identical input (idempotency)
• Reliable event processing in Azure Functions
• Error handling and retries in Azure Functions
• Queue trigger for Azure Functions
• Service Bus trigger for Azure Functions
• Azure Functions host storage considerations
• Function app timeout duration
• Durable Functions overview