Cluster Autoscaler Issues¶

1. Summary¶

Use this playbook when pending pods do not trigger scale-out, empty nodes do not scale in, or autoscaler events show repeated failures. In AKS, the most common causes are tight node pool min and max limits, unsupported scheduling constraints, exhausted subnet or quota limits, or incorrect expectations about daemonsets and disruption budgets.

Typical incident window: 10-20 minutes to establish whether the issue is workload-specific, node-specific, or cluster-wide. Time to resolution: 30 minutes to several hours depending on whether the fix is manifest-level, node-level, or Azure control-plane level.

Symptoms¶

• Pods remain Pending with scheduling messages even though autoscaler is enabled.
• kubectl describe pod shows CPU, memory, or node affinity constraints that no current node satisfies.
• AKS activity or autoscaler logs mention scale-up rejection, quota issues, or subnet capacity exhaustion.
• Scale-in never happens because PDBs, local storage, or daemonset overhead keep nodes non-empty.

Diagnostic flowchart¶

flowchart TD
    A[Reported symptom] --> B{Can the object be reproduced now?}
    B -->|No| C[Use recent events, Container Insights, and rollout history]
    B -->|Yes| D[Capture current state with kubectl and Azure CLI]
    D --> E{Is the issue isolated to one workload or node pool?}
    E -->|Workload| F[Check image, probes, config, and service wiring]
    E -->|Node pool| G[Check node health, autoscaler, subnet, and VMSS state]
    F --> H[Validate with KQL and controller logs]
    G --> H
    H --> I[Apply targeted fix and verify telemetry returns to baseline]

2. Common Misreadings¶

3. Competing Hypotheses¶

4. What to Check First¶

1. Confirm the current object state from Kubernetes

   kubectl get pods \
       --all-namespaces \
       --output wide
   

2. Describe the affected object to capture recent events

   kubectl describe pod <pod-name> \
       --namespace <namespace>
   

3. Check AKS cluster and node pool configuration from Azure

   az aks show \
       --resource-group "$RG" \
       --name "$CLUSTER_NAME" \
       --query "{name:name,provisioningState:provisioningState,kubernetesVersion:kubernetesVersion,nodeResourceGroup:nodeResourceGroup}" \
       --output json
   

4. List node pools and autoscaler settings

   az aks nodepool list \
       --resource-group "$RG" \
       --cluster-name "$CLUSTER_NAME" \
       --output table
   

5. Run a fast Container Insights control query

   az monitor log-analytics query \
       --workspace "$WORKSPACE_ID" \
       --analytics-query "KubePodInventory | where TimeGenerated > ago(15m) | summarize Restarts=sum(ContainerRestartCount) by Namespace | order by Restarts desc" \
       --timespan "PT15M"
   

5. Evidence to Collect¶

5.1 KQL Queries¶

KubePodInventory
| where TimeGenerated > ago(30m)
| summarize Restarts=max(ContainerRestartCount), LastSeen=max(TimeGenerated) by ClusterName, Namespace, PodName, ContainerName
| order by Restarts desc

ContainerLogV2
| where TimeGenerated > ago(30m)
| summarize LogLines=count(), LastSeen=max(TimeGenerated) by Namespace, PodName
| order by LastSeen desc

KubeEvents
| where TimeGenerated > ago(30m)
| where Reason in ("Failed", "BackOff", "Unhealthy", "NodeNotReady", "FailedScheduling")
| project TimeGenerated, Namespace, Name, Reason, Message
| order by TimeGenerated desc

5.2 CLI Investigation¶

kubectl logs <pod-name> \
    --namespace <namespace> \
    --previous

Interpretation: previous logs are usually more valuable than current logs during restart loops because they contain the container exit path.

kubectl get events \
    --all-namespaces \
    --sort-by=.lastTimestamp

Interpretation: look for probe failures, image pull errors, FailedScheduling, NodeNotReady, or backend controller warnings near the incident start time.

az vmss list-instances \
    --resource-group "$NODE_RESOURCE_GROUP" \
    --name "$VMSS_NAME" \
    --query "[].{instanceId:instanceId,provisioningState:provisioningState,latestModelApplied:latestModelApplied}" \
    --output table

Interpretation: when the problem is node- or ingress-related, VMSS state and model drift provide important Azure-side evidence.

6. Validation and Disproof by Hypothesis¶

Node pool limits or autoscaler profile settings prevent scale-out¶

Proves if: Kubernetes events, previous logs, and Azure-side state all align around this hypothesis.

Disproves if: Another signal explains the timing more directly or the expected discriminator is missing.

kubectl describe pod <pod-name> \
    --namespace <namespace>

Pod constraints cannot be satisfied by any node pool¶

Proves if: Kubernetes events, previous logs, and Azure-side state all align around this hypothesis.

Disproves if: Another signal explains the timing more directly or the expected discriminator is missing.

kubectl describe pod <pod-name> \
    --namespace <namespace>

Azure quota or subnet capacity prevents new nodes¶

Proves if: Kubernetes events, previous logs, and Azure-side state all align around this hypothesis.

Disproves if: Another signal explains the timing more directly or the expected discriminator is missing.

kubectl describe pod <pod-name> \
    --namespace <namespace>

Scale-in blocked by workload protections¶

Proves if: Kubernetes events, previous logs, and Azure-side state all align around this hypothesis.

Disproves if: Another signal explains the timing more directly or the expected discriminator is missing.

kubectl describe pod <pod-name> \
    --namespace <namespace>

7. Likely Root Cause Patterns¶

8. Immediate Mitigations and Step-by-Step Resolution¶

1. Inspect pending pod events and the target node pool autoscaler settings together.
2. Increase max counts or add a suitable pool only after proving that workload constraints are legitimate.
3. Resolve quota or subnet exhaustion before retrying scale actions.
4. Tune PDBs, requests, and daemonset placement so scale-in can happen safely.
5. Review cost and reliability implications after every autoscaler policy change.

Example resolution commands:

kubectl rollout restart deployment/<deployment-name> \
    --namespace <namespace>

az aks nodepool update \
    --resource-group "$RG" \
    --cluster-name "$CLUSTER_NAME" \
    --name "$NODEPOOL_NAME" \
    --max-count 10

9. Prevention Checklist¶

• [ ] Create saved Container Insights queries for the symptom family and link them in the team runbook.
• [ ] Require long-flag CLI examples and standardized evidence capture in incident response docs.
• [ ] Review ingress, autoscaler, probes, and node pool settings during every production readiness review.
• [ ] Alert on restart spikes, NotReady nodes, and FailedScheduling events before customers report impact.
• [ ] Document which changes require platform-team approval, especially around networking, ingress, and security policy.

See Also¶

• Scaling Failure
• Cost Optimization
• Lab 05: AKS Disaster Recovery

Sources¶

• Troubleshoot / Azure / Azure Kubernetes / Welcome Azure Kubernetes
• Azure / Aks / Cluster Autoscaler
• Azure / Azure Monitor / Containers / Container Insights Overview
• Azure / Aks / Concepts Network