Disk Full¶

1. Summary¶

An EB instance runs out of usable disk and the environment begins failing writes, deployments, health checks, or application requests.

flowchart TD
    A[Disk Full] --> B{Primary branch}
    B --> C1[Log growth exceeded rotation capacity]
    C1 --> D1[Collect logs, metrics, and platform signals]
    B --> C2[Temporary file accumulation]
    C2 --> D2[Collect logs, metrics, and platform signals]
    B --> C3[Deployment artifact buildup]
    C3 --> D3[Collect logs, metrics, and platform signals]
    B --> C4[Application writes unexpected local data]
    C4 --> D4[Collect logs, metrics, and platform signals]

2. Common Misreadings¶

• Average latency is enough to judge user impact.
• Moderate CPU means there is no bottleneck.
• A restart proves the root cause is fixed.
• Only one slow route cannot affect the whole environment.
• Scale-out timing never matters once new instances appear.

3. Competing Hypotheses¶

• • H1: Log growth exceeded rotation capacity — Primary evidence should confirm or disprove whether log growth exceeded rotation capacity.
• • H2: Temporary file accumulation — Primary evidence should confirm or disprove whether temporary file accumulation.
• • H3: Deployment artifact buildup — Primary evidence should confirm or disprove whether deployment artifact buildup.
• • H4: Application writes unexpected local data — Primary evidence should confirm or disprove whether application writes unexpected local data.

4. What to Check First¶

Metrics¶

• Check one-minute TargetResponseTime and traffic in the same window.
• Check per-instance CPU, memory, and health rather than only environment averages.
• Check whether the issue appears only under concurrency or also at baseline.

Logs¶

• Read nginx/access.log for slow requests and status codes.
• Read web.stdout.log for timeouts, slow paths, pool waits, or OOM clues.
• Read nginx/error.log when saturation reaches the proxy layer.

Platform Signals¶

• Run eb health --environment-name $ENV_NAME --refresh to identify whether the issue is one instance or the fleet.
• Record whether health moves from Ok to Warning, Degraded, or Severe.
• Compare the incident window to one known-good baseline window.

5. Evidence to Collect¶

Required Evidence¶

• First symptom timestamp in UTC.
• One healthy comparison sample if available.
• Relevant EB health color transitions (Ok, Warning, Degraded, Severe).
• Exact app version, platform branch, and environment name.

Useful Context¶

• Whether the symptom started after deploy, config change, platform update, or traffic change.
• Whether the issue is isolated to one instance, one batch, one subnet, or the full environment.
• Any recent changes to health checks, listeners, routes, worker counts, dependencies, or deployment policy.

CLI Investigation Commands¶

1. Correlate latency, health, and traffic¶

eb health --environment-name $ENV_NAME --refresh
aws elasticbeanstalk describe-environment-health --environment-name $ENV_NAME --attribute-names All
aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name TargetResponseTime --dimensions Name=LoadBalancer,Value=$LOAD_BALANCER_DIMENSION --statistics Average p95 --period 60 --start-time $START_TIME --end-time $END_TIME

Example output:

instance-id           status   cause
i-xxxxxxxxxxxxxxxxx   Warning  Application requests are failing or timing out.
i-yyyyyyyyyyyyyyyyy   Ok       No data
Average: 1.42
p95: 4.87

2. Pull proxy and application logs¶

eb logs --environment-name $ENV_NAME --all
aws logs start-query --log-group-name "/aws/elasticbeanstalk/$ENV_NAME/var/log/nginx/access.log" --start-time $START_EPOCH --end-time $END_EPOCH --query-string "fields @timestamp, @message | limit 20"

Example output:

Logs were saved to /var/folders/.../logs-20260407.zip
queryId: 12345678-90ab-cdef-1234-567890abcdef

3. Inspect scaling and host pressure¶

aws autoscaling describe-scaling-activities --auto-scaling-group-name $ASG_NAME --max-items 20
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUUtilization --dimensions Name=AutoScalingGroupName,Value=$ASG_NAME --statistics Average Maximum --period 60 --start-time $START_TIME --end-time $END_TIME

Example output:

Activities:
  - Description: Setting desired capacity to 8. StatusCode: Successful
Average: 71.4
Maximum: 92.1

Evidence Timeline¶

sequenceDiagram
    participant USER as Incoming traffic
    participant APP as App instances
    participant DEP as Dependency or host resource
    USER->>APP: Traffic rises or concentrates
    APP->>DEP: Consume CPU, memory, disk, or connections
    DEP-->>APP: Slow responses, waits, or saturation
    Note over USER,DEP: Capture the first point where request time expands or resources stop keeping pace

Sample Log Patterns¶

2026-04-07T14:03:11.934Z WARN request exceeded expected latency budget
2026-04-07T14:03:12.110Z ERROR connection pool timeout after 2000 ms
2026/04/07 14:03:13 [error] 4110#4110: *311 upstream timed out (110: Connection timed out) while reading response header from upstream
2026-04-07T14:03:14.220Z WARN host pressure increased on i-xxxxxxxxxxxxxxxxx

CloudWatch Logs Insights Queries with Example Output¶

Query 1. Find the earliest incident evidence¶

fields @timestamp, @message
| filter @message like /no space left on device|ENOSPC/
| sort @timestamp asc
| limit 20

Example results:

Query 2. Find the most visible failure signatures¶

fields @timestamp, @message
| filter @message like /extracting source bundle|failed to rotate/
| sort @timestamp desc
| limit 20

Example results:

6. Validation and Disproof by Hypothesis¶

H1: Log growth exceeded rotation capacity¶

Confirm: - Logs, metrics, and platform state all point directly at this branch. - The first failing timestamp lines up with evidence expected for Log growth exceeded rotation capacity.

Disprove: - The expected log or state change for this branch never appears. - Another branch has earlier, stronger, and more direct evidence.

H2: Temporary file accumulation¶

Confirm: - Logs, metrics, and platform state all point directly at this branch. - The first failing timestamp lines up with evidence expected for Temporary file accumulation.

Disprove: - The expected log or state change for this branch never appears. - Another branch has earlier, stronger, and more direct evidence.

H3: Deployment artifact buildup¶

Confirm: - Logs, metrics, and platform state all point directly at this branch. - The first failing timestamp lines up with evidence expected for Deployment artifact buildup.

Disprove: - The expected log or state change for this branch never appears. - Another branch has earlier, stronger, and more direct evidence.

H4: Application writes unexpected local data¶

Confirm: - Logs, metrics, and platform state all point directly at this branch. - The first failing timestamp lines up with evidence expected for Application writes unexpected local data.

Disprove: - The expected log or state change for this branch never appears. - Another branch has earlier, stronger, and more direct evidence.

7. Likely Root Cause Patterns¶

• A recent change shifted the failure into this playbook's domain.
• The earliest warning was ignored and later symptoms obscured the first cause.
• A platform, configuration, or dependency assumption drifted from the known-good state.
• The environment had too little safety margin for rollout, load, or path changes.

8. Immediate Mitigations¶

1. Preserve the first-failure evidence before retrying or restarting anything.
2. Contain user impact with the smallest safe rollback, scale, or routing change.
3. Change only one suspected variable at a time and re-check health colors, logs, and metrics.
4. Confirm that the symptom, not just the dashboard noise, has improved.

9. Prevention¶

• Keep environment configuration, health checks, and rollout assumptions under version control.
• Test the same path in staging with the same platform branch and deployment policy.
• Alert on the earliest signal for this failure mode, not only the final outage symptom.
• Review baselines regularly so abnormal behavior is obvious during incidents.

See Also¶

• Troubleshooting Playbooks Hub
• Health Turns Red After Successful Deploy
• Load Balancer Returns 5xx Errors

Sources¶

• Elastic Beanstalk monitoring with Amazon CloudWatch
• Elastic Beanstalk enhanced health reporting
• Elastic Beanstalk logs
• Application Load Balancer CloudWatch metrics