lb/Unhealthy Backends

Product: Load balancing Kind: Debugging Tree

Description

This runbook helps investigate why backends in a load balancer are unhealthy. It confirms and summarizes the current health status of the backends, aiding in identifying any unhealthy instances.

Key Investigation Areas:

• Firewalls:
  • Verifies if firewall rules are properly configured to allow health check traffic.
• Port Configuration:
  • Validates if the health check and serving ports are configured correctly, ensuring they are not mismatched.
• Logging:
  • Checks if health check logging is enabled to aid in troubleshooting.
• Health Check Logs (if enabled):
  • Analyzes the latest health check logs to identify the specific reasons for backend unhealthiness:
    • Timeouts: Identifies if the backend is timing out and provides potential causes and remediation steps.
    • Unhealthy: Indicates that the backend is reachable but doesn’t meet the health check’s criteria. It provides guidance on the expected health check behavior and suggests configuration checks.
    • Unknown: Explains the potential reasons for the “UNKNOWN” health state and suggests actions like adjusting timeouts or checking for Google Cloud outages.
• Past Health Check Success:
  • Checks if the health check has worked successfully in the past to determine if the issue is recent or ongoing.

Executing this runbook

gcpdiag runbook lb/unhealthy-backends \
  -p project_id=value \
  -p backend_service_name=value \
  -p region=value

Parameters

Get help on available commands

gcpdiag runbook --help

Potential Steps

• Unhealthy Backends Start

• Verify Health Check Logging Enabled

• Analyze Latest Health Check Log

• Check Past Health Check Success

• Validate Backend Service Port Configuration

• Verify Firewall Rules

• Check Vm Performance

• High Vm Memory Utilization

• High Vm Disk Utilization

• High Vm Cpu Utilization

• Unhealthy Backends End