2

When an application suddenly becomes unreachable in Google Cloud, one of the most common causes is a failed load balancer health check. Even when servers appear to be running normally, a health check failure can cause Google Cloud Load Balancers to stop routing traffic to backend instances, resulting in downtime, error messages, and poor user experience.

Understanding why health checks fail and knowing how to troubleshoot them quickly can significantly reduce downtime and improve application availability.

In this guide, we’ll explore the most common causes of Google Cloud Load Balancer health check failures, how to diagnose them, and the Linux commands that can help identify issues efficiently.

What Are Google Cloud Health Checks?

Google Cloud health checks are monitoring probes used by load balancers to determine whether backend instances or services are healthy enough to receive traffic.

Health checks verify several factors, including:

• Server availability
• Application responsiveness
• Open and accessible ports
• HTTP/HTTPS response status codes
• TCP connectivity

If a backend repeatedly fails health checks, Google Cloud automatically removes it from traffic rotation until it becomes healthy again.

Common Symptoms of Health Check Failures

You may be experiencing a health check issue if you notice any of the following:

• Website or application becomes inaccessible
• Frequent 502 or 503 errors
• Backend instances marked as unhealthy
• Uneven traffic distribution
• Sudden drop in incoming requests
• Auto-scaling behaving unexpectedly

Common Causes of Google Cloud Health Check Failures

1. Firewall Rules Blocking Health Check Traffic

One of the most common causes of health check failures is firewall rules blocking Google’s health check IP ranges.

Google Cloud health checks typically originate from:

• 35.191.0.0/16
• 130.211.0.0/22

If these IP ranges cannot access your backend service, health checks will fail even when the application is functioning properly.

Create a Firewall Rule

gcloud compute firewall-rules create allow-health-checks \
--network default \
--allow tcp:80,tcp:443 \
--source-ranges 35.191.0.0/16,130.211.0.0/22

Always verify that required ports are accessible from Google Cloud health check sources.

2. Application Not Listening on the Correct Port

Health checks must target the same port on which your application is actively listening.

For example:

• Backend service configured on port 80
• Application running on port 8080

This mismatch causes health checks to fail.

Check Listening Ports

sudo netstat -tulnp

or

sudo ss -tulnp

Verify that the configured health check port matches the application’s listening port.

3. Web Server or Application Service Is Down

If the backend application or web server has stopped or crashed, health checks will immediately fail.

Check Service Status

For NGINX:

sudo systemctl status nginx

For Apache:

sudo systemctl status httpd

For custom services:

sudo systemctl status your-service-name

Restart the Service

sudo systemctl restart nginx

Always review service logs to identify the root cause of service failures.

4. Incorrect Health Check Path

HTTP and HTTPS health checks typically expect a successful 200 OK response.

If the configured health check path returns:

• 404 Not Found
• 403 Forbidden
• 500 Internal Server Error

the backend will be marked unhealthy.

Test the Health Check Endpoint

curl -I http://localhost/health

or

curl -I http://localhost

A healthy response should return:

HTTP/1.1 200 OK

If errors occur, update either the application endpoint or the Google Cloud health check configuration.

5. High CPU, Memory, or Disk Utilization

Overloaded servers often respond slowly to health check requests, causing timeout failures.

Check CPU and Memory Usage

top

or

htop

Check Disk Usage

df -h

Check Memory Usage

free -m

If resource utilization is consistently high, consider scaling resources or optimizing application performance.

6. Slow Backend Startup Time

Applications that require significant initialization time may fail health checks immediately after startup.

This commonly occurs in:

• Managed Instance Groups (MIGs)
• Auto-scaling environments
• Kubernetes workloads
• Large application deployments

Check Startup Logs

journalctl -xe

or

sudo journalctl -u nginx

Consider increasing health check timeout values during startup if necessary.

7. Incorrect Health Check Configuration

Even a healthy application can fail health checks due to configuration mistakes.

Common issues include:

• Incorrect protocol selection
• Wrong port number
• Invalid request path
• Timeout values that are too short
• Improper healthy or unhealthy thresholds

View Existing Health Checks

gcloud compute health-checks list

Describe a Health Check

gcloud compute health-checks describe HEALTH_CHECK_NAME

Verify:

• Port number
• Request path
• Check interval
• Timeout settings
• Healthy threshold
• Unhealthy threshold

8. SSL or HTTPS Certificate Issues

HTTPS health checks may fail due to SSL-related problems such as:

• Expired certificates
• TLS version mismatches
• Broken HTTPS redirects
• SSL handshake failures

Verify SSL Certificate

openssl s_client -connect yourdomain.com:443

Test HTTPS Endpoint

curl -Iv https://yourdomain.com

Look for certificate errors, expiration warnings, or handshake failures.

9. DNS Resolution Problems

Applications that rely on DNS services may experience health check failures when domain resolution stops working correctly.

Verify DNS Resolution

nslookup yourdomain.com

or

dig yourdomain.com

Ensure all required domains resolve correctly and consistently.

10. Local Firewall or Security Software Blocking Traffic

Linux firewall tools such as iptables or firewalld may accidentally block health check traffic.

Check iptables Rules

sudo iptables -L -n

Check firewalld Configuration

sudo firewall-cmd --list-all

Verify that health check IP ranges and required ports are properly allowed.

Useful Google Cloud Commands for Troubleshooting

Check Backend Health

gcloud compute backend-services get-health BACKEND_SERVICE_NAME --global

List Backend Services

gcloud compute backend-services list

View Compute Instances

gcloud compute instances list

Review Logs

gcloud logging read "resource.type=gce_instance"

These commands can quickly help determine whether the issue originates from networking, application configuration, or infrastructure.

Best Practices to Prevent Health Check Failures

Preventing health check failures is easier than troubleshooting them during an outage.

-> Use Dedicated Health Check Endpoints

Create lightweight endpoints specifically for health checks.

-> Avoid Complex Health Checks

Minimize dependencies on databases or third-party services whenever possible.

-> Monitor Infrastructure Proactively

Use monitoring tools such as:

• Google Cloud Monitoring
• Prometheus
• Grafana
• Nagios

-> Enable Auto-Healing

Configure health checks alongside auto-healing policies to automatically recover failed instances.

-> Document Firewall and Network Rules

Maintain clear documentation for:

• Allowed ports
• Firewall rules
• Health check IP ranges
• Backend configurations

This significantly speeds up troubleshooting during incidents.

Conclusion

Google Cloud Load Balancer health checks play a critical role in maintaining application availability and performance. When health checks fail, backend services can be removed from traffic rotation, leading to downtime and service disruptions.

By understanding common failure causes such as firewall restrictions, incorrect ports, application crashes, SSL issues, resource bottlenecks, and configuration errors, administrators can quickly identify and resolve problems before they impact users.

Regular monitoring, proper health check configuration, and proactive infrastructure management are essential for maintaining a highly available and reliable Google Cloud environment.

Need Help Troubleshooting Google Cloud Infrastructure Issues?

SupportPRO’s cloud engineers provide expert Google Cloud support, load balancer troubleshooting, performance optimization, and 24/7 infrastructure management. Contact us today to keep your cloud applications running reliably and efficiently.