Haproxy ошибка 503

I’m new to this load balancing scenario and I’m being tasked to figure out how to make this load balancing works.

My environment:

Centos 6.4 64 Bit
Webserver: Lighttpd
All running in ESXI
virtual IP: 192.168.1.6
LB1: 192.168.1.4
LB2: 192.168.1.5
Webserver 1: 192.168.1.12
Webserver 2: 192.168.1.13
Gateway: 192.168.1.1

Trying to run a test in the lab prior production with HAproxy and keepalived. Here is what I have on my keepalived setting:

! Configuration File for keepalived

global_defs {
   notification_email {
     myemail@gmail.com
   }
   notification_email_from keepalived@domain.local
   smtp_server 192.168.1.4
   smtp_connect_timeout 30
   router_id 192.168.1.1
}

vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 1                     # check every second
weight 2                       # add 2 points of prio if OK
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101  #priority 101 for master
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.6

    }

    track_script {
    chk_haproxy
            }

            }

and here is my setting for HAproxy

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main *:80
#    acl url_static       path_beg       -i /static /images /javascript /stylesheets
#    acl url_static       path_end       -i .jpg .gif .png .css .js

#    use_backend static          if url_static
#    default_backend             view

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
#backend static
#    balance     roundrobin
#    server      static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app
mode tcp
    balance     roundrobin
    server  server1 192.168.1.12:80 check inter 2000 rise 2 fall 5
    server  server2 192.168.1.13:80 check inter 2000 rise 2 fall 5

When I started the HAproxy, I got thiss error and I’m not quite sure where to start looking to fix it. Perhaps someone who has done this many times can help me shed some light?

503 Service Unavailable No server is available to handle this request. 

However manually connecting to the webserver1 and webserver2 works just fine.

All I want is just a simple load balancing for the webserver that sit behind the HAproxy. Any advise or suggestion is absolutely appreciated. Please help? Many thanks.

Здраствуйте , У меня возник интересная ситуация , есть сайт которые работет через haproxy с ssl терминацией на самом haproxy , написал простой скрипт который проверяет состояние сайта и haproxy по коду ответа http если кож ответа отличается от 200, то скрипт запускает определенные действии , Проблема в том что иногда при нормальном работающем сайте, haproxy отвечает на запросы (через curl) кодом 503, и наоборот если для проверки изменить бакэнд сайта на другой ip , и заново запускать скрипт или команду curl , то иногда я получаю 200 как будто все нормально,но на самом деле он должен отвечать кодом 503, и к стати в логах тоже самое , Пожалуйста вот логи

Apr 10 04:50:15 haproxy1 haproxy[1917]: 192.168.1.154:6440 [10/Apr/2023:04:50:15.545] frontend_01~ example.com/s2 2/0/1/2/5 200 202 - - ---- 1/1/0/0/0 0/0 "HEAD / HTTP/1.1"
Apr 10 04:50:20 haproxy1 haproxy[1959]: 192.168.1.154:6448 [10/Apr/2023:04:50:17.622] frontend_01~ example.com/s2 2/0/-1/-1/3009 503 108 - - SC-- 1/1/0/0/3 0/0 "HEAD / HTTP/1.1"
Apr 10 04:50:24 haproxy1 haproxy[23458]: 192.168.1.153:6882 [10/Apr/2023:04:50:24.719] stats stats/<STATS> 0/0/0/-1/0 400 251 - - LR-- 1/1/0/0/0 0/0 "HEAD /stat HTTP/1.1"
Apr 10 04:50:25 haproxy1 haproxy[23465]: 192.168.1.154:6456 [10/Apr/2023:04:50:22.721] frontend_01~ example.com/s2 3/0/-1/-1/3010 503 108 - - SC-- 1/1/0/0/3 0/0 "HEAD / HTTP/1.1"
Apr 10 04:50:27 haproxy1 haproxy[1917]: 192.168.1.154:43170 [10/Apr/2023:04:50:27.798] frontend_01~ example.com/s2 2/0/1/2/5 200 202 - - ---- 1/1/0/0/0 0/0 "HEAD / HTTP/1.1"
Apr 10 04:50:32 haproxy1 haproxy[23458]: 192.168.1.154:43172 [10/Apr/2023:04:50:29.893] frontend_01~ example.com/s2 2/0/-1/-1/3009 503 108 - - SC-- 1/1/0/0/3 0/0 "HEAD / HTTP/1.1"
Apr 10 04:50:37 haproxy1 haproxy[1959]: 192.168.1.154:41422 [10/Apr/2023:04:50:34.953] frontend_01~ example.com/s2 2/0/-1/-1/3009 503 108 - - SC-- 1/1/0/0/3 0/0 "HEAD / HTTP/1.1"
Apr 10 04:50:39 haproxy1 haproxy[23465]: 192.168.1.153:32656 [10/Apr/2023:04:50:39.724] stats stats/<STATS> 0/0/0/-1/0 400 251 - - LR-- 1/1/0/0/0 0/0 "HEAD /stat HTTP/1.1"
Apr 10 04:50:43 haproxy1 haproxy[23458]: 192.168.1.154:41438 [10/Apr/2023:04:50:40.023] frontend_01~ example.com/s2 2/0/-1/-1/3008 503 108 - - SC-- 1/1/0/0/3 0/0 "HEAD / HTTP/1.1"

Спасибо за внимание !!

Try making the timers bigger and check that the server is reachable.

From the HAproxy docs:

It can happen from many reasons:

The status code is always 3-digit. The first digit indicates a general status :
 - 1xx = informational message to be skipped (eg: 100, 101)
 - 2xx = OK, content is following   (eg: 200, 206)
 - 3xx = OK, no content following   (eg: 302, 304)
 - 4xx = error caused by the client (eg: 401, 403, 404)
 - 5xx = error caused by the server (eg: 500, 502, 503)

  503  when no server was available to handle the request, or in response to
        monitoring requests which match the "monitor fail" condition

  When a server's maxconn is reached, connections are left pending in a queue
  which may be server-specific or global to the backend. In order not to wait
  indefinitely, a timeout is applied to requests pending in the queue. If the
  timeout is reached, it is considered that the request will almost never be
  served, so it is dropped and a 503 error is returned to the client.

if you see SC in the logs:

SC   The server or an equipment between it and haproxy explicitly refused
          the TCP connection (the proxy received a TCP RST or an ICMP message
          in return). Under some circumstances, it can also be the network
          stack telling the proxy that the server is unreachable (eg: no route,
          or no ARP response on local network). When this happens in HTTP mode,
          the status code is likely a 502 or 503 here.

Check ACLs, check timeouts… and check the logs, that’s the most important…

Learn how to resolve the 503 Service Unavailable HAProxy error. Our HAProxy Support team is here to help you with your questions and concerns.

503 Service Unavailable HAProxy – Troubleshooting Tips

503 Service Unavailable is an HTTP status code. This error message lets us know that the server is not able to handle the request currently. If the HAProxy load balancer returns this status code it indicates that the backend servers are overloaded or offline.

HAProxy is a well-known open-source load balancer. It is responsible for distributing incoming traffic across multiple servers to boost performance and reliability. When a client sends a request to HAProxy, it gets forwarded to one of the backend servers based on the configured load-balancing algorithm.

In case the selected backend server is unavailable or unable to handle the request, HAProxy returns the 503 Service Unavailable error message to the client.

Let’s take a quick look at some of the common reasons behind this error message:

• Backend Server Overload:

  An increase in the traffic to the backend servers can result in overloading. This prevents the processing of incoming requests. As a result, HAProxy will receive a large number of connection requests that cannot be fulfilled, leading to a 503 error.
• Backend Server Maintenance:

  In case a backend server is currently under maintenance or down, it will not be able to handle incoming requests. As a result, HAProxy will detect that the server is not responding and return a 503 error to the client.

• Connection Limits:

  If the number of connections permitted to the backend server exceeds the set limit, the server will start rejecting new connections. Hence, HAProxy will receive a large number of failed requests.

• Configuration Errors:

  If there are errors in the HAProxy configuration file it will lead to unexpected behavior and cause the servers to return a 503 error.

How to troubleshoot 503 Service Unavailable Error

1. First, check if the backend servers are online and responding to requests.
2. Next, verify the server load to ensure that the backend servers are not overloaded and can handle incoming traffic.
3. If the above two tips did not help, try increasing the number of allowed connections to the backend servers.
4. If the error persists, we must review the HAProxy configuration file for any errors or misconfigurations.

Let us know in the comments if you need further help with this specific error.

[Need assistance with a different issue? Our team is available 24/7.]

Conclusion

To conclude, our Support Techs demonstrated several different ways to resolve the 503 Service Unavailable HAProxy error.