The following web load balancer configuration is made up of Nginx, CentOS 6.5 and Keepalived.
Nginx is a highly scalable web server.
Keepalived’s website : The main goal of this project is to provide simple and robust facilities for loadbalancing and high-availability to Linux system and Linux based infrastructures. Loadbalancing framework relies on well-known and widely used Linux Virtual Server (IPVS) kernel module providing Layer4 loadbalancing.
Check out my previous post for compiling and configuring Keepalived
Load balancers:
lb01.lab.net - 172.16.1.2 lb02.lab.net - 172.16.1.3 virtual IP - 172.16.1.4
Web Servers:
web01.lab.net - 172.16.1.20 web02.lab.net - 172.16.1.30
Syslog Server:
syslog.lab.net - 172.16.1.31
STEP 1. Configure Keepalived
Primary Keepalived configuration file: /etc/keepalived/keepalived.conf
Once Keepalived.conf is configured you can scp ./keepalived.conf user@lb02:/etc/keepalived/keepalived.conf
On lb02.lab.net change the line “state BACKUP” to “state MASTER”. One server will act as the master and the other as backup.
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server smtp.lab.net
smtp_connect_timeout 30
router_id loadbalancer01
}
vrrp_script chk_proxy {
script '/usr/bin/killall -0 nginx';
interval 2
weight 2
}
vrrp_instance VI_169 {
state BACKUP
interface eth0
virtual_router_id 169
priority 150
advert_int 5
smtp_alert
authentication {
auth_type PASS
auth_pass 9999
}
virtual_ipaddress {
172.16.1.4/24 brd 172.16.1.255 dev eth0
}
track_script {
chk_proxy
}
preempt_delay 300
}
STEP 2. Backup Keepalive config file: /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server smtp.lab.net
smtp_connect_timeout 30
router_id loadbalancer02
}
vrrp_script chk_proxy {
script '/usr/bin/killall -0 nginx';
interval 2
weight 2
}
vrrp_instance VI_169 {
state BACKUP
interface eth0
virtual_router_id 169
priority 100
advert_int 5
smtp_alert
authentication {
auth_type PASS
auth_pass 9999
}
virtual_ipaddress {
172.16.1.4/24 brd 172.16.1.255 dev eth0
}
track_script {
chk_proxy
}
nopreempt
preempt_delay 300
}
STEP 3. Start Keepalived
On lb01 and lb02 run the following to start keepalived
service keepalived start
STEP 4. Verify Virtual IP is UP
Verify that the virtual IP address is up by running ip addr show.
You should see a secondary IP on eth0.
[root@lb01 ctatro]$ ip addr show
1: lo: LOOPBACK,UP,LOWER_UP mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:89:30:54 brd ff:ff:ff:ff:ff:ff
inet 172.16.1.2/24 brd 10.196.63.255 scope global eth0
inet 172.16.1.4/24 brd 10.196.63.255 scope global secondary eth0
inet6 fe80::250:56ff:fe89:3054/64 scope link
valid_lft forever preferred_lft forever
STEP 5. Niginx.conf configuration
Nginx global configuration file: /etc/nginx/nginx.conf
The nginx.conf file should be identical on lb01.lab.net and lb02.lab.net.
# configure 1 worker_processes per core
user nginx;
worker_processes 4;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
# worker_rlimit_nofile load balancer: worker_rlimit_nofile * worker_connections =
# make sure your ulimit nofile in limits.conf is set high enough
worker_rlimit_nofile 40960;
events {
use epoll;
# accept X connections per worker process
worker_connections 10240;
# accept more than 1 connection at a time
multi_accept on;
}
timer_resolution 500ms;
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr\t$remote_user\t[$time_local]\t$request\t'
'$status\t$body_bytes_sent\t$http_referer\t'
'$http_user_agent\t$http_x_forwarded_for\t'
'Req_Time\t$request_time\tWeb_srv_rsp_time\t$upstream_response_time\tWeb_srv_IP:\t$upstream_addr\tReq_size:\t$request_length
\tHTTP_content_size:\t$http_content_length\tSSL_cipher:\t$ssl_cipher\tSSL_protocol:\t$ssl_protocol';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
# timeout during which a keep-alive client connection will stay open on the server side
keepalive_timeout 30;
#Sets the maximum number of requests that can be served through one keep-alive
#connection. After the maximum number of requests are made, the connection is closed.
keepalive_requests 100000;
tcp_nodelay on;
# Caches information about open FDs, freqently accessed files.
open_file_cache max=10000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
[root@lb01 ctatro]#
STEP 6. Nginx loadbalancer configuration
The cluster01.conf file should be identical on lb01.lab.net and lb02.lab.net.
nginx site configuration file: /etc/nginx/conf.d/cluster01.conf
In this configuration file we have HTTP and HTTPS backend web servers configured.
HTTP requests will be forwarded on to port 80 on the upstream servers. And HTTPS requests will be terminated by Nginx then re encrypted and forwarded on to the upstread HTTPS web servers.
# HTTPS WEB POOL
upstream secureProd {
server web01.lab.net:443 weight=10 max_fails=3 fail_timeout=3s;
server web02.lab.net:443 weight=10 max_fails=3 fail_timeout=3s;
keepalive 100;
}
# HTTP WEB POOL
upstream httpProd {
server web01.lab.net:80 weight=10 max_fails=3 fail_timeout=3s;
server web02.lab.net:80 weight=10 max_fails=3 fail_timeout=3s;
keepalive 100;
}
### START HTTPS SERVER
server {
# backlog = worker_connections setting
listen 172.16.1.4:443 ssl backlog=65535;
ssl on;
ssl_certificate /etc/ssl/certs/cert.pem;
ssl_certificate_key /etc/ssl/certs/cert_private_key.pem;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers RC4:HIGH:!aNULL:!MD5:!kEDH;
ssl_prefer_server_ciphers on;
server_name webcluster.corp.domain.net;
access_log /var/log/nginx/webcluster_https.access.log main;
error_log /var/log/nginx/webcluster_https.error.log debug;
access_log syslog:server=syslog.lab.net main;
# do not transfer http request to next server on timeout
proxy_next_upstream off;
client_max_body_size 10m;
proxy_buffering on;
client_body_buffer_size 10m;
proxy_buffer_size 32k;
proxy_buffers 1024 32k;
large_client_header_buffers 20 8k;
location / {
index index.html
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_max_temp_file_size 0;
proxy_pass https://secureProd;
} #end location
}
### END HTTPS SERVER
### START HTTP SERVER
server {
# backlog = worker_connections setting
listen 172.16.1.4:80 backlog=65535;
server_name webcluster.corp.footlocker.net;
access_log /var/log/nginx/webcluster_http.access.log main;
error_log /var/log/nginx/webcluster_http.error.log debug;
access_log syslog:server=syslog.lab.net main;
# do not transfer http request to next server on timeout
proxy_next_upstream off;
client_max_body_size 10m;
proxy_buffering on;
client_body_buffer_size 10m;
proxy_buffer_size 32k;
proxy_buffers 1024 32k;
large_client_header_buffers 20 8k;
location / {
index index.html
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_max_temp_file_size 0;
proxy_pass http://httpProd;
} #end location
}
### END HTTP SERVER
STEP 7. Start Nginx
On lb01 and lb02 run the following to start keepalived
service nginx start
STEP 8. Tune kernel parameters
I also tuned some of the Linux kernel parameters to open up the TCP/IP stack. The TCP/IP stack on line is fairly restricted and needs to be opened up if the system will be handling a high number of TCP connections. I was able to substantially increase the throughput of my load balancer by tuning these settings.
[root@lb01 ~]$ sysctl -p net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 net.ipv4.ip_nonlocal_bind = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_orphans = 60000 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2 net.ipv4.icmp_echo_ignore_broadcasts = 1 kernel.exec-shield = 1 kernel.randomize_va_space = 1 net.ipv4.icmp_ignore_bogus_error_responses = 1 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.all.secure_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv4.conf.default.secure_redirects = 0 net.ipv4.ip_local_port_range = 1024 65535 net.ipv4.tcp_window_scaling = 1 net.core.somaxconn = 65535 net.core.netdev_max_backlog = 16384 net.ipv4.tcp_max_syn_backlog = 65536 net.ipv4.tcp_max_tw_buckets = 1440000 net.core.rmem_default = 8388608 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_congestion_control = cubic fs.file-max = 3000000 net.ipv4.tcp_slow_start_after_idle = 0 net.ipv4.tcp_fin_timeout = 15
Tater's Tech Blog 