How to add HA networking with Keepalived
Sep 15, 2015 19:00
Overview
This post outlines the HA network configuration I’ve been using in my homelab to failover HAProxy VIPs in case of network issues on my primary HAProxy node. All my work has been done inside of virtual machines but the same concepts will apply to physical hosts. I’ll assume you have a working HAProxy configuration. There is however, one item related to HAProxy which we’ll cover later.
Assumptions
- MASTER (primary HAProxy node) and BACKUP (secondary HAProxy node) are both running CentOS 7 with SELinux in “enforcing” mode.
- Both HAProxy nodes are attached to 3 distinct networks:
- Network 1 (VIP Net) on eth0 on which the layer 7 requests are entering. eth0 is also where the VIPs are being added during failover and failback.
- Network 2 (VRRP Net) on eth1 on which the two HAProxy nodes are sending Unicast VRRP traffic to one another
- Both nodes need to have a static address configured on eth1
- If you’re running VMs on OpenStack, make sure your security groups for MASTER and BACKUP are set to “-1”
- Network 3 (LB Net) on eth2 on which the HAProxy nodes are communicating with the backend nodes
Diagram
The network layout is probably best demonstrated with a diagram:
***********
* REQUEST *
***********
|
|
V
(VIP Net)
eth0............................eth0
| |
| |
---------- (VRRP Net) ----------
| MASTER |--eth1 <------> eth1--| BACKUP |
---------- ----------
| |
| |
eth2 eth2
| |
|...............................|
|
|
(LB Net)
|
|
************
* BACKENDS *
************
Implementation
Both nodes
# yum install keepalived -y
# cd /etc/keepalived; mv keepalived.conf{,.original}
# setsebool -P domain_kernel_load_modules=true
- Create /etc/keepalived/keepalived.conf with the following content:
MASTER node
vrrp_instance HAPROXY_VIPS {
state MASTER
interface eth1 # interface connected to VRRP net
virtual_router_id 51
priority 101 # number here must be > than on BACKUP
advert_int 1
unicast_src_ip <MASTER-vrrp-net-ip>
unicast_peer {
<backup-vrrp-net-ip>
}
authentication {
auth_type PASS
auth_pass 1111 # make it whatever you want, but only the first 8 chars will be used
}
virtual_ipaddress {
X.X.X.X/32 brd X.X.X.255 dev eth0 label eth0:0
X.X.X.Y/32 brd X.X.X.255 dev eth0 label eth0:1
X.X.X.Z/32 brd X.X.X.255 dev eth0 label eth0:2
}
}
BACKUP node
vrrp_instance HAPROXY_VIPS {
state BACKUP
interface eth1
virtual_router_id 51
priority 100 # must be lower than MASTER
advert_int 1
unicast_src_ip <BACKUP-vrrp-net-ip>
unicast_peer {
<master-vrrp-net-ip>
}
authentication {
auth_type PASS
auth_pass 1111 # same as MASTER
}
virtual_ipaddress {
X.X.X.X/32 brd X.X.X.255 dev eth0 label eth0:0
X.X.X.Y/32 brd X.X.X.255 dev eth0 label eth0:1
X.X.X.Z/32 brd X.X.X.255 dev eth0 label eth0:2
}
}
Optional
Prepend something like the following to keepalived.conf on both nodes if you want to receive an email when failover/failback occurs:
global_defs {
notification_email {
user@email.tld # replace with your email address
}
notification_email_from keepalived@localhost # replace with desired "FROM" address
smtp_server 127.0.0.1 # replace
smtp_connect_timeout 30
}
Testing
After the configuration is all in place, start up keepalived for the first time on both nodes:
# systemctl start keepalived
You should see your VIPs on eth0 on MASTER:
# ip a show eth0
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
...
inet X.X.X.X/32 brd X.X.X.255 scope global eth0:0
valid_lft forever preferred_lft forever
inet X.X.X.Y/32 brd X.X.X.X.255 scope global eth0:1
valid_lft forever preferred_lft forever
inet X.X.X.Z/32 brd X.X.X.255 scope global eth0:2
valid_lft forever preferred_lft forever
...
If the VIPs show up twice, it’s probably because you have network configuration files in place which are managing the IPs. The following commands should fix that:
# ifdown eth0:0 && mv /etc/sysconfig/network-scripts/ifcfg-eth0:0 /etc/sysconfig/network-scripts/ifcfg-eth0:0.bak
# ifdown eth0:1 && mv /etc/sysconfig/network-scripts/ifcfg-eth0:1 /etc/sysconfig/network-scripts/ifcfg-eth0:1.bak
# ifdown eth0:2 && mv /etc/sysconfig/network-scripts/ifcfg-eth0:2 /etc/sysconfig/network-scripts/ifcfg-eth0:2.bak
# systemctl restart keepalived
BACKUP should not have any of the VIPs on eth0
Failover
On BACKUP follow the logs:
# tailf /var/log/messages
On MASTER, take down eth1 to simulate a network failure:
# ifdown eth1
If all goes as planned, you should see something like the following in the log output on BACKUP:
...
Sep 15 19:41:47 BACKUP Keepalived_vrrp[12345]: VRRP_Instance(HAPROXY_VIPS) Transition to MASTER STATE
Sep 15 19:41:48 BACKUP Keepalived_vrrp[12345]: VRRP_Instance(HAPROXY_VIPS) Entering MASTER STATE
Sep 15 19:41:48 BACKUP Keepalived_vrrp[12345]: VRRP_Instance(HAPROXY_VIPS) setting protocol VIPs.
Sep 15 19:41:48 BACKUP Keepalived_vrrp[12345]: VRRP_Instance(HAPROXY_VIPS) Sending gratuitous ARPs on eth0 for X.X.X.X
Sep 15 19:41:48 BACKUP Keepalived_vrrp[12345]: VRRP_Instance(HAPROXY_VIPS) Sending gratuitous ARPs on eth0 for X.X.X.Y
Sep 15 19:41:48 BACKUP Keepalived_vrrp[12345]: VRRP_Instance(HAPROXY_VIPS) Sending gratuitous ARPs on eth0 for X.X.X.Z
...
You should also see the VIPs on BACKUP now. MASTER should no longer have any of the VIPs.
Failback
On MASTER:
ifup eth2
On BACKUP:
# tail /var/log/messages
...
Sep 15 19:41:56 BACKUP Keepalived_vrrp[12345]: VRRP_Instance(HAPROXY_VIPS) Received higher prio advert
Sep 15 19:41:56 BACKUP Keepalived_vrrp[12345]: VRRP_Instance(HAPROXY_VIPS) Entering BACKUP STATE
Sep 15 19:41:56 BACKUP Keepalived_vrrp[12345]: VRRP_Instance(HAPROXY_VIPS) removing protocol VIPs.
The VIPs should now show up on eth0 on MASTER and should no longer be on BACKUP.
Other Items
Keep in mind that Keepalived doesn’t manage the HAProxy service as would cluster software like Pacemaker. This means you’ll need to ensure that HAProxy is both enabled and started if you want to continue serving requests via HAProxy after the failover occurs. As such, HAProxy will need to be listening on the VIPs before failover. Normally, a service cannot bind to or listen on an IP that isn’t associated with any interface. This would be the case with BACKUP before failover and on MASTER after failover. To get around this, will need make a slight kernel parameter change on both nodes:
# sysctl -w net.ipv4.ip_nonlocal_bind=1 # immediate change # echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf # persistent changeYou may have noticed the unicast parameters in keepalived.conf. Many tutorials simply use multicast (the default) for keepalived communication. I chose unicast because on many networks (including mine) ACLs on the firewall drop all multicast traffic. Unicast ensures that the VRRP communication is happening only between the two nodes running keepalived and all other hosts on the network (VRRP net) don’t see any of the VRRP advertisements.
If you find you’re still unable to get a working configuration, it may be the case that IPTables on either node is dropping VRRP communication. You could add a blanket allow rule on both nodes to resolve this issue or rule it out:
# iptables -I INPUT -i eth1 -j ACCEPT -m comment --comment "VRRP communication"