企业 keepalived 高可用项目实战

最新推荐文章于 2024-06-03 10:27:06 发布

转载最新推荐文章于 2024-06-03 10:27:06 发布 · 442 阅读

2 ·

本内容遵循CC 4.0 BY-SA版权协议

原文链接：https://www.zhihu.com/org/qian-feng-yun-ji-suan-xue-yuan/posts?page=1

收录于

1092151971：群云计算交流

本文介绍了Keepalived的工作原理，特别是其VRRP协议在防止单点故障中的作用。讲解了如何利用Nginx结合Keepalived实现七层负载均衡，包括轮询、会话保持等策略。同时，详细阐述了Keepalived在LVS_Director场景下的应用，配置了LVS路由模式和健康检查。此外，还涉及了Keepalived在MySQL主从复制高可用中的角色，以及脑裂问题的解决方案。

1、Keepalived VRRP 介绍

 keepalived是什么
     keepalived是集群管理中保证集群高可用的一个服务软件，用来防止单点故障。
 
 keepalived工作原理
     keepalived是以VRRP协议为实现基础的，VRRP全称Virtual Router Redundancy Protocol，即虚拟路由冗余协议。
 
     虚拟路由冗余协议，可以认为是实现路由器高可用的协议，即将N台提供相同功能的路由器组成一个路由器组，这个组里面有一个master和多个backup，master上面有一个对外提供服务的vip（该路由器所在局域网内其他机器的默认路由为该vip），master会发组播，当backup收不到vrrp包时就认为master宕掉了，这时就需要根据VRRP的优先级来选举一个backup当master。这样的话就可以保证路由器的高可用了。
 
 keepalived主要有三个模块，分别是core、check和vrrp。core模块为keepalived的核心，负责主进程的启动、维护以及全局配置文件的加载和解析。check负责健康检查，包括常见的各种检查方式。vrrp模块是来实现VRRP协议的。
 ==============================================
 脑裂：
 Keepalived的BACKUP主机在收到不MASTER主机报文后就会切换成为master，如果是它们之间的通信线路出现问题，无法接收到彼此的组播通知，但是两个节点实际都处于正常工作状态，这时两个节点均为master强行绑定虚拟IP，导致不可预料的后果，这就是脑裂。
 解决方式:
 1、添加更多的检测手段，比如ping对方等等。尽量减少"裂脑"发生机会。(指标不治本，只是提高了检测到的概率)；
 2、爆头，将master停掉。然后检查机器之间的防火墙。网络之间的通信

2、Nginx+keepalived实现七层的负载均衡(同类服务)

Nginx通过Upstream模块实现负载均衡

upstream 支持的负载均衡算法

 轮询（默认）:可以通过weight指定轮询的权重，权重越大，被调度的次数越多
 ip_hash：可以实现会话保持，将同一客户的IP调度到同一样后端服务器，可以解决session的问题，不能使用weight
 fair：可以根据请求页面的大小和加载时间长短进行调度，使用第三方的upstream_fair模块
 url_hash：按请求的url的hash进行调度，从而使每个url定向到同一服务器，使用第三方的url_hash模块
 配置安装nginx 所有的机器，关闭防火墙和selinux
 [root@nginx-proxy ~]# cd /etc/yum.repos.d/
 [root@nginx-proxy yum.repos.d]# vim nginx.repo
 [nginx-stable]
 name=nginx stable repo
 baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
 gpgcheck=0
 enabled=1
 [root@nginx-proxy yum.repos.d]# yum install yum-utils -y
 [root@nginx-proxy yum.repos.d]# yum install nginx -y
 调度到不同组后端服务器
 网站分区进行调度
 =================================================================================
 
 拓扑结构
 
                             [vip: 20.20.20.20]
 
                         [LB1 Nginx]     [LB2 Nginx]
                         192.168.1.2     192.168.1.3
 
         [index]     [milis]      [videos]      [images]       [news]
          1.11        1.21          1.31           1.41         1.51
          1.12        1.22          1.32           1.42         1.52
          1.13        1.23          1.33           1.43         1.53
          ...         ...            ...           ...           ...
          /web     /web/milis    /web/videos     /web/images   /web/news
       index.html  index.html     index.html      index.html   index.html
 一、实施过程 
 1、选择两台nginx服务器作为代理服务器。
 2、给两台代理服务器安装keepalived制作高可用生成VIP
 3、配置nginx的负载均衡
 以上两台nginx服务器配置文件一致
 根据站点分区进行调度
 配置upstream文件
 [root@nginx-proxy ~]# cd /etc/nginx/conf.d/
 [root@nginx-proxy conf.d]# cp default.conf proxy.conf
 [root@nginx-proxy conf.d]# mv default.conf default.conf.bak
 [root@nginx-proxy conf.d]# vim upstream.conf
     upstream index {
         server 192.168.246.162:80 weight=1 max_fails=2 fail_timeout=2;
         server 192.168.246.163:80 weight=2 max_fails=2 fail_timeout=2;
        }       
 [root@nginx-proxy conf.d]# vim proxy.conf
 server {
         listen 80;
         server_name     localhost;
         access_log  /var/log/nginx/host.access.log  main;
         
         location / {
          proxy_pass http://index;
          proxy_redirect default;
          proxy_set_header Host $http_host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         }
 }
 将nginx的配置文件拷贝到另一台代理服务器中:
 [root@nginx-proxy conf.d]# scp proxy.conf 192.168.246.161:/etc/nginx/conf.d/ 
 [root@nginx-proxy conf.d]# scp upstream.conf 192.168.246.161:/etc/nginx/conf.d/
 二、Keepalived实现调度器HA
 注：主/备调度器均能够实现正常调度
 1. 主/备调度器安装软件
 [root@nginx-proxy-master ~]# yum install -y keepalived
 [root@nginx-proxy-slave ~]# yum install -y keepalived
 [root@nginx-proxy-master ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
 [root@nginx-proxy-master ~]# vim /etc/keepalived/keepalived.conf
 ! Configuration File for keepalived
 
 global_defs {
    router_id directory1   #辅助改为directory2
 }
 
 vrrp_instance VI_1 {
     state MASTER        #定义主还是备
     interface ens33     #VIP绑定接口
     virtual_router_id 80  #整个集群的调度器一致
     priority 100         #back改为50
     advert_int 1         #检查间隔，默认为1s
     authentication {
         auth_type PASS
         auth_pass 1111
     }
     virtual_ipaddress {
         192.168.246.16/24
     }
 }
 
 [root@nginx-porxy-slave ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
 [root@nginx-proxy-slave ~]# vim /etc/keepalived/keepalived.conf
 ! Configuration File for keepalived
 
 global_defs {
    router_id directory2
 }
 
 vrrp_instance VI_1 {
     state BACKUP    #设置为backup
     interface ens33
     virtual_router_id 80
     priority 50   #辅助改为50
     advert_int 1
     authentication {
         auth_type PASS
         auth_pass 1111
     }
     virtual_ipaddress {
         192.168.246.16/24
     }
 }
 3. 启动KeepAlived（主备均启动）
 [root@nginx-proxy-master ~]# chkconfig keepalived on
 [root@nginx-proxy-master ~]# service keepalived start
 [root@nginx-proxy-master ~]# ip addr
 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet6 ::1/128 scope host 
        valid_lft forever preferred_lft forever
 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
     link/ether 00:0c:29:48:07:7d brd ff:ff:ff:ff:ff:ff
     inet 192.168.246.169/24 brd 192.168.246.255 scope global dynamic ens33
        valid_lft 1726sec preferred_lft 1726sec
     inet 192.168.246.16/24 scope global ens33
        valid_lft forever preferred_lft forever
     inet6 fe80::23e9:de18:1e67:f152/64 scope link 
        valid_lft forever preferred_lft forever
 到此：
 可以解决心跳故障keepalived
 不能解决Nginx服务故障
 4. 扩展对调度器Nginx健康检查（可选）两台都设置
 思路：
 让Keepalived以一定时间间隔执行一个外部脚本，脚本的功能是当Nginx失败，则关闭本机的Keepalived
 (1) script
 [root@nginx-proxy-master ~]# vim /etc/keepalived/check_nginx_status.sh
 #!/bin/bash                                                     
 /usr/bin/curl -I http://localhost &>/dev/null   
 if [ $? -ne 0 ];then                                            
 #   /etc/init.d/keepalived stop
     systemctl stop keepalived
 fi                                                                      
 [root@nginx-proxy-master ~]# chmod a+x /etc/keepalived/check_nginx_status.sh
 
 (2). keepalived使用script
 ! Configuration File for keepalived
 
 global_defs {
    router_id director1
 }
 vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx_status.sh"
    interval 5
 }
 
 vrrp_instance VI_1 {
     state MASTER
     interface ens33
     virtual_router_id 80
     priority 100
     advert_int 1
     authentication {
         auth_type PASS
         auth_pass 1111
     }
     virtual_ipaddress {
         192.168.246.16/24
     }
     track_script {
         check_nginx
     }
 }
 注：必须先启动nginx，再启动keepalived
 
 测试访问：

3、LVS_Director + KeepAlived

 LVS_Director + KeepAlived
 
 KeepAlived在该项目中的功能：
 1. 管理IPVS的路由表（包括对RealServer做健康检查）
 2. 实现调度器的HA
 http://www.keepalived.org
 
 Keepalived所执行的外部脚本命令建议使用绝对路径
 =================================================================================
 实施步骤：
 1. 主/备调度器安装软件
 [root@lvs-keepalived-master ~]# yum -y install ipvsadm keepalived 
 [root@lvs-keepalived-slave ~]# yum -y install ipvsadm keepalived
 2. Keepalived
 lvs-master
 [root@ha-proxy-master ~]# vim /etc/keepalived/keepalived.conf
 ! Configuration File for keepalived
 
 global_defs {
    router_id lvs-keepalived-master    #辅助改为lvs-backup
 }
 
 vrrp_instance VI_1 {
     state MASTER
     interface ens33                #VIP绑定接口
     virtual_router_id 80         #VRID 同一组集群，主备一致          
     priority 100            #本节点优先级，辅助改为50
     advert_int 1            #检查间隔，默认为1s
     authentication {
         auth_type PASS
         auth_pass 1111
     }
     virtual_ipaddress {
         192.168.246.110/32
     }
 }
 
 virtual_server 192.168.246.110 80 {    #LVS配置
     delay_loop 6   #健康检查rs时间间隔
     lb_algo rr     #LVS调度算法
     lb_kind DR     #LVS集群模式（路由模式）
     protocol TCP      #健康检查使用的协议
     real_server 192.168.246.162 80 {
         weight 1
         inhibit_on_failure   #当该节点失败时，把权重设置为0，而不是从IPVS中删除
         TCP_CHECK {          #健康检查
             connect_port 80   #检查的端口
             connect_timeout 3  #连接超时的时间
             }
         }
     real_server 192.168.246.163 80 {
         weight 1
         inhibit_on_failure
         TCP_CHECK {
             connect_timeout 3
             connect_port 80
             }
         }
 }
 
 [root@lvs-keepalived-slave ~]# vim /etc/keepalived/keepalived.conf
 ! Configuration File for keepalived
 
 global_defs {
    router_id lvs-keepalived-slave
 }
 
 vrrp_instance VI_1 {
     state BACKUP
     interface ens33
     virtual_router_id 80
     priority 50
     advert_int 1
     authentication {
         auth_type PASS
         auth_pass 1111
     }
     virtual_ipaddress {
         192.168.246.110/32
     }
 }
 virtual_server 192.168.246.110 80 {
     delay_loop 6
     lb_algo rr
     lb_kind DR
     protocol TCP
     real_server 192.168.246.162 80 {
         weight 1
         inhibit_on_failure
         TCP_CHECK {
             connect_port 80
             connect_timeout 3
             }
         }
     real_server 192.168.246.163 80 {
         weight 1
         inhibit_on_failure
         TCP_CHECK {
             connect_timeout 3
             connect_port 80
             }
         }
 }
 3. 启动KeepAlived（主备均启动）
 [root@lvs-keepalived-master ~]# systemctl start keepalived
 [root@lvs-keepalived-master ~]# systemctl enable keepalived
 
 [root@lvs-keepalived-master ~]# ipvsadm -Ln
 IP Virtual Server version 1.2.1 (size=4096)
 Prot LocalAddress:Port Scheduler Flags
   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
 TCP  192.168.246.110:80 rr persistent 20
   -> 192.168.246.162:80           Route   1      0          0         
   -> 192.168.246.163:80           Route   0      0          0
 
 4. 所有RS配置(nginx1,nginx2)
 配置好网站服务器，测试所有RS
 [root@test-nginx1 ~]# yum install -y nginx
 [root@test-nginx2 ~]# yum install -y nginx
 [root@test-nginx1 ~]# ip addr add dev lo 192.168.246.110/32
 [root@test-nginx1 ~]# echo "net.ipv4.conf.all.arp_ignore = 1" >> /etc/sysctl.conf
 [root@test-nginx1 ~]# sysctl -p
 [root@test-nginx1 ~]# echo "web1..." >> /usr/share/nginx/html/index.html
 [root@test-nginx1 ~]# systemctl start nginx

3.测试数据库高可用

 1.两台机器安装mysql
 [root@mysql-keep-master ~]# wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
 [root@mysql-keep-master ~]# rpm -ivh mysql80-community-release-el7-3.noarch.rpm
 [root@mysql-keep-master ~]# yum -y install yum-utils #安装yum工具包
 [root@mysql-keep-master ~]# vim /etc/yum.repos.d/mysql-community.repo

 安装mysql
 [root@mysql-keep-master ~]# yum install -y mysql-community-server
 [root@mysql-keep-master ~]# systemctl start mysqld 
 [root@mysql-keep-master ~]# systemctl enable mysqld
 修改密码
 [root@mysql-keep-master ~]# grep pass /var/log/mysqld.log 
 2020-02-18T13:54:25.706635Z 1 [Note] A temporary password is generated for root@localhost: %baWi=4!NsT)
 [root@mysql-keep-master ~]# mysqladmin -uroot -p'%baWi=4!NsT)' password 'QianFeng@123!'
 制作互为主从:
 [root@mysql-keep-master ~]# vim /etc/my.cnf   #编辑配置文件开启gtid
 log-bin=m1.log  
 server-id=1 
 gtid_mode = ON    #开启gtid
 enforce_gtid_consistency=1  #强制gtid移植性
 [root@mysql-keep-master ~]# systemctl restart mysqld 
 [root@mysql-keep-master ~]# mysql -uroot -p'QianFeng@123!'  #登录数据库
 mysql> GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'QianFeng@123';  #创建远程登录用户
 mysql> flush privileges;
 mysql> grant replication  slave,reload,super  on *.*  to  'slave'@'%'  identified by  'QianFeng@123';  ---创建主从授权用户
 mysql> flush privileges;  #刷新授权
 mysql> \e   #指定主服务器信息
 CHANGE MASTER TO
 MASTER_HOST='192.168.58.136',      #指定主服务器的ip或者主机名
 MASTER_USER='slave',           #授权用户
 MASTER_PASSWORD='QianFeng@123',  #授权用户密码
 master_auto_position=1
 -> ;
 启动slave
 mysql> start slave;
 [root@mysql-keep-backup ~]# vim /etc/my.cnf
 log-bin=m2.log
 server-id=2
 gtid_mode = ON
 enforce_gtid_consistency=1
 [root@mysql-keep-backup ~]# systemctl restart mysqld 
 [root@mysql-keep-backup ~]# mysql -uroot -p'QianFeng@123!'
 mysql> GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'QianFeng@123';
 mysql> flush privileges;
 mysql> grant replication  slave,reload,super  on *.*  to  'slave'@'%'  identified by  'QianFeng@123';
 mysql> flush privileges;
 指定主服务器信息
 mysql> \e
 CHANGE MASTER TO
 MASTER_HOST='192.168.58.135',
 MASTER_USER='slave',
 MASTER_PASSWORD='QianFeng@123',
 master_auto_position=1
 -> ;
 启动slave
 mysql> start slave;

两台机器验证是否为yes

 mysql> show slave status\G

两台机器安装keepalived

 [root@mysql-keep-master ~]# yum -y install keepalived
 [root@mysql-keep-master ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
 [root@mysql-keep-master ~]# vim /etc/keepalived/keepalived.conf
 ! Configuration File for keepalived
 
 global_defs {
    router_id master1
 }
 
 vrrp_instance VI_1 {
     state MASTER
     interface ens33
     virtual_router_id 80
     priority 100
     advert_int 1
     authentication {
         auth_type PASS
         auth_pass 1111
     }
     virtual_ipaddress {
         192.168.58.16/24
     }
 }
 将配置文件拷贝到backu机器
 [root@mysql-keep-backup ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
 [root@mysql-keep-master ~]# scp /etc/keepalived/keepalived.conf 192.168.58.136:/etc/keepalived/
 [root@mysql-keep-backup ~]# vim /etc/keepalived/keepalived.conf  #修改如下
 ! Configuration File for keepalived
 
 global_defs {
    router_id backup1
 }
 
 vrrp_instance VI_1 {
     state BACKUP
     interface ens33
     virtual_router_id 80
     priority 50
     advert_int 1
     authentication {
         auth_type PASS
         auth_pass 1111
     }
     virtual_ipaddress {
         192.168.58.16/24
     }
 }
 两台机器启动keepalived
 [root@mysql-keep-master ~]# systemctl start keepalived
 [root@mysql-keep-backup ~]# systemctl start keepalived
 [root@mysql-keep-master ~]# ip a   #vip在master上面
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet6 ::1/128 scope host 
        valid_lft forever preferred_lft forever
 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
     link/ether 00:0c:29:92:0a:64 brd ff:ff:ff:ff:ff:ff
     inet 192.168.58.135/24 brd 192.168.58.255 scope global dynamic ens33
        valid_lft 1729sec preferred_lft 1729sec
     inet 192.168.58.16/24 scope global secondary ens33
        valid_lft forever preferred_lft forever
     inet6 fe80::86f4:1d74:ad05:c61c/64 scope link 
        valid_lft forever preferred_lft forever
 测试：
 [root@mysql-keep-master ~]# mysql -uroot -p'QianFeng@123' -h 192.168.58.16  #登录vip
 mysql> create database test1;
 Query OK, 1 row affected (0.07 sec)
 
 mysql> show databases;
 +--------------------+
 | Database           |
 +--------------------+
 | information_schema |
 | mysql              |
 | performance_schema |
 | sys                |
 | test1              |
 +--------------------+
 5 rows in set (0.08 sec)
 验证
 [root@mysql-keep-backup ~]# mysql -uroot -p'QianFeng@123!'
 mysql> show databases;
 +--------------------+
 | Database           |
 +--------------------+
 | information_schema |
 | mysql              |
 | performance_schema |
 | sys                |
 | test1              |
 +--------------------+
 5 rows in set (0.12 sec)
 [root@mysql-keep-master ~]# systemctl stop keepalived #关闭master
 [root@mysql-keep-backup ~]# ip a  #vip切换backup机器
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet6 ::1/128 scope host 
        valid_lft forever preferred_lft forever
 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
     link/ether 00:0c:29:a2:0f:5c brd ff:ff:ff:ff:ff:ff
     inet 192.168.58.136/24 brd 192.168.58.255 scope global dynamic ens33
        valid_lft 1547sec preferred_lft 1547sec
     inet 192.168.58.16/24 scope global secondary ens33
        valid_lft forever preferred_lft forever
     inet6 fe80::af6e:2eff:e2d1:99cb/64 scope link 
        valid_lft forever preferred_lft forever
 测试登录
 [root@mysql-keep-master ~]# mysql -uroot -p'QianFeng@123' -h 192.168.58.16