使用Keepalived如何實(shí)現(xiàn)SFTP服務(wù)的高可用

更新時(shí)間：2024年04月12日 09:22:18 作者：Meepoljd

這篇文章主要介紹了使用Keepalived如何實(shí)現(xiàn)SFTP服務(wù)的高可用問(wèn)題,具有很好的參考價(jià)值,希望對(duì)大家有所幫助,如有錯(cuò)誤或未考慮完全的地方,望不吝賜教

背景

這個(gè)事情的背景是生產(chǎn)環(huán)境的數(shù)據(jù)采集流程時(shí)不時(shí)會(huì)出問(wèn)題（這個(gè)也是不可避免的），目前的處理手段是：所有的數(shù)據(jù)接口服務(wù)器（也就是存放原始數(shù)據(jù)等待采集的服務(wù)器）都部署一模一樣的2臺(tái)，數(shù)據(jù)也傳的一模一樣，然后當(dāng)采集程序采集當(dāng)前節(jié)點(diǎn)的數(shù)據(jù)異常的時(shí)候，由運(yùn)維人員去改配置手動(dòng)的切換。

這樣操作面臨的問(wèn)題不用多說(shuō)，首先就是時(shí)效性的問(wèn)題，就算數(shù)據(jù)斷了能夠及時(shí)發(fā)出告警，等到運(yùn)維人員處理完成那也是至少幾十分鐘后了，所以高可用的實(shí)現(xiàn)還是很有必要的。

簡(jiǎn)單調(diào)研下來(lái)還是只能用Keepalived來(lái)做這個(gè)軟負(fù)載，畢竟客戶不愿意出錢（鐵公雞）去購(gòu)買F5設(shè)備這些東西做硬負(fù)載，所以就基于這個(gè)目標(biāo)開干。

由于是做高可用，且我們的使用場(chǎng)景不是主備，應(yīng)該是角色相同的兩個(gè)服務(wù)器，所以不使用Keepalived的搶占式機(jī)制，改為非搶占。

準(zhǔn)備工作

服務(wù)器和VIP

準(zhǔn)備2臺(tái)服務(wù)器和一個(gè)VIP：

服務(wù)器A：172.18.0.26，sftp等服務(wù)提前裝好
服務(wù)器B：172.18.0.27，sftp等服務(wù)提前裝好
VIP：172.18.0.78，虛擬IP，用于對(duì)外訪問(wèn)，在AB之間漂移

Keepalived軟件

因?yàn)楣倬W(wǎng)提供的是源碼包的下載，為了方便后續(xù)實(shí)施人員在環(huán)境上做安裝操作，做成RPM包更穩(wěn)妥點(diǎn)，Centos7的官方倉(cāng)庫(kù)RPM包也是很老的版本，好像是1.3.X的，最新版已經(jīng)2.2.8了，所以這里要自己打一下包，先寫SPEC文件，保存為keepalived.spec：

%bcond_without snmp
%bcond_without vrrp
%bcond_without sha1
%bcond_with profile
%bcond_with debug
%if 0%{?rhel} && 0%{?rhel} <= 6
%bcond_with nftables
%bcond_with track_process
%bcond_with libiptc
%else
%bcond_without nftables
%bcond_without track_process
%bcond_without libiptc
%endif

%global _hardened_build 1

Name: keepalived
Summary: High Availability monitor built upon LVS, VRRP and service pollers
Version: 2.2.8
Release: 1%{?dist}
License: GPLv2+
URL: http://www.keepalived.org/
Group: System Environment/Daemons

Source0: http://www.keepalived.org/software/keepalived-%{version}.tar.gz
Source1: keepalived.service
Source2: keepalived.init

# distribution specific definitions
%define use_systemd (0%{?fedora} && 0%{?fedora} >= 18) || (0%{?rhel} && 0%{?rhel} >= 7) || (0%{?suse_version} == 1315)

%if %{use_systemd}
Requires(post): systemd
Requires(preun): systemd
Requires(postun): systemd
%else
Requires(post): /sbin/chkconfig
Requires(preun): /sbin/chkconfig
Requires(preun): /sbin/service
Requires(postun): /sbin/service
%endif

BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
%if %{with snmp}
BuildRequires: net-snmp-devel
%endif
%if %{use_systemd}
BuildRequires: systemd-units
%endif
BuildRequires: openssl-devel
BuildRequires: libnl3-devel
BuildRequires: ipset-devel
BuildRequires: iptables-devel
BuildRequires: libnfnetlink-devel

%if (0%{?rhel} && 0%{?rhel} >= 7)
Requires: ipset-libs
%endif

%description
Keepalived provides simple and robust facilities for load balancing
and high availability to Linux system and Linux based infrastructures.
The load balancing framework relies on well-known and widely used
Linux Virtual Server (IPVS) kernel module providing Layer4 load
balancing. Keepalived implements a set of checkers to dynamically and
adaptively maintain and manage load-balanced server pool according
their health. High availability is achieved by VRRP protocol. VRRP is
a fundamental brick for router failover. In addition, keepalived
implements a set of hooks to the VRRP finite state machine providing
low-level and high-speed protocol interactions. Keepalived frameworks
can be used independently or all together to provide resilient
infrastructures.

%prep
%setup -q

%build
%configure \
    %{?with_debug:--enable-debug} \
    %{?with_profile:--enable-profile} \
    %{!?with_vrrp:--disable-vrrp} \
    %{?with_snmp:--enable-snmp --enable-snmp-rfc} \
    %{?with_sha1:--enable-sha1} \
    %{!?with_nftables:--disable-nftables} \
    %{!?with_track_process:--disable-track-process} \
    %{!?with_libiptc:--disable-libiptc}
%{__make} %{?_smp_mflags} STRIP=/bin/true

%install
rm -rf %{buildroot}
make install DESTDIR=%{buildroot}
rm -rf %{buildroot}%{_sysconfdir}/keepalived/samples/
rm -rf %{buildroot}%{_defaultdocdir}/keepalived/
%if %{use_systemd}
rm -rf %{buildroot}%{_initrddir}/
%{__install} -p -D -m 0644 %{SOURCE1} %{buildroot}%{_unitdir}/keepalived.service
%else
rm %{buildroot}%{_sysconfdir}/init/keepalived.conf
%{__install} -p -D -m 0755 %{SOURCE2} %{buildroot}%{_initrddir}/keepalived
%endif
mkdir -p %{buildroot}%{_libexecdir}/keepalived

%clean
rm -rf %{buildroot}

%post
%if %{use_systemd}
%systemd_post keepalived.service
%else
/sbin/chkconfig --add keepalived
%endif

%preun
%if %{use_systemd}
%systemd_preun keepalived.service
%else
if [ "$1" -eq 0 ]; then
    /sbin/service keepalived stop >/dev/null 2>&1
    /sbin/chkconfig --del keepalived
fi
%endif

%postun
%if %{use_systemd}
%systemd_postun_with_restart keepalived.service
%else
if [ "$1" -eq 1 ]; then
    /sbin/service keepalived condrestart >/dev/null 2>&1 || :
fi
%endif

%files
%defattr(-,root,root,-)
%attr(0755,root,root) %{_sbindir}/keepalived
%config(noreplace) %attr(0644,root,root) %{_sysconfdir}/sysconfig/keepalived
%config(noreplace) %attr(0644,root,root) %{_sysconfdir}/keepalived/keepalived.conf.sample
%doc AUTHOR ChangeLog CONTRIBUTORS COPYING README README.md TODO
%doc doc/keepalived.conf.SYNOPSIS doc/samples/keepalived.conf.*
%dir %{_sysconfdir}/keepalived/
%dir %{_libexecdir}/keepalived/
%if %{with snmp}
%{_datadir}/snmp/mibs/KEEPALIVED-MIB.txt
%{_datadir}/snmp/mibs/VRRP-MIB.txt
%{_datadir}/snmp/mibs/VRRPv3-MIB.txt
%endif
%{_bindir}/genhash
%if %{use_systemd}
%{_unitdir}/keepalived.service
%else
%{_initrddir}/keepalived
%endif
%{_mandir}/man1/genhash.1*
%{_mandir}/man5/keepalived.conf.5*
%{_mandir}/man8/keepalived.8*

把這個(gè)spec文件放在rpmbuild/SPECS下，把官網(wǎng)下載的源碼包放在rpmbuild/SOURCES下然后執(zhí)行編譯命令：

rpmbuild -bb ~/rpmbuild/SPECS/keepalived.spec

該命令成功后會(huì)在rpmbuild/RPMS/x86_64/目錄下生成這兩個(gè)rpm包：

我們只需要用keepalived-2.2.8-1.el7.x86_64.rpm包就行了。

實(shí)施

安裝Keepalived軟件

rpm包拷貝到服務(wù)器A和服務(wù)器B上做安裝，或者自己會(huì)做yum就做成yum裝，要方便些，不用到處scp：

rpm -ivh keepalived-2.2.8-1.el7.x86_64.rpm

如果報(bào)了缺少libnetsnmp之類的依賴，需要安裝幾個(gè)依賴軟件：

yum install -y net-snmp-libs net-snmp-agent-libs

準(zhǔn)備健康檢查腳本

準(zhǔn)備以下健康腳本用來(lái)檢查服務(wù)的狀態(tài)，這個(gè)腳本可以根據(jù)實(shí)際情況來(lái)改動(dòng)：

#!/bin/bash

# 檢查SSH服務(wù)是否正在運(yùn)行
ssh_status=$(systemctl is-active sshd)

# 判斷SSH服務(wù)狀態(tài)
if [ "$ssh_status" = "active" ]; then
    exit 0
else
    systemctl stop keepalived
    exit 1
fi

當(dāng)sshd服務(wù)異常的時(shí)候，sftp自然不能用了，keepalived也就沒(méi)有必要啟動(dòng)了，于是執(zhí)行stop邏輯

配置服務(wù)器

上面說(shuō)到，我們使用的是非搶占式的模式，所以配置文件這樣寫，只要注意改動(dòng)幾個(gè)特別說(shuō)明的字段就可以：

! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_script check_sftp {
    script "/etc/keepalived/scripts/check_sftp.sh"
    interval 2
    timeout 5
    fall 2
    rise 1
}


# 節(jié)點(diǎn)配置內(nèi)容
vrrp_instance VI_1 {
    state BACKUP
    interface p1p2 # 綁定VIP的網(wǎng)卡
    nopreempt # 配置為非搶占式
    virtual_router_id 53
    mcast_src_ip 172.18.0.26
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.18.0.78
    }

    track_script {
        check_sftp
    }
}

# 注意此處
virtual_server 172.18.0.78 22 { # 虛擬服務(wù)
    delay_loop 6
    lb_algo rr
    lb_kind DR
    nat_mask 255.255.255.0    
    persistence_timeout 0
    protocol TCP
    
    real_server 172.18.0.26 22 { # 實(shí)際對(duì)應(yīng)的服務(wù)，這是A服務(wù)器的
        weight 1
        TCP_CHECK {
            connect_timeout 8
            nb_get_retry 3
            delay_before_retry 3
            connect_port 22 # 服務(wù)端口
        }
    }

    real_server 172.18.0.27 22 { # 實(shí)際對(duì)應(yīng)的服務(wù)，這是B服務(wù)器的
        weight 1
        TCP_CHECK {
            connect_timeout 8
            nb_get_retry 3
            delay_before_retry 3
            connect_port 22 # 服務(wù)端口
        }
    }
}

按照上述配置配置好2臺(tái)服務(wù)器，然后分別啟動(dòng)keepalived服務(wù)：

systemctl start keepalived
systemctl status keepalived

我們可以通過(guò)ip addr查看當(dāng)前vip綁定的機(jī)器是服務(wù)器B

測(cè)試驗(yàn)證

接下來(lái)測(cè)試驗(yàn)證一下高可用的能力，為了方便區(qū)分，首先在兩個(gè)服務(wù)器的root目錄下放不同的文件，如果使用別的用戶測(cè)試就放在對(duì)應(yīng)用戶的默認(rèn)目錄下就行，編寫以下的測(cè)試腳本：

import time
import paramiko

host = "172.18.0.78"
username = "root"
password = "xxxxx"

print("開始運(yùn)行測(cè)試腳本")
ssh_client = paramiko.SSHClient()
print("首次建立ssh和sftp連接")
ssh_client.set_missing_host_key_policy(paramiko.WarningPolicy)
ssh_client.connect(hostname=host, username=username, password=password)
sftp = ssh_client.open_sftp()
while True:
    try:
        tran = ssh_client.get_transport()
        if tran.is_active():
            print("檢測(cè)到ssh連接已經(jīng)建立，直接執(zhí)行測(cè)試邏輯")
            # 如果連接已經(jīng)建立
            print(sftp.listdir())
        else:

            ssh_client.connect(
                hostname=host, username=username, password=password)
            sftp = ssh_client.open_sftp()
    except Exception as e:
        print("檢測(cè)到ssh發(fā)生主備切換，重新建立sftp連接")
        ssh_client.connect(hostname=host, username=username, password=password)
        sftp = ssh_client.open_sftp()
    time.sleep(10)

腳本會(huì)每隔十秒就在sftp上面列出以下當(dāng)前目錄，運(yùn)行起來(lái)：