lvs+ldirectord+pacemaker+corosync+mysql实现高可用负载均衡(一)

忘是亡心i 2022-09-19 00:17 150阅读 0赞

author：skate  
tiome：2013/12/06

**lvs+ldirectord+pacemaker+corosync+mysql实现高可用负载均衡**

环境说明  
vip：192.168.213.97  
rip1：192.168.213.91  
rip2：192.168.213.92  
client：192.168.213.93

os：centos6.3  
kernel：2.6.32-279.el6.x86\_64  
mysql version：5.5.28

1. ldirectord安装与配置

ldirectord下载  
下载地址：[http://horms.net/projects/ldirectord/download.shtml][http_horms.net_projects_ldirectord_download.shtml]

\# unzip resource-agents-master.zip  
\# cd resource-agents-master  
\# ./autogen.sh  
\# ./configure  
\# make && make install

此时ldirectord还是不能使用，比如查看状态，如下：

\[root@localhost resource-agents-master\]\# service ldirectord status  
Can't locate Socket6.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor\_perl /usr/share/perl5/vendor\_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/sbin/ldirectord line 838.  
BEGIN failed--compilation aborted at /usr/sbin/ldirectord line 838.  
\[root@localhost resource-agents-master\]\#

需要安装一些perl的包

\[root@localhost resource-agents-master\]\# yum install perl-Socket6 perl-IO-Socket-INET6 perl-Email-Date-Format perl-TimeDate perl-Pod-Escapes perl-Pod-Simple perl-Test-Pod perl-MailTools perl-libwww-perl  
....  
....  
Dependencies Resolved

====================================================================================================================================  
 Package                                      Arch                   Version                             Repository            Size  
====================================================================================================================================  
Installing:  
 perl-IO-Socket-INET6                         noarch                 2.56-4.el6                          base                  17 k  
 perl-Socket6                                 x86\_64                 0.23-4.el6                          base                  27 k  
 perl-Test-Pod                                noarch                 1.40-1.el6                          base                  14 k  
Updating:  
 perl-Pod-Escapes                             x86\_64                 1:1.04-136.el6                      base                  32 k  
 perl-Pod-Simple                              x86\_64                 1:3.13-136.el6                      base                 212 k  
Installing for dependencies:  
 db4-cxx                                      x86\_64                 4.7.25-18.el6\_4                     base                 588 k  
 db4-devel                                    x86\_64                 4.7.25-18.el6\_4                     base                 6.6 M  
 gdbm-devel                                   x86\_64                 1.8.0-36.el6                        base                  25 k  
 perl-CGI                                     x86\_64                 3.51-136.el6                        base                 209 k  
Updating for dependencies:  
 db4                                          x86\_64                 4.7.25-18.el6\_4                     base                 563 k  
 db4-utils                                    x86\_64                 4.7.25-18.el6\_4                     base                 130 k  
 perl                                         x86\_64                 4:5.10.1-136.el6                    base                  10 M  
 perl-Archive-Extract                         x86\_64                 1:0.38-136.el6                      base                  40 k  
 perl-Archive-Tar                             x86\_64                 1.58-136.el6                        base                  73 k  
 perl-CPAN                                    x86\_64                 1.9402-136.el6                      base                 246 k  
 perl-CPANPLUS                                x86\_64                 0.88-136.el6                        base                 307 k  
 perl-Compress-Raw-Bzip2                      x86\_64                 2.021-136.el6                       base                  45 k  
 perl-Compress-Raw-Zlib                       x86\_64                 1:2.021-136.el6                     base                  69 k  
 perl-Compress-Zlib                           x86\_64                 2.021-136.el6                       base                  45 k  
 perl-Digest-SHA                              x86\_64                 1:5.47-136.el6                      base                  64 k  
 perl-ExtUtils-CBuilder                       x86\_64                 1:0.27-136.el6                      base                  48 k  
 perl-ExtUtils-Embed                          x86\_64                 1.28-136.el6                        base                  31 k  
 perl-ExtUtils-MakeMaker                      x86\_64                 6.55-136.el6                        base                 293 k  
 perl-ExtUtils-ParseXS                        x86\_64                 1:2.2003.0-136.el6                  base                  45 k  
 perl-File-Fetch                              x86\_64                 0.26-136.el6                        base                  41 k  
 perl-IO-Compress-Base                        x86\_64                 2.021-136.el6                       base                  69 k  
 perl-IO-Compress-Bzip2                       x86\_64                 2.021-136.el6                       base                  48 k  
 perl-IO-Compress-Zlib                        x86\_64                 2.021-136.el6                       base                 135 k  
 perl-IO-Zlib                                 x86\_64                 1:1.09-136.el6                      base                  33 k  
 perl-IPC-Cmd                                 x86\_64                 1:0.56-136.el6                      base                  46 k  
 perl-Locale-Maketext-Simple                  x86\_64                 1:0.18-136.el6                      base                  31 k  
 perl-Log-Message                             x86\_64                 1:0.02-136.el6                      base                  46 k  
 perl-Log-Message-Simple                      x86\_64                 0.04-136.el6                        base                  28 k  
 perl-Module-Build                            x86\_64                 1:0.3500-136.el6                    base                 229 k  
 perl-Module-CoreList                         x86\_64                 2.18-136.el6                        base                  65 k  
 perl-Module-Load                             x86\_64                 1:0.16-136.el6                      base                  28 k  
 perl-Module-Load-Conditional                 x86\_64                 0.30-136.el6                        base                  34 k  
 perl-Module-Loaded                           x86\_64                 1:0.02-136.el6                      base                  27 k  
 perl-Module-Pluggable                        x86\_64                 1:3.90-136.el6                      base                  40 k  
 perl-Object-Accessor                         x86\_64                 1:0.34-136.el6                      base                  37 k  
 perl-Package-Constants                       x86\_64                 1:0.02-136.el6                      base                  26 k  
 perl-Params-Check                            x86\_64                 1:0.26-136.el6                      base                  35 k  
 perl-Parse-CPAN-Meta                         x86\_64                 1:1.40-136.el6                      base                  29 k  
 perl-Term-UI                                 x86\_64                 0.20-136.el6                        base                  39 k  
 perl-Test-Harness                            x86\_64                 3.17-136.el6                        base                 231 k  
 perl-Test-Simple                             x86\_64                 0.92-136.el6                        base                 112 k  
 perl-Time-HiRes                              x86\_64                 4:1.9721-136.el6                    base                  48 k  
 perl-Time-Piece                              x86\_64                 1.15-136.el6                        base                  46 k  
 perl-core                                    x86\_64                 5.10.1-136.el6                      base                  23 k  
 perl-devel                                   x86\_64                 4:5.10.1-136.el6                    base                 423 k  
 perl-libs                                    x86\_64                 4:5.10.1-136.el6                    base                 578 k  
 perl-parent                                  x86\_64                 1:0.221-136.el6                     base                  27 k  
 perl-suidperl                                x86\_64                 4:5.10.1-136.el6                    base                  50 k  
 perl-version                                 x86\_64                 3:0.77-136.el6                      base                  51 k  
 zlib                                         x86\_64                 1.2.3-29.el6                        base                  73 k  
 zlib-devel                                   x86\_64                 1.2.3-29.el6                        base                  44 k

Transaction Summary  
====================================================================================================================================  
Install       7 Package(s)  
Upgrade      49 Package(s)

Total download size: 23 M

....  
....

继续看看ldirectord是否可用

\[root@localhost resource-agents-master\]\# service ldirectord status  
Can not find ipvsadm at /usr/sbin/ldirectord line 887.  
\[root@localhost resource-agents-master\]\#

发现没有安装ipvsadm，安装如下：

\# yum install ipvsadm  
....  
Dependencies Resolved

Transaction Summary  
====================================================================================================================================  
Install       1 Package(s)

Total download size: 41 k  
....  
....

\[root@localhost resource-agents-master\]\# service ldirectord status  
Config file ldirectord.cf not found

想查看配置文件放在何处，可以查看文件“/etc/init.d/ldirectord”和“/usr/sbin/ldirectord”；默认是放在“/etc/ha.d/ldirectord.cf”。把在安装目录的样本配置复制到“/etc/ha.d/”下，然后再修改。这时ldirectord服务才可以正常工作

这回可用了

\[root@localhost resource-agents-master\]\# service ldirectord status  
ldirectord is stopped for /etc/ha.d/ldirectord.cf  
\[root@localhost resource-agents-master\]\# service ldirectord status  
ldirectord is stopped for /etc/ha.d/ldirectord.cf  
\[root@localhost resource-agents-master\]\# service ldirectord start  
Starting ldirectord... success  
\[root@localhost resource-agents-master\]\# service ldirectord status  
ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 28911  
\[root@localhost resource-agents-master\]\#

安装参考：  
[http://blog.chinaunix.net/uid-17268883-id-3145428.html][http_blog.chinaunix.net_uid-17268883-id-3145428.html]

2. lvs安装

LVS软件包括二部分：

A.IPVS模块，LVS已经是Linux标准内核的一部分,直接被编译在内核中!  
 B.IPVS管理工具ipvsadm

检测IPVS模块是否真的编译到内核中去了  
\[root@master mysqlinstall\]\# modprobe -l |grep ipvs  
kernel/net/netfilter/ipvs/ip\_vs.ko  
kernel/net/netfilter/ipvs/ip\_vs\_rr.ko  
kernel/net/netfilter/ipvs/ip\_vs\_wrr.ko  
kernel/net/netfilter/ipvs/ip\_vs\_lc.ko  
kernel/net/netfilter/ipvs/ip\_vs\_wlc.ko  
kernel/net/netfilter/ipvs/ip\_vs\_lblc.ko  
kernel/net/netfilter/ipvs/ip\_vs\_lblcr.ko  
kernel/net/netfilter/ipvs/ip\_vs\_dh.ko  
kernel/net/netfilter/ipvs/ip\_vs\_sh.ko  
kernel/net/netfilter/ipvs/ip\_vs\_sed.ko  
kernel/net/netfilter/ipvs/ip\_vs\_nq.ko  
kernel/net/netfilter/ipvs/ip\_vs\_ftp.ko

ipvsadm管理工具安装上面介绍了，通过yum安装比较方便，可以解决很多依赖关系，如下：  
\# yum install ipvsadm

安装参考：  
[http://beyondhdf.blog.51cto.com/229452/1331874][http_beyondhdf.blog.51cto.com_229452_1331874]  
[http://soulboy.blog.51cto.com/4007306/1299896][http_soulboy.blog.51cto.com_4007306_1299896]  
[http://zh.linuxvirtualserver.org/node/98][http_zh.linuxvirtualserver.org_node_98]  
[http://itnihao.blog.51cto.com/1741976/744237][http_itnihao.blog.51cto.com_1741976_744237]

Directorer的lvsdr脚本

\[root@localhost mysqlinstall\]\# more lvsdr.sh  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
\#Director script:  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
\#!/bin/bash  
\#  
\# LVS script for VS/DR  
\#  
\# Startup script handle the initialisation of LVS  
\# chkconfig: - 28 72  
\# description: Initialise director of the Linux Virtual Server  
\#  
\# Author: zxg  
\# CreateTime: 2013/12/06  
\#  
\# Modificator:   
\#             zxg: repair bug  
\#  
\#  
\# Source function library  
. /etc/rc.d/init.d/functions  
\#  
VIP=192.168.213.97  
RIP1=192.168.213.91  
RIP2=192.168.213.92  
ETHx=eth0:3  
PORT=3306  
\#

start() \{  
      /sbin/ifconfig $ETHx $VIP broadcast $VIP netmask 255.255.255.255 up  
      /sbin/route add -host $VIP dev $ETHx  
\# Since this is the Director we must be able to forward packets  
     echo 1 > /proc/sys/net/ipv4/ip\_forward  
\# Clear all iptables rules.  
     /sbin/iptables -F  
\# Reset iptables counters.  
     /sbin/iptables -Z  
\# Clear all ipvsadm rules/services.  
     /sbin/ipvsadm -C  
\# Add an IP virtual service for VIP 192.168.0.219 port $PORT  
\# In this recipe, we will use the round-robin scheduling method.  
\# In production, however, you should use a weighted, dynamic scheduling method.  
    /sbin/ipvsadm -A -t $VIP:$PORT -s wrr  
\# Now direct packets for this VIP to  
\# the real server IP (RIP) inside the cluster  
    /sbin/ipvsadm -a -t $VIP:$PORT -r $RIP1 -g -w 1  
    /sbin/ipvsadm -a -t $VIP:$PORT -r $RIP2 -g -w 3  
    /bin/touch /var/lock/subsys/ipvsadm &> /dev/null  
\}

stop() \{  
\# Stop forwarding packets  
    echo 0 > /proc/sys/net/ipv4/ip\_forward  
\# Reset ipvsadm  
    /sbin/ipvsadm -C  
\# Bring down the VIP interface  
    /sbin/ifconfig $ETHx down  
    /sbin/route del $VIP  
    /bin/rm -f /var/lock/subsys/ipvsadm  
    echo "ipvs stopped..."

status() \{  
  if \[ ! -e /var/lock/subsys/ipvsadm \]; then  
    echo "ipvsadm stopped ..."  
  else  
    echo "ipvs is running ..."  
    ipvsadm -L -n  
  fi  
\}

restart() \{  
  stop  
  start  
\}

case "$1" in  
  start)  
    \[ -f "/var/lock/subsys/ipvsadm" \] && exit 0  
    start          
  ;;  
  stop)  
    stop  
  ;;

restart)  
    restart  
  ;;  
  status)  
    
   status  
  ;;  
  \*)  
  echo "Usage: $0 \{start|stop|status|restart\}"  
  ;;  
esac

realserver的运行脚本

\[root@master mysqlinstall\]\# more lvsrl.sh  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
\#RealServer脚本:  
   
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
\#!/bin/bash  
\#  
\# Script to start LVS DR real server.  
\# description: LVS DR real server  
\#  
\# Author: zxg  
\# CreateTime: 2013/12/06  
\#  
\# Modificator:   
\#             zxg: repair bug  
\#  
\#  
.  /etc/rc.d/init.d/functions  
VIP=192.168.213.97  
host=\`/bin/hostname\`  
case "$1" in  
start)  
       \# Start LVS-DR real server on this machine.  
        /sbin/ifconfig lo down  
        /sbin/ifconfig lo up  
        echo 1 > /proc/sys/net/ipv4/conf/lo/arp\_ignore  
        echo 2 > /proc/sys/net/ipv4/conf/lo/arp\_announce  
        echo 1 > /proc/sys/net/ipv4/conf/all/arp\_ignore  
        echo 2 > /proc/sys/net/ipv4/conf/all/arp\_announce  
        /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up  
        /sbin/route add -host $VIP dev lo:0  
;;  
stop)  
        \# Stop LVS-DR real server loopback device(s).  
        /sbin/ifconfig lo:0 down  
        echo 0 > /proc/sys/net/ipv4/conf/lo/arp\_ignore  
        echo 0 > /proc/sys/net/ipv4/conf/lo/arp\_announce  
        echo 0 > /proc/sys/net/ipv4/conf/all/arp\_ignore  
        echo 0 > /proc/sys/net/ipv4/conf/all/arp\_announce  
;;  
status)  
        \# Status of LVS-DR real server.  
        islothere=\`/sbin/ifconfig lo:0 | grep $VIP\`  
        isrothere=\`netstat -rn | grep "lo:0" | grep $VIP\`  
        if \[ ! "$islothere" -o ! "isrothere" \];then  
            \# Either the route or the lo:0 device  
            \# not found.  
            echo "LVS-DR real server Stopped."  
        else  
            echo "LVS-DR real server Running."  
        fi  
;;  
\*)  
            \# Invalid entry.  
            echo "$0: Usage: $0 \{start|status|stop\}"  
            exit 1  
;;  
esac

**先把脚本lvsrl.sh部署到所有的realserver上，并启动  
然后把脚本lvsdr.sh部署到director上，并启动**

可以用如下两个命令查看lvs的运行状态  
\# ip addr show dev eth0     //查看ip的漂移情况  
\# ipvsadm -ln               //查看转发配置表

例如：

\[root@localhost mysqlinstall\]\# ip addr show dev eth0  
2: eth0: <BROADCAST,MULTICAST,UP,LOWER\_UP> mtu 1500 qdisc pfifo\_fast state UP qlen 1000  
    link/ether 00:50:56:b8:7a:78 brd ff:ff:ff:ff:ff:ff  
    inet 192.168.213.90/24 brd 192.168.213.255 scope global eth0  
    inet 192.168.213.97/32 brd 192.168.213.97 scope global eth0:3  
    inet6 fe80::250:56ff:feb8:7a78/64 scope link  
       valid\_lft forever preferred\_lft forever  
\[root@localhost mysqlinstall\]\# ipvsadm -ln  
IP Virtual Server version 1.2.1 (size=4096)  
Prot LocalAddress:Port Scheduler Flags  
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn  
TCP  192.168.213.97:3306 rr persistent 2  
  -> 192.168.213.91:3306          Route   1      0          0          
  -> 192.168.213.92:3306          Route   1      0          40         
\[root@localhost mysqlinstall\]\#

在客户端写个循环连接的脚本

\[root@slave93 mysqlinstall\]\#  more load\_mysql.sh  
while (true)  
do  
 mysql -uroot -proot -h192.168.213.97 -e "select count(\*) from test.a;"       
 sleep 3   
done

在客户端执行脚本  
\[root@slave93 mysqlinstall\]\# sh load\_mysql.sh

在director上查看连接情况  
\# ipvsadm -lcn

**上面的lvs是采用的持久连接的方式，轮询的均衡算法；对于使用数据库大都会使用持久连接（300s），可以采用wrr的均衡算法**

上面的director是独立的主机，为了节省成本，可以把director和某一台realserver部署同一台上，比如部署到91上

\[root@master mysqlinstall\]\# sh lvsdr.sh status  
ipvs is running ...  
IP Virtual Server version 1.2.1 (size=4096)  
Prot LocalAddress:Port Scheduler Flags  
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn  
TCP  192.168.213.97:3306 wrr  
  -> 192.168.213.91:3306          **Local  ** 1      0          0          
  -> 192.168.213.92:3306          Route   3      0          0

到现在，lvs可以正常工作，可以实现负载均衡，但后端某个realserver有问题时，lvs把请求分配器上，就会有问题，这时就需要配置  
前面我们安装的ldirectord服务，其可以自动监控后端realserver的健康状态，发现有问题时就更新director的虚拟服务器和RS链表。即自动从集群中摘除，待以后故障恢复，ldirectord服务会自动把其加入集群中

按如下修改ldirectord的配置文件  
\[root@master mysqlinstall\]\# more /etc/ha.d/ldirectord.cf  
\#  
\# Sample ldirectord configuration file to configure various virtual services.  
\#  
\# Ldirectord will connect to each real server once per second and request  
\# /index.html. If the data returned by the server does not contain the  
\# string "Test Message" then the test fails and the real server will be  
\# taken out of the available pool. The real server will be added back into  
\# the pool once the test succeeds. If all real servers are removed from the  
\# pool then localhost:80 is added to the pool as a fallback measure.

\# Global Directives  
checktimeout=3   //超时  
checkinterval=1  //check的间隔时间  
\#fallback=127.0.0.1:80  
\#fallback6=\[::1\]:80  
autoreload=yes   //实时读取配置文件  
\#logfile="/var/log/ldirectord.log"   //定义日志文件位置  
\#logfile="local0"  
\#emailalert="[admin@x.y.z][admin_x.y.z]"  
\#emailalertfreq=3600  
\#emailalertstatus=all  
quiescent=no   //真正的把故障realserver从lvs映射表中删除，以保证集群对外的不间断服务

\#Sample configuration for a MySQL virtual service.  
virtual = 192.168.213.97:3306  
        \#real=master->slave92:3306 gate 10  
        real=192.168.213.91 gate 10  
        real=192.168.213.92 gate 10  
        fallback=127.0.0.1:3306  
        service=mysql  
        scheduler=wrr  
        \#persistent=600  
        \#netmask=255.255.255.255  
        protocol=tcp  
        checktype=negotiate  
        login="root"  
        passwd="root"  
        database="test"  
        request="SELECT \* FROM a"

然后重启ldirectord服务  
\[root@master mysqlinstall\]\# service ldirectord restart   
\[root@master mysqlinstall\]\# service ldirectord status

然后查看log看ldirectord的工作状况

\[root@master mysqlinstall\]\# tail -f /var/log/ldirectord.log

测试ldirectord是否可以对realserver做健康检查，为了便于测试，我把均衡算法改为rr，并去掉持久连接

测试前：

\[root@master mysqlinstall\]\# ipvsadm -ln  
IP Virtual Server version 1.2.1 (size=4096)  
Prot LocalAddress:Port Scheduler Flags  
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn  
TCP  192.168.213.97:3306 rr  
  -> 192.168.213.91:3306          Local   10     0          59         
  -> 192.168.213.92:3306          Route   10     0          59         
\[root@master mysqlinstall\]\#

启动客户端测试脚本，每秒执行一次数据库查询操作  
\# sh load\_mysql.sh

停止92的mysql

查看结果

查看ldirectord的log  
\[root@master mysqlinstall\]\# tail -f /var/log/ldirectord.log  
\[Sat Dec  7 01:52:22 2013|ldirectord.cf|7445\] Added real server: 192.168.213.92:0 (192.168.213.97:3306) (Weight set to 10)  
\[Sat Dec  7 01:52:22 2013|ldirectord|26081\] system(/sbin/ipvsadm -a -t 192.168.213.97:3306 -r 192.168.213.92:0 -g -w 10) failed:  
\[Sat Dec  7 01:52:22 2013|ldirectord|26081\] Added real server: 192.168.213.92:0 (192.168.213.97:3306) (Weight set to 10)

\[Sat Dec  7 02:02:53 2013|ldirectord.cf|7445\] Deleted real server: 192.168.213.92:3306 mapped from 192.168.213.92:0 (192.168.213.97:3306)  
\[Sat Dec  7 02:02:53 2013|ldirectord|26081\] system(/sbin/ipvsadm -d -t 192.168.213.97:3306 -r 192.168.213.92:3306) failed:  
\[Sat Dec  7 02:02:53 2013|ldirectord|26081\] Deleted real server: 192.168.213.92:3306 mapped from 192.168.213.92:0 (192.168.213.97:3306)

前端的服务有短暂1s的影响

.....  
\+----------+  
\+----------+  
| count(\*) |  
\+----------+  
|        2 |  
\+----------+  
ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.213.97' (111)  
\+----------+  
| count(\*) |  
\+----------+  
|        2 |  
\+----------+  
......

到目前为止，lvs可以负载均衡前端的请求，并可以通过ldirectord服务实现对后端的realserver的健康检查，但是如过lvs和ldirectord所在的机器故障怎么办呢，就没办法高可用了，这时就需要corosync和pacemaker来做HA。

会继续下面会继续........

\-------续---------

[http_horms.net_projects_ldirectord_download.shtml]: http://horms.net/projects/ldirectord/download.shtml
[http_blog.chinaunix.net_uid-17268883-id-3145428.html]: http://blog.chinaunix.net/uid-17268883-id-3145428.html
[http_beyondhdf.blog.51cto.com_229452_1331874]: http://beyondhdf.blog.51cto.com/229452/1331874
[http_soulboy.blog.51cto.com_4007306_1299896]: http://soulboy.blog.51cto.com/4007306/1299896
[http_zh.linuxvirtualserver.org_node_98]: http://zh.linuxvirtualserver.org/node/98
[http_itnihao.blog.51cto.com_1741976_744237]: http://itnihao.blog.51cto.com/1741976/744237
[admin_x.y.z]: mailto:admin@x.y.z