MHA 手动故障转移

ゞ浴缸里的玫瑰 2022-08-05 12:22 210阅读 0赞

MHA提供了3种方式用于实现故障转移，分别自动故障转移，需要启用MHA监控；在无监控的情况下的手动故障转移以及基于在线手动切换。三种方式可以应对MySQL主从故障的任意场景。本文主要描述在无监控的情形是手动实现故障转移。供大家参考。

有关MHA的其他两种切换方式，可以参考：  
            [MHA 在线切换过程  
][MHA _]            [MHA 自动故障转移步骤及过程剖析][MHA]

1、手动故障转移的特点  
    a、在监控节点未启用masterha\_manager  
    b、master库已经宕机或者转移到高性能服务器  
    c、手动故障转移支持交互或非交互两种模式  
    d、切换样例：$ masterha\_master\_switch --master\_state=dead --conf=/etc/app1.cnf --dead\_master\_host=host1

2、masterha\_master\_switch切换的几个参数  
\--master\_state=dead  
      强制参数为"dead" 或者 "alive". dead为手动故障转移，alive为在线切换。  
    
\--dead\_master\_host=(hostname)  
      强制参数为主机名，另2个--dead\_master\_ip --dead\_master\_port(缺省3306)可选。  
    
\--new\_master\_host=(hostname)  
      可选参数，用于指定新master，如果未指定则按candidate\_master参数设定值。  
    
\--interactive=(0|1)  
      可选参数，指定是否交互。缺省为1，表明交互。

3、演示手动故障转移  
\#\#\#环境    
\[root@vdbsrv1 ~\]\# more /etc/hosts  
127.0.0.1    localhost.localdomain localhost  
192.168.1.6  vdbsrv1  \#master  
192.168.1.7  vdbsrv2  \#slave1  
192.168.1.8  vdbsrv3  \#slave2  
192.168.1.12 vdbsrv4  \#manager

\#\#\#master开启VIP  
\[root@vdbsrv4 ~\]\# ssh vdbsrv1 "/sbin/ifconfig eth0:0 192.168.1.13 netmask 255.255.255.0 up"

\#\#\#检测MHA监控是否运行，如果已启动，可以使用masterha\_check\_stop先停止  
\[root@vdbsrv4 ~\]\# masterha\_check\_status --conf=/etc/masterha/app1.cnf  
app1 is stopped(2:NOT\_RUNNING).

\#\#\#模拟从库滞后  
\[root@vdbsrv4 ~\]\# ssh vdbsrv2 "mysql -e 'stop slave io\_thread'"  
\[root@vdbsrv4 ~\]\# ssh vdbsrv2 "mysql -e 'show slave status\\G' | egrep 'Slave\_IO|Slave\_SQL'"  
               Slave\_IO\_State:  
             Slave\_IO\_Running: No  
            Slave\_SQL\_Running: Yes  
      Slave\_SQL\_Running\_State: Slave has read all relay log; waiting for the slave I/O thread to update it

\#\#\#主库上插入新记录  
mysql> insert into tb(val,dt) values('john',current\_date());  
Query OK, 1 row affected (0.01 sec)

\#\#\#模拟master异常宕机  
\[root@vdbsrv4 ~\]\# ssh vdbsrv1 "killall -r mysqld"

\#\#\#开始手工故障转移  
\[root@vdbsrv4 ~\]\# masterha\_master\_switch --master\_state=dead --conf=/etc/masterha/app1.cnf --dead\_master\_host=vdbsrv1 \\  
>  --dead\_master\_port=3306 --new\_master\_host=vdbsrv3 --new\_master\_port=3306 --ignore\_last\_failover  
\--dead\_master\_ip=<dead\_master\_ip> is not set. Using 192.168.1.6.  
Wed Apr 22 11:48:39 2015 - \[warning\] Global configuration file /etc/masterha\_default.cnf not found. Skipping.  
Wed Apr 22 11:48:39 2015 - \[info\] Reading application default configuration from /etc/masterha/app1.cnf..  
Wed Apr 22 11:48:39 2015 - \[info\] Reading server configuration from /etc/masterha/app1.cnf..  
Wed Apr 22 11:48:39 2015 - \[info\] MHA::MasterFailover version 0.56.  
Wed Apr 22 11:48:39 2015 - \[info\] Starting master failover.  
Wed Apr 22 11:48:39 2015 - \[info\]  
Wed Apr 22 11:48:39 2015 - \[info\] \* Phase 1: Configuration Check Phase..  
Wed Apr 22 11:48:39 2015 - \[info\]  
Wed Apr 22 11:48:39 2015 - \[info\] GTID failover mode = 0  
Wed Apr 22 11:48:39 2015 - \[info\] Dead Servers:  
Wed Apr 22 11:48:39 2015 - \[info\]   vdbsrv1(192.168.1.6:3306)  
Wed Apr 22 11:48:39 2015 - \[info\] Checking master reachability via MySQL(double check)...  
Wed Apr 22 11:48:39 2015 - \[info\]  ok.  
Wed Apr 22 11:48:39 2015 - \[info\] Alive Servers:  
Wed Apr 22 11:48:39 2015 - \[info\]   vdbsrv2(192.168.1.7:3306)  
Wed Apr 22 11:48:39 2015 - \[info\]   vdbsrv3(192.168.1.8:3306)  
Wed Apr 22 11:48:39 2015 - \[info\] Alive Slaves:  
Wed Apr 22 11:48:39 2015 - \[info\]   vdbsrv2(192.168.1.7:3306)  Version=5.6.22-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 22 11:48:39 2015 - \[info\]     Replicating from 192.168.1.6(192.168.1.6:3306)  
Wed Apr 22 11:48:39 2015 - \[info\]   vdbsrv3(192.168.1.8:3306)  Version=5.6.22-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 22 11:48:39 2015 - \[info\]     Replicating from 192.168.1.6(192.168.1.6:3306)  
Master vdbsrv1(192.168.1.6:3306) is dead. Proceed? (yes/NO): yes  
Wed Apr 22 11:48:54 2015 - \[info\] Starting Non-GTID based failover.  
Wed Apr 22 11:48:54 2015 - \[info\]  
Wed Apr 22 11:48:54 2015 - \[info\] \*\* Phase 1: Configuration Check Phase completed.  
Wed Apr 22 11:48:54 2015 - \[info\]  
Wed Apr 22 11:48:54 2015 - \[info\] \* Phase 2: Dead Master Shutdown Phase..  
Wed Apr 22 11:48:54 2015 - \[info\]  
Wed Apr 22 11:48:54 2015 - \[info\] HealthCheck: SSH to vdbsrv1 is reachable.  
Wed Apr 22 11:48:55 2015 - \[info\] Forcing shutdown so that applications never connect to the current master..  
Wed Apr 22 11:48:55 2015 - \[info\] Executing master IP deactivation script:  
Wed Apr 22 11:48:55 2015 - \[info\]   /tmp/master\_ip\_failover --orig\_master\_host=vdbsrv1 --orig\_master\_ip=192.168.1.6  
  --orig\_master\_port=3306 --command=stopssh --ssh\_user=root

IN SCRIPT TEST====/sbin/ifconfig eth0:0 down==/sbin/ifconfig eth0:0 192.168.1.13/24===

Disabling the VIP on old master: vdbsrv1  
Wed Apr 22 11:48:55 2015 - \[info\]  done.  
Wed Apr 22 11:48:55 2015 - \[warning\] shutdown\_script is not set. Skipping explicit shutting down of the dead master.  
Wed Apr 22 11:48:55 2015 - \[info\] \* Phase 2: Dead Master Shutdown Phase completed.  
Wed Apr 22 11:48:55 2015 - \[info\]  
Wed Apr 22 11:48:55 2015 - \[info\] \* Phase 3: Master Recovery Phase..  
Wed Apr 22 11:48:55 2015 - \[info\]  
Wed Apr 22 11:48:55 2015 - \[info\] \* Phase 3.1: Getting Latest Slaves Phase..  
Wed Apr 22 11:48:55 2015 - \[info\]  
Wed Apr 22 11:48:55 2015 - \[info\] The latest binary log file/position on all slaves is mysql-bin.000027:432  
Wed Apr 22 11:48:55 2015 - \[info\] Latest slaves (Slaves that received relay log files to the latest):  
Wed Apr 22 11:48:55 2015 - \[info\]   vdbsrv3(192.168.1.8:3306)  Version=5.6.22-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 22 11:48:55 2015 - \[info\]     Replicating from 192.168.1.6(192.168.1.6:3306)  
Wed Apr 22 11:48:55 2015 - \[info\] The oldest binary log file/position on all slaves is mysql-bin.000027:120  
Wed Apr 22 11:48:55 2015 - \[info\] Oldest slaves:  
Wed Apr 22 11:48:55 2015 - \[info\]   vdbsrv2(192.168.1.7:3306)  Version=5.6.22-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 22 11:48:55 2015 - \[info\]     Replicating from 192.168.1.6(192.168.1.6:3306)  
Wed Apr 22 11:48:55 2015 - \[info\]  
Wed Apr 22 11:48:55 2015 - \[info\] \* Phase 3.2: Saving Dead Master's Binlog Phase..  
Wed Apr 22 11:48:55 2015 - \[info\]  
Wed Apr 22 11:48:55 2015 - \[info\] Fetching dead master's binary logs..  
Wed Apr 22 11:48:55 2015 - \[info\] Executing command on the dead master vdbsrv1(192.168.1.6:3306): save\_binary\_logs --command=save  
  --start\_file=mysql-bin.000027  --start\_pos=432 --binlog\_dir=/data/mysqldata  
  --output\_file=/var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog --handle\_raw\_binlog=1 --disable\_log\_bin=0  
  --manager\_version=0.56  
  Creating /var/tmp if not exists..    ok.  
 Concat binary/relay logs from mysql-bin.000027 pos 432 to mysql-bin.000027 EOF into  
   /var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog ..  
 Binlog Checksum enabled  
  Dumping binlog format description event, from position 0 to 120.. ok.  
  Dumping effective binlog data from /data/mysqldata/mysql-bin.000027 position 432 to tail(455).. ok.  
 Binlog Checksum enabled  
 Concat succeeded.  
saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog                100%  143     0.1KB/s   00:00     
Wed Apr 22 11:48:56 2015 - \[info\] scp from root@192.168.1.6:/var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog to  
  local:/var/log/masterha/app1/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog succeeded.  
Wed Apr 22 11:48:56 2015 - \[info\] HealthCheck: SSH to vdbsrv2 is reachable.  
Wed Apr 22 11:48:56 2015 - \[info\] HealthCheck: SSH to vdbsrv3 is reachable.  
Wed Apr 22 11:48:56 2015 - \[info\]  
Wed Apr 22 11:48:56 2015 - \[info\] \* Phase 3.3: Determining New Master Phase..  
Wed Apr 22 11:48:56 2015 - \[info\]  
Wed Apr 22 11:48:56 2015 - \[info\] Finding the latest slave that has all relay logs for recovering other slaves..  
Wed Apr 22 11:48:56 2015 - \[info\] Checking whether vdbsrv3 has relay logs from the oldest position..  
Wed Apr 22 11:48:56 2015 - \[info\] Executing command: apply\_diff\_relay\_logs --command=find --latest\_mlf=mysql-bin.000027 --latest\_rmlp=432  
   --target\_mlf=mysql-bin.000027 --target\_rmlp=120 --server\_id=1002 --workdir=/var/tmp --timestamp=20150422114839 --manager\_version=0.56  
   --relay\_log\_info=/data/mysqldata/relay-log.info  --relay\_dir=/data/mysqldata/  :  
    Opening /data/mysqldata/relay-log.info ... ok.  
    Relay log found at /data/mysqldata, up to vdbsrv3-relay-bin.000002  
 Fast relay log position search succeeded.  
 Target relay log file/position found. start\_file:vdbsrv3-relay-bin.000002, start\_pos:283.  
Target relay log FOUND!  
Wed Apr 22 11:48:56 2015 - \[info\] OK. vdbsrv3 has all relay logs.  
Wed Apr 22 11:48:56 2015 - \[info\] vdbsrv3 can be new master.  
Wed Apr 22 11:48:56 2015 - \[info\] New master is vdbsrv3(192.168.1.8:3306)  
Wed Apr 22 11:48:56 2015 - \[info\] Starting master failover..  
Wed Apr 22 11:48:56 2015 - \[info\]  
From:  
vdbsrv1(192.168.1.6:3306) (current master)  
 +--vdbsrv2(192.168.1.7:3306)  
 +--vdbsrv3(192.168.1.8:3306)

To:  
vdbsrv3(192.168.1.8:3306) (new master)  
 +--vdbsrv2(192.168.1.7:3306)

Starting master switch from vdbsrv1(192.168.1.6:3306) to vdbsrv3(192.168.1.8:3306)? (yes/NO): yes  
Wed Apr 22 11:48:58 2015 - \[info\] New master decided manually is vdbsrv3(192.168.1.8:3306)  
Wed Apr 22 11:48:58 2015 - \[info\]  
Wed Apr 22 11:48:58 2015 - \[info\] \* Phase 3.3: New Master Diff Log Generation Phase..  
Wed Apr 22 11:48:58 2015 - \[info\]  
Wed Apr 22 11:48:58 2015 - \[info\]  This server has all relay logs. No need to generate diff files from the latest slave.  
Wed Apr 22 11:48:58 2015 - \[info\] Sending binlog..  
     saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog          100%  143     0.1KB/s   00:00     
Wed Apr 22 11:48:58 2015 - \[info\] scp from local:/var/log/masterha/app1/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog to  
  root@vdbsrv3:/var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog succeeded.  
Wed Apr 22 11:48:58 2015 - \[info\]  
Wed Apr 22 11:48:58 2015 - \[info\] \* Phase 3.4: Master Log Apply Phase..  
Wed Apr 22 11:48:58 2015 - \[info\]  
Wed Apr 22 11:48:58 2015 - \[info\] \*NOTICE: If any error happens from this phase, manual recovery is needed.  
Wed Apr 22 11:48:58 2015 - \[info\] Starting recovery on vdbsrv3(192.168.1.8:3306)..  
Wed Apr 22 11:48:58 2015 - \[info\]  Generating diffs succeeded.  
Wed Apr 22 11:48:58 2015 - \[info\] Waiting until all relay logs are applied.  
Wed Apr 22 11:48:58 2015 - \[info\]  done.  
Wed Apr 22 11:48:58 2015 - \[info\] Getting slave status..  
Wed Apr 22 11:48:58 2015 - \[info\] This slave(vdbsrv3)'s Exec\_Master\_Log\_Pos equals to Read\_Master\_Log\_Pos(mysql-bin.000027:432).  
  No need to recover from Exec\_Master\_Log\_Pos.  
Wed Apr 22 11:48:58 2015 - \[info\] Connecting to the target slave host vdbsrv3, running recover script..  
Wed Apr 22 11:48:58 2015 - \[info\] Executing command: apply\_diff\_relay\_logs --command=apply --slave\_user='mha' --slave\_host=vdbsrv3  
   --slave\_ip=192.168.1.8  --slave\_port=3306 --apply\_files=/var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog  
   --workdir=/var/tmp --target\_version=5.6.22-log --timestamp=20150422114839 --handle\_raw\_binlog=1 --disable\_log\_bin=0  
   --manager\_version=0.56 --slave\_pass=xxx  
Wed Apr 22 11:48:59 2015 - \[info\]  
Applying differential binary/relay log files /var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog on vdbsrv3:3306.  
  This may take long time...  
Applying log files succeeded.  
Wed Apr 22 11:48:59 2015 - \[info\]  All relay logs were successfully applied.  
Wed Apr 22 11:48:59 2015 - \[info\] Getting new master's binlog name and position..  
Wed Apr 22 11:48:59 2015 - \[info\]  mysql-bin.000017:259187555  
Wed Apr 22 11:48:59 2015 - \[info\]  All other slaves should start replication from here. Statement should be:  
   CHANGE MASTER TO MASTER\_HOST='vdbsrv3 or 192.168.1.8', MASTER\_PORT=3306, MASTER\_LOG\_FILE='mysql-bin.000017',  
   MASTER\_LOG\_POS=259187555, MASTER\_USER='repl', MASTER\_PASSWORD='xxx';  
Wed Apr 22 11:48:59 2015 - \[info\] Executing master IP activate script:  
Wed Apr 22 11:48:59 2015 - \[info\]   /tmp/master\_ip\_failover --command=start --ssh\_user=root --orig\_master\_host=vdbsrv1  
  --orig\_master\_ip=192.168.1.6 --orig\_master\_port=3306 --new\_master\_host=vdbsrv3 --new\_master\_ip=192.168.1.8 --new\_master\_port=3306  
  --new\_master\_user='mha' --new\_master\_password='xxx'   
Unknown option: new\_master\_user  
Unknown option: new\_master\_password

IN SCRIPT TEST====/sbin/ifconfig eth0:0 down==/sbin/ifconfig eth0:0 192.168.1.13/24===

Enabling the VIP - 192.168.1.13/24 on the new master - vdbsrv3  
Wed Apr 22 11:48:59 2015 - \[info\]  OK.  
Wed Apr 22 11:48:59 2015 - \[info\] \*\* Finished master recovery successfully.  
Wed Apr 22 11:48:59 2015 - \[info\] \* Phase 3: Master Recovery Phase completed.  
Wed Apr 22 11:48:59 2015 - \[info\]  
Wed Apr 22 11:48:59 2015 - \[info\] \* Phase 4: Slaves Recovery Phase..  
Wed Apr 22 11:48:59 2015 - \[info\]  
Wed Apr 22 11:48:59 2015 - \[info\] \* Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..  
Wed Apr 22 11:48:59 2015 - \[info\]  
Wed Apr 22 11:48:59 2015 - \[info\] -- Slave diff file generation on host vdbsrv2(192.168.1.7:3306) started, pid: 17143.  
   Check tmp log /var/log/masterha/app1/vdbsrv2\_3306\_20150422114839.log if it takes time..  
Wed Apr 22 11:49:00 2015 - \[info\]  
Wed Apr 22 11:49:00 2015 - \[info\] Log messages from vdbsrv2 ...  
Wed Apr 22 11:49:00 2015 - \[info\]  
Wed Apr 22 11:48:59 2015 - \[info\] Server vdbsrv2 received relay logs up to: mysql-bin.000027:120  
Wed Apr 22 11:48:59 2015 - \[info\] Need to get diffs from the latest slave(vdbsrv3) up to: mysql-bin.000027:432 (using the latest slave's relay logs)  
Wed Apr 22 11:48:59 2015 - \[info\] Connecting to the latest slave host vdbsrv3, generating diff relay log files..  
Wed Apr 22 11:48:59 2015 - \[info\] Executing command: apply\_diff\_relay\_logs --command=generate\_and\_send --scp\_user=root  
   --scp\_host=192.168.1.7 --latest\_mlf=mysql-bin.000027 --latest\_rmlp=432 --target\_mlf=mysql-bin.000027 --target\_rmlp=120  
   --server\_id=1002 --diff\_file\_readtolatest=/var/tmp/relay\_from\_read\_to\_latest\_vdbsrv2\_3306\_20150422114839.binlog --workdir=/var/tmp  
   --timestamp=20150422114839 --handle\_raw\_binlog=1 --disable\_log\_bin=0 --manager\_version=0.56 --relay\_log\_info=/data/mysqldata/relay-log.info  
   --relay\_dir=/data/mysqldata/  
Wed Apr 22 11:49:00 2015 - \[info\]  
    Opening /data/mysqldata/relay-log.info ... ok.  
    Relay log found at /data/mysqldata, up to vdbsrv3-relay-bin.000002  
 Fast relay log position search succeeded.  
 Target relay log file/position found. start\_file:vdbsrv3-relay-bin.000002, start\_pos:283.  
 Concat binary/relay logs from vdbsrv3-relay-bin.000002 pos 283 to vdbsrv3-relay-bin.000002 EOF into  
   /var/tmp/relay\_from\_read\_to\_latest\_vdbsrv2\_3306\_20150422114839.binlog ..  
 Binlog Checksum enabled  
 Binlog Checksum enabled  
  Dumping binlog format description event, from position 0 to 283.. ok.  
  Dumping effective binlog data from /data/mysqldata/vdbsrv3-relay-bin.000002 position 283 to tail(595).. ok.  
 Binlog Checksum enabled  
 Binlog Checksum enabled  
 Concat succeeded.  
 Generating diff relay log succeeded. Saved at /var/tmp/relay\_from\_read\_to\_latest\_vdbsrv2\_3306\_20150422114839.binlog .  
 scp vdbsrv3:/var/tmp/relay\_from\_read\_to\_latest\_vdbsrv2\_3306\_20150422114839.binlog to root@192.168.1.7(22) succeeded.  
Wed Apr 22 11:49:00 2015 - \[info\]  Generating diff files succeeded.  
Wed Apr 22 11:49:00 2015 - \[info\] End of log messages from vdbsrv2.  
Wed Apr 22 11:49:00 2015 - \[info\] -- Slave diff log generation on host vdbsrv2(192.168.1.7:3306) succeeded.  
Wed Apr 22 11:49:00 2015 - \[info\] Generating relay diff files from the latest slave succeeded.  
Wed Apr 22 11:49:00 2015 - \[info\]  
Wed Apr 22 11:49:00 2015 - \[info\] \* Phase 4.2: Starting Parallel Slave Log Apply Phase..  
Wed Apr 22 11:49:00 2015 - \[info\]  
Wed Apr 22 11:49:00 2015 - \[info\] -- Slave recovery on host vdbsrv2(192.168.1.7:3306) started, pid: 17149. Check tmp log  
  /var/log/masterha/app1/vdbsrv2\_3306\_20150422114839.log if it takes time..  
saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog               100%  143     0.1KB/s   00:00     
Wed Apr 22 11:49:00 2015 - \[info\]  
Wed Apr 22 11:49:00 2015 - \[info\] Log messages from vdbsrv2 ...  
Wed Apr 22 11:49:00 2015 - \[info\]  
Wed Apr 22 11:49:00 2015 - \[info\] Sending binlog..  
Wed Apr 22 11:49:00 2015 - \[info\] scp from local:/var/log/masterha/app1/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog  
   to root@vdbsrv2:/var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog succeeded.  
Wed Apr 22 11:49:00 2015 - \[info\] Starting recovery on vdbsrv2(192.168.1.7:3306)..  
Wed Apr 22 11:49:00 2015 - \[info\]  Generating diffs succeeded.  
Wed Apr 22 11:49:00 2015 - \[info\] Waiting until all relay logs are applied.  
Wed Apr 22 11:49:00 2015 - \[info\]  done.  
Wed Apr 22 11:49:00 2015 - \[info\] Getting slave status..  
Wed Apr 22 11:49:00 2015 - \[info\] This slave(vdbsrv2)'s Exec\_Master\_Log\_Pos equals to Read\_Master\_Log\_Pos(mysql-bin.000027:120).  
  No need to recover from Exec\_Master\_Log\_Pos.  
Wed Apr 22 11:49:00 2015 - \[info\] Connecting to the target slave host vdbsrv2, running recover script..  
Wed Apr 22 11:49:00 2015 - \[info\] Executing command: apply\_diff\_relay\_logs --command=apply --slave\_user='mha' --slave\_host=vdbsrv2  
  --slave\_ip=192.168.1.7  --slave\_port=3306 --apply\_files=/var/tmp/relay\_from\_read\_to\_latest\_vdbsrv2\_3306\_20150422114839.binlog,  
  /var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog --workdir=/var/tmp --target\_version=5.6.22-log  
  --timestamp=20150422114839 --handle\_raw\_binlog=1 --disable\_log\_bin=0 --manager\_version=0.56 --slave\_pass=xxx  
Wed Apr 22 11:49:00 2015 - \[info\]  
 Concat all apply files to /var/tmp/total\_binlog\_for\_vdbsrv2\_3306.20150422114839.binlog ..  
 Copying the first binlog file /var/tmp/relay\_from\_read\_to\_latest\_vdbsrv2\_3306\_20150422114839.binlog to  
   /var/tmp/total\_binlog\_for\_vdbsrv2\_3306.20150422114839.binlog.. ok.  
  Dumping binlog head events (rotate events), skipping format description events from  
    /var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog..  Binlog Checksum enabled  
dumped up to pos 120. ok.  
 /var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog has effective binlog events from pos 120.  
  Dumping effective binlog data from /var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog position 120 to tail(143).. ok.  
 Concat succeeded.  
All apply target binary logs are concatinated at /var/tmp/total\_binlog\_for\_vdbsrv2\_3306.20150422114839.binlog .  
Applying differential binary/relay log files /var/tmp/relay\_from\_read\_to\_latest\_vdbsrv2\_3306\_20150422114839.binlog,  
  /var/tmp/saved\_master\_binlog\_from\_vdbsrv1\_3306\_20150422114839.binlog on vdbsrv2:3306. This may take long time...  
Applying log files succeeded.  
Wed Apr 22 11:49:00 2015 - \[info\]  All relay logs were successfully applied.  
Wed Apr 22 11:49:00 2015 - \[info\]  Resetting slave vdbsrv2(192.168.1.7:3306) and starting replication from the new master vdbsrv3(192.168.1.8:3306)..  
Wed Apr 22 11:49:00 2015 - \[info\]  Executed CHANGE MASTER.  
Wed Apr 22 11:49:00 2015 - \[info\]  Slave started.  
Wed Apr 22 11:49:00 2015 - \[info\] End of log messages from vdbsrv2.  
Wed Apr 22 11:49:00 2015 - \[info\] -- Slave recovery on host vdbsrv2(192.168.1.7:3306) succeeded.  
Wed Apr 22 11:49:00 2015 - \[info\] All new slave servers recovered successfully.  
Wed Apr 22 11:49:00 2015 - \[info\]  
Wed Apr 22 11:49:00 2015 - \[info\] \* Phase 5: New master cleanup phase..  
Wed Apr 22 11:49:00 2015 - \[info\]  
Wed Apr 22 11:49:00 2015 - \[info\] Resetting slave info on the new master..  
Wed Apr 22 11:49:00 2015 - \[info\]  vdbsrv3: Resetting slave info succeeded.  
Wed Apr 22 11:49:00 2015 - \[info\] Master failover to vdbsrv3(192.168.1.8:3306) completed successfully.  
Wed Apr 22 11:49:00 2015 - \[info\]

\----- Failover Report -----

app1: MySQL Master failover vdbsrv1(192.168.1.6:3306) to vdbsrv3(192.168.1.8:3306) succeeded

Master vdbsrv1(192.168.1.6:3306) is down!

Check MHA Manager logs at vdbsrv4 for details.

Started manual(interactive) failover.  
Invalidated master IP address on vdbsrv1(192.168.1.6:3306)  
The latest slave vdbsrv3(192.168.1.8:3306) has all relay logs for recovery.  
Selected vdbsrv3(192.168.1.8:3306) as a new master.  
vdbsrv3(192.168.1.8:3306): OK: Applying all logs succeeded.  
vdbsrv3(192.168.1.8:3306): OK: Activated master IP address.  
vdbsrv2(192.168.1.7:3306): Generating differential relay logs up to vdbsrv3(192.168.1.8:3306)succeeded.  
Generating relay diff files from the latest slave succeeded.  
vdbsrv2(192.168.1.7:3306): OK: Applying all logs succeeded. Slave started, replicating from vdbsrv3(192.168.1.8:3306)  
vdbsrv3(192.168.1.8:3306): Resetting slave info succeeded.  
Master failover to vdbsrv3(192.168.1.8:3306) completed successfully.

[MHA _]: http://blog.csdn.net/leshami/article/details/45189825
[MHA]: http://blog.csdn.net/leshami/article/details/45165929