GA

2014/11/06

MySQL Fabricつらい(障害ノードの戻しとか)

マスター昇格編 でマスターをkillしても、スレーブがちゃんとマスターに昇格してくれるところまでは良かったんですが、


$ mysqlfabric group lookup_servers my_first_fabric
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                         server_uuid         address  status       mode weight
------------------------------------ --------------- ------- ---------- ------
47cf54df-63fc-11e4-942e-fa163e020fd0 127.0.0.1:20886  FAULTY READ_WRITE    1.0
4b7036a9-63fc-11e4-942e-fa163e020fd0 127.0.0.1:20887 PRIMARY READ_WRITE    1.0



(旧)マスターを起動したあとでも、待てど暮らせどファームには勝手に戻ってこない。


$ ./m
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.6.21-log Source distribution

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

master [localhost] {msandbox} ((none)) > show slave status\G
Empty set (0.00 sec)


$ mysqlfabric group promote my_first_fabric
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

GroupError: There is no valid candidate that can be automatically chosen in group (my_first_fabric). Please, choose one manual
ly.

killしてもう一度起動したマスターには、当然今のマスターの情報はない。生きてる間にpromoteならFabricサーバーがログインしてゴニョゴニョしてくれるんだろうけど、そもそも接続できなかったんだから当たり前。スレーブがいないんだからもちろんpromoteもできない。


$ mysqlfabric group demote my_first_fabric
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                                uuid finished success result
------------------------------------ -------- ------- ------
dec31088-00b1-4a33-bf36-622a9e1f302a        1       1      1

state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2   1.41526e+09 Triggered by <mysql .fabric.events.event="" 0x26c38d0="" at="" object="">.
    4       2   1.41526e+09                       Executing action (_block_write_demote).
    5       2   1.41526e+09                        Executed action (_block_write_demote).
    3       2   1.41526e+09 Triggered by <mysql .fabric.events.event="" 0x26c39d0="" at="" object="">.
    4       2   1.41526e+09                       Executing action (_wait_slaves_demote).
    5       2   1.41526e+09                        Executed action (_wait_slaves_demote).


$ mysqlfabric group health my_first_fabric
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                                uuid is_alive    status is_not_running is_not_configured io_not_running sql_not_running io_err
or sql_error
------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- ------
-- ---------
47cf54df-63fc-11e4-942e-fa163e020fd0        1    FAULTY              0                 1              0               0    Fal
se     False
4b7036a9-63fc-11e4-942e-fa163e020fd0        1 SECONDARY              0                 1              0               0    Fal
se     False

issue
-----


$ mysqlfabric group lookup_servers my_first_fabric
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                         server_uuid         address    status       mode weight
------------------------------------ --------------- --------- ---------- ------
47cf54df-63fc-11e4-942e-fa163e020fd0 127.0.0.1:20886    FAULTY READ_WRITE    1.0
4b7036a9-63fc-11e4-942e-fa163e020fd0 127.0.0.1:20887 SECONDARY  READ_ONLY    1.0


$ mysqlfabric group promote my_first_fabric
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                                uuid finished success result
------------------------------------ -------- ------- ------
01a40c48-347d-4fa8-887c-2cde33a353ad        1       1      1

state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2   1.41526e+09 Triggered by <mysql .fabric.events.event="" 0x2552e50="" at="" object="">.
    4       2   1.41526e+09                      Executing action (_define_ha_operation).
    5       2   1.41526e+09                       Executed action (_define_ha_operation).
    3       2   1.41526e+09 Triggered by <mysql .fabric.events.event="" 0x26bc7d0="" at="" object="">.
    4       2   1.41526e+09                      Executing action (_find_candidate_fail).
    5       2   1.41526e+09                       Executed action (_find_candidate_fail).
    3       2   1.41526e+09 Triggered by <mysql .fabric.events.event="" 0x26bc710="" at="" object="">.
    4       2   1.41526e+09                     Executing action (_check_candidate_fail).
    5       2   1.41526e+09                      Executed action (_check_candidate_fail).
    3       2   1.41526e+09 Triggered by <mysql .fabric.events.event="" 0x26bc750="" at="" object="">.
    4       2   1.41526e+09                          Executing action (_wait_slave_fail).
    5       2   1.41526e+09                           Executed action (_wait_slave_fail).
    3       2   1.41526e+09 Triggered by <mysql .fabric.events.event="" 0x26c3890="" at="" object="">.
    4       2   1.41526e+09                      Executing action (_change_to_candidate).
    5       2   1.41526e+09                       Executed action (_change_to_candidate).


$ mysqlfabric group health my_first_fabric
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                                uuid is_alive  status is_not_running is_not_configured io_not_running sql_not_running io_error
 sql_error
------------------------------------ -------- ------- -------------- ----------------- -------------- --------------- --------
 ---------
47cf54df-63fc-11e4-942e-fa163e020fd0        1  FAULTY              0                 1              0               0    False
     False
4b7036a9-63fc-11e4-942e-fa163e020fd0        1 PRIMARY              0                 0              0               0    False
     False

issue
-----



demote/promoteしてみたけどコレジャナイ
じゃあ、remove/addで登録しなおすか。


$ mysqlfabric group remove my_first_fabric 47cf54df-63fc-11e4-942e-fa163e020fd0
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                                uuid finished success result
------------------------------------ -------- ------- ------
51509e3b-d1a7-4876-a52d-e25364c46d8d        1       1      1

state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2   1.41526e+09 Triggered by .
    4       2   1.41526e+09                            Executing action (_remove_server).
    5       2   1.41526e+09                             Executed action (_remove_server).


$ mysqlfabric group lookup_servers my_first_fabric
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                         server_uuid         address  status       mode weight
------------------------------------ --------------- ------- ---------- ------
4b7036a9-63fc-11e4-942e-fa163e020fd0 127.0.0.1:20887 PRIMARY READ_WRITE    1.0


$ mysqlfabric group add my_first_fabric 127.0.0.1:20886
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                                uuid finished success result
------------------------------------ -------- ------- ------
f50a682b-dab6-4094-9f54-72c72569e71a        1       1      1

state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2   1.41526e+09 Triggered by .
    4       2   1.41526e+09                               Executing action (_add_server).
    5       2   1.41526e+09                                Executed action (_add_server).


$ mysqlfabric group lookup_servers my_first_fabric                            [896/2897]
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                         server_uuid         address    status       mode weight
------------------------------------ --------------- --------- ---------- ------
47cf54df-63fc-11e4-942e-fa163e020fd0 127.0.0.1:20886 SECONDARY  READ_ONLY    1.0
4b7036a9-63fc-11e4-942e-fa163e020fd0 127.0.0.1:20887   PRIMARY READ_WRITE    1.0


$ ./check_slaves.a
master
              Master_Log_File: mysql-bin.000003
          Read_Master_Log_Pos: 1548
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
          Exec_Master_Log_Pos: 1548
slave # 1
             File: mysql-bin.000003
         Position: 1548

あ、ちゃんとスレーブとしてファームに組み込まれた。


$ make_multiple_sandbox 5.6.21
installing node 1
installing node 2
installing node 3
group directory installed in $HOME/sandboxes/multi_msb_5_6_21

$ ./use_all "select @@port"
# server: 1:
@@port
14722
# server: 2:
@@port
14723
# server: 3:
@@port
14724

$ vim node?/my.sandbox.cnf
..
log-slave-updates
gtid-mode= ON
enforce-gtid-consistency

$ export MYCLIENT_OPTIONS="-uroot"
$ ./use_all "grant all on *.* to fabric@localhost identified by 'fabric_password'"
$ ./restart_all

$ mysqlfabric group add my_first_fabric 127.0.0.1:14722
$ mysqlfabric group add my_first_fabric 127.0.0.1:14723
$ mysqlfabric group add my_first_fabric 127.0.0.1:14724

$ mysqlfabric group health my_first_fabric
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                                uuid is_alive    status is_not_running is_not_configured io_not_running sql_not_running io_err
or sql_error
------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- ------
-- ---------
01250ed8-6584-11e4-9e28-fa163e020fd0        1 SECONDARY              0                 0              0               0    Fal
se     False
47cf54df-63fc-11e4-942e-fa163e020fd0        1 SECONDARY              0                 0              0               0    Fal
se     False
4b7036a9-63fc-11e4-942e-fa163e020fd0        1   PRIMARY              0                 0              0               0    Fal
se     False
f9fe3cee-6583-11e4-9e28-fa163e020fd0        1 SECONDARY              0                 0              0               0    Fal
se     False
fd8c1c45-6583-11e4-9e28-fa163e020fd0        1 SECONDARY              0                 0              0               0    Fal
se     False

issue
-----


$ ./n1
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 28
Server version: 5.6.21-log Source distribution

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

node1 [localhost] {root} ((none)) > show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| d1                 |
| mysql              |
| performance_schema |
| test               |
+--------------------+
5 rows in set (0.01 sec)

おおー。GTIDだから、マスターにバイナリーログさえ残ってればmysqlfabric group addするより前のバイナリーログもちゃんと適用される。
ジェネラルログを覗くと、


76 Connect   fabric@localhost on
                   76 Query     SET NAMES 'utf8' COLLATE 'utf8_general_ci'
                   76 Query     SET @@session.autocommit = ON
                   76 Query     SELECT @@GLOBAL.SERVER_UUID as SERVER_UUID
                   76 Query     SELECT @@GLOBAL.SERVER_ID as SERVER_ID
                   76 Query     SELECT @@GLOBAL.VERSION as VERSION
                   76 Query     SELECT @@GLOBAL.GTID_MODE as GTID_MODE
                   76 Query     SELECT @@GLOBAL.LOG_BIN as LOG_BIN
                   76 Query     SELECT @@GLOBAL.READ_ONLY as READ_ONLY
                   76 Query     SHOW GRANTS
                   76 Query     STOP SLAVE
                   76 Query     SHOW SLAVE STATUS
                   76 Query     SHOW SLAVE STATUS
                   76 Query     CHANGE MASTER TO MASTER_HOST = '127.0.0.1' MASTER_USER = 'fabric' MASTER_PASSWORD =  MASTER_PORT = 20887
                   76 Query     SET @@GLOBAL.READ_ONLY = ON
                   76 Query     SELECT @@GLOBAL.READ_ONLY as READ_ONLY
                   76 Query     START SLAVE
                   76 Query     SHOW SLAVE STATUS
                   76 Query     SHOW SLAVE STATUS

うむ。シャードをゴニョゴニョするときはmysqldumpを使う(/etc/mysql/fabric.cfgにmysqldumpのパスを書くパラメーターがある)けど、単に追加するだけの時はFabricサーバーは特に面倒を見ないと。とはいえGTID前提だから、XtraBackupでサクッと作ってサクッとmysqlfabric group addでおしまいと。

( ´-`).oO(や、便利なんだけどその前提条件がな。。


【2014/11/06 18:16】
途中からだいぶ脱線していた気がするけれど、もともとの目的だった"status: FAULTYからstatus: SECONDARYに戻す"は、




$ mysqlfabric server set_status 4b7036a9-63fc-11e4-942e-fa163e020fd0 SPARE
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                                uuid finished success result
------------------------------------ -------- ------- ------
aa09ecf8-2f55-4c11-bc25-53b0ae19b3b0        1       1      1

state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2   1.41527e+09 Triggered by .
    4       2   1.41527e+09                        Executing action (_set_server_status).
    5       2   1.41527e+09                         Executed action (_set_server_status).


$ mysqlfabric group health my_first_fabric
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                                uuid is_alive  status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- ------- -------------- ----------------- -------------- --------------- -------- ---------
47cf54df-63fc-11e4-942e-fa163e020fd0        1 PRIMARY              0                 0              0               0    False     False
4b7036a9-63fc-11e4-942e-fa163e020fd0        1   SPARE              0                 0              0               0    False     False

issue
-----


$ mysqlfabric server set_status 4b7036a9-63fc-11e4-942e-fa163e020fd0 SECONDARY
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                                uuid finished success result
------------------------------------ -------- ------- ------
1e8eb7c0-2233-4087-86cb-bedc9e331f1e        1       1      1

state success          when                                                   description
----- ------- ------------- -------------------------------------------------------------
    3       2   1.41527e+09 Triggered by .
    4       2   1.41527e+09                        Executing action (_set_server_status).
    5       2   1.41527e+09                         Executed action (_set_server_status).


$ mysqlfabric group health my_first_fabric
Fabric UUID:  5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

                                uuid is_alive    status is_not_running is_not_configured io_not_running sql_not_running io_error sql_error
------------------------------------ -------- --------- -------------- ----------------- -------------- --------------- -------- ---------
47cf54df-63fc-11e4-942e-fa163e020fd0        1   PRIMARY              0                 0              0               0    False     False
4b7036a9-63fc-11e4-942e-fa163e020fd0        1 SECONDARY              0                 0              0               0    False     False

issue
-----



Σ(゚д゚lll) できたーっ!

0 件のコメント :

コメントを投稿