我有一个四节点的Percona XtraDB集群。 每晚,一个节点closures备份,然后重新join群集。 现在运行良好。
昨晚,备份节点拒绝重新join群集。 其他三个节点嗡嗡作响,但任何join失败节点的尝试都不成功。
幸运的是,我们使用Ansible来提供所有的节点,所以我们有一个可重复使用的方法来吹掉失败的节点,并用一个全新的,从零开始的数据库节点来取代它。
幸运的是,新的节点也失败了。 最近没有对Ansible脚本进行更改,而且我们已经成功启动了实例。 这些都在亚马逊AWS上运行在一个二维码的VPC上,如果这样做有什么区别的话。
新的服务器的MySQL日志说:
131226 04:44:50 mysqld_safe Starting mysqld daemon with databases from /data/mysql 131226 04:44:50 mysqld_safe Skipping wsrep-recover for empty datadir: /data/mysql 131226 04:44:50 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position 131226 4:44:50 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1' 131226 4:44:50 [Note] WSREP: Read nil XID from storage engines, skipping position init 131226 4:44:50 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so' 131226 4:44:50 [Note] WSREP: wsrep_load(): Galera 3.2(r170) by Codership Oy <[email protected]> loaded successfully. 131226 4:44:50 [Note] WSREP: CRC-32C: using hardware acceleration. 131226 4:44:50 [Warning] WSREP: Could not open saved state file for reading: /data/mysql//grastate.dat 131226 4:44:50 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 131226 4:44:50 [Note] WSREP: Passing config to GCS: base_host = 10.0.1.226; base_port = 4567; cert.log_conflicts = no; gcache.dir = /data/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /data/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.proto_max = 5 131226 4:44:50 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 131226 4:44:50 [Note] WSREP: wsrep_sst_grab() 131226 4:44:50 [Note] WSREP: Start replication 131226 4:44:50 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 131226 4:44:50 [Note] WSREP: protonet asio version 0 131226 4:44:50 [Note] WSREP: Using CRC-32C (optimized) for message checksums. 131226 4:44:50 [Note] WSREP: backend: asio 131226 4:44:50 [Note] WSREP: GMCast version 0 131226 4:44:50 [Note] WSREP: (7497477a-6de8-11e3-9a71-7af4d47d85be, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 131226 4:44:50 [Note] WSREP: (7497477a-6de8-11e3-9a71-7af4d47d85be, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 131226 4:44:50 [Note] WSREP: EVS version 0 131226 4:44:50 [Note] WSREP: PC version 0 131226 4:44:50 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.0.2.180:,10.0.1.226:,10.0.1.21:,10.0.2.236:' 131226 4:44:50 [Warning] WSREP: (7497477a-6de8-11e3-9a71-7af4d47d85be, 'tcp://0.0.0.0:4567') address 'tcp://10.0.1.226:4567' points to own listening address, blacklisting 131226 4:44:53 [Warning] WSREP: no nodes coming from prim view, prim not possible 131226 4:44:53 [Note] WSREP: view(view_id(NON_PRIM,7497477a-6de8-11e3-9a71-7af4d47d85be,1) memb { 7497477a-6de8-11e3-9a71-7af4d47d85be,0 } joined { } left { } partitioned { }) 131226 4:44:54 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.5049S), skipping check 131226 4:45:23 [Note] WSREP: view((empty)) 131226 4:45:23 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) at gcomm/src/pc.cpp:connect():141 131226 4:45:23 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():196: Failed to open backend connection: -110 (Connection timed out) 131226 4:45:23 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1291: Failed to open channel 'my_wsrep_cluster' at 'gcomm://10.0.2.180,10.0.1.226,10.0.1.21,10.0.2.236': -110 (Connection timed out) 131226 4:45:23 [ERROR] WSREP: gcs connect failed: Connection timed out 131226 4:45:23 [ERROR] WSREP: wsrep::connect() failed: 7 131226 4:45:23 [ERROR] Aborting 131226 4:45:23 [Note] WSREP: Service disconnected. 131226 4:45:24 [Note] WSREP: Some threads may fail to exit. 131226 4:45:24 [Note] /usr/sbin/mysqld: Shutdown complete 131226 04:45:24 mysqld_safe mysqld from pid file /data/mysql/mysql.pid ended
新节点尝试join时,群集中的现有节点将吐出以下内容:
131226 4:45:23 [Warning] WSREP: unserialize error invalid flags 2: 71 (Protocol error) at gcomm/src/gcomm/datagram.hpp:unserialize():101
有没有人遇到过这个? 谷歌search显示很less,这让我很担心,我们可重复的服务器configuration目前不可重复。
这原来是Galera 2.x – > 3.x开关的副作用。
新服务器得到了较新的Galera 3.x软件包。 在混合集群中,它需要:
wsrep_provider_options = 'socket.checksum = 1;'
在/etc/my.cnf为3.x节点,所以他们可以和2.x节点通话。
非常感谢在Launchpad上的Alex Yurchenko(ayurchen)。
互联网上唯一引用这个特定错误的地方是错误1153727的注释9,它被标记为固定的。 我build议将“function”节点升级到最新版本的Percona XtraDB集群。 然后,您应该能够join该最新版本的新节点。