redis集群介绍

多个redis节点实现网络互连，数据共享

所有节点都是一主一从(或一主多从)，其中从不提供服务，仅作为备份备用，保证redis数据的安全性不支持同时处理多个键，redis集群写入数据是向集群里随机写入的(比如raid5磁盘阵列格式)，因为redis集群需要把键平均分布在各个节点上，并发量很高的情况下同时创建键值会降低性能并会导致不可预估的行为redis集群支持在线增加、删除节点，在扩充redis的情况下，是非常容易扩容集群服务数量的客户端可以连接任意一个主节点进行读写

redis集群配置

redis群集场景设定

使用两台机器，分别开启三个redis服务(基于不同端口)服务器ip为192.168.1.223和192.168.1.234192.168.1.234上开启7000、7002、7002三个端口，设置角色全部为主192.168.1.223上开启7001、7003、7005三个端口，设置角色全部为从两台机器都要编译安装redis，编辑三个不同的redis.conf，分别设置不同端口号、dir等参数，还需要增加cluster相关参数，启动为六个redis服务首先在192.168.1.234机器上拷贝三个不同的redis配置文件，修改配置文件的端口、存储数据文件的路径等，并创建配置文件中指定存储数据文件的目录在192.168.1.234机器上创建三个基于不同端口的配置文件，如：

[root@nfs1 /]# cat /etc/redis_7000.conf port 7000bind 192.168.1.234daemonize yespidfile /var/run/redis_7000.piddir /data/redis_data/7000cluster-enabled yescluster-config-file nodes_7000.confcluster-node-timeout 10100appendonly yes[root@nfs1 /]# cat /etc/redis_7002.conf port 7002bind 192.168.1.234daemonize yespidfile /var/run/redis_7002.piddir /data/redis_data/7002cluster-enabled yescluster-config-file nodes_7002.confcluster-node-timeout 10100appendonly yes[root@nfs1 /]# cat /etc/redis_7004.conf port 7004bind 192.168.1.234daemonize yespidfile /var/run/redis_7004.piddir /data/redis_data/7004cluster-enabled yescluster-config-file nodes_7004.confcluster-node-timeout 10100appendonly yes

启动这三个基于不同端口的redis服务，并查看其启动端口信息

[root@nfs1 /]# /usr/local/bin/redis-server /etc/redis_7000.conf 91981:C 14 Nov 2018 18:31:38.224 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo91981:C 14 Nov 2018 18:31:38.224 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=91981, just started91981:C 14 Nov 2018 18:31:38.224 # Configuration loaded[root@nfs1 /]# /usr/local/bin/redis-server /etc/redis_7002.conf 91986:C 14 Nov 2018 18:31:46.752 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo91986:C 14 Nov 2018 18:31:46.752 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=91986, just started91986:C 14 Nov 2018 18:31:46.752 # Configuration loaded[root@nfs1 /]# /usr/local/bin/redis-server /etc/redis_7004.conf 91991:C 14 Nov 2018 18:31:50.830 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo91991:C 14 Nov 2018 18:31:50.830 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=91991, just started91991:C 14 Nov 2018 18:31:50.830 # Configuration loaded[root@nfs1 /]# netstat -ntlp |grep redistcp        0      0 192.168.1.234:7000      0.0.0.0:*               LISTEN      91982/redis-server  tcp        0      0 192.168.1.234:7002      0.0.0.0:*               LISTEN      91987/redis-server  tcp        0      0 192.168.1.234:7004      0.0.0.0:*               LISTEN      91992/redis-server  tcp        0      0 192.168.1.234:17000     0.0.0.0:*               LISTEN      91982/redis-server  tcp        0      0 192.168.1.234:17002     0.0.0.0:*               LISTEN      91987/redis-server  tcp        0      0 192.168.1.234:17004     0.0.0.0:*               LISTEN      91992/redis-server 192.168.1.223的redis也是按照这种方式启动多个redis服务的[root@Huaching /]# /usr/local/bin/redis-server /etc/redis_7001.conf 26608:C 15 Nov 2018 00:00:10.297 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo26608:C 15 Nov 2018 00:00:10.297 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=26608, just started26608:C 15 Nov 2018 00:00:10.297 # Configuration loaded[root@Huaching /]# /usr/local/bin/redis-server /etc/redis_7003.conf 26622:C 15 Nov 2018 00:00:22.984 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo26622:C 15 Nov 2018 00:00:22.985 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=26622, just started26622:C 15 Nov 2018 00:00:22.985 # Configuration loaded[root@Huaching /]# /usr/local/bin/redis-server /etc/redis_7005.conf 26627:C 15 Nov 2018 00:00:27.128 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo26627:C 15 Nov 2018 00:00:27.128 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=26627, just started26627:C 15 Nov 2018 00:00:27.128 # Configuration loaded[root@Huaching /]# netstat -antp |grep redistcp        0      0 192.168.1.200:7003          0.0.0.0:*                   LISTEN      26623/redis-server  tcp        0      0 192.168.1.200:7005          0.0.0.0:*                   LISTEN      26628/redis-server  tcp        0      0 192.168.1.200:17001         0.0.0.0:*                   LISTEN      26609/redis-server  tcp        0      0 192.168.1.200:17003         0.0.0.0:*                   LISTEN      26623/redis-server  tcp        0      0 192.168.1.200:17005         0.0.0.0:*                   LISTEN      26628/redis-server  tcp        0      0 192.168.1.200:7001          0.0.0.0:*                   LISTEN      26609/redis-server  

启动多个redis服务后接下来安装ruby中间件实现群集功能

redis群集安装ruby

安装ruby只需要在一台服务器上进行安装运行

安装Development的软件组，安装过程省略

[root@nfs1 redis_data]# yum -y groupinstall "Development Tools"

安装需要支持的库，安装过程省略

[root@nfs1 redis_data]# yum install gdbm-devel linbdb4-devel libffi-devel libyaml libyaml-devel ncurses-devel openssl_devel readline-devel tcl-deve

创建编译rpm包的目录，下载ruby的源码包文件，将源码包下载只刚刚创建的目录下

如果下载有rpm编译器，则执行编译器来生成rpm包进行安装，如果没有就使用./configure编译安装

[root@nfs1 ~]# mkdir -p /root/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}[root@nfs1 ~]# rpmbuild -bb /root/rpmbuild/SPECS/ruby22x.spec[root@nfs1 ~]# yum -y localinstall rpmbuild/RPMS/x86_64/ruby-2.4.5-1.e17.centos.x86_64.rpm

由于这里我无法找到ruby的rpm编译器，所以无法编译成rpm包。这里依旧使用解压源码包，使用./configure来编译安装

[root@nfs1 ruby-2.4.5]# ./configure[root@nfs1 ruby-2.4.5]# make -----------------------------编译内容  Files: 890  Classes: 1312 ( 549 undocumented)  Modules: 275 ( 110 undocumented)  Constants: 2143 ( 538 undocumented)  Attributes: 1053 ( 247 undocumented)  Methods: 9837 (2106 undocumented)  Total: 14620 (3550 undocumented)   75.72% documented  Elapsed: 86.3s[root@nfs1 ruby-2.4.5]# make install-----------------------------编译内容installing bundle gems:       /usr/local/lib/ruby/gems/2.4.0 (build_info, cache, doc, extensions, gems, specifications)                              minitest 5.10.1                              xmlrpc 0.2.1                              test-unit 3.2.3                              net-telnet 0.1.1                              rake 12.0.0                              did_you_mean 1.1.0                              power_assert 0.4.1installing rdoc:              /usr/local/share/ri/2.4.0/systeminstalling capi-docs:         /usr/local/share/doc/ruby

编译安装完成后可以查看下ruby的版本号，验证是否编译安装正确

[root@nfs1 ruby-2.4.5]# ruby -vruby 2.4.5p335 (2018-10-18 revision 65137) [x86_64-linux]安装redis的ruby集群的相关命令[root@nfs1 ruby-2.4.5]# gem install redisFetching: redis-4.0.3.gem (100%)Successfully installed redis-4.0.3Parsing documentation for redis-4.0.3Installing ri documentation for redis-4.0.3Done installing documentation for redis after 1 seconds1 gem installed

拷贝redis的ruby命令到/usr/local/bin目录下，新版本舍弃了redis-trib.rb的这个命令，如果安装新版本，可以不执行此步操作

[root@nfs1 ruby-2.4.5]# cp /usr/local/src/redis-5.0.0/src/redis-trib.rb /usr/local/bin/

在新版本的redis中，使用redis-trib.rb来创建redis的群集会提示版本命令不可用，因为redis-trib.rb命令被舍弃不再使用。具体信息如下：

WARNING: redis-trib.rb is not longer available!You should use redis-cli instead.All commands and features belonging to redis-trib.rb have been movedto redis-cli.In order to use them you should call redis-cli with the --clusteroption followed by the subcommand name, arguments and options.Use the following syntax:redis-cli --cluster SUBCOMMAND [ARGUMENTS] [OPTIONS]Example:redis-cli --cluster create 192.168.1.234:700 192.168.1.234:7002 192.168.1.234:7004 192.168.1.200:7001 192.168.1.200:7003 192.168.1.200:7005 --cluster-replicas 1To get help about all subcommands, type:redis-cli --cluster help

从上面提示可以了解到redis-trub.rb命令已经被redis-cli代替使用

那么我们现在使用redis-cli来创建群集格式为redis-cli --cluster create  ip:prot  ip:port ...... --cluster-replicas 1这里我创建的群集如下，在确认好群集信息后，会提示你是否按照这个主从指定的方式保存

[root@nfs1 ruby-2.4.5]# redis-cli --cluster create 192.168.1.234:7000 192.168.1.234:7002 192.168.1.234:7004 192.168.1.200:7001 192.168.1.200:7003 192.168.1.200:7005 --cluster-replicas 1>>> Performing hash slots allocation on 6 nodes...Master[0] -> Slots 0 - 5460Master[1] -> Slots 5461 - 10922Master[2] -> Slots 10923 - 16383Adding replica 192.168.1.200:7003 to 192.168.1.234:7000                 这里指定了主从关系：从redis-->主redisAdding replica 192.168.1.234:7004 to 192.168.1.200:7001Adding replica 192.168.1.200:7005 to 192.168.1.234:7002M: 3af55accd31b6a057945c4b9e9378c7932d5f1f8 192.168.1.234:7000          输出信息中的M为master   S则是slave   slots:[0-5460] (5461 slots) masterM: d8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002            slots:[10923-16383] (5461 slots) masterS: 2a642a2e14d15ae8cb77b3ef04f58d32cfb09933 192.168.1.234:7004   replicates 3e8c289a64638da55b0e812f924aaa242efbc315M: 3e8c289a64638da55b0e812f924aaa242efbc315 192.168.1.200:7001   slots:[5461-10922] (5462 slots) masterS: 9882ec7d971ec8b159ff126ac8e1ce480a6ba9e0 192.168.1.200:7003   replicates 3af55accd31b6a057945c4b9e9378c7932d5f1f8S: a3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005   replicates d8418ab100f8f32539fc60a7c8ead5927591f52aCan I set the above configuration? (type 'yes' to accept): yes

在上面确认交互的地方确认保存后会输出两个ok信息，可以判断redis群集创建成功

Can I set the above configuration? (type 'yes' to accept): yes[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.

redis创建集群

redis集群创建好后，登录redis时需要指定登入ip和端口，创建几个key值查看redis存储的redis信息

-c  登入redis集群模式 -h 指定集群中的一个redis

[root@nfs1 ruby-2.4.5]# redis-cli -c -h 192.168.1.234 -p 7000192.168.1.234:7000> set key1 123-> Redirected to slot [9189] located at 192.168.1.200:7001OK192.168.1.200:7001> set key3 abc-> Redirected to slot [935] located at 192.168.1.234:7000OK192.168.1.234:7000> set key2 ABCOK192.168.1.234:7000> set key4 ABC-> Redirected to slot [13120] located at 192.168.1.234:7002OK192.168.1.234:7002> set key5 ABC-> Redirected to slot [9057] located at 192.168.1.200:7001OK192.168.1.200:7001> set key6 ABC-> Redirected to slot [4866] located at 192.168.1.234:7000OK

查看群集状态信息，redis-cli代替了redis-trib.rb，所以这里使用新的命令格式。可以看出群集的redis信息状态

[root@nfs1 ruby-2.4.5]# redis-cli --cluster check 192.168.1.234:7000192.168.1.234:7000 (3af55acc...) -> 4 keys | 5461 slots | 1 slaves.192.168.1.234:7002 (d8418ab1...) -> 1 keys | 5461 slots | 1 slaves.192.168.1.200:7001 (3e8c289a...) -> 2 keys | 5462 slots | 1 slaves.[OK] 7 keys in 3 masters.0.00 keys per slot on average.>>> Performing Cluster Check (using node 192.168.1.234:7000)M: 3af55accd31b6a057945c4b9e9378c7932d5f1f8 192.168.1.234:7000   slots:[0-5460] (5461 slots) master   1 additional replica(s)S: a3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005   slots: (0 slots) slave   replicates d8418ab100f8f32539fc60a7c8ead5927591f52aM: d8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002   slots:[10923-16383] (5461 slots) master   1 additional replica(s)M: 3e8c289a64638da55b0e812f924aaa242efbc315 192.168.1.200:7001   slots:[5461-10922] (5462 slots) master   1 additional replica(s)S: 2a642a2e14d15ae8cb77b3ef04f58d32cfb09933 192.168.1.234:7004   slots: (0 slots) slave   replicates 3e8c289a64638da55b0e812f924aaa242efbc315S: 9882ec7d971ec8b159ff126ac8e1ce480a6ba9e0 192.168.1.200:7003   slots: (0 slots) slave   replicates 3af55accd31b6a057945c4b9e9378c7932d5f1f8[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.

redis群集的管理命令

cluster nodes   列出集群节点

[root@nfs1 ruby-2.4.5]# redis-cli -c -h 192.168.1.234 -p 7000         登入任意一个redis中192.168.1.234:7000> CLUSTER NODESa3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005@17005 slave d8418ab100f8f32539fc60a7c8ead5927591f52a 0 1542270025000 6 connectedd8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002@17002 master - 0 1542270024896 2 connected 10923-163833e8c289a64638da55b0e812f924aaa242efbc315 192.168.1.200:7001@17001 master - 0 1542270027918 4 connected 5461-109222a642a2e14d15ae8cb77b3ef04f58d32cfb09933 192.168.1.234:7004@17004 slave 3e8c289a64638da55b0e812f924aaa242efbc315 0 1542270026906 4 connected3af55accd31b6a057945c4b9e9378c7932d5f1f8 192.168.1.234:7000@17000 myself,master - 0 1542270024000 1 connected 0-54609882ec7d971ec8b159ff126ac8e1ce480a6ba9e0 192.168.1.200:7003@17003 slave 3af55accd31b6a057945c4b9e9378c7932d5f1f8 0 1542270028928 5 connected

cluster info   查看集群信息

192.168.1.234:7000> cluster infocluster_state:okcluster_slots_assigned:16384cluster_slots_ok:16384cluster_slots_pfail:0cluster_slots_fail:0cluster_known_nodes:6cluster_size:3cluster_current_epoch:6cluster_my_epoch:1cluster_stats_messages_ping_sent:1594cluster_stats_messages_pong_sent:1603cluster_stats_messages_sent:3197cluster_stats_messages_ping_received:1598cluster_stats_messages_pong_received:1594cluster_stats_messages_meet_received:5cluster_stats_messages_received:3197

所有的新添加的redis节点，都是为master角色

cluster meet   ip  port在192.168.1.200上新启动一个redis，监听端口为7007。登入群集中执行redis节点添加命令

[root@nfs1 redis_data]# redis-cli -c -h 192.168.1.234 -p 7000192.168.1.234:7000> cluster meet 192.168.1.200 7007OK192.168.1.234:7000> cluster nodes                  //第一条就是新添加的节点信息cff62edb40a4df11d8390bc28d254abca16188ba 192.168.1.200:7007@17007 master - 0 1542270675527 0 connecteda3c7b8a393f2b4d3b9838753fb6496c3861bbb69 192.168.1.200:7005@17005 slave d8418ab100f8f32539fc60a7c8ead5927591f52a 0 1542270677550 6 connectedd8418ab100f8f32539fc60a7c8ead5927591f52a 192.168.1.234:7002@17002 master - 0 1542270678560 2 connected 10923-16383----------------省略

再添加一个新的redis节点，并将新节点指定为从节点

cluster reolicate  master node info            //登入需要设定为从节点的redis，执行命令指定主节点指定从节点需要登入到那个redis节点当中操作，而不是在集群中指定，在登入节点中指定本节点的master再次添加一个redis节点

192.168.1.234:7000> cluster meet 192.168.1.234 7006OKcff62edb40a4df11d8390bc28d254abca16188ba 192.168.1.200:7007@17007 master - 0 1542270675527 0 connected24f1c355bc92a0ab15a6077b1bbf4bd510dfa132 192.168.1.234:7006@17006 master - 0 1542270869918 7 connected

退出节点登入端口为7006的节点，

192.168.1.234:7000> exit[root@nfs1 redis_data]# redis-cli -c -h 192.168.1.234 -p 7006192.168.1.234:7006> CLUSTER REPLICATE cff62edb40a4df11d8390bc28d254abca16188baOK

再次查看群集节点状态信息，可以看到端口7006的redis已经成为7007redis的slave节点了

192.168.1.234:7006> cluster nodescff62edb40a4df11d8390bc28d254abca16188ba 192.168.1.200:7007@17007 master - 0 1542271290173 0 connected24f1c355bc92a0ab15a6077b1bbf4bd510dfa132 192.168.1.234:7006@17006 myself,slave cff62edb40a4df11d8390bc28d254abca16188ba 0 1542271292000 7 connected

移除一个redis节点

cluster forget    node_id在移除某个redis节点之前，首先不能在登入该节点当中，否则不能正常移除该节点，当前处于7006的redis当中，移除节点提示信息：

24f1c355bc92a0ab15a6077b1bbf4bd510dfa132 192.168.1.234:7006@17006 myself,slave cff62edb40a4df11d8390bc28d254abca16188ba 0 1542271292000 7 connected192.168.1.234:7006> cluster forget 24f1c355bc92a0ab15a6077b1bbf4bd510dfa132(error) ERR I tried hard but I can't forget myself...

必须登入其他节点移除端口为7006的redis节点

[root@nfs1 redis_data]# redis-cli -c -h 192.168.1.234 -p 7000

移除7006节点之前查看所有节点信息

192.168.1.234:7000> CLUSTER NODES

拷贝7006节点的node_id，移除7006节点

192.168.1.234:7000> cluster forget 24f1c355bc92a0ab15a6077b1bbf4bd510dfa132OK

移除7006节点后

192.168.1.234:7000> CLUSTER NODES

保存redis集群的配置文件

cluster saveconfig  保存配置文件，会在当前登入的redis数据存储路径下增加redis配置内容

192.168.1.234:7000> cluster saveconfigOK192.168.1.234:7000> exit