我的产品Redis服务器已经疯了,并占用了100%的CPU使用率。
我已经尽我所能从升级redis服务器到重新启动机器。
我无法弄清楚是什么原因造成的。
这是redis.log
> [851] 17 Jun 13:13:15.290 *后台保存以成功结束 > [851] 6月17日13:14:16.061 * 60秒内发生10000次变化。 保存... > [851] 17 Jun 13:14:16.270 *后台保存由pid 32451开始 > [32451] 17 Jun 13:14:25.265 * DB保存在磁盘上[32451] > 13:14:25.279 * RDB:写入时使用的内存为5 MB [851] > 13:14:25.535 *保存背景成功[851] > 13:15:26.025 * 10000在60秒内改变。 拯救... [851] > 13:15:26.238 *由pid开始的后台保存32452 [32452] > 13:15:36.587 * DB保存在磁盘上[32452] 17 Jun 13:15:36.601 * RDB:5 >写入时使用的内存MB [851] 17 Jun 13:15:36.675 * >后台保存成功终止[851] 17 Jun 13:16:37.079 * 在60秒内> 10000次更改。 节省... [851] 17 Jun 13:16:37.294 * >后台保存由pid 1210开始[1210] 17 Jun 13:16:45.960 * DB >保存在磁盘上[1210] 17 Jun 13:16:45.975 * RDB:使用5 MB的内存 >写时复制[851] 17 Jun 13:16:46.051 *后台保存已终止 >成功
这里是redis-cli信息
>➜redis-cli > redis 127.0.0.1:6379> info >#服务器redis_version:2.6.13 redis_git_sha1:00000000 redis_git_dirty:0 redis_mode:独立os:Linux 3.2.0-36-virtual > x86_64 arch_bits:64 multiplexing_api:epoll gcc_version:4.6.3 > process_id:851 run_id:21c90a7be41353c4616203cdb5e6cc2af5c47337 > tcp_port:6379 uptime_in_seconds:7809 uptime_in_days:0 hz:10 > lru_clock:832635 > >#客户端connected_clients:84 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 > >#内存used_memory:709903928 used_memory_human:677.02M used_memory_rss:726933504 used_memory_peak:710305600 > used_memory_peak_human:677.40M used_memory_lua:37888 > mem_fragmentation_ratio:1.02 mem_allocator:jemalloc-3.3.1 > >#持久性加载:0 rdb_changes_since_last_save:2164 rdb_bgsave_in_progress:0 rdb_last_save_time:1371475146 > rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:9 > rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 > aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 > aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok > >#统计total_connections_received:1351 total_commands_processed:3210273 instantaneous_ops_per_sec:606 > rejected_connections:0 expired_keys:122 evicted_keys:0 > keyspace_hits:626012 keyspace_misses:1057334 pubsub_channels:0 > pubsub_patterns:0 latest_fork_usec:210633 > >#复制angular色:master connected_slaves:0 > >#CPU used_cpu_sys:75.42 used_cpu_user:6280.34 used_cpu_sys_children:74.85 used_cpu_user_children:426.74 > >#keyspace db0:keys = 33999,expires = 5 db5:keys = 95,expires = 13 db15:keys = 21221,expires = 1 redis 127.0.0.1:6379>
PoormansProfiler
> 200 > pthread_cond_wait @@ GLIBC_2.3.2,bioProcessBackgroundJobs,start_thread,clone,?? > 26,sdscmp,compareStringObjects,equalStringObjects,lremCommand,call,luaRedisGenericCommand,??,??,??,??,lua_pcall,evalGenericCommand,call,processCommand,processInputBuffer,readQueryFromClient,aeProcessEvents,aeMain,main > 19 listTypeNext,lremCommand,call,luaRedisGenericCommand,??,??,??,??,lua_pcall,evalGenericCommand,call,processCommand,processInputBuffer,readQueryFromClient,aeProcessEvents,aeMain,main > 16 sdscmp,compareStringObjects,equalStringObjects,lremCommand,call,luaRedisGenericCommand,??,??,??,??,lua_pcall,evalGenericCommand,call,processCommand,processInputBuffer,readQueryFromClient,aeProcessEvents,aeMain,main > 16 compareStringObjects,equalStringObjects,lremCommand,call,luaRedisGenericCommand,??,??,??,??,lua_pcall,evalGenericCommand,call,processCommand,processInputBuffer,readQueryFromClient,aeProcessEvents,aeMain,main > 8 listTypeEqual,lremCommand,call,luaRedisGenericCommand,??,??,??,??,lua_pcall,evalGenericCommand,call,processCommand,processInputBuffer,readQueryFromClient,aeProcessEvents,aeMain,main > 5 memcmp @ plt,sdscmp,compareStringObjects,equalStringObjects,lremCommand,call,luaRedisGenericCommand,??,??,??,??,lua_pcall,evalGenericCommand,call,processCommand,processInputBuffer,readQueryFromClient,aeProcessEvents,aeMain,main > 5 epoll_wait,aeProcessEvents,aeMain,main > 2 lremCommand,call,luaRedisGenericCommand,??,??,??,??,lua_pcall,evalGenericCommand,call,processCommand,processInputBuffer,readQueryFromClient,aeProcessEvents,aeMain,main > 2 equalStringObjects,lremCommand,call,luaRedisGenericCommand,??,??,??,??,lua_pcall,evalGenericCommand,call,processCommand,processInputBuffer,readQueryFromClient,aeProcessEvents,aeMain,main > 1 gettimeofday,ustime,call,luaRedisGenericCommand,??,??,??,??,lua_pcall,evalGenericCommand,call,processCommand,processInputBuffer,readQueryFromClient,aeProcessEvents,aeMain,main > 1
我用轨3.2.13使用sidekiq 2.13.1
任何帮助将非常感激。
[修正]在sidekiq 2.13.4和sidekiq-limit_fetch 1.7中修复的一个不好的组合是罪魁祸首