昨晚在午夜左右,我们的应用程序崩溃了,我试图确定为什么。 我们目前在EC2上有一个nginx前端服务器和2个独angular兽工作者(app)服务器。
我们麒麟的工作人员几乎不停地计时,因此被主人重新启动。
从我所看到的,我们现在没有crontabs或任何设置运行,所以我有点难住。
当我醒来(经过6个小时的宕机)之后,我杀死了独angular兽进程并重新执行了独angular兽二进制文件,于是我能够在今天早上得到这个应用程序。 (unicorn_rails -c unicorn.rb等)
任何想法在哪里看? 事实上,这两个应用程序服务器宕机让我觉得它可能是数据库(RDS)?
日志填满了以下(6小时..等);
E, [2013-02-28T00:07:40.367981 #11097] ERROR -- : worker=2 PID:26941 timeout (31s > 30s), killing E, [2013-02-28T00:07:40.468495 #11097] ERROR -- : reaped #<Process::Status: pid 26941 SIGKILL (signal 9)> worker=2 I, [2013-02-28T00:07:40.756724 #28319] INFO -- : worker=2 ready E, [2013-02-28T00:07:44.519818 #11097] ERROR -- : worker=1 PID:11292 timeout (31s > 30s), killing E, [2013-02-28T00:07:44.626362 #11097] ERROR -- : worker=0 PID:26933 timeout (31s > 30s), killing E, [2013-02-28T00:07:44.726936 #11097] ERROR -- : reaped #<Process::Status: pid 11292 SIGKILL (signal 9)> worker=1 E, [2013-02-28T00:07:44.727254 #11097] ERROR -- : worker=0 PID:26933 timeout (31s > 30s), killing E, [2013-02-28T00:07:44.932858 #11097] ERROR -- : reaped #<Process::Status: pid 26933 SIGKILL (signal 9)> worker=0 I, [2013-02-28T00:07:45.661356 #28329] INFO -- : worker=1 ready I, [2013-02-28T00:07:45.828289 #28334] INFO -- : worker=0 ready E, [2013-02-28T00:08:11.113970 #11097] ERROR -- : worker=2 PID:28319 timeout (31s > 30s), killing E, [2013-02-28T00:08:11.214770 #11097] ERROR -- : reaped #<Process::Status: pid 28319 SIGKILL (signal 9)> worker=2 I, [2013-02-28T00:08:11.518723 #28368] INFO -- : worker=2 ready E, [2013-02-28T00:08:16.270463 #11097] ERROR -- : worker=1 PID:28329 timeout (31s > 30s), killing E, [2013-02-28T00:08:16.371067 #11097] ERROR -- : worker=0 PID:28334 timeout (31s > 30s), killing E, [2013-02-28T00:08:16.471684 #11097] ERROR -- : reaped #<Process::Status: pid 28329 SIGKILL (signal 9)> worker=1 E, [2013-02-28T00:08:16.471983 #11097] ERROR -- : reaped #<Process::Status: pid 28334 SIGKILL (signal 9)> worker=0 I, [2013-02-28T00:08:17.038915 #28376] INFO -- : worker=0 ready I, [2013-02-28T00:08:17.128931 #28379] INFO -- : worker=1 ready E, [2013-02-28T00:08:42.628665 #11097] ERROR -- : worker=2 PID:28368 timeout (31s > 30s), killing E, [2013-02-28T00:08:42.729290 #11097] ERROR -- : reaped #<Process::Status: pid 28368 SIGKILL (signal 9)> worker=2 I, [2013-02-28T00:08:43.015140 #28390] INFO -- : worker=2 ready E, [2013-02-28T00:08:48.778221 #11097] ERROR -- : worker=0 PID:28376 timeout (31s > 30s), killing E, [2013-02-28T00:08:48.878530 #11097] ERROR -- : worker=1 PID:28379 timeout (31s > 30s), killing
pipe理解决这个家伙。 进一步的调查显示,凌晨12点到凌晨4点之间的networkingstream量(和CPU使用率!)很大。 事实certificate,在这段时间之间,我们的网站站长设置为“高”,导致bingbot疯狂,绝对锤独angular兽。 (可怜的独angular兽)。