服务器一小时没有响应，所以我重新启动并检查了/ var / log / messages
并发现这一点。 谁能指出最新的错误？
Sep 28 07:39:35 www kernel: INFO: task mysqld:22749 blocked for more than 120 seconds. Sep 28 07:39:35 www kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 28 07:39:35 www kernel: mysqld D ffff810001015120 0 22749 3266 22792 22659 (NOTLB) Sep 28 07:39:35 www kernel: ffff810139d21e58 0000000000000086 ffff810036217000 ffffffff8000f758 Sep 28 07:39:35 www kernel: ffff81020dfd1408 0000000000000007 ffff8101cfbaf7e0 ffff81020fca5080 Sep 28 07:39:35 www kernel: 00017a451524782a 00000000000043b2 ffff8101cfbaf9c8 0000000280009a22 Sep 28 07:39:35 www kernel: Call Trace: Sep 28 07:39:35 www kernel: [<ffffffff8000f758>] generic_permission+0x52/0xca Sep 28 07:39:35 www kernel: [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b Sep 28 07:39:35 www kernel: [<ffffffff8000cea2>] do_path_lookup+0x294/0x310 Sep 28 07:39:35 www kernel: [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14 Sep 28 07:39:35 www kernel: [<ffffffff8003c618>] do_unlinkat+0x66/0x141 Sep 28 07:39:35 www kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0 Sep 28 07:39:57 www kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 28 07:39:58 www kernel: Sep 28 07:39:59 www kernel: INFO: task httpd:22679 blocked for more than 120 seconds. Sep 28 07:40:04 www kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 28 07:40:08 www kernel: httpd D ffff81000100caa0 0 22679 22413 22680 22678 (NOTLB) Sep 28 07:40:51 www kernel: ffff81018b0dbc78 0000000000000086 ffff81018b0dbc88 0000004480063002 Sep 28 07:41:52 www kernel: ffff81000001cc00 0000000000000007 ffff81013ac5e860 ffff81020fc96100 Sep 28 07:43:10 www kernel: 00017a44de6376c8 000000000000a89f ffff81013ac5ea48 000000010001cc00 Sep 28 07:43:38 www kernel: Call Trace: Sep 28 07:44:06 www kernel: [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b Sep 28 07:44:09 www kernel: [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14 Sep 28 07:44:10 www kernel: [<ffffffff8000d0b2>] do_lookup+0x90/0x1e6 Sep 28 07:44:13 www kernel: [<ffffffff8000a2e9>] __link_path_walk+0xa3a/0xfd1 Sep 28 07:44:16 www kernel: [<ffffffff8000eb8e>] link_path_walk+0x45/0xb8 Sep 28 07:44:16 www kernel: [<ffffffff8000cea2>] do_path_lookup+0x294/0x310 Sep 28 07:44:29 www kernel: [<ffffffff800129ad>] getname+0x15b/0x1c2 Sep 28 07:44:38 www kernel: [<ffffffff80023b60>] __user_walk_fd+0x37/0x4c Sep 28 07:44:42 www kernel: [<ffffffff80028ada>] vfs_stat_fd+0x1b/0x4a Sep 28 07:44:43 www kernel: [<ffffffff8003c69a>] do_unlinkat+0xe8/0x141 Sep 28 07:45:02 www kernel: [<ffffffff80023890>] sys_newstat+0x19/0x31 Sep 28 07:46:18 www kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0 Sep 28 07:46:43 www kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 28 07:46:55 www kernel: Sep 28 07:46:58 www kernel: INFO: task php:28906 blocked for more than 120 seconds. Sep 28 07:46:59 www kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 28 07:47:00 www kernel: php D ffff810165127000 0 28906 28905 (NOTLB) Sep 28 07:47:37 www kernel: ffff810078431e58 0000000000000082 ffff810165127000 ffffffff8000f758 Sep 28 07:48:29 www kernel: ffff81020dfd1408 0000000000000007 ffff8101247b9860 ffff810207d0e100 Sep 28 07:48:36 www kernel: 00017a4218932fae 0000000000377111 ffff8101247b9a48 0000000280009a22 Sep 28 07:48:37 www kernel: Call Trace: Sep 28 07:48:37 www kernel: [<ffffffff8000f758>] generic_permission+0x52/0xca Sep 28 07:48:37 www kernel: [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b Sep 28 07:48:37 www kernel: [<ffffffff8000cea2>] do_path_lookup+0x294/0x310 Sep 28 07:48:41 www kernel: [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14 Sep 28 07:48:41 www kernel: [<ffffffff8003c618>] do_unlinkat+0x66/0x141 Sep 28 07:48:42 www kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0 Sep 28 07:48:42 www kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 28 07:48:42 www kernel: Sep 28 07:48:43 www kernel: INFO: task php:29032 blocked for more than 120 seconds. Sep 28 07:48:45 www kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 28 07:48:46 www kernel: php D 0000000000000004 0 29032 1 29050 29024 (NOTLB) Sep 28 07:48:46 www kernel: ffff81006b465dc8 0000000000000086 ffff81020dfd1408 ffffffff80009a22 Sep 28 07:48:46 www kernel: 0000000000000000 0000000000000007 ffff81002946e860 ffff81003c943100 Sep 28 07:48:46 www kernel: 00017a4211450766 000000000024be3d ffff81002946ea48 000000020e42b300 Sep 28 07:48:52 www kernel: Call Trace: Sep 28 07:48:54 www kernel: [<ffffffff80009a22>] __link_path_walk+0x173/0xfd1 Sep 28 07:48:54 www kernel: [<ffffffff8002cc58>] mntput_no_expire+0x19/0x89 Sep 28 07:48:55 www kernel: [<ffffffff8000ebf5>] link_path_walk+0xac/0xb8 Sep 28 07:48:55 www kernel: [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b Sep 28 07:48:55 www kernel: [<ffffffff80023974>] __path_lookup_intent_open+0x56/0x97 Sep 28 07:48:55 www kernel: [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14 Sep 28 07:48:55 www kernel: [<ffffffff8001b260>] open_namei+0xea/0x718 Sep 28 07:48:59 www kernel: [<ffffffff80067235>] do_page_fault+0x4cc/0x842 Sep 28 07:49:01 www kernel: [<ffffffff80027726>] do_filp_open+0x1c/0x38 Sep 28 07:49:01 www kernel: [<ffffffff8001a09c>] do_sys_open+0x44/0xbe Sep 28 07:49:02 www kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 28 07:49:03 www kernel: Sep 28 07:49:07 www kernel: INFO: task mysqld:22749 blocked for more than 120 seconds. Sep 28 07:49:09 www kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 28 07:49:09 www kernel: mysqld D ffff810001015120 0 22749 3266 22792 22659 (NOTLB) Sep 28 07:49:14 www kernel: ffff810139d21e58 0000000000000086 ffff810036217000 ffffffff8000f758 Sep 28 07:49:14 www kernel: ffff81020dfd1408 0000000000000007 ffff8101cfbaf7e0 ffff81020fca5080 Sep 28 07:49:15 www kernel: 00017a451524782a 00000000000043b2 ffff8101cfbaf9c8 0000000280009a22 Sep 28 07:49:15 www kernel: Call Trace: Sep 28 07:49:22 www kernel: [<ffffffff8000f758>] generic_permission+0x52/0xca Sep 28 07:49:23 www kernel: [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b Sep 28 07:49:23 www kernel: [<ffffffff8000cea2>] do_path_lookup+0x294/0x310 Sep 28 07:49:23 www kernel: [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14 Sep 28 07:49:23 www kernel: [<ffffffff8003c618>] do_unlinkat+0x66/0x141 Sep 28 07:49:23 www kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0 Sep 28 07:49:23 www kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 28 07:49:23 www kernel: Sep 28 07:49:23 www kernel: INFO: task php:29024 blocked for more than 120 seconds. Sep 28 07:49:23 www kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 28 07:49:24 www kernel: php D ffff8101920a0000 0 29024 1 29032 29001 (NOTLB) Sep 28 07:49:26 www kernel: ffff8101cca8fe58 0000000000000086 ffff8101920a0000 ffffffff8000f758 Sep 28 07:49:26 www kernel: ffff81020dfd1408 0000000000000007 ffff81000b64b040 ffff8101e05337e0 Sep 28 07:49:26 www kernel: 00017a552aef9f35 0000000000009513 ffff81000b64b228 0000000180009a22 Sep 28 07:49:27 www kernel: Call Trace: Sep 28 07:49:27 www kernel: [<ffffffff8000f758>] generic_permission+0x52/0xca Sep 28 07:49:27 www kernel: [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b Sep 28 07:49:27 www kernel: [<ffffffff8000cea2>] do_path_lookup+0x294/0x310 Sep 28 07:49:27 www kernel: [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14 Sep 28 07:49:27 www kernel: [<ffffffff8003c618>] do_unlinkat+0x66/0x141 Sep 28 07:49:27 www kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0 Sep 28 07:49:27 www kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 28 07:49:27 www kernel: Sep 28 07:49:27 www kernel: INFO: task php:29050 blocked for more than 120 seconds. Sep 28 07:49:28 www kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 28 07:49:28 www kernel: php D ffff810201d95000 0 29050 1 29032 (NOTLB) Sep 28 07:49:28 www kernel: ffff810051e45e58 0000000000000086 ffff810201d95000 ffffffff8000f758 Sep 28 07:49:28 www kernel: ffff81020dfd1408 0000000000000007 ffff81001c23f080 ffff81020f5e2080 Sep 28 07:49:29 www kernel: 00017a5d0bc2aa75 0000000000d0ecfe ffff81001c23f268 0000000280009a22 Sep 28 07:49:29 www kernel: Call Trace: Sep 28 07:49:29 www kernel: [<ffffffff8000f758>] generic_permission+0x52/0xca Sep 28 07:49:29 www kernel: [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b Sep 28 07:49:29 www kernel: [<ffffffff8000cea2>] do_path_lookup+0x294/0x310 Sep 28 07:49:34 www kernel: [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14 Sep 28 07:49:35 www kernel: [<ffffffff8003c618>] do_unlinkat+0x66/0x141 Sep 28 07:49:37 www kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0 Sep 28 07:49:37 www kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 28 07:49:37 www kernel: Sep 28 07:49:37 www kernel: INFO: task php:29064 blocked for more than 120 seconds. Sep 28 07:49:37 www kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 28 07:49:37 www kernel: php D ffff81009c231000 0 29064 29057 (NOTLB) Sep 28 07:49:38 www kernel: ffff8100a5dc7e58 0000000000000086 ffff81009c231000 ffffffff8000f758 Sep 28 07:49:38 www kernel: ffff81020dfd1408 0000000000000007 ffff81000a850820 ffff8102038037a0 Sep 28 07:49:38 www kernel: 00017a5bb5c6846e 000000000000861a ffff81000a850a08 0000000080009a22 Sep 28 07:49:38 www kernel: Call Trace: Sep 28 07:49:38 www kernel: [<ffffffff8000f758>] generic_permission+0x52/0xca Sep 28 07:49:38 www kernel: [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b Sep 28 07:49:38 www kernel: [<ffffffff8000cea2>] do_path_lookup+0x294/0x310 Sep 28 07:49:38 www kernel: [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14 Sep 28 07:49:38 www kernel: [<ffffffff8003c618>] do_unlinkat+0x66/0x141 Sep 28 07:49:38 www kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0 Sep 28 07:49:40 www kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 28 07:49:42 www kernel: Sep 28 07:49:42 www kernel: INFO: task mysqld:24612 blocked for more than 120 seconds. Sep 28 07:49:43 www kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 28 07:49:46 www kernel: mysqld D ffff81020dfd14c0 0 24612 3266 19643 3599 (NOTLB) Sep 28 07:49:46 www kernel: ffff81019e517c78 0000000000000086 ffff81019e517c88 ffffffff80063002 Sep 28 07:49:47 www kernel: ffff810201966558 0000000000000009 ffff81015fa560c0 ffff8101c263b860 Sep 28 07:49:51 www kernel: 00017a9d113e27fe 0000000000008d5a ffff81015fa562a8 000000018006ec9f Sep 28 07:49:52 www kernel: Call Trace: Sep 28 07:49:52 www kernel: [<ffffffff80063002>] thread_return+0x62/0xfe Sep 28 07:49:52 www kernel: [<ffffffff8005a46a>] getnstimeofday+0x10/0x29 Sep 28 07:49:53 www kernel: [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b Sep 28 07:49:54 www kernel: [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14 Sep 28 07:49:54 www kernel: [<ffffffff8000d0b2>] do_lookup+0x90/0x1e6 Sep 28 07:49:56 www kernel: [<ffffffff8000a2e9>] __link_path_walk+0xa3a/0xfd1 Sep 28 07:50:00 www kernel: [<ffffffff8000eb8e>] link_path_walk+0x45/0xb8 Sep 28 07:50:03 www kernel: [<ffffffff8000cea2>] do_path_lookup+0x294/0x310 Sep 28 07:50:04 www kernel: [<ffffffff800129ad>] getname+0x15b/0x1c2 Sep 28 07:50:06 www kernel: [<ffffffff80023b60>] __user_walk_fd+0x37/0x4c Sep 28 07:50:06 www kernel: [<ffffffff8003f013>] vfs_lstat_fd+0x18/0x47 Sep 28 07:50:08 www kernel: [<ffffffff8002ad91>] sys_newlstat+0x19/0x31 Sep 28 07:50:10 www kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0 Sep 28 07:50:15 www kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 28 07:50:19 www kernel: Sep 28 07:50:19 www kernel: INFO: task php:29178 blocked for more than 120 seconds. Sep 28 07:50:23 www kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 28 07:50:23 www kernel: php D 0000000000000003 0 29178 29123 (NOTLB) Sep 28 07:50:23 www kernel: ffff81004a95bdc8 0000000000000086 ffff81020dfd1408 ffffffff80009a22 Sep 28 07:50:24 www kernel: ffffffff800a2fd0 0000000000000007 ffff8101937a4040 ffff81010bde27a0 Sep 28 07:50:26 www kernel: 00017aa3a1d89c9b 000000000000d66e ffff8101937a4228 000000020e42b300 Sep 28 07:50:26 www kernel: Call Trace: Sep 28 07:50:26 www kernel: [<ffffffff80009a22>] __link_path_walk+0x173/0xfd1 Sep 28 07:50:27 www kernel: [<ffffffff800a2fd0>] wake_bit_function+0x0/0x23 Sep 28 07:50:27 www kernel: [<ffffffff8002cc58>] mntput_no_expire+0x19/0x89 Sep 28 07:50:27 www kernel: [<ffffffff8000ebf5>] link_path_walk+0xac/0xb8 Sep 28 07:50:28 www kernel: [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b Sep 28 07:50:32 www kernel: [<ffffffff80023974>] __path_lookup_intent_open+0x56/0x97 Sep 28 07:50:32 www kernel: [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14 Sep 28 07:50:34 www kernel: [<ffffffff8001b260>] open_namei+0xea/0x718 Sep 28 07:50:34 www kernel: [<ffffffff80067235>] do_page_fault+0x4cc/0x842 Sep 28 07:50:35 www kernel: [<ffffffff80027726>] do_filp_open+0x1c/0x38 Sep 28 07:50:35 www kernel: [<ffffffff8001a09c>] do_sys_open+0x44/0xbe Sep 28 07:50:35 www kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Sep 28 07:50:35 www kernel: Sep 28 07:56:41 www kernel: proftpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Sep 28 07:56:41 www kernel: Sep 28 07:56:41 www kernel: Call Trace: Sep 28 07:56:41 www kernel: [<ffffffff800c9f35>] out_of_memory+0x8e/0x2f3 Sep 28 07:56:41 www kernel: [<ffffffff800a2fa2>] autoremove_wake_function+0x0/0x2e Sep 28 07:56:41 www kernel: [<ffffffff8000f67d>] __alloc_pages+0x27f/0x308 Sep 28 07:56:41 www kernel: [<ffffffff80013047>] __do_page_cache_readahead+0x96/0x17b Sep 28 07:56:41 www kernel: [<ffffffff80013984>] filemap_nopage+0x14c/0x360 Sep 28 07:56:41 www kernel: [<ffffffff80008972>] __handle_mm_fault+0x1fd/0x103b Sep 28 07:56:41 www kernel: [<ffffffff800a4fe1>] ktime_get_ts+0x1a/0x4e Sep 28 07:56:41 www kernel: [<ffffffff80067202>] do_page_fault+0x499/0x842 Sep 28 07:56:41 www kernel: [<ffffffff8003ad91>] hrtimer_try_to_cancel+0x4a/0x53 Sep 28 07:58:10 www kernel: [<ffffffff80033541>] do_setitimer+0xd0/0x689 Sep 28 08:26:22 www syslogd 1.4.1: restart. Sep 28 08:26:22 www kernel: klogd 1.4.1, log source = /proc/kmsg started. Sep 28 08:26:22 www kernel: Linux version 2.6.18-274.17.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-51)) #1 SMP Tue Jan 10 17:25:58 EST 2012 Sep 28 08:26:22 www kernel: Command line: ro root=LABEL=/ 正如萨法多所说，你的记忆力已经耗尽，导致了杀手锏和closures事情。 我也有这个问题 。
我采取了以下行动：
这使我在login到服务器时，事情开始看起来不稳定，并检查什么是使用所有的内存。
在我的情况下，这是Apache。 我重新configuration它以减less备用线程和服务器的数量，问题就消失了。
主要的一点是，当这样的事情发生在你身上时，监控确实会有所帮助。
你有任何服务器监控软件保持logging的服务器标志？ 特别是RAM？ 在07:56:41，被称为oom-killer，这意味着你可能是内存不足，这可以解释你与httpd和mysqld的问题，造成不响应的行为。