服务器星期天凌晨6点崩溃 – 内存不足

我有一个奇怪的问题:每个星期天上午6点,我的LAMP服务器崩溃。
看看日志,我看到当时有500个apache2进程(这是一个没有任何负载的testing服务器 – 特别是不是在上午6点)

系统日志声明如下:

May 19 06:00:11 myserver kernel: [313742.304291] Out of memory: Kill process 912 (mysqld) score 31 or sacrifice child May 19 06:00:11 myserver kernel: [313742.304311] Killed process 912 (mysqld) total-vm:816528kB, anon-rss:6240kB, file-rss:0kB 

看来服务器内存不足,从而导致一些进程中断。

可能是什么问题呢? 这可能与每周的crontab有关吗?

以下是系统日志中的更多行:

 May 19 06:00:11 myserver kernel: [313742.290517] oom_kill_process: 3 callbacks suppressed May 19 06:00:11 myserver kernel: [313742.290526] apache2 invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0 May 19 06:00:11 myserver kernel: [313742.290534] apache2 cpuset=/ mems_allowed=0 May 19 06:00:11 myserver kernel: [313742.290541] Pid: 1884, comm: apache2 Not tainted 3.2.0-29-generic #46-Ubuntu May 19 06:00:11 myserver kernel: [313742.290546] Call Trace: May 19 06:00:11 myserver kernel: [313742.290561] [<ffffffff810bf9ad>] ? cpuset_print_task_mems_allowed+0x9d/0xb0 May 19 06:00:11 myserver kernel: [313742.290570] [<ffffffff8111a7e1>] dump_header+0x91/0xe0 May 19 06:00:11 myserver kernel: [313742.290577] [<ffffffff8111ab65>] oom_kill_process+0x85/0xb0 May 19 06:00:11 myserver kernel: [313742.290584] [<ffffffff8111af0a>] out_of_memory+0xfa/0x220 May 19 06:00:11 myserver kernel: [313742.290592] [<ffffffff8112098f>] __alloc_pages_nodemask+0x80f/0x820 May 19 06:00:11 myserver kernel: [313742.290603] [<ffffffff8115937a>] alloc_pages_vma+0x9a/0x150 May 19 06:00:11 myserver kernel: [313742.290611] [<ffffffff811399cc>] do_anonymous_page.isra.38+0x7c/0x2f0 May 19 06:00:11 myserver kernel: [313742.290618] [<ffffffff8113d3f1>] handle_pte_fault+0x1e1/0x200 May 19 06:00:11 myserver kernel: [313742.290625] [<ffffffff8113d7c8>] handle_mm_fault+0x1f8/0x350 May 19 06:00:11 myserver kernel: [313742.290634] [<ffffffff8165d3e0>] do_page_fault+0x150/0x520 May 19 06:00:11 myserver kernel: [313742.290642] [<ffffffff81177d1d>] ? vfs_read+0x10d/0x180 May 19 06:00:11 myserver kernel: [313742.290649] [<ffffffff8165a035>] page_fault+0x25/0x30 May 19 06:00:11 myserver kernel: [313742.290653] Mem-Info: May 19 06:00:11 myserver kernel: [313742.290657] Node 0 DMA per-cpu: May 19 06:00:11 myserver kernel: [313742.290663] CPU 0: hi: 0, btch: 1 usd: 0 May 19 06:00:11 myserver kernel: [313742.290666] Node 0 DMA32 per-cpu: May 19 06:00:11 myserver kernel: [313742.290672] CPU 0: hi: 186, btch: 31 usd: 124 May 19 06:00:11 myserver kernel: [313742.290682] active_anon:73974 inactive_anon:73976 isolated_anon:0 May 19 06:00:11 myserver kernel: [313742.290684] active_file:305 inactive_file:3393 isolated_file:0 May 19 06:00:11 myserver kernel: [313742.290687] unevictable:0 dirty:11 writeback:4 unstable:0 May 19 06:00:11 myserver kernel: [313742.290689] free:12251 slab_reclaimable:2341 slab_unreclaimable:19263 May 19 06:00:11 myserver kernel: [313742.290692] mapped:1006 shmem:37 pagetables:59627 bounce:0 May 19 06:00:11 myserver kernel: [313742.290697] Node 0 DMA free:4652kB min:684kB low:852kB high:1024kB active_anon:4380kB inactive_anon:4380kB active_file:0kB inactive_file:36kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15656kB mlocked:0kB dirty:0kB writeback:8kB mapped:0kB shmem:0kB slab_reclaimable:200kB slab_unreclaimable:212kB kernel_stack:0kB pagetables:2024kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? yes May 19 06:00:11 myserver kernel: [313742.290720] lowmem_reserve[]: 0 991 991 991 May 19 06:00:11 myserver kernel: [313742.290728] Node 0 DMA32 free:44352kB min:44368kB low:55460kB high:66552kB active_anon:291516kB inactive_anon:291524kB active_file:1220kB inactive_file:13536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1014992kB mlocked:0kB dirty:44kB writeback:8kB mapped:4024kB shmem:148kB slab_reclaimable:9164kB slab_unreclaimable:76840kB kernel_stack:5112kB pagetables:236484kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5925 all_unreclaimable? yes May 19 06:00:11 myserver kernel: [313742.290752] lowmem_reserve[]: 0 0 0 0 May 19 06:00:11 myserver kernel: [313742.290759] Node 0 DMA: 11*4kB 18*8kB 39*16kB 46*32kB 3*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4652kB May 19 06:00:11 myserver kernel: [313742.290778] Node 0 DMA32: 54*4kB 119*8kB 121*16kB 79*32kB 49*64kB 26*128kB 12*256kB 7*512kB 3*1024kB 1*2048kB 5*4096kB = 44352kB May 19 06:00:11 myserver kernel: [313742.290797] 10895 total pagecache pages May 19 06:00:11 myserver kernel: [313742.290801] 7149 pages in swap cache May 19 06:00:11 myserver kernel: [313742.290805] Swap cache stats: add 1460822, delete 1453673, find 653694/726620 May 19 06:00:11 myserver kernel: [313742.290809] Free swap = 0kB May 19 06:00:11 myserver kernel: [313742.290812] Total swap = 2097084kB May 19 06:00:11 myserver kernel: [313742.299856] 261856 pages RAM May 19 06:00:11 myserver kernel: [313742.299860] 7335 pages reserved May 19 06:00:11 myserver kernel: [313742.299863] 291314 pages shared May 19 06:00:11 myserver kernel: [313742.299866] 239474 pages non-shared 

这个问题似乎是由一个糟糕的apache服务器引起的。 你不应该让apache资源的增长超过你的内存或CPU。

这个引用真的很有趣,可能值得一看: http : //drupal.org/node/215516

我的一个vSservers有一个完全相同的问题。

很难确定由内存运行引起的崩溃的确切原因,但时间指向了crontab.weekly。

在从不同的服务器上学习crontab.weekly的内容后,我发现问题服务器有一个脚本,工作服务器没有:

的apt-Xapian的指数

看起来“Debian软件包的Xapian索引维护工具”在小型服务器和已经使用的计算机上引起了很多问题。 做了一些Googlesearch后,我决定从crontab.weekly中删除脚本,现在问题似乎已经消失。

我build议你试图从crontab.weekly中删除这个和任何其他重量级的脚本,看看是否有助于你的问题:)

问题是Apache的设置包含MinSpareServers = 500 ,导致服务器在单个核心testing服务器上一直加载>> 10。