我有一个nginx实例,它被设置为logging访问/var/log/nginx/access.log和错误到/var/log/nginx/errors.log,但是一旦logrotate每周运行一次,文件就会移动到* .log.1和新的* .log文件被创build,但nginx继续login到log.1文件而不是新的.log文件(没有任何东西被压缩)。 我第一次注意到这一点,自从轮伐日志变成3周后,日志变得越来越大了。 运行kill -HUP `cat /run/nginx.pid`让nginx重新开始logging到正确的地方,但是这个问题在下个星期又开始了。
这是令人沮丧的更重要的原因是我有日志设置为通过rsyslog上传到Loggly,当nginx停止logging到文件我有rsyslog轮询,然后停止上传,我没有得到任何警报。
我怀疑这与重新启动nginx或重新加载configuration有关,因为它没有启动,直到我做了一个configuration更改,并以我认为是正常的方式重新加载configuration。 我试着运行kill -USR1 `cat /run/nginx.pid`但是文件继续logging到错误的位置,直到我运行kill -HUP `cat /run/nginx.pid` ,而我已经知道这并不能解决问题。
任何想法是怎么回事? 我承认我不是logrotate或者nginxpipe理方面的专家,但是我的Google在这个方面失败了。
这里是我的nginx logrotate脚本,让我知道如果还有什么你可能想看到的。 除了定义输出位置之外,nginx.conf在日志方面没有什么特别之处。
/var/log/nginx/*.log { weekly missingok rotate 52 compress delaycompress notifempty create 0640 www-data adm sharedscripts prerotate if [ -d /etc/logrotate.d/httpd-prerotate ]; then \ run-parts /etc/logrotate.d/httpd-prerotate; \ fi \ endscript postrotate [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid` endscript }
编辑:我想我发现了这个问题。 这是在debugging模式下运行logrotate的输出:
$ sudo logrotate --force -d /etc/logrotate.d/nginx reading config file /etc/logrotate.d/nginx Handling 1 logs rotating pattern: /var/log/nginx/*.log forced from command line (52 rotations) empty log files are not rotated, old logs are removed considering log /var/log/nginx/access.log log needs rotating considering log /var/log/nginx/error.log log needs rotating rotating log /var/log/nginx/access.log, log->rotateCount is 52 dateext suffix '-20141023' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' previous log /var/log/nginx/access.log.1 does not exist renaming /var/log/nginx/access.log.52.gz to /var/log/nginx/access.log.53.gz (rotatecount 52, logstart 1, i 52), renaming /var/log/nginx/access.log.51.gz to /var/log/nginx/access.log.52.gz (rotatecount 52, logstart 1, i 51), renaming /var/log/nginx/access.log.50.gz to /var/log/nginx/access.log.51.gz (rotatecount 52, logstart 1, i 50), renaming /var/log/nginx/access.log.49.gz to /var/log/nginx/access.log.50.gz (rotatecount 52, logstart 1, i 49), renaming /var/log/nginx/access.log.48.gz to /var/log/nginx/access.log.49.gz (rotatecount 52, logstart 1, i 48), renaming /var/log/nginx/access.log.47.gz to /var/log/nginx/access.log.48.gz (rotatecount 52, logstart 1, i 47), renaming /var/log/nginx/access.log.46.gz to /var/log/nginx/access.log.47.gz (rotatecount 52, logstart 1, i 46), renaming /var/log/nginx/access.log.45.gz to /var/log/nginx/access.log.46.gz (rotatecount 52, logstart 1, i 45), renaming /var/log/nginx/access.log.44.gz to /var/log/nginx/access.log.45.gz (rotatecount 52, logstart 1, i 44), renaming /var/log/nginx/access.log.43.gz to /var/log/nginx/access.log.44.gz (rotatecount 52, logstart 1, i 43), renaming /var/log/nginx/access.log.42.gz to /var/log/nginx/access.log.43.gz (rotatecount 52, logstart 1, i 42), renaming /var/log/nginx/access.log.41.gz to /var/log/nginx/access.log.42.gz (rotatecount 52, logstart 1, i 41), renaming /var/log/nginx/access.log.40.gz to /var/log/nginx/access.log.41.gz (rotatecount 52, logstart 1, i 40), renaming /var/log/nginx/access.log.39.gz to /var/log/nginx/access.log.40.gz (rotatecount 52, logstart 1, i 39), renaming /var/log/nginx/access.log.38.gz to /var/log/nginx/access.log.39.gz (rotatecount 52, logstart 1, i 38), renaming /var/log/nginx/access.log.37.gz to /var/log/nginx/access.log.38.gz (rotatecount 52, logstart 1, i 37), renaming /var/log/nginx/access.log.36.gz to /var/log/nginx/access.log.37.gz (rotatecount 52, logstart 1, i 36), renaming /var/log/nginx/access.log.35.gz to /var/log/nginx/access.log.36.gz (rotatecount 52, logstart 1, i 35), renaming /var/log/nginx/access.log.34.gz to /var/log/nginx/access.log.35.gz (rotatecount 52, logstart 1, i 34), renaming /var/log/nginx/access.log.33.gz to /var/log/nginx/access.log.34.gz (rotatecount 52, logstart 1, i 33), renaming /var/log/nginx/access.log.32.gz to /var/log/nginx/access.log.33.gz (rotatecount 52, logstart 1, i 32), renaming /var/log/nginx/access.log.31.gz to /var/log/nginx/access.log.32.gz (rotatecount 52, logstart 1, i 31), renaming /var/log/nginx/access.log.30.gz to /var/log/nginx/access.log.31.gz (rotatecount 52, logstart 1, i 30), renaming /var/log/nginx/access.log.29.gz to /var/log/nginx/access.log.30.gz (rotatecount 52, logstart 1, i 29), renaming /var/log/nginx/access.log.28.gz to /var/log/nginx/access.log.29.gz (rotatecount 52, logstart 1, i 28), renaming /var/log/nginx/access.log.27.gz to /var/log/nginx/access.log.28.gz (rotatecount 52, logstart 1, i 27), renaming /var/log/nginx/access.log.26.gz to /var/log/nginx/access.log.27.gz (rotatecount 52, logstart 1, i 26), renaming /var/log/nginx/access.log.25.gz to /var/log/nginx/access.log.26.gz (rotatecount 52, logstart 1, i 25), renaming /var/log/nginx/access.log.24.gz to /var/log/nginx/access.log.25.gz (rotatecount 52, logstart 1, i 24), renaming /var/log/nginx/access.log.23.gz to /var/log/nginx/access.log.24.gz (rotatecount 52, logstart 1, i 23), renaming /var/log/nginx/access.log.22.gz to /var/log/nginx/access.log.23.gz (rotatecount 52, logstart 1, i 22), renaming /var/log/nginx/access.log.21.gz to /var/log/nginx/access.log.22.gz (rotatecount 52, logstart 1, i 21), renaming /var/log/nginx/access.log.20.gz to /var/log/nginx/access.log.21.gz (rotatecount 52, logstart 1, i 20), renaming /var/log/nginx/access.log.19.gz to /var/log/nginx/access.log.20.gz (rotatecount 52, logstart 1, i 19), renaming /var/log/nginx/access.log.18.gz to /var/log/nginx/access.log.19.gz (rotatecount 52, logstart 1, i 18), renaming /var/log/nginx/access.log.17.gz to /var/log/nginx/access.log.18.gz (rotatecount 52, logstart 1, i 17), renaming /var/log/nginx/access.log.16.gz to /var/log/nginx/access.log.17.gz (rotatecount 52, logstart 1, i 16), renaming /var/log/nginx/access.log.15.gz to /var/log/nginx/access.log.16.gz (rotatecount 52, logstart 1, i 15), renaming /var/log/nginx/access.log.14.gz to /var/log/nginx/access.log.15.gz (rotatecount 52, logstart 1, i 14), renaming /var/log/nginx/access.log.13.gz to /var/log/nginx/access.log.14.gz (rotatecount 52, logstart 1, i 13), renaming /var/log/nginx/access.log.12.gz to /var/log/nginx/access.log.13.gz (rotatecount 52, logstart 1, i 12), renaming /var/log/nginx/access.log.11.gz to /var/log/nginx/access.log.12.gz (rotatecount 52, logstart 1, i 11), renaming /var/log/nginx/access.log.10.gz to /var/log/nginx/access.log.11.gz (rotatecount 52, logstart 1, i 10), renaming /var/log/nginx/access.log.9.gz to /var/log/nginx/access.log.10.gz (rotatecount 52, logstart 1, i 9), renaming /var/log/nginx/access.log.8.gz to /var/log/nginx/access.log.9.gz (rotatecount 52, logstart 1, i 8), renaming /var/log/nginx/access.log.7.gz to /var/log/nginx/access.log.8.gz (rotatecount 52, logstart 1, i 7), renaming /var/log/nginx/access.log.6.gz to /var/log/nginx/access.log.7.gz (rotatecount 52, logstart 1, i 6), renaming /var/log/nginx/access.log.5.gz to /var/log/nginx/access.log.6.gz (rotatecount 52, logstart 1, i 5), renaming /var/log/nginx/access.log.4.gz to /var/log/nginx/access.log.5.gz (rotatecount 52, logstart 1, i 4), renaming /var/log/nginx/access.log.3.gz to /var/log/nginx/access.log.4.gz (rotatecount 52, logstart 1, i 3), renaming /var/log/nginx/access.log.2.gz to /var/log/nginx/access.log.3.gz (rotatecount 52, logstart 1, i 2), renaming /var/log/nginx/access.log.1.gz to /var/log/nginx/access.log.2.gz (rotatecount 52, logstart 1, i 1), renaming /var/log/nginx/access.log.0.gz to /var/log/nginx/access.log.1.gz (rotatecount 52, logstart 1, i 0), rotating log /var/log/nginx/error.log, log->rotateCount is 52 dateext suffix '-20141023' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' previous log /var/log/nginx/error.log.1 does not exist renaming /var/log/nginx/error.log.52.gz to /var/log/nginx/error.log.53.gz (rotatecount 52, logstart 1, i 52), renaming /var/log/nginx/error.log.51.gz to /var/log/nginx/error.log.52.gz (rotatecount 52, logstart 1, i 51), renaming /var/log/nginx/error.log.50.gz to /var/log/nginx/error.log.51.gz (rotatecount 52, logstart 1, i 50), renaming /var/log/nginx/error.log.49.gz to /var/log/nginx/error.log.50.gz (rotatecount 52, logstart 1, i 49), renaming /var/log/nginx/error.log.48.gz to /var/log/nginx/error.log.49.gz (rotatecount 52, logstart 1, i 48), renaming /var/log/nginx/error.log.47.gz to /var/log/nginx/error.log.48.gz (rotatecount 52, logstart 1, i 47), renaming /var/log/nginx/error.log.46.gz to /var/log/nginx/error.log.47.gz (rotatecount 52, logstart 1, i 46), renaming /var/log/nginx/error.log.45.gz to /var/log/nginx/error.log.46.gz (rotatecount 52, logstart 1, i 45), renaming /var/log/nginx/error.log.44.gz to /var/log/nginx/error.log.45.gz (rotatecount 52, logstart 1, i 44), renaming /var/log/nginx/error.log.43.gz to /var/log/nginx/error.log.44.gz (rotatecount 52, logstart 1, i 43), renaming /var/log/nginx/error.log.42.gz to /var/log/nginx/error.log.43.gz (rotatecount 52, logstart 1, i 42), renaming /var/log/nginx/error.log.41.gz to /var/log/nginx/error.log.42.gz (rotatecount 52, logstart 1, i 41), renaming /var/log/nginx/error.log.40.gz to /var/log/nginx/error.log.41.gz (rotatecount 52, logstart 1, i 40), renaming /var/log/nginx/error.log.39.gz to /var/log/nginx/error.log.40.gz (rotatecount 52, logstart 1, i 39), renaming /var/log/nginx/error.log.38.gz to /var/log/nginx/error.log.39.gz (rotatecount 52, logstart 1, i 38), renaming /var/log/nginx/error.log.37.gz to /var/log/nginx/error.log.38.gz (rotatecount 52, logstart 1, i 37), renaming /var/log/nginx/error.log.36.gz to /var/log/nginx/error.log.37.gz (rotatecount 52, logstart 1, i 36), renaming /var/log/nginx/error.log.35.gz to /var/log/nginx/error.log.36.gz (rotatecount 52, logstart 1, i 35), renaming /var/log/nginx/error.log.34.gz to /var/log/nginx/error.log.35.gz (rotatecount 52, logstart 1, i 34), renaming /var/log/nginx/error.log.33.gz to /var/log/nginx/error.log.34.gz (rotatecount 52, logstart 1, i 33), renaming /var/log/nginx/error.log.32.gz to /var/log/nginx/error.log.33.gz (rotatecount 52, logstart 1, i 32), renaming /var/log/nginx/error.log.31.gz to /var/log/nginx/error.log.32.gz (rotatecount 52, logstart 1, i 31), renaming /var/log/nginx/error.log.30.gz to /var/log/nginx/error.log.31.gz (rotatecount 52, logstart 1, i 30), renaming /var/log/nginx/error.log.29.gz to /var/log/nginx/error.log.30.gz (rotatecount 52, logstart 1, i 29), renaming /var/log/nginx/error.log.28.gz to /var/log/nginx/error.log.29.gz (rotatecount 52, logstart 1, i 28), renaming /var/log/nginx/error.log.27.gz to /var/log/nginx/error.log.28.gz (rotatecount 52, logstart 1, i 27), renaming /var/log/nginx/error.log.26.gz to /var/log/nginx/error.log.27.gz (rotatecount 52, logstart 1, i 26), renaming /var/log/nginx/error.log.25.gz to /var/log/nginx/error.log.26.gz (rotatecount 52, logstart 1, i 25), renaming /var/log/nginx/error.log.24.gz to /var/log/nginx/error.log.25.gz (rotatecount 52, logstart 1, i 24), renaming /var/log/nginx/error.log.23.gz to /var/log/nginx/error.log.24.gz (rotatecount 52, logstart 1, i 23), renaming /var/log/nginx/error.log.22.gz to /var/log/nginx/error.log.23.gz (rotatecount 52, logstart 1, i 22), renaming /var/log/nginx/error.log.21.gz to /var/log/nginx/error.log.22.gz (rotatecount 52, logstart 1, i 21), renaming /var/log/nginx/error.log.20.gz to /var/log/nginx/error.log.21.gz (rotatecount 52, logstart 1, i 20), renaming /var/log/nginx/error.log.19.gz to /var/log/nginx/error.log.20.gz (rotatecount 52, logstart 1, i 19), renaming /var/log/nginx/error.log.18.gz to /var/log/nginx/error.log.19.gz (rotatecount 52, logstart 1, i 18), renaming /var/log/nginx/error.log.17.gz to /var/log/nginx/error.log.18.gz (rotatecount 52, logstart 1, i 17), renaming /var/log/nginx/error.log.16.gz to /var/log/nginx/error.log.17.gz (rotatecount 52, logstart 1, i 16), renaming /var/log/nginx/error.log.15.gz to /var/log/nginx/error.log.16.gz (rotatecount 52, logstart 1, i 15), renaming /var/log/nginx/error.log.14.gz to /var/log/nginx/error.log.15.gz (rotatecount 52, logstart 1, i 14), renaming /var/log/nginx/error.log.13.gz to /var/log/nginx/error.log.14.gz (rotatecount 52, logstart 1, i 13), renaming /var/log/nginx/error.log.12.gz to /var/log/nginx/error.log.13.gz (rotatecount 52, logstart 1, i 12), renaming /var/log/nginx/error.log.11.gz to /var/log/nginx/error.log.12.gz (rotatecount 52, logstart 1, i 11), renaming /var/log/nginx/error.log.10.gz to /var/log/nginx/error.log.11.gz (rotatecount 52, logstart 1, i 10), renaming /var/log/nginx/error.log.9.gz to /var/log/nginx/error.log.10.gz (rotatecount 52, logstart 1, i 9), renaming /var/log/nginx/error.log.8.gz to /var/log/nginx/error.log.9.gz (rotatecount 52, logstart 1, i 8), renaming /var/log/nginx/error.log.7.gz to /var/log/nginx/error.log.8.gz (rotatecount 52, logstart 1, i 7), renaming /var/log/nginx/error.log.6.gz to /var/log/nginx/error.log.7.gz (rotatecount 52, logstart 1, i 6), renaming /var/log/nginx/error.log.5.gz to /var/log/nginx/error.log.6.gz (rotatecount 52, logstart 1, i 5), renaming /var/log/nginx/error.log.4.gz to /var/log/nginx/error.log.5.gz (rotatecount 52, logstart 1, i 4), renaming /var/log/nginx/error.log.3.gz to /var/log/nginx/error.log.4.gz (rotatecount 52, logstart 1, i 3), renaming /var/log/nginx/error.log.2.gz to /var/log/nginx/error.log.3.gz (rotatecount 52, logstart 1, i 2), renaming /var/log/nginx/error.log.1.gz to /var/log/nginx/error.log.2.gz (rotatecount 52, logstart 1, i 1), renaming /var/log/nginx/error.log.0.gz to /var/log/nginx/error.log.1.gz (rotatecount 52, logstart 1, i 0), running prerotate script running script with arg /var/log/nginx/*.log : " if [ -d /etc/logrotate.d/httpd-prerotate ]; then \ run-parts /etc/logrotate.d/httpd-prerotate; \ fi \ " renaming /var/log/nginx/access.log to /var/log/nginx/access.log.1 creating new /var/log/nginx/access.log mode = 0640 uid = 33 gid = 4 renaming /var/log/nginx/error.log to /var/log/nginx/error.log.1 creating new /var/log/nginx/error.log mode = 0640 uid = 33 gid = 4 running postrotate script running script with arg /var/log/nginx/*.log : " [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid` " removing old log /var/log/nginx/access.log.53.gz error: error opening /var/log/nginx/access.log.53.gz: No such file or directory
但是,只有大约* .log.8.gz的档案,所以logrotate在尝试与/var/log/nginx/access.log.53.gz交互时失败。 为什么要这样做呢? 我想我需要接触假文件来填写它? 不知何故这似乎是错误的。
嗯,经过很长时间的挖掘,我终于find了答案。 在我的情况下,问题不在于logrotate失败。 该错误信息是好的,并不实际上停止logrotate。 问题是nginx在收到kill -USR1信号后没有释放文件句柄到日志文件。 长话短说,没有重新加载日志文件的原因是/var/log/nginx文件夹不是由nginx工作进程(由www-data,在web下运行)拥有。 我不知道如何改变(可能是因为这台服务器最近重拍),但改变文件夹由同一个用户拥有的Nginx工作进程(和固定logrotate文件,使新的日志为networking)解决了这个问题。
我在Ubuntu 14.04有同样的问题,但在阅读你的答案后,我检查了我的nginx运行在万维网数据,谁拥有该文件夹,所以…
但是,我发现这个bug ,指出/etc/logrotate.d/nginx中postrotate部分的错误。
要解决它,你可以注释掉
invoke-rc.d nginx rotate >/dev/null 2>&1
并使用以下任何选项(从该错误中的注释),而不是:
此问题的另一个原因可能是磁盘已满 。
注意: 这可能甚至会在访问日志突然填满时发生! 可能吗?
为了重现这一点,您可能会让您的访问日志突然充满多次请求,一旦logrotate激活,它会将access.log移动到access.log.1 ,但将无法压缩下一个文件,因此无法发送USR1信号给nginx。
你如何解决这个问题?