即使服务器仍处于活动状态,nginx proxy_pass最终会返回GATEWAY_TIMEOUT

我已经设置了一个nginx代理直通function,当nginx初始加载时工作正常,但最终(也许在一天左右?,不太确定在这一点上),代理直通的位置只返回一个504 GATEWAY_TIMEOUT响应。 奇怪的是,我已经设置了多个proxy_pass位置,并且随机发生什么事情,并保持什么状态。

这里是nginxconfiguration文件:

 server { listen 80; server_name example.com; access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; root /srv/www/frontend/current/; location / { try_files $uri $uri/ /index.html; } # Block all svn access if ($request_uri ~* ^.*\.svn.*$) { return 404; } # Block all git access if ($request_uri ~* ^.*\.git.*$) { return 404; } location /nginx_status { stub_status on; access_log off; allow 127.0.0.1; deny all; } location /api/karma/ { proxy_pass <server A>; } location /api/auth/ { proxy_pass <server B>; } location /api/search/ { proxy_pass <server C>; } location /api/history/ { proxy_pass <server D>; } } 

(我显然匿名服务器的位置)

例如,当nginx最初启动时,我可以做到

curl -i example.com/api/history/health_check

(其中health_check是一个已知的终点)并得到一个200响应,但过了一段时间,相同的确切命令返回504 GATEWAY_TIMEOUT。

底层的服务器和http服务器仍然是完全可以运行的,我可以直接对服务器的地址进行curl命令,并从我的本地机器上获取响应,并且当连接到nginx服务器。

开始返回GATEWAY_TIMEOUT的位置前缀是随机的,因为已经设置了proxy_pass在这个或那个方面失败的四个位置中的每一个。

所有东西都托pipe在AWS t2_micro上Ubuntu 14.04 Linux实例和nginx本身是v1.4.6。 底层的服务器/应用程序没有被重启,而且AWS实例本身也一直处于启动状态。 底层http服务器是Go应用程序,但是当我开始获得这些GATEWAY_TIMEOUT时,应用程序本身也是完全可操作的。

错误日志中没有什么有趣的事情发生时,但是我刚刚打开了错误日志上的debugging日志logging,下面是一个破坏的请求:

 2015/07/13 19:56:31 [debug] 3523#0: *12096 post event 00000000022542A0 2015/07/13 19:56:31 [debug] 3523#0: *12096 delete posted event 00000000022542A0 2015/07/13 19:56:31 [debug] 3523#0: *12096 http wait request handler 2015/07/13 19:56:31 [debug] 3523#0: *12096 posix_memalign: 000000000222F6A0:256 @16 2015/07/13 19:56:31 [debug] 3523#0: *12096 malloc: 0000000002225CA0:1024 2015/07/13 19:56:31 [debug] 3523#0: *12096 recv: fd:23 1018 of 1024 2015/07/13 19:56:31 [debug] 3523#0: *12096 reusable connection: 0 2015/07/13 19:56:31 [debug] 3523#0: *12096 posix_memalign: 00000000022260B0:4096 @16 2015/07/13 19:56:31 [debug] 3523#0: *12096 http process request line 2015/07/13 19:56:31 [debug] 3523#0: *12096 http request line: "POST <url> HTTP/1.1" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http uri: "<url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http args: "" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http exten: "" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http process request header line 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "host: <url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "Accept: application/json, text/plain, */*" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "Accept-Encoding: gzip, deflate" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "Accept-Language: en-US,en;q=0.8" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "Authorization: <auth token>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "Content-Type: application/json;charset=UTF-8" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "Origin: <url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "Referer: <url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.132 Safari/537.36" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "X-Forwarded-For: <ip address>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "X-Forwarded-Port: 443" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "X-Forwarded-Proto: https" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "Content-Length: 209" 2015/07/13 19:56:31 [debug] 3523#0: *12096 posix_memalign: 00000000022270C0:4096 @16 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header: "Connection: keep-alive" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http header done 2015/07/13 19:56:31 [debug] 3523#0: *12096 event timer del: 23: 1436817428740 2015/07/13 19:56:31 [debug] 3523#0: *12096 generic phase: 0 2015/07/13 19:56:31 [debug] 3523#0: *12096 rewrite phase: 1 ** snip ( bunch of logs about trying to match the url to a route ) 2015/07/13 19:56:31 [debug] 3523#0: *12096 http cl:209 max:4194304 2015/07/13 19:56:31 [debug] 3523#0: *12096 rewrite phase: 3 2015/07/13 19:56:31 [debug] 3523#0: *12096 post rewrite phase: 4 2015/07/13 19:56:31 [debug] 3523#0: *12096 generic phase: 5 2015/07/13 19:56:31 [debug] 3523#0: *12096 generic phase: 6 2015/07/13 19:56:31 [debug] 3523#0: *12096 generic phase: 7 2015/07/13 19:56:31 [debug] 3523#0: *12096 access phase: 8 2015/07/13 19:56:31 [debug] 3523#0: *12096 access phase: 9 2015/07/13 19:56:31 [debug] 3523#0: *12096 post access phase: 10 2015/07/13 19:56:31 [debug] 3523#0: *12096 try files phase: 11 2015/07/13 19:56:31 [debug] 3523#0: *12096 http client request body preread 209 2015/07/13 19:56:31 [debug] 3523#0: *12096 http request body content length filter 2015/07/13 19:56:31 [debug] 3523#0: *12096 http body new buf t:1 f:0 0000000002225FC9, pos 0000000002225FC9, size: 209 file: 0, size: 0 2015/07/13 19:56:31 [debug] 3523#0: *12096 http init upstream, client timer: 0 2015/07/13 19:56:31 [debug] 3523#0: *12096 epoll add event: fd:23 op:3 ev:80000005 2015/07/13 19:56:31 [debug] 3523#0: *12096 http script copy: "Host: " 2015/07/13 19:56:31 [debug] 3523#0: *12096 http script var: "<aws url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http script copy: " " 2015/07/13 19:56:31 [debug] 3523#0: *12096 http script copy: "Connection: close " 2015/07/13 19:56:31 [debug] 3523#0: *12096 http script copy: "Content-Length: " 2015/07/13 19:56:31 [debug] 3523#0: *12096 http script var: "209" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http script copy: " " 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Accept: application/json, text/plain, */*" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Accept-Encoding: gzip, deflate" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Accept-Language: en-US,en;q=0.8" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Authorization: <auth>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Content-Type: application/json;charset=UTF-8" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Origin: <url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Referer: <url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.132 Safari/537.36" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "X-Forwarded-For: 96.45.202.230" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "X-Forwarded-Port: 443" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "X-Forwarded-Proto: https" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "POST /v1/trip HTTP/1.0 Host: <url> Connection: close Content-Length: 209 Accept: application/json, text/plain, */* Accept-Encoding: gzip, deflate Accept-Language: en-US,en;q=0.8 Authorization: <auth> Content-Type: application/json;charset=UTF-8 Origin: <url> Referer: <url> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.132 Safari/537.36 X-Forwarded-For: 96.45.202.230 X-Forwarded-Port: 443 X-Forwarded-Proto: https " 2015/07/13 19:56:31 [debug] 3523#0: *12096 http cleanup add: 0000000002227C40 2015/07/13 19:56:31 [debug] 3523#0: *12096 get rr peer, try: 2 2015/07/13 19:56:31 [debug] 3523#0: *12096 get rr peer, current: 1 0 2015/07/13 19:56:31 [debug] 3523#0: *12096 socket 24 2015/07/13 19:56:31 [debug] 3523#0: *12096 epoll add connection: fd:24 ev:80000005 2015/07/13 19:56:31 [debug] 3523#0: *12096 connect to 54.174.36.245:80, fd:24 #12097 2015/07/13 19:56:31 [debug] 3523#0: *12096 http upstream connect: -2 2015/07/13 19:56:31 [debug] 3523#0: *12096 posix_memalign: 000000000222F7B0:128 @16 2015/07/13 19:56:31 [debug] 3523#0: *12096 event timer add: 24: 60000:1436817451902 2015/07/13 19:56:31 [debug] 3523#0: *12096 http finalize request: -4, "<url>" a:1, c:2 2015/07/13 19:56:31 [debug] 3523#0: *12096 http request count:2 blk:0 2015/07/13 19:56:31 [debug] 3523#0: *12096 post event 000000000226E2B0 2015/07/13 19:56:31 [debug] 3523#0: *12096 delete posted event 000000000226E2B0 2015/07/13 19:56:31 [debug] 3523#0: *12096 http run request: "<url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http upstream check client, write event:1, "<url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http upstream recv(): -1 (11: Resource temporarily unavailable) 2015/07/13 19:56:31 [debug] 3523#0: post event 0000000002253E90 2015/07/13 19:56:31 [debug] 3523#0: delete posted event 0000000002253E90 2015/07/13 19:56:31 [debug] 3523#0: accept on 0.0.0.0:80, ready: 0 2015/07/13 19:56:31 [debug] 3523#0: posix_memalign: 000000000222F840:256 @16 2015/07/13 19:56:31 [debug] 3523#0: *12098 accept: 172.31.15.160 fd:25 2015/07/13 19:56:31 [debug] 3523#0: *12098 event timer add: 25: 60000:1436817451902 2015/07/13 19:56:31 [debug] 3523#0: *12098 reusable connection: 1 2015/07/13 19:56:31 [debug] 3523#0: *12098 epoll add event: fd:25 op:1 ev:80000001 2015/07/13 19:56:31 [debug] 3523#0: post event 0000000002253E90 2015/07/13 19:56:31 [debug] 3523#0: delete posted event 0000000002253E90 2015/07/13 19:56:31 [debug] 3523#0: accept on 0.0.0.0:80, ready: 0 2015/07/13 19:56:31 [debug] 3523#0: posix_memalign: 000000000222F950:256 @16 2015/07/13 19:56:31 [debug] 3523#0: *12099 accept: 172.31.15.160 fd:26 2015/07/13 19:56:31 [debug] 3523#0: *12099 event timer add: 26: 60000:1436817451902 2015/07/13 19:56:31 [debug] 3523#0: *12099 reusable connection: 1 2015/07/13 19:56:31 [debug] 3523#0: *12099 epoll add event: fd:26 op:1 ev:80000001 2015/07/13 19:56:31 [debug] 3523#0: *12096 post event 000000000226E040 2015/07/13 19:56:31 [debug] 3523#0: *12096 delete posted event 000000000226E040 2015/07/13 19:56:31 [debug] 3523#0: *12096 http upstream request: "<url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http upstream send request handler 2015/07/13 19:56:31 [debug] 3523#0: *12096 http upstream send request 2015/07/13 19:56:31 [debug] 3523#0: *12096 chain writer buf fl:0 s:830 2015/07/13 19:56:31 [debug] 3523#0: *12096 chain writer buf fl:1 s:209 2015/07/13 19:56:31 [debug] 3523#0: *12096 chain writer in: 0000000002227C98 2015/07/13 19:56:31 [debug] 3523#0: *12096 writev: 1039 2015/07/13 19:56:31 [debug] 3523#0: *12096 chain writer out: 0000000000000000 2015/07/13 19:56:31 [debug] 3523#0: *12096 event timer del: 24: 1436817451902 2015/07/13 19:56:31 [debug] 3523#0: *12096 event timer add: 24: 60000:1436817451904 2015/07/13 19:56:31 [debug] 3523#0: *12096 post event 0000000002254030 2015/07/13 19:56:31 [debug] 3523#0: *12096 post event 000000000226E040 2015/07/13 19:56:31 [debug] 3523#0: *12096 delete posted event 000000000226E040 2015/07/13 19:56:31 [debug] 3523#0: *12096 http upstream request: "<url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http upstream dummy handler 2015/07/13 19:56:31 [debug] 3523#0: *12096 delete posted event 0000000002254030 2015/07/13 19:56:31 [debug] 3523#0: *12096 http upstream request: "<url>" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http upstream process header 2015/07/13 19:56:31 [debug] 3523#0: *12096 malloc: 00000000022280D0:4096 2015/07/13 19:56:31 [debug] 3523#0: *12096 recv: fd:24 141 of 4096 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy status 200 "200 OK" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Content-Type: text/plain" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Date: Mon, 13 Jul 2015 19:56:31 GMT" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Server: nginx/1.6.2" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Content-Length: 0" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header: "Connection: Close" 2015/07/13 19:56:31 [debug] 3523#0: *12096 http proxy header done 2015/07/13 19:56:31 [debug] 3523#0: *12096 xslt filter header 2015/07/13 19:56:31 [debug] 3523#0: *12096 HTTP/1.1 200 OK Server: nginx/1.4.6 (Ubuntu) Date: Mon, 13 Jul 2015 19:56:31 GMT Content-Type: text/plain Content-Length: 0 Connection: keep-alive 

有趣的线(对我来说)是:

 2015/07/13 19:56:31 [debug] 3523#0: *12096 http upstream recv(): -1 (11: Resource temporarily unavailable) 

我尝试在谷歌查找string,但说实话,我不明白的答案,他们似乎没有关系到我遇到的问题。

瓶颈可能在你的t2_micro数量的文件描述符中,但也很可能是networking带宽问题。

有两种通用的方法可以在不增加带宽的情况下解决这个问题:1)增加系统打开文件的限制(数量)和/或nginx的ulimit 2)升级到nginx 1.7.5或更高版本

如果增加文件描述符不能解决问题,请记住t2_micro通常是低networking带宽产品。 尽pipe您可能无法获得更好的networking性能,但您可以尝试使用一些“更加准确”的代理响应health_check,以最大限度地减lesst2_micro的弱点:

  • 默认情况下,缓冲处于打开状态,因此请确认proxy_temp_path指定的缺省pathproxy_temp_path存在,并且是否有足够的磁盘空间,因为如果不是,缓冲的消息可能偶尔为空或未命中,并创build一个等待响应的超时条件。
  • 您需要1.7.5或更高版本才能调整proxy_next_upstream_timeoutproxy_next_upstream_tries ,让nginx能够在低带宽服务器上更“智能化”地处理代理请求。 例如,您可以在传递错误之前定义一些重试次数。

更新:要configuration版本1.7.5及更高版本中出现错误之前的代理尝试次数,请将此条目添加到nginx.conf中:

proxy_next_upstream_tries 3;

这会给你两次重试 (一次尝试,再加两次重试)。

将此添加到nginx.conf:

 proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; send_timeout 600; 

然后执行'service nginx reload'(如果你使用的是systemd,则使用'systemctl nginx reload')