NGINX内部负载平衡+ PHP-FPM上游导致随机双重请求/提交

我们遇到了一个非常严重的问题,在看似随机的时候,我们得到了一个由我们的应用程序处理的重复请求。 通常情况下,用户提交表单,偶尔会将其内容保存两次。

我们已经消除了这个问题被JS驱动的双重提交的可能性。 我们有一台networking分析仪,certificate只有一个请求正在进行。 但是,我们也已经certificate,PHP应用程序当然是在整个执行两次。 彻底调查后,应用程序中没有任何逻辑问题会导致这种双重保存行为。

编辑:我们删除了“Keepalive 8” 来自NGINX conf,我们不再得到双重意见。 相反,在有问题的请求中我们得到了一个504

我将不胜感激任何人谁可以看看下面,让我知道,如果有什么突出的 – 谢谢!

我们的NGINX和PHP-FPMconfiguration如下:

/etc/nginx/nginx.conf

user nginx; worker_processes 1; worker_rlimit_nofile 10240; error_log /var/log/nginx/error.log; pid /var/run/nginx.pid; events { worker_connections 1024; multi_accept on; use epoll; } http { server_tokens off; add_header 'Access-Control-Allow-Origin' http://$host; add_header 'Access-Control-Allow-Methods' 'GET, POST'; add_header 'X-Powered-By' 'smartCMS'; upstream php_fpm { least_conn; server 127.0.0.1:9000 max_fails=3 fail_timeout=15s; keepalive 8; } include /etc/nginx/mime.types; default_type application/octet-stream; log_not_found off; access_log /var/log/nginx/access.log combined buffer=16k; open_file_cache max=200000 inactive=20s; open_file_cache_valid 30s; open_file_cache_min_uses 2; open_file_cache_errors on; sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_requests 200; keepalive_timeout 65; gzip on; gzip_static on; gzip_http_version 1.0; gzip_comp_level 6; gzip_proxied any; gzip_types application/javascript application/x-javascript application/xhtml+xml application/xml application/xml+rss image/svg+xml text/css text/javascript text/plain text/xml; gzip_vary on; gzip_disable "MSIE [1-6].(?!.*SV1)"; client_max_body_size 12m; client_body_buffer_size 128k; client_body_timeout 60; client_header_timeout 10; large_client_header_buffers 4 16k; send_timeout 60; server_names_hash_bucket_size 64; include /etc/nginx/conf.d/*.conf; include /etc/nginx/sites-enabled/*; } 

/etc/php-fpm.conf

 ;;;;;;;;;;;;;;;;;;;;; ; FPM Configuration ; ;;;;;;;;;;;;;;;;;;;;; ; All relative paths in this configuration file are relative to PHP's install ; prefix. ; Include one or more files. If glob(3) exists, it is used to include a bunch of ; files from a glob(3) pattern. This directive can be used everywhere in the ; file. include=/etc/php-fpm.d/pools/*.conf ;;;;;;;;;;;;;;;;;; ; PHP INI ; ;;;;;;;;;;;;;;;;;; php_admin_value[upload_max_filesize] = 10M; php_admin_value[post_max_size] = 12M; php_admin_value[max_execution_time] = 60; php_admin_value[expose_php] = Off; ;;;;;;;;;;;;;;;;;; ; Global Options ; ;;;;;;;;;;;;;;;;;; [global] ; Pid file ; Default Value: none pid = /var/run/php-fpm/php-fpm.pid ; Error log file ; Default Value: /var/log/php-fpm.log error_log = /var/log/php-fpm/error.log ; Log level ; Possible Values: alert, error, warning, notice, debug ; Default Value: notice log_level = warning ; If this number of child processes exit with SIGSEGV or SIGBUS within the time ; interval set by emergency_restart_interval then FPM will restart. A value ; of '0' means 'Off'. ; Default Value: 0 emergency_restart_threshold = 1 ; Interval of time used by emergency_restart_interval to determine when ; a graceful restart will be initiated. This can be useful to work around ; accidental corruptions in an accelerator's shared memory. ; Available Units: s(econds), m(inutes), h(ours), or d(ays) ; Default Unit: seconds ; Default Value: 0 emergency_restart_interval = 1m ; Time limit for child processes to wait for a reaction on signals from master. ; Available units: s(econds), m(inutes), h(ours), or d(ays) ; Default Unit: seconds ; Default Value: 0 process_control_timeout = 60s ; Send FPM to background. Set to 'no' to keep FPM in foreground for debugging. ; Default Value: yes daemonize = yes ;;;;;;;;;;;;;;;;;;;; ; Pool Definitions ; ;;;;;;;;;;;;;;;;;;;; ; See /etc/php-fpm.d/pools/*.conf 

/etc/php-fpm.d/pools/www0.conf

 ; Start a new pool named 'www0'. [www0] ; pool_id0php_fpm_service_namephp-fpmtemplatepool.conf.erbnamewwwenabletrue ; The address on which to accept FastCGI requests. ; Valid syntaxes are: ; 'ip.add.re.ss:port' - to listen on a TCP socket to a specific address on ; a specific port; ; 'port' - to listen on a TCP socket to all addresses on a ; specific port; ; '/path/to/unix/socket' - to listen on a unix socket. ; Note: This value is mandatory. listen = 127.0.0.1:9000 ; Set listen(2) backlog. A value of '-1' means unlimited. ; Default Value: -1 listen.backlog = 4096 ; List of ipv4 addresses of FastCGI clients which are allowed to connect. ; Equivalent to the FCGI_WEB_SERVER_ADDRS environment variable in the original ; PHP FCGI (5.2.2+). Makes sense only with a tcp listening socket. Each address ; must be separated by a comma. If this value is left blank, connections will be ; accepted from any ip address. ; Default Value: any listen.allowed_clients = 127.0.0.1 ; Set permissions for unix socket, if one is used. In Linux, read/write ; permissions must be set in order to allow connections from a web server. Many ; BSD-derived systems allow connections regardless of permissions. ; Default Values: user and group are set as the running user ; mode is set to 0666 ;listen.owner = nobody ;listen.group = nobody ;listen.mode = 0666 listen.owner = nginx listen.group = nginx listen.mode = 0660 ; Unix user/group of processes ; Note: The user is mandatory. If the group is not set, the default user's group ; will be used. ; RPM: apache Choosed to be able to access some dir as httpd user = nginx ; RPM: Keep a group allowed to write in log dir. group = nginx ; Choose how the process manager will control the number of child processes. ; Possible Values: ; static - a fixed number (pm.max_children) of child processes; ; dynamic - the number of child processes are set dynamically based on the ; following directives: ; pm.max_children - the maximum number of children that can ; be alive at the same time. ; pm.start_servers - the number of children created on startup. ; pm.min_spare_servers - the minimum number of children in 'idle' ; state (waiting to process). If the number ; of 'idle' processes is less than this ; number then some children will be created. ; pm.max_spare_servers - the maximum number of children in 'idle' ; state (waiting to process). If the number ; of 'idle' processes is greater than this ; number then some children will be killed. ; Note: This value is mandatory. pm = static ; The number of child processes to be created when pm is set to 'static' and the ; maximum number of child processes to be created when pm is set to 'dynamic'. ; This value sets the limit on the number of simultaneous requests that will be ; served. Equivalent to the ApacheMaxClients directive with mpm_prefork. ; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP ; CGI. ; Note: Used when pm is set to either 'static' or 'dynamic' ; Note: This value is mandatory. pm.max_children = 48 ; The number of requests each child process should execute before respawning. ; This can be useful to work around memory leaks in 3rd party libraries. For ; endless request processing specify '0'. Equivalent to PHP_FCGI_MAX_REQUESTS. ; Default Value: 0 pm.max_requests = 10000 ; The URI to view the FPM status page. If this value is not set, no URI will be ; recognized as a status page. By default, the status page shows the following ; information: ; accepted conn - the number of request accepted by the pool; ; pool - the name of the pool; ; process manager - static or dynamic; ; idle processes - the number of idle processes; ; active processes - the number of active processes; ; total processes - the number of idle + active processes. ; The values of 'idle processes', 'active processes' and 'total processes' are ; updated each second. The value of 'accepted conn' is updated in real time. ; Example output: ; accepted conn: 12073 ; pool: www ; process manager: static ; idle processes: 35 ; active processes: 65 ; total processes: 100 ; By default the status page output is formatted as text/plain. Passing either ; 'html' or 'json' as a query string will return the corresponding output ; syntax. Example: ; http://www.foo.bar/status ; http://www.foo.bar/status?json ; http://www.foo.bar/status?html ; Note: The value must start with a leading slash (/). The value can be ; anything, but it may not be a good idea to use the .php extension or it ; may conflict with a real PHP file. ; Default Value: not set ;pm.status_path = /status ; The ping URI to call the monitoring page of FPM. If this value is not set, no ; URI will be recognized as a ping page. This could be used to test from outside ; that FPM is alive and responding, or to ; - create a graph of FPM availability (rrd or such); ; - remove a server from a group if it is not responding (load balancing); ; - trigger alerts for the operating team (24/7). ; Note: The value must start with a leading slash (/). The value can be ; anything, but it may not be a good idea to use the .php extension or it ; may conflict with a real PHP file. ; Default Value: not set ;ping.path = /ping ; This directive may be used to customize the response of a ping request. The ; response is formatted as text/plain with a 200 response code. ; Default Value: pong ;ping.response = pong ; The timeout for serving a single request after which the worker process will ; be killed. This option should be used when the 'max_execution_time' ini option ; does not stop script execution for some reason. A value of '0' means 'off'. ; Available units: s(econds)(default), m(inutes), h(ours), or d(ays) ; Default Value: 0 request_terminate_timeout = 60s ; The timeout for serving a single request after which a PHP backtrace will be ; dumped to the 'slowlog' file. A value of '0s' means 'off'. ; Available units: s(econds)(default), m(inutes), h(ours), or d(ays) ; Default Value: 0 request_slowlog_timeout = 20s ; The log file for slow requests ; Default Value: not set ; Note: slowlog is mandatory if request_slowlog_timeout is set slowlog = /var/log/php-fpm/www-slow.log ; Set open file descriptor rlimit. ; Default Value: system defined value ;rlimit_files = 1024 ; Set max core size rlimit. ; Possible Values: 'unlimited' or an integer greater or equal to 0 ; Default Value: system defined value ;rlimit_core = 0 ; Chroot to this directory at the start. This value must be defined as an ; absolute path. When this value is not set, chroot is not used. ; Note: chrooting is a great security feature and should be used whenever ; possible. However, all PHP paths will be relative to the chroot ; (error_log, sessions.save_path, ...). ; Default Value: not set ;chroot = ; Chdir to this directory at the start. This value must be an absolute path. ; Default Value: current directory or / when chroot ;chdir = /var/www ; Redirect worker stdout and stderr into main error log. If not set, stdout and ; stderr will be redirected to /dev/null according to FastCGI specs. ; Default Value: no ;catch_workers_output = yes ; Limits the extensions of the main script FPM will allow to parse. This can ; prevent configuration mistakes on the web server side. You should only limit ; FPM to .php extensions to prevent malicious users to use other extensions to ; exectute php code. ; Note: set an empty value to allow all extensions. ; Default Value: .php ;security.limit_extensions = .php .php3 .php4 .php5 ; Pass environment variables like LD_LIBRARY_PATH. All $VARIABLEs are taken from ; the current environment. ; Default Value: clean env ;env[HOSTNAME] = $HOSTNAME ;env[PATH] = /usr/local/bin:/usr/bin:/bin ;env[TMP] = /tmp ;env[TMPDIR] = /tmp ;env[TEMP] = /tmp ; Additional php.ini defines, specific to this pool of workers. These settings ; overwrite the values previously defined in the php.ini. The directives are the ; same as the PHP SAPI: ; php_value/php_flag - you can set classic ini defines which can ; be overwritten from PHP call 'ini_set'. ; php_admin_value/php_admin_flag - these directives won't be overwritten by ; PHP call 'ini_set' ; For php_*flag, valid values are on, off, 1, 0, true, false, yes or no. ; Defining 'extension' will load the corresponding shared extension from ; extension_dir. Defining 'disable_functions' or 'disable_classes' will not ; overwrite previously defined php.ini values, but will append the new value ; instead. ; Default Value: nothing is defined by default except the values in php.ini and ; specified at startup with the -d argument ;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f [email protected] php_flag[display_errors] = off php_admin_value[error_log] = /var/log/php-fpm/www-error.log php_admin_flag[log_errors] = on php_admin_value[memory_limit] = 256M ; Set session path to a directory owned by process user ;php_value[session.save_handler] = files ;php_value[session.save_path] = /var/lib/php/session 

您在nginx中使用least_conn负载平衡策略来pipe理上游PHP-FPM。 这意味着对于位于一个IP地址的一个用户,可能会由不同的PHP-FPM进程提供服务。

如果这两个PHP-FPM进程之间没有共享用户之间的所有必要的状态,那么奇怪的事情就会发生。 例如,如果用户会话状态对于PHP-FPM节点是本地的,那么一旦用户在login时碰到另一个服务器就会被注销。

为了避免这种情况,你需要用ip_hashreplaceip_hash 。 这确保了来自一个IP地址的所有连接都被发送到相同的PHP-FPM节点。 理论上这使得负载平衡更加不均衡,但实际上没有区别。

这可能不是你遇到的问题的原因。

我认为删除Keepalive会向您显示潜在的问题,我怀疑是您configuration的超时的组合,以及后端负载的响应程度。

更具体地说,我认为这是你的问题:

 upstream php_fpm { least_conn; server 127.0.0.1:9000 max_fails=3 fail_timeout=15s; keepalive 8; } 

我会尝试以下内容:

 upstream php_fpm { least_conn; server 127.0.0.1:9000 max_fails=3 fail_timeout=60s; keepalive 8; } 

我认为发生的事情是PHP-FPM设置为在60秒后终止处理,但nginx认为请求在15秒后失败。

https://nginx.org/en/docs/http/ngx_http_upstream_module.html#server

fail_timeout = time设置指定次数的不成功尝试与服务器进行通信的时间,以此来考虑服务器不可用; 以及服务器将被视为不可用的时间段。 默认情况下,该参数设置为10秒。

也许你也想检查一下你的高峰负载是什么样的,并考虑缩放你的后端来吸收它。