在`Nginx < – > PHP < – > Postgres`的基础结构中检测瓶颈

我们已经build立了一个基础设施作为我们正在build设的移动应用程序的后端,包括以下几个部分:

  1. 一台NGINX服务器作为负载均衡器(512MB RAM – 1个核心CPU)
  2. 2个PHP-NGINX服务器主机脚本(2GB内存 – 2个核心CPI,8GB内存 – 4个核心CPU)
  3. 1个PostgreSql服务器作为主数据库(8GB RAM – 4个核心CPU)

现在我正试图在压力下testing这个环境的性能。 我正在使用loader.io来达到这个目的。

初步testing(每分钟1000个客户端)表明我有近40%的错误率。 现在我需要检测这个env的哪个部分导致了这么多失败?

我已经检查了PHP,NGINX和Postgres的日志,但不能拿出任何有用的信息。 这是池configuration文件:

[arioo] user = arioo group = arioo listen = /run/php/php7.0-fpm-arioo.sock listen.owner = www-data listen.group = www-data listen.mode = 0666 pm = dynamic pm.max_children = 10 pm.start_servers = 10 pm.min_spare_servers = 5 pm.max_spare_servers = 10 pm.max_requests = 1000 pm.status_path = /$pool-fpm 

对于Postgres

 data_directory = '/var/lib/postgresql/9.5/main' # use data in another directory hba_file = '/etc/postgresql/9.5/main/pg_hba.conf' # host-based authentication file ident_file = '/etc/postgresql/9.5/main/pg_ident.conf' # ident configuration file external_pid_file = '/var/run/postgresql/9.5-main.pid' # write an extra PID file listen_addresses = 'localhost, 10.136.22.22, 198.199.86.83' # what IP address(es) to listen on; port = 5432 # (change requires restart) max_connections = 10000 # (change requires restart) unix_socket_directories = '/var/run/postgresql' # comma-separated list of directories ssl = true # (change requires restart) ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem' # (change requires restart) ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key' # (change requires restart) shared_buffers = 4096MB # min 128kB temp_buffers = 128MB # min 800kB work_mem = 64MB # min 64kB dynamic_shared_memory_type = posix # the default is the first option effective_cache_size = 4GB log_line_prefix = '%t [%p-%l] %q%u@%d ' # special values: log_timezone = 'UTC' track_activities = on track_counts = on track_io_timing = on track_activity_query_size = 1024 # (change requires restart) stats_temp_directory = '/var/run/postgresql/9.5-main.pg_stat_tmp' statement_timeout = 20000 # in milliseconds, 0 is disable datestyle = 'iso, mdy' timezone = 'UTC' lc_messages = 'en_US.UTF-8' # locale for system error message lc_monetary = 'en_US.UTF-8' # locale for monetary formatting lc_numeric = 'en_US.UTF-8' # locale for number formatting lc_time = 'en_US.UTF-8' # locale for time formatting default_text_search_config = 'pg_catalog.english' 

这里是日志的一部分:

 [28-Jun-2017 10:02:53.082868] NOTICE: pid 32494, fpm_children_make(), line 421: [pool arioo] child 18283 started [28-Jun-2017 10:02:53.085758] NOTICE: pid 32494, fpm_children_bury(), line 252: [pool arioo] child 18281 exited with code 0 after 0.012442 seconds from start [28-Jun-2017 10:02:53.086183] NOTICE: pid 32494, fpm_children_make(), line 421: [pool arioo] child 18284 started [28-Jun-2017 10:02:53.089738] NOTICE: pid 32494, fpm_children_bury(), line 252: [pool arioo] child 18282 exited with code 0 after 0.015568 seconds from start [28-Jun-2017 10:02:53.090257] NOTICE: pid 32494, fpm_children_make(), line 421: [pool arioo] child 18285 started [28-Jun-2017 10:02:53.096626] NOTICE: pid 32494, fpm_children_bury(), line 252: [pool arioo] child 18285 exited with code 0 after 0.006375 seconds from start [28-Jun-2017 10:02:53.097161] NOTICE: pid 32494, fpm_children_make(), line 421: [pool arioo] child 18286 started [28-Jun-2017 10:02:53.103822] NOTICE: pid 32494, fpm_children_bury(), line 252: [pool arioo] child 18286 exited with code 0 after 0.006668 seconds from start [28-Jun-2017 10:02:53.104293] NOTICE: pid 32494, fpm_children_make(), line 421: [pool arioo] child 18287 started [28-Jun-2017 10:02:53.105056] NOTICE: pid 32494, fpm_children_bury(), line 252: [pool arioo] child 18284 exited with code 0 after 0.018883 seconds from start [28-Jun-2017 10:02:53.105515] NOTICE: pid 32494, fpm_children_make(), line 421: [pool arioo] child 18288 started [28-Jun-2017 10:02:53.110855] NOTICE: pid 32494, fpm_children_bury(), line 252: [pool arioo] child 18283 exited with code 0 after 0.027996 seconds from start [28-Jun-2017 10:02:53.111360] NOTICE: pid 32494, fpm_children_make(), line 421: [pool arioo] child 18289 started [28-Jun-2017 10:02:53.111382] NOTICE: pid 32494, fpm_children_bury(), line 252: [pool arioo] child 18287 exited with code 0 after 0.007095 seconds from start [28-Jun-2017 10:02:53.111827] NOTICE: pid 32494, fpm_children_make(), line 421: [pool arioo] child 18290 started [28-Jun-2017 10:02:53.121067] NOTICE: pid 32494, fpm_children_bury(), line 252: [pool arioo] child 18288 exited with code 0 after 0.015559 seconds from start [28-Jun-2017 10:02:53.121681] NOTICE: pid 32494, fpm_children_make(), line 421: [pool arioo] child 18291 started [28-Jun-2017 10:02:53.135962] NOTICE: pid 32494, fpm_children_bury(), line 252: [pool arioo] child 18290 exited with code 0 after 0.024143 seconds from start [28-Jun-2017 10:02:53.136721] NOTICE: pid 32494, fpm_children_make(), line 421: [pool arioo] child 18292 started [28-Jun-2017 10:02:53.147137] NOTICE: pid 32494, fpm_children_bury(), line 252: [pool arioo] child 18291 exited with code 0 after 0.025461 seconds from start 

和postgres日志:

 2017-06-28 10:02:59 UTC [23399-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:02:59 UTC [23399-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:03:08 UTC [23404-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:03:08 UTC [23404-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:09:43 UTC [23712-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:09:43 UTC [23712-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:09:55 UTC [23722-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:09:55 UTC [23722-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:09:57 UTC [23727-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:09:57 UTC [23727-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 root@arioo2-db:~# tail /var/log/postgresql/postgresql-9.5-main.log -n 50 2017-06-28 09:49:23 UTC [22112-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 09:49:23 UTC [22112-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 09:49:33 UTC [22120-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 09:49:33 UTC [22120-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 09:49:35 UTC [22122-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 09:49:35 UTC [22122-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 09:49:38 UTC [22127-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 09:49:38 UTC [22127-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 09:49:40 UTC [22129-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 09:49:40 UTC [22129-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 09:49:41 UTC [22131-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 09:49:41 UTC [22131-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 09:49:41 UTC [22132-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 09:49:41 UTC [22132-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 09:49:42 UTC [22133-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 09:49:42 UTC [22133-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 09:49:46 UTC [7002-2] LOG: received fast shutdown request 2017-06-28 09:49:46 UTC [7002-3] LOG: aborting any active transactions 2017-06-28 09:49:46 UTC [22134-1] neshan-dev@neshan-dev FATAL: terminating connection due to administrator command 2017-06-28 09:49:46 UTC [22128-1] neshan-dev@neshan-dev FATAL: terminating connection due to administrator command 2017-06-28 09:49:46 UTC [22130-1] neshan-dev@neshan-dev FATAL: terminating connection due to administrator command 2017-06-28 09:49:46 UTC [22103-1] neshan-dev@neshan-dev FATAL: terminating connection due to administrator command 2017-06-28 09:49:46 UTC [7009-2] LOG: autovacuum launcher shutting down 2017-06-28 09:49:46 UTC [22141-1] neshan-dev@neshan-dev FATAL: terminating connection due to administrator command 2017-06-28 09:49:46 UTC [7006-1] LOG: shutting down 2017-06-28 09:49:46 UTC [7006-2] LOG: database system is shut down 2017-06-28 09:49:48 UTC [22169-1] LOG: database system was shut down at 2017-06-28 09:49:46 UTC 2017-06-28 09:49:48 UTC [22171-1] [unknown]@[unknown] LOG: incomplete startup packet 2017-06-28 09:49:48 UTC [22170-1] neshan-dev@neshan-dev FATAL: the database system is starting up 2017-06-28 09:49:48 UTC [22169-2] LOG: MultiXact member wraparound protections are now enabled 2017-06-28 09:49:48 UTC [22175-1] LOG: autovacuum launcher started 2017-06-28 09:49:48 UTC [22168-1] LOG: database system is ready to accept connections 2017-06-28 10:01:04 UTC [23322-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:01:04 UTC [23322-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:01:58 UTC [23362-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:01:58 UTC [23362-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:02:48 UTC [23390-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:02:48 UTC [23390-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:02:57 UTC [23398-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:02:57 UTC [23398-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:02:59 UTC [23399-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:02:59 UTC [23399-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:03:08 UTC [23404-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:03:08 UTC [23404-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:09:43 UTC [23712-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:09:43 UTC [23712-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:09:55 UTC [23722-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:09:55 UTC [23722-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5 2017-06-28 10:09:57 UTC [23727-1] postgres@arioo ERROR: column "birthDate" of relation "profiles" does not exist at character 55 2017-06-28 10:09:57 UTC [23727-2] postgres@arioo STATEMENT: update "profiles" set "nickname" = $1, "gender" = $2, "birthDate" = $3, "updated_at" = $4 where "id" = $5