来自AMQP的石墨指标会引发错误,需要很长时间才能重新连接

我正试图通过RabbitMQ交换机将一些指标转化为Graphite。 我已经让我的发布商愉快地将数据发布到称为metrics的交换,并且已经使用以下方法configuration了carbon.conf

 ENABLE_AMQP = True AMQP_HOST = hostname AMQP_PORT = 5672 AMQP_VHOST = /vhost AMQP_USER = user AMQP_PASSWORD = password AMQP_EXCHANGE = metrics AMQP_METRIC_NAME_IN_BODY = True 

rMQ安装是haproxy后面的双节点群集。

当这个工作,它的作品很好。 但是,碳往往会引发以下问题:

 02/05/2013 15:13:14 :: [console] Unhandled error in Deferred: 02/05/2013 15:13:14 :: [console] Unhandled Error Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 421, in errback self._startRunCallbacks(fail) File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 488, in _startRunCallbacks self._runCallbacks() File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 575, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1126, in gotResult _inlineCallbacks(r, g, deferred) --- <exception caught here> --- File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1068, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/opt/graphite/lib/carbon/amqp_listener.py", line 70, in connectionMade yield self.receive_loop() File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1068, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/local/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/opt/graphite/lib/carbon/amqp_listener.py", line 102, in receive_loop msg = yield queue.get() File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 575, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/local/lib/python2.7/dist-packages/txamqp/queue.py", line 32, in _raiseIfClosed raise Closed() txamqp.queue.Closed: 02/05/2013 15:13:14 :: [console] <twisted.internet.tcp.Connector instance at 0x2219f80> will retry in 1976 seconds 02/05/2013 15:13:14 :: [console] Stopping factory <carbon.amqp_listener.AMQPReconnectingFactory instance at 0x2214ab8> 

不知何故,连接被中断了。 更糟糕的是,它想要在半小时内重新连接!

我如何

  1. 找出为什么它断开连接?
  2. 大量减less重新连接时间?

软件:

 txAMQP==0.6.2 graphite 0.9.11 RabbitMQ 3.1.0 Haproxy 1.4.18 

我们今天经历过同样的问题。 我不知道#1,但我相信第二个问题是重新连接延迟从未在amqp_listener.py中重置,并且应该在协议在buildProtocol中生成之前完成。 我在这里提交了一个请求: https : //github.com/graphite-project/carbon/pull/102 。 希望这可以帮助。

更改之前(省略例外):

console.log.2013_5_2:02/05/2013 17:11:14 ::将在2秒内重试console.log.2013_5_2:02/05/2013 17:11:16 ::将在5秒内重试console.log。 2013_5_2:02/05/2013 17:41:18 ::将在12秒内重试console.log.2013_5_2:02/05/2013 18:11:22 ::将在28秒内重试console.log.2013_5_2:02 / 05/2013 18:41:26 ::将在77秒内重试console.log.2013_5_2:02/05/2013 19:11:32 ::将在178秒内重试console.log.2013_5_2:02/05/2013 19 :41:39 ::将在455秒重试console.log.2013_5_2:02/05/2013 20:11:48 ::将在967秒内重试console.log.2013_5_2:02/05/2013 20:42:01 ::将在1831年重试console.log.2013_5_2:02/05/2013 21:22:13 ::将在3375秒重试

更改后(省略例外):

console.log.2013_5_2:02/05/2013 21:42:21 ::将在2秒内重试console.log.2013_5_2:02/05/2013 21:42:24 ::将在9秒内重试console.log。 2013_5_2:02/05/2013 22:12:18 ::将在2秒内重试console.log.2013_5_2:02/05/2013 22:12:21 ::将在9秒内重试console.log.2013_5_2:02 / 05/2013 22:42:32 ::将在2秒后重试console.log.2013_5_2:02/05/2013 22:42:35 ::将在7秒内重试console.log.2013_5_2:02/05/2013 23 :12:29 ::将在2秒内重试console.log.2013_5_2:02/05/2013 23:12:32 ::将在5秒内重试console.log.2013_5_2:02/05/2013 23:42:38 ::将在2秒内重试console.log.2013_5_2:02/05/2013 23:42:41 ::将在6秒内重试