rabbitmq 消費者消費消息異常,無限重試導致日誌過大,磁盤被佔用滿
問題的排查以及解決方式
消費者模塊日誌查看
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener threw exception
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1506) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1417) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1337) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at sun.reflect.GeneratedMethodAccessor446.invoke(Unknown Source) ~[na:na]
2020/1/19 上午3:13:38 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111-internal]
2020/1/19 上午3:13:38 at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111-internal]
2020/1/19 上午3:13:38 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) ~[spring-aop-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
2020/1/19 上午3:13:38 at brave.spring.rabbit.TracingRabbitListenerAdvice.invoke(TracingRabbitListenerAdvice.java:84) ~[brave-instrumentation-spring-rabbit-5.1.0.jar!/:na]
2020/1/19 上午3:13:38 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.$Proxy220.invokeListener(Unknown Source) ~[na:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1324) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1303) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:817) [spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:801) [spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:77) [spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1042) [spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111-internal]
2020/1/19 上午3:13:38Caused by: org.springframework.amqp.AmqpException: No method found for class java.lang.String
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.adapter.DelegatingInvocableHandler.getHandlerForPayload(DelegatingInvocableHandler.java:147) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.adapter.DelegatingInvocableHandler.getMethodNameFor(DelegatingInvocableHandler.java:250) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.getMethodAsString(HandlerAdapter.java:70) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:190) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38 at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:120) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
針對上述日誌中產生的問題的解決方案
- 在消費者的類上面刪除掉註解
@RabbitListener(queues = OfficeQueue.OFFICE_QUEUE)
- 在消費者的方法上面加上註解
@RabbitListener(queues = OfficeQueue.OFFICE_QUEUE)
- 在消費者的方法上面去掉註解
@RabbitHandler
上面的方法處理完後又出現異常(忘記記錄)
- 總的意思是在說,消費者在隊列中獲取這條消息時,生產者那邊生產的 message 是空的;
問題排查
對服務裏面的各個模塊生產消息的地方進行代碼檢查:
- 發現有幾處生產消息的地方,message 發送出去時,內容是空的;
解決方式
- 將所有生產者 message 爲空的地方作出修改,不允許消息爲空還發出消息的情況;
本次異常引申出的一些問題
- 日誌非常非常的大,本來服務器 180G 的磁盤空間,已經被佔滿了 152G
- 由於磁盤空間越來越少,導致其他服務器再發布時,無法發佈成功