ActiveMQ问题排查：磁盘使用激增、启动报错[references corrupt locations]

最新推荐文章于 2022-08-06 12:14:37 发布

原创最新推荐文章于 2022-08-06 12:14:37 发布 · 2.9k 阅读

3 ·

本内容遵循CC 4.0 BY-SA版权协议

标签

#activemq #apache

ActiveMQ 专栏收录该内容

1 篇文章

订阅专栏

解决ActiveMQ因清理死信队列导致的启动失败问题，通过修改配置文件启用错误容忍机制。

故障原因：

ActiveMQ服务器监控报警，发现最近磁盘使用率节节攀升，随即开始定位问题发现，是业务系统升级导致一个废弃的queue中消息没有被消费，导致数据一直被写进kahadb中，然后持久化在磁盘上，导致该问题发生。

故障处理：

最优解：业务系统不再使用已废弃的queue；但是升级需要周期，不能立即解决问题。

次优解：手动清除queue中消息，释放磁盘空间。

登陆ActiveMQ管理页面控制台，具体端口见配置文件'activemq_home/conf/jetty.xml'

<bean id="jettyPort" class="org.apache.activemq.web.WebConsolePort" init-method="start">
	<!-- the default port number for the web console -->
	<property name="host" value="0.0.0.0"/>
	<property name="port" value="8161"/>
</bean>

使用IP 端口，打开管理页面。

点击【Manage ActiveMQ broker】进入管理页面，此时会弹出一个登陆窗口，用户名和密码配置在'/activemq_home/conf/credentials.properties'中，得到

activemq.username=admin
activemq.password=admin

选择需要清除的queue，点击Operations 中【Purge】即可清除。

本来至此，本文应该结束的，但是后续又遇到了新的问题，这里记录一下解决方案。

queue中有个一个默认存在的死信队列（Dead Letter Queue），我好奇的点了一下Purge操作，出现了个一个Error页面，然后ActiveMQ就挂了，重新启动依然失败，报错信息如下。

2022-01-10 13:23:22,461 | INFO  | KahaDB is version 6 | org.apache.activemq.store.kahadb.MessageDatabase | main
2022-01-10 13:23:22,515 | WARN  | Some journal files are missing: [1, 2, 258, 259, 4, 261, 6, 263, 519, 520, 265, 524, 525, 273, 274, 530, 531, 278, 279, 535, 536, 281, 283, 285, 287, 289, 553, 554, 299, 300, 556, 302, 304, 560, 561, 306, 563, 310, 311, 567, 568, 313, 315, 317, 573, 574, 319, 321, 323, 579, 325, 327, 336, 337, 341, 86, 342, 344, 346, 348, 350, 354, 99, 355, 357, 103, 359, 361, 365, 367, 369, 380, 383, 384, 386, 388, 390, 136, 392, 138, 394, 140, 396, 142, 398, 400, 404, 149, 151, 153, 155, 157, 416, 161, 417, 162, 419, 164, 166, 423, 168, 424, 170, 426, 172, 428, 174, 430, 432, 434, 179, 180, 436, 182, 438, 184, 186, 442, 188, 190, 192, 194, 196, 455, 456, 202, 458, 204, 206, 462, 463, 208, 210, 467, 212, 468, 214, 216, 472, 473, 218, 220, 226, 227, 484, 229, 485, 231, 233, 489, 490, 235, 237, 239, 495, 496, 243, 500, 501, 506, 251, 507, 252, 254, 766] | org.apache.activemq.store.kahadb.MessageDatabase | main
2022-01-10 13:23:22,546 | ERROR | [0:ActiveMQ.DLQ] references corrupt locations. 7742 messages affected. | org.apache.activemq.store.kahadb.MessageDatabase | main
2022-01-10 13:23:22,548 | ERROR | Failed to start Apache ActiveMQ ([localhost, null], java.io.IOException: Detected missing/corrupt journal files referenced by:[0:ActiveMQ.DLQ] 7742 messages affected.) | org.apache.activemq.broker.BrokerService | main
2022-01-10 13:23:22,598 | INFO  | Apache ActiveMQ 5.14.0 (localhost, null) is shutting down | org.apache.activemq.broker.BrokerService | main
...
...
...
2022-01-10 13:23:22,659 | INFO  | Apache ActiveMQ 5.14.0 (localhost, null) is shutdown | org.apache.activemq.broker.BrokerService | main
2022-01-10 13:23:22,660 | INFO  | Closing org.apache.activemq.xbean.XBeanBrokerFactory$1@3fee9989: startup date [Mon Jan 10 13:23:21 CST 2022]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2022-01-10 13:23:22,663 | WARN  | Exception thrown from LifecycleProcessor on context close | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
java.lang.IllegalStateException: LifecycleProcessor not initialized - call 'refresh' before invoking lifecycle methods via the context: org.apache.activemq.xbean.XBeanBrokerFactory$1@3fee9989: startup date [Mon Jan 10 13:23:21 CST 2022]; root of context hierarchy
	at org.springframework.context.support.AbstractApplicationContext.getLifecycleProcessor(AbstractApplicationContext.java:357)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:884)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:843)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.apache.activemq.hooks.SpringContextHook.run(SpringContextHook.java:30)[activemq-spring-5.14.0.jar:5.14.0]
	at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:875)[activemq-broker-5.14.0.jar:5.14.0]
	at org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122)[activemq-spring-5.14.0.jar:5.14.0]
	at org.apache.activemq.broker.BrokerService.start(BrokerService.java:629)[activemq-broker-5.14.0.jar:5.14.0]
	at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.14.0.jar:5.14.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_301]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_301]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_301]
	at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_301]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1700)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1639)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1568)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
	at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18]
	at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18]
	at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.14.0.jar:5.14.0]
	at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.14.0.jar:5.14.0]
	at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.14.0.jar:5.14.0]
	at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.14.0.jar:5.14.0]
	at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.14.0.jar:5.14.0]
	at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.14.0.jar:5.14.0]
	at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.0.jar:5.14.0]
	at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.14.0.jar:5.14.0]
	at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.0.jar:5.14.0]
	at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.14.0.jar:5.14.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_301]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_301]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_301]
	at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_301]
	at org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.14.0]
	at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.0]
2022-01-10 13:23:22,669 | WARN  | Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.io.IOException: Detected missing/corrupt journal files referenced by:[0:ActiveMQ.DLQ] 7742 messages affected. | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main

出现【警告】日志，| WARN | Some journal files are missing...，然后就是错误日志，[0:ActiveMQ.DLQ] references corrupt locations.

经查发现，是kahadb中对应db文件缺失，然后ActiveMQ默认的死信队列依然引用着已经不存在的文件，导致启动异常。该异常与手动删除kahadb中宁db文件，再启动ActiveMQ的异常一致，由此确定问题。

解决方案为：

把/activemq_home/conf/activemq.xml文件内以下配置从：

<persistenceAdapter>
	<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>

改为：

<persistenceAdapter>
	<kahaDB directory="${activemq.data}/kahadb"  
        ignoreMissingJournalfiles="true" 
        checkForCorruptJournalFiles="true" 
        checksumJournalFiles="true"/>
</persistenceAdapter>

ignoreMissingJournalfiles 默认值：false 忽略丢失的消息文件；当丢失了消息文件，启动异常。
checkForCorruptJournalFiles 默认值：false 检查消息文件是否损坏；true，检查发现损坏会尝试修复
checksumJournalFiles 默认值：false 产生一个checksum，以便能够检测journal文件是否损坏。

此时再启动ActiveMQ，日志显示正常，服务也正常启动。