hbase重启报错: passed in file status is for something other than a regular file
背景:由于hbase有几个region一直是Regions in Transition导入部分表数据无法使用,然后搜索给的解决方案重启最简单。重启完regionserver后问题依然在,又把master也重启了,之后regionserver一直挂,报错。
2020-05-07 08:33:00,709 FATAL [RS_LOG_REPLAY_OPS-host_name:6002-1] regionserver.HRegionServer: ABORTING region server host_name,6002,1588811577747: Caught throwable while processing event RS_LOG_REPLAY
java.lang.IllegalArgumentException: passed in file status is for something other than a regular file.
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:272)
at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:236)
at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-05-07 08:33:00,709 FATAL [RS_LOG_REPLAY_OPS-10.67.2.160:6002-1] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2020-05-07 08:33:00,725 WARN [SplitLogWorker-10.67.2.160:6002] coordination.ZkSplitLogWorkerCoordination: Interrupted while yielding for other region servers
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.grabTask(ZkSplitLogWorkerCoordination.java:272)
at org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.taskLoop(ZkSplitLogWorkerCoordination.java:432)
at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:142)
at java.lang.Thread.run(Thread.java:748)
2020-05-07 08:33:02,046 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2677)
看日志报错信息有点摸不到头脑,搜了一堆没有相关的解决方案。


去hbase监控页面发现有一个task一直处于等待状态,查看hdfs文件发现wal日志下的这个文件夹包含一个空文件,与上面报错的passed in file status is for something other than a regular file相关,将空文件移除后,regionserver启动正常,数据也恢复。
在尝试解决HBase中Regions in Transition问题时,执行重启操作后,regionserver仍出现问题,日志显示'passed in file status is for something other than a regular file'错误。通过监控页面发现一个等待状态的task,进一步调查发现HDFS的WAL日志目录下存在一个空文件,该文件与错误信息相关。删除空文件后,regionserver成功启动,数据恢复正常。

1658

被折叠的 条评论
为什么被折叠?



