Hadoop Hdfs Edit log 加载错误，重启无效问题解决过程

最新推荐文章于 2023-11-08 18:12:21 发布

原创

最新推荐文章于 2023-11-08 18:12:21 发布 · 2.8k 阅读

标签

#hadoop #hdfs #big data

收录于

在CDH 5.13版本中，遇到HDFS启动后无法通过HDFS连接的问题。错误源于大量数据导致NameNode内存不足，加载edit log失败。解决方案是将NameNode的Java堆大小从4GB增加到16GB，JournalNode的Java堆大小从1GB提升到8GB，调整后重新启动HDFS服务，问题得到解决。

cdh 问题，启动显示正常，但是不能通过hdfs连接。

版本：5.13

问题：hdfs 启动错误，提示下面日志

2021-12-20 00:14:56,817 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack of redundant storage directories!
2021-12-20 06:37:16,571 WARN org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 19629ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=20129ms
2021-12-20 06:37:33,855 WARN org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 15722ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=17280ms
2021-12-20 06:37:33,858 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage: Error replaying edit log at offset 6457157.  Expected transaction ID was 750743826
Recent opcode offsets: 67913