Failed to initialize compiler: object scala.runtime in compiler mirror-Spark Shell由于Scala编译器原因不能正常启动

最新推荐文章于 2023-11-29 09:23:00 发布

原创最新推荐文章于 2023-11-29 09:23:00 发布 · 1.1k 阅读

标签

#spark #spark-shell #spark-submit #scala.runtime #scala

报错记录专栏收录该内容

14 篇文章 ¥19.90 ¥99.00

订阅专栏

超级会员免费看

在尝试启动Spark Shell时遇到'Failed to initialize compiler: object scala.runtime in compiler mirror'的错误。虽然其他Spark命令如`spark-submit`正常，但问题出在环境变量配置。原来在`spark-env`中设置了`SPARK_SUBMIT_OPTS`，导致`SPARK_SUBMIT_OPTS`被重写，丢失了`-Dscala.usejavacp=true`配置，进而引发问题。解决方案是移除或调整`SPARK_SUBMIT_OPTS`的设置，确保-Dscala.usejavacp=true生效。通过日志验证，问题得到根本解决。

有日子没用之前搭的测试集群,今天想用来测试下 Spark一个问题，发现spark-shell启动失败了，报错信息如下：

hadoop@master:/software/servers/spark$ spark-shell
Listening for transport dt_socket at address: 9001
19/11/06 09:34:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/06 09:34:26 INFO SignalUtils: Registered signal handler for INT
[init] error: error while loading Object, Missing dependency 'object scala in compiler mirror', required by /data0/jdk/jdk1.8.0_161/jre/lib/rt.jar(java/lang/Object.class)

Failed to initialize compiler: object scala in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
**

了解本专栏

超级会员免费看