spark2 任务报错

最新推荐文章于 2025-06-04 09:00:43 发布

原创最新推荐文章于 2025-06-04 09:00:43 发布 · 648 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

标签

#spark

博客内容讨论了在使用Spark2执行任务时遇到的两个常见错误：错误1和错误2。通过对错误的详细分析，提供了可能的原因及解决策略，帮助用户理解和解决Spark2任务执行过程中的问题。

spark2 任务报错1：

ERROR Datastore.Schema: Failed initialising database.
[2020-12-10 09:45:27,624] {bash_operator.py:126} INFO - Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
[2020-12-10 09:45:27,624] {bash_operator.py:126} INFO - java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@4b988cb2, see the next exception for details.


Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@4b988cb2, see the next exception for details.
[2020-12-10 09:45:27,630] {bash_operator.py:126} INFO - 	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
[2020-12-10 09:45:27,630] {bash_operator.py:126} INFO - 	at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
[2020-12-10 09:45:27,630] {bash_operator.py:126} INFO - 	... 183 more
[2020-12-10 09:45:27,630] {bash_operator.py:126} INFO - Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /home/workrd/metastore_db.



解决：

由于提交的是spark任务，在提交spark任务的节点，找到/home/workrd/metastore_db，将其备份或者删除即可

spark2 任务报错2：

 javax.jdo.JDODataStoreException: Error executing SQL query "select "DB_ID" from "DBS"".


解决：

提示找不到元数据库中的dbs库，由于spark2任务找不到hive的配置文件，查看/etc/spark2/conf下是否有hive-site.xml文件，如果没有复制一份即可