使用idea编辑sparkSql的程序运行出现java.lang.IllegalArgumentException: java.net.URISyntaxException

最新推荐文章于 2023-08-05 12:24:36 发布

原创最新推荐文章于 2023-08-05 12:24:36 发布 · 4.3k 阅读

1 ·

本内容遵循CC 4.0 BY-SA版权协议

收录于

spark

在使用IntelliJ IDEA（Idea）执行SparkSql程序时遇到了`java.lang.IllegalArgumentException: java.net.URISyntaxException`的错误。通过修改SparkSession配置，特别是设置`spark.sql.warehouse.dir`为有效的文件路径（例如：file:///D://test），可以成功解决问题。

这个是具体错误：

Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:D:/idea/work/sparkTest/spark-warehouse
    at org.apache.hadoop.fs.Path.initialize(Path.java:206)
    at org.apache.hadoop.fs.Path.<init>(Path.java:172)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.makeQualifiedPath(SessionCatalog.scala:114)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createDatabase(SessionCatalog.scala:145)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.<init>(SessionCatalog.scala:89)
    at org.apache.spark.sql.internal.SessionState.catalog$lzycompute(SessionState.scala:95)
    at org.apache.spark.sql.internal.SessionState.catalog(SessionState.scala:95)
    at org.apache.spark.sql.internal.SessionState$$anon$1.<init>(SessionState.scala:112)
    at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:112)
    at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:111)
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
    at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:252)
    at org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:278)
    at com.Sql_on_spar$.main(Sql_on_spar.scala:20)
    at com.Sql_on_spar.main(Sql_on_spar.scala)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: file:D:/idea/work/sparkTest/spark-warehouse
    at java.net.URI.checkPath(URI.java:1823)
    at java.net.URI.<init>(URI.java:745)
    at org.apache.hadoop.fs.Path.initialize(Path.java:203)
    ... 15 more

虽然我不知道问什么这么做，但能解决：

var spark=SparkSession.builder().master(“spark://spark02:7077”).appName(“spark_mllib”).config(“spark.sql.warehouse.dir”,”file:///D://test”).getOrCreate()
//需要设置spark.sql.warehouse.dir的路径

标签

#idea #spark