spark yarn 提交任务异常怎么办

这篇文章主要讲解了“spark yarn 提交任务异常怎么办”，文中的讲解内容简单清晰，易于学习与理解，下面请大家跟着小编的思路慢慢深入，一起来研究和学习“spark yarn 提交任务异常怎么办”吧！

Exception in thread "main" java.lang.IllegalStateException: Library directory 'C:\eclipse-workspace\SparkDemo\assembly\target\scala-2.11\jars' does not exist; make sure Spark is built.
	at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
	at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:342)
	at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
	at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:545)
	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:865)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:179)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:188)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:501)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at my.test.WordCountYarn.main(WordCountYarn.java:34)
[Thread-1] [INFO] [2021-04-12 18:12:44] Shutdown hook called 
[Thread-1] [INFO] [2021-04-12 18:12:44] Deleting directory C:\Users\zengming.zhang\AppData\Local\Temp\spark-ed6b5d87-e618-4375-9ee4-264c0d832447

SparkConf conf = new SparkConf().setAppName("WordCountYarn").setMaster("yarn-client")
                .set("spark.yarn.dist.files", "C:\\eclipse-workspace\\SparkDemo\\src\\main\\resources\\yarn-site.xml")

//添加此参数解决
                .set("spark.yarn.jars", "*.jars")

https://blog.csdn.net/qq_40651753/article/details/86239630

感谢各位的阅读，以上就是“spark yarn 提交任务异常怎么办”的内容了，经过本文的学习后，相信大家对spark yarn 提交任务异常怎么办这一问题有了更深刻的体会，具体使用情况还需要大家实践验证。这里是亿速云，小编将为大家推送更多相关知识点的文章，欢迎关注！