一个Spark maven项目打包并使用spark-submit运行

最新推荐文章于 2026-05-04 06:30:54 发布

原创最新推荐文章于 2026-05-04 06:30:54 发布 · 8.4k 阅读

9 ·

本内容遵循CC 4.0 BY-SA版权协议

HDFS 同时被 3 个专栏收录

11 篇文章

订阅专栏

Spark

9 篇文章

订阅专栏

Scala

4 篇文章

订阅专栏

本文介绍了如何将一个使用Scala编写的Spark项目打包为jar，并通过`spark-submit`命令在本地及YARN集群上运行。项目源代码位于`stubs`包下的`CountJPGs.scala`文件，主要功能是处理HDFS上的web日志文件。打包后，jar文件存储在`target`目录。在YARN上运行时，可以通过配置文件传递参数。

项目目录名 countjpgs
pom.xml文件（位于项目目录下）
countjpgs => src => main => scala => stubs => CountJPGs.scala
weblogs文件存放在HDFS的/loudacre目录下，是一个包含各种请求的web日志文件。

pom.xml文件内容：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.cloudera.training.dev1</groupId>
  <artifactId>countjpgs</artifactId>
  <version>1.0</version>
  <packaging>jar</packaging>
  <name>"Count JPGs"</name>
  
  <properties>
    <spark-assembly>/usr/lib/spark/lib/spark-assembly.jar</spark-assembly>
    <hadoop-mapreduce-client-common>/usr/lib/hadoop/client/hadoop-mapreduce-client-common.jar</hadoop-mapreduce-client-common>
    <hadoop-mapreduce-client-core>/usr/lib/hadoop/client/hadoop-mapreduce-client-core.jar</hadoop-mapreduce-client-core>
    <hadoop-common>/usr/lib/hadoop/client/hadoop-common.jar</hadoop-common>
    <avro>/usr/lib/hadoop/client/avro.jar</avro>
    <commons-lang>/usr/lib/hadoop/client/commons-lang.jar</commons-lang>
    <guava>/usr/lib/hadoop/client/guava.jar</guava>
    <slf4j-api>/usr/lib/hadoop/client/slf4j-api.jar</slf4j-api>
    <slf4j-log4j12>/usr/lib/hadoop/client/slf4j-log4j12.jar</slf4j-log4j12>
    <hadoop-common>/usr/lib/hadoop/client/hadoop-common.jar</hadoop-common>
    <hadoop-annotations>/usr/lib/hadoop/client/hadoop-annotations.jar</hadoop-annotations>
  </properties>
  
  <repositories>
    <repository>
      <id>apache-repo</id>
      <name>Apache Repository</name>
      <url>https://repository.apache.org/content/repositories/releases</url>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </repository>
   <repository>
     <id>cloudera-repo-releases</id>
     <url>https://repository.cloudera.com/artifactory/repo/</url>
   </repository> 
  </repositories>

  <build>
    <plugins>
      <plugin>
        <groupId>org.scala-tools</groupId>
        <artifactId>maven-scala-plugin</artifactId>
	    <version>2.15.2</version>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
	    <version>2.5.1</version>
        <configuration>
          <source>1.7</source>
          <target>1.7</target>
        </configuration>
      </plugin>
    </plugins>  
  </build>

  <dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>2.10.5</version>
      <scope>system</scope>
      <systemPath>${spark-assembly}</systemPath>
    </dependency>
    <dependency>
       <groupId>org.apache.spark</groupId>
       <artifactId>spark-core_2.10</artifactId>
       <version>local</version>
       <scope>system</scope>
       <systemPath>${spark-assembly}</systemPath>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>local</version>
        <scope>system</scope>
        <systemPath>${hadoop-common}</systemPath>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-common</artifactId>
        <version>local</version>
        <scope>system</scope>
        <systemPath>${hadoop-mapreduce-client-common}</systemPath>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-annotations</artifactId>
        <version>local</version>
        <scope>system</scope>
        <systemPath>${hadoop-annotations}</systemPath>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>avro</artifactId>
        <version>local</version>
        <scope>system</scope>
        <systemPath>${avro}</systemPath>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>local</version>
        <scope>system</scope>
        <systemPath>${slf4j-log4j12}</systemPath>
    </dependency>

  </dependencies>
</project>

CountJPGs.scala文件内容：

package stubs

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

object CountJPGs {
   def main(args: Array[String]) {
     if (args.length < 1) {
       System.err.println("Usage: CountJPGs <file>")
       System.exit(1)
     }
	//val sc = new SparkContext("hdfs","weblogs")
	val sc = new SparkContext()
	//val filepath = "/loudace/weblogs/*66"
	val logfile = args(0)
	val weblogs = sc.textFile(logfile)
	val weblogsJpg = weblogs.filter(_.contains(".jpg"))
	var weblogsJpgCount = weblogsJpg.count()
	println("JPG Count : "+weblogsJpgCount)
	sc.stop
     //TODO: complete exercise
     println("stub is not implemented")
     System.exit(1)

   }
 }

进入到项目根目录countjpg文件夹下：

$ cd 项目存放路径/countjpgs

打包程序：

$ mvn package

打包成功后，jar包会生成在target文件夹下，名称和项目名类似：

还是进入到项目根目录countjpg文件夹下：

$ cd 项目存放路径/countjpgs

使用spark-submit命令运行程序：

$ spark-submit --class stubs.CountJPGs target/countjpgs-1.0.jar /loudacre/weblogs/*

输出效果：

补充：提交到YARN集群上面运行的命令：

$ spark-submit --class stubs.CountJPGs --master yarn-client --name 'Count JPGs' target/countjpgs-1.0.jar /loudacre/weblogs/*

另外可以在项目根目录创建一个配置文件，以便在使用spark-submit命令时调用：

$ vim myspark.conf

此文件内容：

spark.app.name My Spark App
spark.master yarn-client
spark.executor.memory 400M

启动命令：

$ spark-submit --properties-file myspark.conf --class stubs.CountJPGs target/loudacre/weblogs/*

然后就可以在YARN可视化页面看到相关的配置。