温馨提示×

温馨提示×

您好,登录后才能下订单哦!

密码登录×
登录注册×
其他方式登录
点击 登录注册 即表示同意《亿速云用户服务条款》

SpringBoot怎么集成Flink的部署与打包

发布时间:2022-05-20 11:28:21 来源:亿速云 阅读:2569 作者:iii 栏目:开发技术

本文小编为大家详细介绍“SpringBoot怎么集成Flink的部署与打包”,内容详细,步骤清晰,细节处理妥当,希望这篇“SpringBoot怎么集成Flink的部署与打包”文章能帮助大家解决疑惑,下面跟着小编的思路慢慢深入,一起来学习新知识吧。

一、SpringBoot集成Flink

其实没什么特别的,就把Flink依赖的包在pom引入就行了。只是FlinkTask的写法要小调整下,把相关依赖交给spring管理就行。

然后如果放弃Flink的Dashboard端监控task执行相关信息,那也可以在SpringBoot的启动类里调用就行,但是可能出现task的相关对象没有注入,这种都是小问题(实际就是springboot启动完成再调用,或者通过自动任务调用。也可以在springBoot的入口类用@ComponentScan注解扫描flinkTask所在的目录)。

实际更潇洒一点的做法可以将flinkTask的信息存表,通过springBoot的接口调用restfullApi接口,接口里调用task,甚至可以做task的启停、线程监控(接口里开线程调用task)。

二、FlinkTask写法调整

@Component
@Slf4j
public class JianGongStopCarTask {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStream<JGParkingLotInfo> dataStream = env.addSource(new JGParkingLotInfoSource());
//       获取到数据之后转换格式  此处不做转换
        SingleOutputStreamOperator<JGParkingLotInfo> jgParkingLotInfoSingleOutputStreamOperator = dataStream.map(jgParkingLotInfo -> jgParkingLotInfo);
        jgParkingLotInfoSingleOutputStreamOperator.addSink(new SinkToMySQL());
        env.execute();
    }
}

PS:

实际就是@Component这个注解,然后这个示例里的JGParkingLotInfoSource、SinkToMySQL类都需要加这个注解。其他就没有什么调整了,然后如果还有其他依赖没有,使用hutool的SpringUtil进行get对象就行。

三、打包插件

<plugins>
      <!-- 编译插件 -->
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
          <encoding>UTF-8</encoding>
        </configuration>
      </plugin>
      <!--  spring boot 项目打包
       <plugin>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-maven-plugin</artifactId>
       </plugin>-->
      <!-- Flink打包方式一 -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>3.3.0</version>
        <configuration>
          <archive>
            <manifest>
              <mainClass>com.easylinkin.dc.olap.JianGongStopCarTask</mainClass>
            </manifest>
          </archive>
          <!-- 打包依赖 -->
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
        </configuration>
        <executions>
          <execution>
            <id>make-assembly</id>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <!-- flink打包方式二
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.3.0</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <createDependencyReducedPom>false</createDependencyReducedPom>
              <artifactSet>
                <excludes>
                  <exclude>com.google.code.findbugs:jsr305</exclude>
                  <exclude>org.slf4j:*</exclude>
                  <exclude>log4j:*</exclude>
                </excludes>
              </artifactSet>
              <filters>
                <filter>
                  <artifact>*:*</artifact>
                  <excludes>
                    <exclude>module-info.class</exclude>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                  </excludes>
                </filter>
              </filters>
              <transformers>
                <transformer
                  implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                  <resource>META-INF/spring.handlers</resource>
                  <resource>reference.conf</resource>
                </transformer>
                <transformer
                  implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                  <resource>META-INF/spring.factories</resource>
                </transformer>
                <transformer
                  implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                  <resource>META-INF/spring.schemas</resource>
                </transformer>
                <transformer
                  implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                <transformer
                  implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>com.easylinkin.dc.olap.JianGongStopCarTask</mainClass>
                </transformer>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>
      -->
      <!-- flink打包方式三
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <mainClass>com.easylinkin.dc.olap.JianGongStopCarTask</mainClass>
              <useUniqueVersions>false</useUniqueVersions>
              <addClasspath>true</addClasspath>
              <classpathPrefix>./lib/</classpathPrefix>
            </manifest>
          </archive>
          <excludes>
            <exclude>module-info.class</exclude>
            <exclude>META-INF/*.SF</exclude>
            <exclude>META-INF/*.DSA</exclude>
            <exclude>META-INF/*.RSA</exclude>
            <exclude>com.google.code.findbugs:jsr305</exclude>
            <exclude>org.slf4j:*</exclude>
            <exclude>log4j:*</exclude>
          </excludes>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>3.3.0</version>
        <executions>
          <execution>
            <id>copy-dependencies</id>
            <phase>package</phase>
            <goals>
              <goal>copy-dependencies</goal>
            </goals>
            <configuration>
              <outputDirectory>${project.build.directory}/lib</outputDirectory>
              <excludeTransitive>false</excludeTransitive>
              <stripVersion>false</stripVersion>
            </configuration>
          </execution>
        </executions>
      </plugin>-->
    </plugins>

除了打成SpringBoot用springboot的插件打包,flinkTask的打包有3种方式,方式三适合lib包提前传到task指定的依赖存储目录。这样上传flinkTask就很小。

方式二是官方推荐FlinkTask的打包方式

说一千道一万就是因为打包的META-INF下的MANIFEST.MF文件的内容有区别。springBoot项目的这个文件有自己的JarLauncher。

SpringBoot怎么集成Flink的部署与打包

FlinkTask的jar这个文件内容

SpringBoot怎么集成Flink的部署与打包

四、Flink的上传与运行

1、上传并命令运行

SpringBoot怎么集成Flink的部署与打包

配置好flink环境,命令就是

flink run WordCount.jar

2、Flink管理大屏上传运行

SpringBoot怎么集成Flink的部署与打包

点击“Submit”

SpringBoot怎么集成Flink的部署与打包

读到这里,这篇“SpringBoot怎么集成Flink的部署与打包”文章已经介绍完毕,想要掌握这篇文章的知识点还需要大家自己动手实践使用过才能领会,如果想了解更多相关内容的文章,欢迎关注亿速云行业资讯频道。

向AI问一下细节

免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。

AI