Hudi：问题汇总（1）—Flink-1.12.2消费kafka并插入hudi

原创已于 2023-03-10 15:03:25 修改 · 1.6k 阅读

1 ·

本内容遵循CC 4.0 BY-SA版权协议

标签

#flink #hudi

于 2023-01-13 14:47:38 首次发布

hudi 专栏收录该内容

2 篇文章

订阅专栏

文章列举了在使用Flink过程中遇到的问题，包括找不到Kafka工厂实现、类转换错误、抽象方法错误、jar包冲突以及端口被占用等问题，并提供了相应的解决办法，如更新依赖、排除冲突的jar、正确打包等。

问题1：Flink 1.12 Could not find any factory for identifier 'kafka' that implements 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath

解决方法：

pom.xml文件中加入依赖（也可去如下网站下载对应版本）

https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-kafka_2.11/1.12.1

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-sql-connector-kafka_${scala.binary.version}</artifactId>
<version>${flink.version}</version>

</dependency>

问题2：ByteArraySerializer is not an instance of org.kafka.common.serialization.Deserializer

解决：看下pom里是不是引入了flink-connector-kafka_2.11-1.12.0.jar，这是pom的flink-connector-kafka与服务器的flink/lib/flink-sql-connector-kafka冲突，删掉flink-connector-kafka就行，引入flink-sql-connctor-kafka

问题3：java.lang.ClassCastException: org.codehaus.janino.CompilerFactory cannot be cast to org.codehaus.commons.compiler.ICompilerFactory

解决：本地环境jar包中flink-table-planner包与集群中flink/lib中的 flink-table_2.12-1.14.5.jar 冲突，在打包时在pom里写上<scope>provided</scope>不打flink-table-planner包即可

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner_${scala.version}</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>

问题4：Caused by: java.lang.AbstractMethodError: Method org/apache/hudi/sink/StreamWriteOperatorCoordinator.subtaskReady(ILorg/apache/flink/runtime/operators/coordination/OperatorCoordinator$SubtaskGateway;)V is abstract

解决：服务器上flink与hudi版本不兼容造成，在代码的pom里引入匹配的版本并且打包时把所有依赖打进去，这样在运行时用的是pom的依赖包，否则用的是服务器上flink与hudi的包

问题5：org.apache.flink.client.program.ProgramInvocationException:在jar文件中找不到

原因：打包时没有将pom依赖全打进去

解决：build换成这个

<build>
        <sourceDirectory>src/main/java</sourceDirectory>
        <testSourceDirectory>src/test/java</testSourceDirectory>
        <plugins>
            <!-- 编译插件 -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.18.1</version>
                <configuration>
                    <useFile>false</useFile>
                    <disableXmlReport>true</disableXmlReport>
                    <includes>
                        <include>**/*Test.*</include>
                        <include>**/*Suite.*</include>
                    </includes>
                </configuration>
            </plugin>
            <!-- 打jar包插件(会包含所有依赖) -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>