Flume 1.8.0读取Kafka写HDFS

最新推荐文章于 2023-05-29 18:58:07 发布

原创最新推荐文章于 2023-05-29 18:58:07 发布 · 1.2k 阅读

2 ·

本内容遵循CC 4.0 BY-SA版权协议

Java 同时被 2 个专栏收录

3 篇文章

订阅专栏

Flume

1 篇文章

订阅专栏

本文介绍如何使用Flume 1.8.0从Kafka读取数据并写入HDFS。环境配置包括Java 1.8.0_45和Hadoop 2.6.0-cdh5.5.1。通过下载Flume和必要的Hadoop依赖包，设置Flume配置文件example.conf，实现Kafka数据源到HDFS的流转。

Flume 1.8.0读取Kafka写HDFS

环境准备

java version “1.8.0_45”
Hadoop 2.6.0-cdh5.5.1

安装Flume

wget http://mirrors.tuna.tsinghua.edu.cn/apache/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz
tar -xvf apache-flume-1.8.0-bin.tar.gz

下载Hadoop依赖

从Hadoop集群中获取相关jar包放到flume/lib文件夹下

hadoop-common-2.6.3.jar
hadoop-hdfs-2.6.3.jar
commons-configuration-1.6.jar
hadoop-auth-2.6.3.jar
htrace-core-3.0.4.jar

DEMO

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.kafka.bootstrap.servers = localhost:9092
a1.sources.r1.kafka.consumer.group.id = flume_test
a1.sources.r1.kafka.topics = flume_topic

# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://localhost:8020/rawlog/20%y-%m-%d/%H
a1.sinks.k1.hdfs.useLocalTimeStamp = false
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.filePrefix = test
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.rollInterval = 3600
a1.sinks.k1.hdfs.rollSize =  12800000000
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.threadsPoolSize = 10
a1.sinks.k1.hdfs.batchSize = 2000
a1.sinks.k1.hdfs.threadsPoolSize = 10


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1500000
a1.channels.c1.transactionCapacity = 10000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

运行命令

./bin/flume-ng agent --conf conf --conf-file ./conf/example.conf --name a1 -Dflume.root.logger=INFO,console