flume之kafkachannel

1.安装flume

2.安装kafka

3.测试kafka的topic是否能正常的传递消息

4.都准备好之后开始接通kafkachannel，

1）kafka的no-sink

#定义agent名， source、channel的名称
a0.sources = r1
a0.channels = c1

#具体定义source
a0.sources.r1.type = exec
a0.sources.r1.command = tail -F /data/logs.txt


a0.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
a0.channels.c1.brokerList = 172.16.37.223:9092
a0.channels.c1.zookeeperConnect=172.16.37.107:2181,172.16.37.108:2181,172.16.37.223:2181
a0.channels.c1.topic = FLUME_TEST_TOPIC
#false表示是以纯文本的形式写进入的，true是以event的形式写进入的，以event写进入时，会出现乱码， 默认是true
a0.channels.c1.parseAsFlumeEvent = false
a0.sources.r1.channels = c1

 启动flume之后。开启一个kafka的消费者

./bin/kafka-console-producer.sh --broker-list 172.16.37.223:9092 --topic FLUME_TEST_TOPIC

往flume指定的文件中塞入信息。kafka的消费者可以消费到你塞入的信息

2）kafka的no-source

agent.channels = kafka-channel
agent.sources = no-source
agent.sinks = k1

agent.channels.kafka-channel.type = org.apache.flume.channel.kafka.KafkaChannel

agent.channels.kafka-channel.brokerList = 172.16.37.223:9092
agent.channels.kafka-channel.zookeeperConnect = 172.16.37.107:2181,172.16.37.108:2181,172.16.37.223:2181
agent.channels.kafka-channel.topic = FLUME_TEST_TOPIC
#agent.channels.kafka-channel.consumer.group.id = groupM
agent.channels.kafka-channel.kafka.consumer.timeout.ms = 100
agent.channels.kafka-channel.parseAsFlumeEvent = false

#文件的sink方式
#agent.sinks.k1.type = file_roll
#agent.sinks.k1.sink.directory = /data/kafkachannel


#程序异步处理的方式。普通测试使用文件的方式即可
agent.sinks.k1.type = asynchbase
agent.sinks.k1.table = monstor_mm7mt
agent.sinks.k1.columnFamily = cf1
agent.sinks.k1.batchSize = 5
agent.sinks.k1.serializer = com.caissa.chador_flume.AsyncHbaseAllLogEventSerializer
agent.sinks.k1.serializer.columns = xunqi_number,protocol_type,message_type,submit_number,smsreq_rid,message_number,company_code,user_name,channel_value,billingusers_number,billing_type,aimphone_number,phone_number,aim_phone,appcode,is_status,messagevalid_time,message_sendtime,mobilevalide_number,valid_type,expenses,link_id,tp_pid,tp_udhi,message_format,message_code,mobiledeal_number,moblie_result,titile_length,mmcresouce_id,mmc_titile


agent.sinks.k1.channel = kafka-channel

启动flume之后开启一个kafka的生产者

./kafka-console-consumer.sh --zookeeper 172.16.37.107:2181 --from-beginning --topic FLUME_TEST_TOPIC

然后生产者输入数据。flume就会消费到 

3）kafkachannel之有source和有sink

agent.channels = kafka-channel
#agent.sources = no-source
agent.sources=r1
agent.sinks = k1

#具体定义source
#agent.sources.r1.type = exec
#agent.sources.r1.command = tail -F /data/logs.txt

agent.sources.r1.type = avro
agent.sources.r1.bind = 172.16.37.107
agent.sources.r1.port = 42411


agent.channels.kafka-channel.type = org.apache.flume.channel.kafka.KafkaChannel
agent.channels.kafka-channel.brokerList = 172.16.37.223:9092
agent.channels.kafka-channel.zookeeperConnect = 172.16.37.107:2181,172.16.37.108:2181,172.16.37.223:2181
agent.channels.kafka-channel.topic = FLUME_TEST_TOPIC
#agent.channels.kafka-channel.consumer.group.id = groupM
agent.channels.kafka-channel.kafka.consumer.timeout.ms = 100
agent.channels.kafka-channel.parseAsFlumeEvent = false


#agent.sinks.k1.type = file_roll
#agent.sinks.k1.sink.directory = /data/kafkachannel

agent.sinks.k1.type = asynchbase
agent.sinks.k1.table = monstor_mm7mt
agent.sinks.k1.columnFamily = cf1
agent.sinks.k1.batchSize = 5
agent.sinks.k1.serializer = com.caissa.chador_flume.AsyncHbaseAllLogEventSerializer
agent.sinks.k1.serializer.columns = xunqi_number,protocol_type,message_type,submit_number,smsreq_rid,message_number,company_code,user_name,channel_value,billingusers_number,billing_type,aimphone_number,phone_number,aim_phone,appcode,is_status,messagevalid_time,message_sendtime,mobilevalide_number,valid_type,expenses,link_id,tp_pid,tp_udhi,message_format,message_code,mobiledeal_number,moblie_result,titile_length,mmcresouce_id,mmc_titile


agent.sinks.k1.channel = kafka-channel
agent.channels.kafka-channel.parseAsFlumeEvent = false
agent.sources.r1.channels = kafka-channel

5.遇到的问题。

1）配置好启动flume之后报错：

flume java.lang.NoClassDefFoundError: org/apache/zookeeper/Watcher

解决办法是将kafka的lib下边的zookeeper的jar包拷贝到flume的lib下即可。

2）启动报的另外一个错误：

 org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException

没有解决但是也不影响数据正常的传递。

3）Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Batch Expired

这个是因为默认情况下kafka是广播的localhost，所以如果不是同一个机器需要修改下配置

advertised.listeners=PLAINTEXT://ｉｐ:9092把默认的localhost替换成ＩＰ地址　重新启动下就可以了．

4）接通flume和kafkachannel的时候可以先测试一下topic是否可用。使用命令如下

1.监听消费者

注意kafka版本的高低不同监听命令不同

老版本：

./kafka-console-consumer.sh --zookeeper 172.16.37.112:2181 --from-beginning --topic IENGENE_TASK_CAISSAUICLOG_CHANGE

新版本：

./kafka-console-consumer.sh --bootstrap-server  172.16.37.112:9002 --from-beginning --topic IENGENE_TASK_CAISSAUICLOG_CHANGE

2.监听生产者

./kafka-console-producer.sh --broker-list 172.16.37.112:9002 --topic IENGENE_TASK_CAISSAUICLOG_CHANG

监听中会出现错误：bogon:bogon

解决办法是在hosts文件中加bogon就可以

5）由于flume和kafka版本不匹配造成的问题

 flume1.7和kafka0.9.0.1版本使用kafkachannel时。通道不通。或者出现错误，启动flume发现根本没有启动端口。后经过一番研究，发现是版本不兼容的问题

2018-09-15 00:10:08,502 (kafka-producer-network-thread | producer-1) [ERROR - org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:130)] Uncaught error in kafka producer I/O thread: 
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'throttle_time_ms': java.nio.BufferUnderflowException
        at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:71)
        at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:439)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:265)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
        at java.lang.Thread.run(Thread.java:748)

 解决办法：

(1)flume1.7.0和kafka_2.11-0.10.2.1是可以合成的。

(2)flume1.6.0和kafka_2.10-0.8.2.1是可以合成的。