一、Kafka-Eagle安装部署

1、下载安装

下载地址：https://www.kafka-eagle.org/

官方文档：https://docs.kafka-eagle.org/

  1. 解压目录：tar -zxvf kafka-eagle-bin-2.0.8.tar.gz
  2. 进入目录，解压安装：tar -zxvf /opt/software/efak-web-2.0.8-bin.tar.gz -C /opt/module/
  3. 修改文件名：mv /opt/module/efak-web-2.0.8 /opt/module/efak
  4. 配置并生效环境变量：vim /etc/profile.d/my_env.sh
1
2
3
#KE_HOME
export KE_HOME=/opt/module/efak
export PATH=$PATH:$KE_HOME/bin

2、配置文件

  • 修改eagle配置文件system-config.properties：vim ./conf/system-config.properties
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
######################################
# multi zookeeper & kafka cluster list
# Settings prefixed with 'kafka.eagle.' will be deprecated, use 'efak.' instead
######################################
efak.zk.cluster.alias=cluster1
cluster1.zk.list=hadoop102:2181,hadoop103:2181,hadoop104:2181/kafka

######################################
# kafka offset storage
######################################
cluster1.efak.offset.storage=kafka

# 配置mysql连接
######################################
# kafka mysql jdbc driver address
######################################
efak.driver=com.mysql.cj.jdbc.Driver
efak.url=jdbc:mysql://hadoop102:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
efak.username=root
efak.password=123456
  1. 关闭kafka集群，修改kafka启动文件kafka-server-start.sh（每台kafka节点）：vim ./bin/kafka-server-start.sh
1
2
3
4
5
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    #export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 XX:InitiatingHeapOccupancyPercent=70"
    export JMX_PORT="9999"
fi

二、Kafka-Eagle监控操作

1、启动/停止监控

  • 启动监控：
    • 启动Zookeeper集群
    • 启动Kafka集群
    • 启动监控：ke.sh start
  • 停止监控：ke.sh stop