kafka 基本介绍
概念
一个分布式流处理平台,消息订阅平台,一般有三个特性
适用场景
构建实时流式应用程序,对这些流数据进行转换或者影响。 (就是流处理,通过kafka stream topic和topic之间内部进行变化)
部署安装
kafka 部署安装需要依赖于 zoookper 和java 环境
安装java 环境
yum install java-1.8.0-openjdk* -y
安装zoookper
wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
解压缩到指定位置
cp zoo_sample.cfg zoo.cfg
配置文件如下,创建所需要的目录
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/zookeeper_data
clientPort=2181
cat /etc/systemd/system/zookeeper.service
[Unit]
Description=zookeeper.service
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/zookeeper/bin/zkServer.sh start
ExecStop=/usr/local/zookeeper/bin/zkServer.sh stop
ExecReload=/usr/local/zookeeper/bin/zkServer.sh restart
[Install]
WantedBy=multi-user.target
systemctl start zookeeper
安装kafka
curl -LO https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.1.0/kafka_2.12-2.1.0.tgz
解压缩到指定位置
修改配置文件
server.properties
broker.id=1
port=9092
host.name=localhost
log.dirs=/usr/local/kafka_logs
zookeeper.connect=localhost:2181
zookeeper.properties
dataDir=/usr/local/kafaka_data
clientPort=2181
maxClientCnxns=0
host.name=localhost
producer.properties
metadata.broker.list=localhost:9092
bootstrap.servers=localhost:9092
compression.type=none
consumer.properties
bootstrap.servers=localhost:9092
group.id=test-consumer-group
zookeeper.connect=localhost:2181
* 制作标准服务启动
cat /etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka server (broker)
After=network.target
After=syslog.target
After=zookeeper.target
[Service]
Type=forking
User=root
Group=root
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh
ExecReload=/bin/kill -HUP $MAINPID
KillMode=none
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
systemctl start kafka
创建话题Topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafka01
Created topic "kafka01".
bin/kafka-topics.sh --list --zookeeper localhost:2181
启动消息生产者并发送消息
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kafka01
启动消息消费者并收到消息
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic kafka01 --from-beginning
查看topic 列表 详细信息
bin/kafka-topics.sh --zookeeper localhost:2181 --list
bin/kafka-topics.sh --zookeeper localhost:2181 --topic kafka01 --describe
Topic:kafka01 PartitionCount:1 ReplicationFactor:1 Configs:
Topic: kafka01 Partition: 0 Leader: 1 Replicas: 1 Isr: 1
拓展分区
bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic kafka01 --partitions 2
bin/kafka-topics.sh --zookeeper localhost:2181 --topic kafka01 --describe
Topic:kafka01 PartitionCount:2 ReplicationFactor:1 Configs:
Topic: kafka01 Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: kafka01 Partition: 1 Leader: 1 Replicas: 1 Isr: 1
免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。