最新版ELK日志集群部署

最新推荐文章于 2025-11-06 15:03:13 发布

原创最新推荐文章于 2025-11-06 15:03:13 发布 · 1.4k 阅读 ·

本内容遵循CC 4.0 BY-SA版权协议

代码可运行

框架信息

系统：Almalinux 9.4
结构：
node1：elastic、kibana，192.168.100.10
node2：elastic、fleet，192.168.100.20
node3：fleet、agent，192.168.100.30
版本：v8.17

全部安装 Elasticsearch（全部节点）

mkdir elastic-install-files
cd elastic-install-files
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.17.4-x86_64.rpm
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.17.4-x86_64.rpm.sha512
sudo rpm --install elasticsearch-8.17.4-x86_64.rpm
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service	# 暂时都不启动！！！

输出结果如下

Creating elasticsearch group... OK
Creating elasticsearch user... OK
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : OswLRyfMMbmYkK+AV-D3

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'	# 加入集群命令
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.	# 生成用户密码

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.	# 生成kibana的token

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.	# 生成node的token

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
/usr/lib/tmpfiles.d/elasticsearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/elasticsearch → /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly.
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch.service → /usr/lib/systemd/system/elasticsearch.service.

在这里插入图片描述
可以使用自动化工具批量，也可以使用MobaXterm支持多窗口同时输入的远程连接工具

配置Elasticsearch（node1）

sudo vim /etc/elasticsearch/elasticsearch.yml
...
cluster.name: elasticsearch-demo
network.host: 192.168.100.10
transport.host: 0.0.0.0
...
sudo systemctl start elasticsearch.service	# node1启动其他不要启动也不要配置

验证（node1）

# 生成密码
/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
This tool will reset the password of the [elastic] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y


Password for the [elastic] user successfully reset.
New value: _MFGnbn1LPPGFMdQvYr7

# 第一个节点验证即可
sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:_MFGnbn1LPPGFMdQvYr7 https://localhost:9200
{
  "name" : "node1",
  "cluster_name" : "elasticsearch-demo",
  "cluster_uuid" : "DHbfL9aISwaQdoxPMflGgw",
  "version" : {
    "number" : "8.17.4",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "c63c7f5f8ce7d2e4805b7b3d842e7e792d84dda1",
    "build_date" : "2025-03-20T15:39:59.811110136Z",
    "build_snapshot" : false,
    "lucene_version" : "9.12.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

其他节点加入集群（除node1）

# 生成密码，第一个节点操作即可！！！
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTkyLjE2OC4xMDAuMTA6OTIwMCJdLCJmZ3IiOiI0NjA4N2Q3NWRhOTFkNDQzNTM5NzY2MTc1ZGQzN2IzZTNmM2Y4ZWUwZTgxZTA3ZmFkNTgzMjM0ZGNiN2MyMGEyIiwia2V5IjoiNkw3RjFaVUJxN3I3Tjc4SldCQ3I6cUNsUWZrQU1TTENaSTdaOUE3dGZyQSJ9
# 加入集群
/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTkyLjE2OC4xMDAuMTA6OTIwMCJdLCJmZ3IiOiI0NjA4N2Q3NWRhOTFkNDQzNTM5NzY2MTc1ZGQzN2IzZTNmM2Y4ZWUwZTgxZTA3ZmFkNTgzMjM0ZGNiN2MyMGEyIiwia2V5IjoiNkw3RjFaVUJxN3I3Tjc4SldCQ3I6cUNsUWZrQU1TTENaSTdaOUE3dGZyQSJ9

This node will be reconfigured to join an existing cluster, using the enrollment token that you provided.
This operation will overwrite the existing configuration. Specifically:
  - Security auto configuration will be removed from elasticsearch.yml
  - The [certs] config directory will be removed
  - Security auto configuration related secure settings will be removed from the elasticsearch.keystore
Do you want to continue with the reconfiguration process [y/N]y

# 配置节点
sudo vim /etc/elasticsearch/elasticsearch.yml
...
cluster.name: elasticsearch-demo
network.host: 10.128.0.132
...
# 启动节点
sudo systemctl start elasticsearch.service

# 验证，其中一个节点验证即可，这边使用node2
sudo tail -f /var/log/elasticsearch/elasticsearch-demo.log
[2025-03-27T12:09:43,158][INFO ][o.e.h.AbstractHttpServerTransport] [node2] publish_address {192.168.100.20:9200}, bound_addresses {[::]:9200}
[2025-03-27T12:09:43,179][INFO ][o.e.n.Node               ] [node2] started {node2}{jjSA9N3gQ5aJJ2pieJIwYw}{-CCzdsyDT96HB8RF6lJxGA}{node2}{192.168.100.20}{192.168.100.20:9300}{cdfhilmrstw}{8.17.4}{7000099-8521000}{ml.allocated_processors=4, ml.allocated_processors_double=4.0, ml.max_jvm_size=4013948928, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=8023773184}
[2025-03-27T12:09:43,189][INFO ][o.e.n.j.JdkPosixCLibrary ] [node2] Sending 7 bytes to socket
[2025-03-27T12:09:47,895][INFO ][o.e.c.s.ClusterApplierService] [node2] added {{node3}{qOgItm6_RqSpkPJ1sMGu1g}{XAVeCcPCRiiV7M1A5Tjwzw}{node3}{192.168.100.30}{192.168.100.30:9300}{cdfhilmrstw}{8.17.4}{7000099-8521000}}, term: 1, version: 86, reason: ApplyCommitRequest{term=1, version=86, sourceNode={node1}{6KevyZUPQ_a95ny6I_vt1g}{BXcpVGuaSya_-iCrQeoO1Q}{node1}{192.168.100.10}{192.168.100.10:9300}{cdfhilmrstw}{8.17.4}{7000099-8521000}{ml.allocated_processors=6, ml.allocated_processors_double=6.0, ml.max_jvm_size=4013948928, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=8023789568}}

# 最终检查
sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:_MFGnbn1LPPGFMdQvYr7 https://localhost:9200
{
  "name" : "node2",
  "cluster_name" : "elasticsearch-demo",
  "cluster_uuid" : "DHbfL9aISwaQdoxPMflGgw",
  "version" : {
    "number" : "8.17.4",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "c63c7f5f8ce7d2e4805b7b3d842e7e792d84dda1",
    "build_date" : "2025-03-20T15:39:59.811110136Z",
    "build_snapshot" : false,
    "lucene_version" : "9.12.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

安装 Kibana（node1）

mkdir kibana-install-files
cd kibana-install-files
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.17.4-x86_64.rpm
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.17.4-x86_64.rpm.sha512
sudo rpm --install kibana-8.17.4-x86_64.rpm

输出结果如下

...
Creating kibana group... OK
Creating kibana user... OK
Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see https://www.elastic.co/guide/en/kibana/8.17/production.html#openssl-legacy-provider
Created Kibana keystore in /etc/kibana/kibana.keystore
/usr/lib/tmpfiles.d/elasticsearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/elasticsearch → /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly.

生成 token并配置及启动（node1）

# 生成 token
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTkyLjE2OC4xMDAuMTA6OTIwMCJdLCJmZ3IiOiI0NjA4N2Q3NWRhOTFkNDQzNTM5NzY2MTc1ZGQzN2IzZTNmM2Y4ZWUwZTgxZTA3ZmFkNTgzMjM0ZGNiN2MyMGEyIiwia2V5IjoiNnI3VjFaVUJxN3I3Tjc4SkV4QVA6VmF5aXNJdnZSQ09wUE5GeEVqVlpVQSJ9

# 配置kibana
sudo vim /etc/kibana/kibana.yml
...
server.host: 192.168.100.10
...

# 启动kibana
systemctl daemon-reload
systemctl enable --now kibana.service

验证 kibana

[root@node1 kibana-install-files]# systemctl status kibana.service
● kibana.service - Kibana
     Loaded: loaded (/usr/lib/systemd/system/kibana.service; enabled; preset: disabled)
     Active: active (running) since Thu 2025-03-27 12:25:05 CST; 49s ago
       Docs: https://www.elastic.co
   Main PID: 10993 (node)
      Tasks: 11 (limit: 48711)
     Memory: 241.9M
        CPU: 8.201s
     CGroup: /system.slice/kibana.service
             └─10993 /usr/share/kibana/bin/../node/glibc-217/bin/node /usr/share/kibana/bin/../src/cli/dist

3月 27 12:25:05 node1 kibana[10993]: Native global console methods have been overridden in production environment.
3月 27 12:25:06 node1 kibana[10993]: [2025-03-27T12:25:06.418+08:00][INFO ][root] Kibana is starting
3月 27 12:25:06 node1 kibana[10993]: [2025-03-27T12:25:06.442+08:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
3月 27 12:25:10 node1 kibana[10993]: [2025-03-27T12:25:10.902+08:00][INFO ][plugins-service] The following plugins are disabled: "cloudChat,cloudExperiments,cloudFullStory,dataUsage,investigateApp,investigate>
3月 27 12:25:10 node1 kibana[10993]: [2025-03-27T12:25:10.954+08:00][INFO ][http.server.Preboot] http server running at http://192.168.100.10:5601
3月 27 12:25:11 node1 kibana[10993]: [2025-03-27T12:25:11.070+08:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
3月 27 12:25:11 node1 kibana[10993]: [2025-03-27T12:25:11.080+08:00][INFO ][preboot] "interactiveSetup" plugin is holding setup: Validating Elasticsearch connection configuration…
3月 27 12:25:11 node1 kibana[10993]: [2025-03-27T12:25:11.103+08:00][INFO ][root] Holding setup until preboot stage is completed.
3月 27 12:25:18 node1 kibana[10993]: i Kibana has not been configured.
3月 27 12:25:18 node1 kibana[10993]: Go to http://192.168.100.10:5601/?code=844442 to get started.	# 复制链接浏览器访问即可

浏览器访问

输入之前生成的kibana的token
在这里插入图片描述
密码是之前生成的用户密码

到此kibana就配置完成了~
如果需要配置SSL就此停住，继续参考SSL配置教程~

安装Fleet Server（node2）

创建工作目录

mkdir fleet-install-files
cd fleet-install-files

浏览器配置

返回 Web 浏览器，打开 Kibana 菜单，然后转到 Management → Fleet。Fleet （队列）将打开，并显示一条消息，指出您需要添加 Fleet Server （队列服务器）。
单击 Add fleet Server。此时将打开 Add a Fleet Server 浮出控件。
在弹出控件中，选择 Quick Start （快速启动）选项卡。
为您的 Fleet Server 主机指定一个名称，例如：Fleet Server。
指定 Elastic 代理将到达 Fleet Server 的主机 URL。
单击 Generate Fleet Server policy（生成队列服务器策略）。将创建一个策略，其中包含 Fleet Server 实例的所有配置设置。

手误输成http了，这边输入https！！！
在 Install Fleet Server to a centralized host （将 Fleet Server 安装到集中式主机）步骤中，在此示例中，我们选择 Linux Tar （Linux tar）选项卡，但您可以选择适合要设置 Fleet Server 的主机作系统的选项卡。
复制生成的命令，然后在 Fleet Server 主机上的终端中逐个运行它们。

复制命令安装

# 在刚刚建的工作目录内执行复制的安装代码
pwd
/root/fleet-install-files

# 执行安装命令
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.17.4-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.17.4-linux-x86_64.tar.gz
cd elastic-agent-8.17.4-linux-x86_64
sudo ./elastic-agent install \
  --fleet-server-es=https://192.168.100.10:9200 \
  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3NDMwNTQ2NTQ5OTI6Q1hjVHk5VWxTYm1SZnRkT2dia0Y4dw \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-es-ca-trusted-fingerprint=46087d75da91d443539766175dd37b3e3f3f8ee0e81e07fad583234dcb7c20a2 \
  --fleet-server-port=8220

# 输出如下
...
elastic-agent-8.17.4-linux-x86_64/elastic-agent
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y # 输入y继续安装
[====] Service Started  [0s] Elastic Agent successfully installed, starting enrollment.
[==  ] Waiting For Enroll...  [3s] {"log.level":"info","@timestamp":"2025-03-27T13:58:08.596+0800","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).prepareFleetTLS","file.name":"cmd/enroll_cmd.go","file.line":447},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
[  ==] Waiting For Enroll...  [3s] {"log.level":"info","@timestamp":"2025-03-27T13:58:08.940+0800","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":493},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
[==  ] Waiting For Enroll...  [5s] {"log.level":"info","@timestamp":"2025-03-27T13:58:10.943+0800","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":817},"message":"Fleet Server - Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
[==  ] Waiting For Enroll...  [6s] {"log.level":"info","@timestamp":"2025-03-27T13:58:11.698+0800","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":530},"message":"Starting enrollment to URL: https://node2:8220/","ecs.version":"1.6.0"}
[=== ] Waiting For Enroll...  [12s] {"log.level":"info","@timestamp":"2025-03-27T13:58:18.047+0800","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":493},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-03-27T13:58:18.048+0800","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":311},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[=== ] Done  [12s]
Elastic Agent has been successfully installed.

验证

在这里插入图片描述
三个全绿说明没问题，点击继续注册

说明已经成功安装好了

安装 Elastic Agent（node3）

创建工作目录

mkdir agent-install-files
cd agent-install-files

配置kibana

打开 Kibana 并转到 Management → Fleet。
在 Agents （代理）选项卡上，您应该会看到新的 Fleet Server 策略以运行状况良好运行。
打开 Agents （代理）选项卡，然后选择 Add agent（添加代理）。此时将打开 Add agent （添加代理）浮出控件。
在浮出控件中，选择策略名称，例如：Demo Agent Policy
保持 Collect system logs and metrics （收集系统日志和指标）处于启用状态。这会将 System integration 添加到 Elastic Agent 策略中。
单击 Create policy（创建策略）。
对于 Enroll in Fleet（注册队列）步骤，将 Enroll in Fleet（注册队列）保持选中状态。
在 Install Elastic Agent on your host （在主机上安装 Elastic Agent）步骤中，在此示例中，我们选择 Linux Tar （Linux Tar）选项卡，但您可以选择适合要设置 Fleet Server 的主机作系统的选项卡。
复制命令

复制命令修改安装

# 执行安装命令
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.17.4-linux-x86_64.tar.gz 
tar xzvf elastic-agent-8.17.4-linux-x86_64.tar.gz
cd elastic-agent-8.17.4-linux-x86_64
sudo ./elastic-agent install --url=https://192.168.100.20:8220 --enrollment-token=Qjc0ODFwVUJxN3I3Tjc4SkVTVms6VnFWcmJNa3hRZm11eUhONlNlR0FGZw== --insecure	# 因为没SSL证书所以我们需要添加标志

# 输出如下
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y	# 输入y继续注册
[====] Service Started  [0s] Elastic Agent successfully installed, starting enrollment.
[====] Waiting For Enroll...  [1s] {"log.level":"warn","@timestamp":"2025-03-27T14:21:02.431+0800","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig","file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
[=== ] Waiting For Enroll...  [1s] {"log.level":"info","@timestamp":"2025-03-27T14:21:02.528+0800","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":530},"message":"Starting enrollment to URL: https://192.168.100.20:8220/","ecs.version":"1.6.0"}
[=   ] Waiting For Enroll...  [1s] {"log.level":"warn","@timestamp":"2025-03-27T14:21:02.738+0800","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig","file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
[==  ] Waiting For Enroll...  [2s] {"log.level":"info","@timestamp":"2025-03-27T14:21:03.521+0800","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":493},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-03-27T14:21:03.525+0800","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":311},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[=   ] Done  [2s]
Elastic Agent has been successfully installed.