Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
单机状态elk的搭建部署安装系统:CentOS‑7.7
elasticsearch版本:7.7.0【7.8还在预览版】
kibana版本:7.7.0
blog:zhoufengjie.cn
0、说明简介:
ELK主要由ElasticSearch、Logstash和Kibana三个开源工具组成,还有其他专门由于收集数据的轻量型数据采集器Beats。 Elasticsearch :分布式搜索引擎。
具有高可伸缩、高可靠、易管理等特点。可以用于全文检索、结构化检索和分析,并能将这三者结合起来。
Elasticsearch 是用Java 基于 Lucene 开发,现在使用最广的开源搜索引擎之一,Wikipedia 、StackOverflow、Github等都基于它来构建自己的搜索引擎。 在elasticsearch中,所有节点的数据是均等的。
Logstash :数据收集处理引擎。
支持动态的从各种数据源搜集数据,并对数据进行过滤、分析、丰富、统一格式等操作,然后存储以供后续使用。
Kibana :可视化化平台。
它能够搜索、展示存储在 Elasticsearch 中索引数据。使用它可以很方便的用图表、表格、地图展示和分析数据。
Filebeat:轻量级数据收集引擎。
相对于Logstash所占用的系统资源来说,Filebeat 所占用的系统资源几乎是微乎及微。它是基于原先 Logstash‑fowarder 的源码改造出来。换句话说:Filebeat就是新版的 Logstash‑fowarder,也会是 ELK Stack 在 Agent 的第一选择。
版本说明: Elasticsearch、Logstash、Kibana、Filebeat安装的版本号必须全部一致,不然会出现kibana无法显示web页面。
ELK工作演示图:
A、Filebeat在APP Server端收集日志
B、Logstash处理过滤Filebeat收集过来的日志
C、Elasticsearch存储Logstash提供的处理之后的日志,用以检索、统计
D、Kibana提供web页面,将Elasticsearch的数据可视化的展示出来
1、软件下载:【从下面网址找最新版本】
https://www.elastic.co/guide/en/elasticsearch/reference/7.7/setup.html#jvm‑version
https://www.elastic.co/cn/downloads/elasticsearch
https://www.elastic.co/cn/downloads/kibana
https://www.elastic.co/cn/downloads/logstash
https://www.elastic.co/cn/downloads/beats
也可以从国内的镜像站找软件包,比如:https://mirrors.tuna.tsinghua.edu.cn/elasticstack/yum/elastic‑7.x/7.7.0/
1.1、下载软件:【在我安装的时候,最新版本是7.7.0版本】
下载openjdk需要自己找一下路径:http://openjdk.java.net/
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch‑7.7.0‑x86_64.rpm
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch‑7.7.0‑x86_64.rpm.sha512
wget https://artifacts.elastic.co/downloads/kibana/kibana‑7.7.0‑x86_64.rpm
wget https://artifacts.elastic.co/downloads/logstash/logstash‑7.7.0.rpm
1.2、下面是我准备的软件包:
elk/jdk‑8u191‑linux‑x64.rpm elk/elasticsearch/elasticsearch‑7.7.0‑x86_64.rpm elk/elasticsearch/elasticsearch‑7.7.0‑x86_64.rpm.sha512 elk/Kibana/kibana‑7.7.0‑x86_64.rpm elk/beats/filebeat‑7.7.0‑x86_64.rpm elk/beats/packetbeat‑7.7.0‑x86_64.rpm elk/logstash/logstash‑7.7.0.rpm
2、安装部署ElasticSearch
通常ES是多台设备做集群性部署,我这边单独单机部署了。
2.1、关掉基础服务
#关闭selinux setenforce 0 sed ‑i.bak 's@^SELINUX=\(.*\)@SELINUX=disabled@p' /etc/selinux/config
#关闭防火墙 #Centos7 systemctl disable firewalld systemctl stop firewalld #CentOS6 service iptables stop service iptables disable
2.2、安装jdk:
rpm ‑ivh jdk‑8u191‑linux‑x64.rpm
[root@tyumen elk]# rpm ‑ivh jdk‑8u191‑linux‑x64.rpm warning: jdk‑8u191‑linux‑x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY Preparing... ################################# [100%] Updating / installing... 1:jdk1.8‑2000:1.8.0_191‑fcs ################################# [100%] Unpacking JAR files... tools.jar... plugin.jar... javaws.jar... deploy.jar... rt.jar... jsse.jar... charsets.jar... localedata.jar... [root@tyumen elk]#
声明java路径:
vim /etc/profile.d/jdk.sh
export JAVA_HOME=/usr/java/jdk1.8.0_191‑amd64 export PATH=$JAVA_HOME/jre/bin:$PATH
source /etc/profile.d/jdk.sh
2.3、部署elasticsearch:
官方安装教程为:https://www.elastic.co/guide/en/elasticsearch/reference/7.7/rpm.html#rpm
修改系统配置属性:vim /etc/security/limits.conf
elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited elasticsearch soft nofile 65536 elasticsearch hard nofile 131072
2.3.1、安装elasticsearch:
校验一下elasticsearch文件是否正常:
[root@tyumen elasticsearch]# sha512sum ‑c elasticsearch‑7.7.0‑x86_64.rpm.sha512 elasticsearch‑7.7.0‑x86_64.rpm: OK
安装:rpm ‑ivh elasticsearch‑7.7.0‑x86_64.rpm
[root@tyumen elasticsearch]# rpm ‑ivh elasticsearch‑7.7.0‑x86_64.rpm warning: elasticsearch‑7.7.0‑x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing... ################################# [100%] Creating elasticsearch group... OK Creating elasticsearch user... OK Updating / installing... 1:elasticsearch‑0:7.7.0‑1 ################################# [100%] ### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd sudo systemctl daemon‑reload sudo systemctl enable elasticsearch.service ### You can start elasticsearch service by executing sudo systemctl start elasticsearch.service future versions of Elasticsearch will require Java 11; your Java version from [/usr/java/jdk1.8.0_191‑amd64/jre] does not meet this requirementCreated elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore [root@tyumen elasticsearch]#
设置elasticsearch随机启动:
systemctl daemon‑reload
systemctl enable elasticsearch
启动elasticsearch:
systemctl start elasticsearch
查看进程监听:
[root@tyumen elasticsearch]# netstat ‑tpln Active Internet connections (only servers) Proto Recv‑Q Send‑Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 32931/sshd tcp6 0 0 127.0.0.1:9200 :::* LISTEN 52384/java tcp6 0 0 ::1:9200 :::* LISTEN 52384/java tcp6 0 0 127.0.0.1:9300 :::* LISTEN 52384/java tcp6 0 0 ::1:9300 :::* LISTEN 52384/java [root@tyumen elasticsearch]#
2.3.2、修改elasticsearch配置文件:
停止elasticsearch:
systemctl stop elasticsearch
修改配置文件:
vim /etc/elasticsearch/elasticsearch.yml
#集群名称 cluster.name: els #节点名称 node.name: tyumen #主机名,需要绑定/etc/hosts #数据存放路径 path.data: /data/els_data
#日志存放路径 path.logs: /data/log/els #锁定jvm.options指定的内存,不交换swap内存[这一条不要打开] #bootstrap.memory_lock: true #绑定IP地址 network.host: 192.168.0.97 #端口号 http.port: 9200 配置唯一node节点: cluster.initial_master_nodes: ["tyumen"] #配置集群配置,填写集群节点,会自动发现节点 # discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#其实单节点配置:只需要修改节点名,和绑定ip地址即可
vim /etc/elasticsearch/jvm.options
‑Xms1g #指定占用内存大小,两个数字要一致 都是1g ‑Xmx1g
创建目录并赋权为elasticsearch:
mkdir /data/{els_data,log/els} ‑p
chown ‑R elasticsearch.elasticsearch /data/els_data
chown ‑R elasticsearch.elasticsearch /data/log/els
重新启动elasticsearch
systemctl start elasticsearch
查看进程监听:
[root@tyumen elasticsearch]# netstat ‑tpln Active Internet connections (only servers) Proto Recv‑Q Send‑Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 54472/sshd tcp6 0 0 192.168.0.97:9200 :::* LISTEN 56468/java tcp6 0 0 192.168.0.97:9300 :::* LISTEN 56468/java
查看请求状态:
[root@tyumen elasticsearch]# curl 192.168.0.97:9200 { "name" : "tyumen", "cluster_name" : "els", "cluster_uuid" : "I5WJWVjQSCmkIh‑YXYzNVw", "version" : { "number" : "7.7.0", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf", "build_date" : "2020‑05‑12T02:01:37.602180Z", "build_snapshot" : false, "lucene_version" : "8.5.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0‑beta1" }, "tagline" : "You Know, for Search" }
2.3.3、使用elasticsearch的api:
常见api如下:
Elasticsearch API 集群状态:http://192.168.0.97:9200/_cluster/health?pretty
节点状态:http://192.168.0.97:9200/_nodes/process?pretty
分片状态:http://192.168.0.97:9200/_cat/shards
索引分片存储信息:http://192.168.0.97:9200/index/_shard_stores?pretty
索引状态:http://192.168.0.97:9200/index/_stats?pretty
索引元数据:http://192.168.0.97:9200/index?pretty
具体使用说明参见文章:https://www.e‑learn.cn/content/java/1078247
3、部署Kibana
Kibana是node.js 编写的,不需要java环境。直接安装即可
3.1、安装Kibana
注意,安装的kibana版本要与elasticsearch版本相同
rpm ‑ivh kibana‑7.7.0‑x86_64.rpm
[root@tyumen Kibana]# rpm ‑ivh kibana‑7.7.0‑x86_64.rpm warning: kibana‑7.7.0‑x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing... ################################# [100%] Updating / installing... 1:kibana‑7.7.0‑1 ################################# [100%]
3.2、配置Kibana
理论上应该单独分开部署,我这里单机部署了:
vim /etc/kibana/kibana.yml
server.port: 5601 server.host: "192.168.0.97" elasticsearch.url: "http://192.168.0.97:9200" kibana.index: ".kibana" logging.dest: /data/log/kibana/kibana.log # 配置kibana日志输出到哪里 i18n.locale: "zh‑CN" #配置kibana界面为中文
创建日志目录和文件并赋权:
mkdir ‑p /data/log/kibana/
touch /data/log/kibana/kibana.log
chmod o+rw /data/log/kibana/kibana.log
3.3、启动kibana并检查
配置随机启动:
systemctl daemon‑reload
systemctl enable kibana
启动kabana
systemctl start kibana
查看端口监听:
[root@tyumen Kibana]# netstat ‑tpln Active Internet connections (only servers) Proto Recv‑Q Send‑Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.0.97:5601 0.0.0.0:* LISTEN 58684/node tcp 0 0 0.0.0.0:7346 0.0.0.0:* LISTEN 54472/sshd tcp6 0 0 192.168.0.97:9200 :::* LISTEN 56468/java tcp6 0 0 192.168.0.97:9300 :::* LISTEN 56468/java
访问kibana:
http://192.168.0.97:5601
4、部署Logstash
配置与Elasticsearch相同的Java环境,版本为8以上的Java环境
4.1、安装logstash
rpm ‑ivh logstash‑7.7.0.rpm
[root@nginx‑server logstash]# rpm ‑ivh logstash‑7.7.0.rpm warning: logstash‑7.7.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing... ################################# [100%] Updating / installing... 1:logstash‑1:7.7.0‑1 ################################# [100%] Using provided startup.options file: /etc/logstash/startup.options /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun‑0.0.31/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecatedSuccessfully created system startup script for Logstash
vim /etc/logstash/logstash.yml
node.name: nginx‑server #设置节点名称,一般写主机名 path.data: /data/logstash/plugin‑data #创建logstash 和插件使用的持久化目录 config.reload.automatic: true #开启配置文件自动加载 config.reload.interval: 10 #定义配置文件重载时间周期 http.host: "192.168.0.108" #定义访问主机名,一般为域名或IP http.port: 9600‑9700 path.logs: /var/log/logstash
增加目录权限:[一定要增加权限,不然可能会一直不上传日志]
mkdir /data/logstash/plugin‑data ‑p
chown logstash:logstash /data/logstash ‑R
chmod +w /data/logstash/plugin‑data ‑R
chown logstash:logstash /var/log/logstash ‑R
4.2、配置logstash采集
配置nginx:这里以采集nginx的日志为例,修改nginx的log配置。下面的信息采集对应的nginx的log配置为json格式,便于采集使用;
log_format elk_log_json '{"@timestamp":"$time_iso8601",' '"request_time":"$request_time",' '"clientip":"$remote_addr",' '"status":"$status",' '"size":$body_bytes_sent,' '"method":$request_method,' '"scheme":$scheme,' '"http_host":"$host",' '"url":"$request_uri",' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"upstreamstatus":"$upstream_http_content_type",' '"referer":"$http_referer"}';
access_log /var/log/nginx_access.log elk_log_json
配置logstash收集日志配置文件:
vi /etc/logstash/conf.d/nginx‑log.conf
input { file { type =>"nginx‑log" path => ["/var/log/nginx_access.log"] codec => json start_position => "beginning" sincedb_path => "/dev/null" } }
output { elasticsearch { hosts => ["192.168.0.97:9200"] index => "nginx‑log‑%{+YYYY.MM}" } }
4.3、测试和启动logstash
手动测试logstash是否正常
/usr/share/logstash/bin/logstash ‑‑path.settings /etc/logstash/ ‑f /etc/logstash/conf.d/nginx.conf ‑‑config.test_and_exit
[root@nginx‑server conf]# /usr/share/logstash/bin/logstash ‑‑path.settings /etc/logstash/ ‑f /etc/logstash/conf.d/nginx.conf ‑‑config.test_and_exitSending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2020‑05‑23T09:26:02,478][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/home/logstash/plugin‑data/queue"}[2020‑05‑23T09:26:02,637][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/logstash/plugin‑data/dead_letter_queue"}[2020‑05‑23T09:26:03,533][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified[2020‑05‑23T09:26:05,609][INFO ][org.reflections.Reflections] Reflections took 86 ms to scan 1 urls, producing 21 keys and 41 values Configuration OK [2020‑05‑23T09:26:06,452][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
设置logstash随机启动,并启动logstash
systemctl enable logstash
systemctl start logstash
查看日志:/var/log/logstash/logstash‑plain.log【有比较小的错误可以后续再排查】
详情见:https://www.jianshu.com/p/d7e0a502fd52
5、部署filebeat
我们知道Elastic Stack被称之为ELK (Elasticsearch,Logstash and Kibana)。由于beats的加入,现在很多人把ELK说成为ELKB。这里的B就是代表Beats。Beats在Elasticsearch中扮演很重要的角色,filebeat可以向logstash输入日志,也可以直接向elasticsearch输入日志,并且beats可扩展,支持自定义构建。
因为:Logstash的数据都是从Beats中获取,Logstash已经不需要自己去数据源中获取数据了。 以前我们使用的日志采集工具是logstash,但是logstash占用的资源比较大,没有beats轻量,所以:官方也推荐使用beats来作为日志采集工具。
5.1、安装filebeat
rpm ‑ivh filebeat‑7.7.0‑x86_64.rpm
[root@nginx‑server beats]# rpm ‑ivh filebeat‑7.7.0‑x86_64.rpm warning: filebeat‑7.7.0‑x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing... ################################# [100%] Updating / installing... 1:filebeat‑7.7.0‑1 ################################# [100%]
5.2、配置filebeat
vi /etc/filebeat/filebeat.yml
‑ type: log paths: ‑ /var/log/messages output.elasticsearch: hosts: ["192.168.0.97:9200"]
测试:/usr/bin/filebeat test output
5.3、启动filebeat:
配置随系统启动:
systemctl enable filebeat
启动filebeat:
systemctl start filebeat
5.4、filebeat使用和排障
见:https://www.elastic.co/cn/beats/filebeat
https://www.elastic.co/guide/en/beats/filebeat/current/index.html
6、在kibana上配置索引
日志=>设置
management(管理)=>索引模式=>创建索引模式
在discover里面查看日志是否已经上来了: