10 Filebeat_Logstash 新架构

Filebeat和Logstash说明
Filebeat：轻量级，但不支持正则、不能移除字段等
Logstash：比较重，但支持正则、支持移除字段等
架构：
Filebeat -> Logstash -> Elasticsearch -> Kibana

部署服务介绍（filebeat可以装在其他Nginx服务器上）
192.168.189.83: Kibana、ES
192.168.189.84: Logstash、Filebeat、Nginx
修改filebeat的配置文件：（189.84上操作）
vim /usr/local/filebeat-6.6.0/filebeat.yml
Filebeat配置发往Logstash
filebeat.inputs:
- type: log
  tail_files: true
  backoff: "1s"
  paths:
    - /usr/local/nginx/logs/access.log
output:
  logstash:
  hosts: ["192.168.189.84:5044"]

保存退出。启动filebeat
nohup /usr/local/filebeat-7.1.1/filebeat -e -c /usr/local/filebeat-7.1.1/filebeat.yml >/tmp/filebeat.log 2>&1 &
修改Logstash 配置文件：
Logstash配置监听在5044端口，接收Filebeat发送过来的日志
vim /usr/local/logstash-6.6.0/config/logstash.conf
input {
 beats {
   host => '0.0.0.0'
   port => '5044'
 }
}
filter {
grok {
match => {
"message"=>'(?<clientIP>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - \[(?<requesttime>[^ ]+ \+[0-9]+)\] "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) HTTP/\d.\d" (?<status>[0-9]+) (?<bodysize>[0-9]+) "[^"]+" "(?<ua>[^"]+)"'
}
remove_field => ["message","@version","path"]
}
date {
    match => ["requesttime", "dd/MMM/yyyy:HH:mm:ss Z"]
   target => "@timestamp"
}
}
output {
if "_grokparsefailure" not in [tags] and "_dateparsefailure" not in [tags] {
elasticsearch {
hosts=>["http://192.168.189.83:9200"]
}
}
}

保存退出，启动logstash
nohup /usr/local/logstash-6.6.0/bin/logstash -f /usr/local/logstash-6.6.0/config/logstash.conf >/tmp/logstash.log 2>/tmp/logstash.log &
Kibana上查看数据
清空kibana上的索引：




在一台或多台服务器上，访问 192.168.189.84 Nginx ，触发日志。
查看索引：





Logstash上移除不必要的字段      Filebeat发过来的无用字段比较多


remove_field => ["message","@version","path","log","offset","prospector","source","tags"]
修改logstash配置文件，移除不必要的字段
input {
beats {
host => '0.0.0.0'
port => '5044'
}
}
filter {
grok {
match => {
"message"=>'(?<clientIP>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - \[(?<requesttime>[^ ]+ \+[0-9]+)\] "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) HTTP/\d.\d" (?<status>[0-9]+) (?<bodysize>[0-9]+) "[^"]+" "(?<ua>[^"]+)"'
}
remove_field => ["message","ecs","path","beat","agent","log","offset","prospector","source","tags"]
}
date {
match => ["requesttime", "dd/MMM/yyyy:HH:mm:ss Z"]
target => "@timestamp"
}
}
output {
if "_grokparsefailure" not in [tags] and "_dateparsefailure" not in [tags] {
elasticsearch {
hosts=>["http://192.168.189.83:9200"]
}
}
}
保存退出，重启logstash
Kibana 上清空索引，重新触发日志。在其他电脑上访问192.168.189.84的Nginx


查看索引：