elk添加nginx日志分析

1.修改nginx配置 nginx需要把日志收集成json字符串;类型: nginx.conf文件 注释掉:

#log_format main '$remote_addr - $remote_user [$time_local] "$request" ' # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"'; #access_log logs/access.log main;

换成

log_format json '{ "@timestamp":"$time_iso8601",' '"@source":"$server_addr",' '"time_local":"$time_local",' '"remote_addr":"$remote_addr",' '"remote_port":"$remote_port",' '"remote_user":"$remote_user",' '"server_name":"$server_name",' '"server_port":"$server_port",' '"server_protocol":"$server_protocol",' '"request":"$request",' '"request_uri":"$request_uri",' '"uri":"$uri",' '"request_time":"$request_time",' '"request_method":"$request_method",' '"request_length":$request_length,' '"status":"$status",' '"scheme":"$scheme",' '"body_bytes_sent":"$body_bytes_sent",' '"bytes_sent":"$bytes_sent",' '"request_body":"$request_body",' '"upstream_addr":"$upstream_addr",' '"upstream_response_time":"$upstream_response_time",' '"upstream_status":"$upstream_status",' '"http_host":"$http_host",' '"http_referrer":"$http_referer",' '"http_user_agent":"$http_user_agent",' '"http_x_forwarded_for":"$http_x_forwarded_for",' '"connection":"$connection",' '"connection_requests":"$connection_requests",' '"content_length":"$content_length",' '"content_type":"$content_type",' '"cookie_name":"$cookie_name",' '"limit_rate":"$limit_rate",' '"hostname":"$hostname",' '"args":"$args",' '"https":"$https",' '"http_cookie":"$http_cookie",' '"msec":"$msec",' '"pid":"$pid"}'; access_log /usr/local/nginx/logs/access.log json;

2.添加filebeat文件:

ilebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 1 filebeat.inputs: - type: log enabled: true paths: - "/usr/local/nginx/logs/access.log" fields: app_id: "access.log" - type: log enabled: true paths: - "/usr/local/nginx/logs/error.log" fields: app_id: "error.log" output.redis: hosts: ["192.168.10.46:6380"] password: "123456" db: 2 key: "nginx" keys: - key: "%{[fields.list]}" mappings: app_id: "access.log" app_id: "error.log" worker: 4 timeout: 20 max_retries: 3 codec.json: pretty: false monitoring.enabled: true monitoring.elasticsearch: hosts: ["http://192.168.10.46:9201","http://192.168.10.46:9202","http://192.168.10.46:9203"]

运行容器,将nginx的日志文档映射到filebeat的容器中

docker run -d --name filebeat --hostname localhost --user=root -v /docker/elk/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro -v /usr/local/nginx/logs:/usr/local/nginx/logs:ro -v /var/run/docker.sock:/var/run/docker.sock:ro docker.elastic.co/beats/filebeat:7.2.0

3.修改logstash配置文件

xqkang@xqkang:/docker/elk/logstash/pipeline$ cat docker.conf input { redis { host => "192.168.10.46" port => 6380 db => 2 key => "nginx" password => "123456" data_type => "list" threads => 4 tags => "nginx" } } filter { if "nginx" in [tags] { json { source => "message" } grok { match => [ "message", "%{HTTPDATE:[@metadata][timestamp]}" ] } date { match => [ "[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ] } geoip { source => "remote_addr" database =>"/usr/share/logstash/config/GeoLite2-City.mmdb" fields => ["city_name", "continent_code", "country_code2", "country_code3", "country_name", "dma_code", "ip", "latitude", "longitude", "postal_code", "region_name", "timezone"] target => "geoip" add_field => [ "[geoip][location]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][location]", "%{[geoip][latitude]}" ] } mutate { convert => [ "[geoip][location]", "float"] } } } output { if "nginx" in [tags] { if [fields][app_id] == "access.log" { elasticsearch { hosts => ["192.168.10.46:9201","192.168.10.46:9202","192.168.10.46:9203"] index => "logstash-nginx-access.log-%{+YYYY.MM.dd}" } } if [fields][app_id] == "error.log" { elasticsearch { hosts => ["192.168.10.46:9201","192.168.10.46:9202","192.168.10.46:9203"] index => "logstash-nginx-error.log%{+YYYY.MM.dd}" } } } }

解释:

链接:https://segmentfault.com/a/1190000021471571