Logstash+Redis+Elasticsearch+Kibana日志监控报警系统.md

Logstash+Redis+Elasticsearch+Kibana日志监控报警系统

@(服务监控)[logstash|Elasticsearch|kibana]

工具介绍

Logstash：用于收集、处理、传输日志数据。
Redis：用于实时标识和传输日志数据。
Elasticsearch：用于实时查询和解析数据。
Kibana：用于数据可视化。

工作流程

Logstash收集日志，将日志过滤处理后传输到Redis中，Redis中的数据再次标识过滤输出到Elasticsearch中，并且将异常信息通过邮件、短信发送给相关负责人。Kibana配置上ES服务接口，通过Kibana服务访问即可。

重点配置

日志服务器上Logstash配置

input {
    file {
        path=>"/XX/XX/dianshang.log"
        codec=>multiline {
            pattern => "^\s"
            what=>"previous"
        }
        type=>"dianshang"
        tags=>["XX.XX.XX.XX"]
    }
}
output {
    redis {
        host=>"XX.XX.XX.XX"
        port=>62627
        data_type=>"list"
        key=>"dianshang"
    }
}

Logstash服务器配置

###输入
input {
    redis {
        host => "XX.XX.XX.XX"
         port => 62627
         data_type => "list"
         key => "dianshang"
    }
}
###过滤
filter{
    grok {
    match => ["message","mailmonitor"]
    add_tag => [mailmonitor]
    }
    
    grok {
    match => [ "message", "smsmonitor" ]
    add_tag => [smsmonitor]
    }
    ....
}
###output to es
output {
    elasticsearch {
        host => "XX.XX.XX.XX"
        protocol => http
        index => "dianshang-%{+YYYY.ww}"
        template_name => "diangshanglog"
    }
}
### Java exception log -> mail
output {
    if "multiline" in [tags] and [type] in ["pay","pay-manage"] {
    email{
        to => "xxxx@xxxx.com"
        from => "xxxx@xxxx.com"
        options => [ "smtpIporHost", "XX.XX.XX.XX",
                "port", "25",
                "starttls", "true"
        ]
        subject => "%{@timestamp}-%{host}-%{path}"
        body => "Here is the event line that occured: \n %{message}"
        }
    }
}
### keyworlds  -> mail
output {
if "mailmonitor" in [tags] and [type] in ["pay"] {
    email{
        to => "xxxx@xxxx.com"
        from => "xxxx@xxxx.com"
        options => [ "smtpIporHost", "XX.XX.XX.XX",
                "port", "25",
                "starttls", "true"
        ]
        subject => "%{@timestamp}-%{host}-%{path}"
        body => "Here is the event line that occured: \n %{message}"
        }
    }
}
###keywords -> SMS
output {
    if "smsmonitor" in [tags]  and [type] in ["pay"] {
        email{
            to => "xxxxxxxxx@139.com"
            from => "xxxx@xxxx.com"
            options => [ "smtpIporHost", "XX.XX.XX.XX",
            "port", "25",
            "starttls", "true"
            ]
            subject => "%{@timestamp}-%{host}-%{path}"
            body => "%{message}"
        }
    }
}

Kibana配置

elasticsearch: "http://XX.XX.XX.XX:9200",
default_route: '/dashboard/file/default.json',

成果展示

z-log.jpg