benpolzin

"@timestamp","host.name","elasticsearch.slowlog.took",message
"Sep 18, 2019 @ 00:43:55.256",server04,"1.2s","{""type"": ""index_search_slowlog"", ""timestamp"": ""2019-09-17T19:43:55,256-0500"", ""level"": ""TRACE"", ""component"": ""i.s.s.query"", ""cluster.name"": ""swlogstash"", ""node.name"": ""server04"", ""cluster.uuid"": ""NxY3dzLATVuFgDsd_plg5Q"", ""node.id"": ""MBJ-Z1X8Tp2XrMnqBPLEMg"",  ""message"": ""[indexnameprefix-3.0-2019.09.13-000043][0] took[1.2s], took_millis[1232], total_hits[10000+ hits], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[209], source[{\""size\"":0,\""timeout\"":\""1000ms\"",\""terminate_after\"":100000,\""query\"":{\""match_all\"":{\""boost\"":1.0}},\""aggregations\"":{\""suggestions\"":{\""terms\"":{\""field\"":\""customer.name\"",\""size\"":10,\""shard_size\"":10,\""min_doc_count\"":1,\""shard_min_doc_count\"":0,\""show_term_doc_count_error\"":false,\""execution_hint\"":\""map\"",\""order\"":[{\""_count\"":\""desc\""},{\""_key\"":\""asc\""}],\""include\"":\""

benpolzin

{
  "took" : 23,
  "timed_out" : false,
  "_shards" : {
    "total" : 70,
    "successful" : 70,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {

benpolzin

{
  "took" : 16,
  "timed_out" : false,
  "_shards" : {
    "total" : 70,
    "successful" : 70,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {

benpolzin

POST _cluster/reroute?retry_failed
...
            "4": [
              {
                "state": "INITIALIZING",
                "primary": false,
                "node": "NobLSCZwTbyxzVONd4FDaA",
                "relocating_node": null,
                "shard": 4,
                "index": "app-2017.11.15",

benpolzin

ben@kibana1:~$ sudo ./6-0-post-migration-fix.sh http://localhost:9200
+ URL=http://localhost:9200
+ '[' -n '' ']'
+ AUTH=
+ CURL_CMD='curl  -H '\''Content-Type: application/json'\'' '
+ echo Before
Before
+ curl -H ''\''Content-Type:' 'application/json'\''' -XGET 'http://localhost:9200/_cat/indices?v'
curl: (6) Could not resolve host: application
health status index                             uuid                   pri rep docs.count docs.deleted store.size pri.store.size

benpolzin

curl -GET 'localhost:9200/.kibana?pretty'
{
  ".kibana-6" : {
    "aliases" : {
      ".kibana" : { }
    },
    "mappings" : {
      "visualization" : {
        "dynamic" : "strict",
        "properties" : {

benpolzin

Nov  4 08:04:09 hostName /usr/bin/filebeat[31680]: output.go:87: output worker: publish 40 events 
Nov  4 08:04:09 hostName /usr/bin/filebeat[31680]: client.go:146: Try to publish 40 events to logstash with window size 40 
Nov  4 08:04:09 hostName /usr/bin/filebeat[31680]: output.go:87: output worker: publish 40 events 
Nov  4 08:04:09 hostName /usr/bin/filebeat[31680]: client.go:124: 40 events out of 40 events sent to logstash. Continue sending ... 
Nov  4 08:04:09 hostName /usr/bin/filebeat[31680]: single.go:135: send completed 
Nov  4 08:04:09 hostName /usr/bin/filebeat[31680]: publish.go:104: Events sent: 40 
Nov  4 08:04:09 hostName /usr/bin/filebeat[31680]: registrar.go:115: Processing 40 events 
Nov  4 08:04:09 hostName /usr/bin/filebeat[31680]: registrar.go:146: Write registry file: /.filebeat 
Nov  4 08:04:09 hostName /usr/bin/filebeat[31680]: registrar.go:162: Registry file updated. 11 states written. 
Nov  4 08:04:10 hostName /usr/bin/filebeat[31680]: reader.go:138: End of file reached: /usr/local/

benpolzin

Filebeat Multiline config:

multiline:
  pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  negate: true
  match: after


Log File with entries like the following (only two events here, each begin with datestamp):