Steps to recover Kibana with failed 6.5.0 upgrade

fix_stuck_kibana.md

Fixes this error when starting Kibana 6.5.0 when Security is enabled and using the default kibana user.

log   [17:58:17.221] [error][status][plugin:spaces@6.5.0] Status changed from yellow to red - action [indices:data/read/get] is unauthorized for user [kibana]: [security_exception] action [indices:data/read/get] is unauthorized for user [kibana]
 error  [17:58:17.227] [fatal][root] [security_exception] action [indices:data/read/get] is unauthorized for user [kibana] :: {"path":"/_tasks/09ge3Z3xQliEfF18KTiZ5Q%3A305","query":{},"statusCode":403,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/read/get] is unauthorized for user [kibana]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/read/get] is unauthorized for user [kibana]\"},\"status\":403}"}
    at respond (/Users/jbryan/stack/6.4/kibana-6.5.0-darwin-x86_64-kibana_issue/node_modules/elasticsearch/src/lib/transport.js:308:15)
    at checkRespForFailure (/Users/jbryan/stack/6.4/kibana-6.5.0-darwin-x86_64-kibana_issue/node_modules/elasticsearch/src/lib/transport.js:267:7)
    at HttpConnector.<anonymous> (/Users/jbryan/stack/6.4/kibana-6.5.0-darwin-x86_64-kibana_issue/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)
    at IncomingMessage.wrapper (/Users/jbryan/stack/6.4/kibana-6.5.0-darwin-x86_64-kibana_issue/node_modules/elasticsearch/node_modules/lodash/lodash.js:4949:19)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

 FATAL  [security_exception] action [indices:data/read/get] is unauthorized for user [kibana] :: {"path":"/_tasks/09ge3Z3xQliEfF18KTiZ5Q%3A305","query":{},"statusCode":403,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/read/get] is unauthorized for user [kibana]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/read/get] is unauthorized for user [kibana]\"},\"status\":403}"}

Delete .kibana_1 and .kibana_2

curl -XDELETE "https://localhost:9200/.kibana_1,.kibana_2" -u user:pass

Create kibana_temp role

curl -XPUT "https://localhost:9200/_xpack/security/role/kibana_temp -u user:pass \
  -H 'content-type: application/json' \
  --data '
{
  "indices": [
    {
       "names": [
         ".tasks"
       ],
       "privileges": [
         "create_index",
         "read",
         "write"
       ]
    }
  ]
}'

Create kibana_temp user

curl -XPUT "https://localhost:9200/_xpack/security/user/kibana_temp -u user:pass \
  -H 'content-type: application/json' \
  --data '
{
  "username": "kibana_temp",
  "password": "somepassword",
  "roles": [
      "kibana_system",
      "kibana_temp"
    ]
}'

Update kibana.yml or Kibana secure settings keystore

kibana.yml

Modify elasticsearch.username and elasticsearch.password to reflect the new user.

keystore

bin/kibana-keystore list
elasticsearch.password
elasticsearch.username

bin/kibana-keystore remove elasticsearch.username
bin/kibana-keystore remove elasticsearch.password

echo "kibana_temp" | bin/kibana-keystore add elasticsearch.username --stdin
echo "somepassword" | bin/kibana-keystore add elasticsearch.password --stdin

Start Kibana