Notes to self. Or if anyone can help with this.
Use Elastic’s helm charts as basis (couldn’t find anything more suitable - e.g. operator or Openshift Templates ):
I couldn’t use Helm itself due to permissions issues related to: helm/helm#1918
git clone https://github.com/elastic/helm-charts/
cd helm-charts
helm template elasticsearch -n elastic-stack > elasticsearch.yaml // (1)
oc apply -f elasticsearch.yaml
-
The
helm template
command basically substitutes all values fromvalues.yaml
and creates a yaml with the actual kubernetes descriptors that can be deployed into the openshift cluster.
Attempting to deploy this failed with:
create Pod elasticsearch-master-0 in StatefulSet elasticsearch-master failed error: pods "elasticsearch-master-0" is forbidden: unable to validate against any security context constraint: [fsGroup: Invalid value: []int64{1000}: 1000 is not an allowed group spec.initContainers[0].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1002210000, 1002219999] spec.initContainers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]
so, combination of things:
-
the initContainer tries to run as root. deleting the initContainer obviously solves that problem, but, of course it didn’t do its job.
-
the fsGroup seems to have an invalid value. I am not sure what that should be. Deleting the fsGroup + securityContext does allow statefulset to be created, BUT, again, that’s probably there for a reason (haven’t looked into that in detail yet)
If I do delete / change the above, I then do get a pod up and running, but then I get a bunch of exceptions like:
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])? at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.5.3.jar:6.5.3] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.5.3.jar:6.5.3] Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])? at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:297) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.3.jar:6.5.3] ... 6 more Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0 at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:215) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.3.jar:6.5.3] ... 6 more Caused by: java.io.IOException: Input/output error at sun.nio.ch.FileDispatcherImpl.lock0(Native Method) ~[?:?] at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:96) ~[?:?] at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1167) ~[?:?] at java.nio.channels.FileChannel.tryLock(FileChannel.java:1165) ~[?:?] at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:126) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13] at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:208) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.3.jar:6.5.3] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.3.jar:6.5.3] ... 6 more
I am currently not sure if these failures here are related to:
-
the initContainer missing (probably not?)
-
the fsGroup (could be)
-
the underlying filesystem used for the persistent volumes. My openshift cluster came with both NFS and glusterfs. tried both with same errors, but I do remember recommendations to NOT run elasticsearch on glusterfs a couple of years ago. Perhaps this still holds (and also goes for nfs…?)
I have the same problem. Did you solve it?