Skip to content

Instantly share code, notes, and snippets.

@gsaslis

gsaslis/elasticsearch_openshift.asciidoc

Last active October 22, 2019 12:30
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save gsaslis/b12c57a9b34ced5bff0d1ed1c3e20d35 to your computer and use it in GitHub Desktop.
Save gsaslis/b12c57a9b34ced5bff0d1ed1c3e20d35 to your computer and use it in GitHub Desktop.
Download ZIP
Notes on how to deploy Elasticsearch on OpenShift
Raw
elasticsearch_openshift.asciidoc

Deploying Elasticsearch on Openshift

Notes to self. Or if anyone can help with this.

1. Helm charts ⇒ Openshift

Use Elastic’s helm charts as basis (couldn’t find anything more suitable - e.g. operator or Openshift Templates ):

I couldn’t use Helm itself due to permissions issues related to: helm/helm#1918

git clone https://github.com/elastic/helm-charts/
cd helm-charts
helm template elasticsearch -n elastic-stack > elasticsearch.yaml // (1)
oc apply -f elasticsearch.yaml

  1. The helm template command basically substitutes all values from values.yaml and creates a yaml with the actual kubernetes descriptors that can be deployed into the openshift cluster.

2. Deploy failures

Attempting to deploy this failed with:

create Pod elasticsearch-master-0 in StatefulSet elasticsearch-master failed error: pods "elasticsearch-master-0" is forbidden: unable to validate against any security context constraint: [fsGroup: Invalid value: []int64{1000}: 1000 is not an allowed group spec.initContainers[0].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1002210000, 1002219999] spec.initContainers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

so, combination of things:

  1. the initContainer tries to run as root. deleting the initContainer obviously solves that problem, but, of course it didn’t do its job.

  2. the fsGroup seems to have an invalid value. I am not sure what that should be. Deleting the fsGroup + securityContext does allow statefulset to be created, BUT, again, that’s probably there for a reason (haven’t looked into that in detail yet)

Deploying…​ something

If I do delete / change the above, I then do get a pod up and running, but then I get a bunch of exceptions like:

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.5.3.jar:6.5.3]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.5.3.jar:6.5.3]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:297) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.3.jar:6.5.3]
	... 6 more
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
	at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:215) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.3.jar:6.5.3]
	... 6 more
Caused by: java.io.IOException: Input/output error
	at sun.nio.ch.FileDispatcherImpl.lock0(Native Method) ~[?:?]
	at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:96) ~[?:?]
	at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1167) ~[?:?]
	at java.nio.channels.FileChannel.tryLock(FileChannel.java:1165) ~[?:?]
	at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:126) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
	at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
	at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
	at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:208) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.5.3.jar:6.5.3]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.3.jar:6.5.3]
	... 6 more

I am currently not sure if these failures here are related to:

  1. the initContainer missing (probably not?)

  2. the fsGroup (could be)

  3. the underlying filesystem used for the persistent volumes. My openshift cluster came with both NFS and glusterfs. tried both with same errors, but I do remember recommendations to NOT run elasticsearch on glusterfs a couple of years ago. Perhaps this still holds (and also goes for nfs…​?)

@VanilaBear
Copy link

I have the same problem. Did you solve it?

@gsaslis
Copy link
Author

gsaslis commented Oct 22, 2019

@VanilaBear I don't think I ever did, no.

But I remember I was pretty sure at the time that this was down to (3) the underlying filesystem.... What environment are you deploying in?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment