Skip to content

Instantly share code, notes, and snippets.

@ruanbekker
Created December 27, 2019 17:24
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save ruanbekker/4e10ea7de6be40b5e385feb851dae544 to your computer and use it in GitHub Desktop.
Save ruanbekker/4e10ea7de6be40b5e385feb851dae544 to your computer and use it in GitHub Desktop.
Fix Elasticsearch issues in LXD/LXC containers

Errors like this appear when you start elasticsearch in a container:

$ journalctl -xe
Dec 27 16:50:17 elasticsearch-03 mount[62]: mount: /sys/kernel/config: cannot mount configfs read-only.
Dec 27 16:50:17 elasticsearch-03 systemd-sysctl[63]: Couldn't write '262144' to 'vm/max_map_count', ignoring: Read-only file system
Dec 27 16:50:17 elasticsearch-03 systemd-sysctl[63]: Couldn't write '1' to 'fs/protected_hardlinks', ignoring: Read-only file system
Dec 27 16:50:17 elasticsearch-03 systemd-sysctl[63]: Couldn't write '1' to 'fs/protected_symlinks', ignoring: Read-only file system

$ /etc/init.d/elasticsearch status
Dec 27 16:54:47 elasticsearch-03 systemd[168]: elasticsearch.service: Failed to set up mount namespacing: Permission denied
Dec 27 16:54:47 elasticsearch-03 systemd[168]: elasticsearch.service: Failed at step NAMESPACE spawning /usr/share/elasticsearch/bin/elasticsearch: Permission denied
Dec 27 16:54:47 elasticsearch-03 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=226/NAMESPACE
Dec 27 16:54:47 elasticsearch-03 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Set the kernel parameters on the host:

$ /etc/sysctl.conf
vm.max_map_count=262144
fs.protected_hardlinks=1
fs.protected_symlinks=1

Save:

$ sysctl -p

Stop the container and set a couple of configs (this step is not required, as we set ES_SKIP_SET_KERNEL_PARAMETERS=true:

$ lxc stop elasticsearch-03
$ lxc config set elasticsearch-03 security.privileged true
$ cat <<EOT | lxc config set elasticsearch-03 raw.lxc -
lxc.cgroup.devices.allow = a
lxc.cap.drop =
EOT
$ lxc start elasticsearch-03

Set the following values in the container:

$ cat /etc/systemd/system/multi-user.target.wants/elasticsearch.service
[Service]
...
PrivateTmp=false
NoNewPrivileges=yes
Environment=ES_SKIP_SET_KERNEL_PARAMETERS=true

Reload systemd:

$ systemctl daemon-reload

Restart elasticsearch:

$ systemctl restart elasticsearch
$ /etc/init.d/elasticsearch status
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-12-27 17:11:28 UTC; 1min 17s ago
     Docs: http://www.elastic.co
 Main PID: 617 (java)
    Tasks: 39 (limit: 4915)
   Memory: 1.2G
   CGroup: /system.slice/elasticsearch.service
           ├─617 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.net…
           └─693 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Dec 27 17:11:28 elasticsearch-03 systemd[1]: Started Elasticsearch.
Dec 27 17:11:29 elasticsearch-03 elasticsearch[617]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be remov…ture release.
Hint: Some lines were ellipsized, use -l to show in full.

Resource:

@dariuskoissi
Copy link

merci bcp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment