Elasticsearch 6.x install for Centmin Mod LEMP stacks for purpose of use with Xenforo forums.
- https://www.elastic.co/support/matrix#matrix_jvm
- https://www.elastic.co/guide/en/elasticsearch/reference/6.6/install-elasticsearch.html
- https://www.elastic.co/guide/en/elasticsearch/reference/6.6/settings.html
- https://www.elastic.co/guide/en/elasticsearch/reference/6.6/important-settings.html
- https://www.elastic.co/guide/en/elasticsearch/reference/6.6/system-config.html
- https://www.elastic.co/guide/en/elasticsearch/reference/current/query-cache.html
- https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-request-cache.html
- https://www.elastic.co/guide/en/elasticsearch/reference/6.6/rpm.html
- https://www.elastic.co/guide/en/elasticsearch/reference/6.6/executable-jna-tmpdir.html
- https://www.elastic.co/guide/en/elasticsearch/reference/6.6/starting-elasticsearch.html
- https://www.elastic.co/guide/en/elasticsearch/reference/6.6/reindex-upgrade.html
- https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html
- https://discuss.elastic.co/t/where-can-i-find-info-on-the-status-of-g1gc-in-lucene-and-es/149526
- elastic/elasticsearch#33685 (https://writequit.org/org/es/g1gc-testing.html#h:20efff85-5479-4a94-99b2-32557ecabb2f)
- https://dzone.com/articles/easy-change-to-reduce-memory-by-20
- https://medium.com/naukri-engineering/garbage-collection-in-elasticsearch-and-the-g1gc-16b79a447181
- openvz systems https://stackoverflow.com/questions/50314586/elasticsearch-fails-to-start-config-seccomp-and-config-seccomp-filter-are-neede
- https://community.centminmod.com/threads/how-to-install-elasticsearch-6-x-for-xenforo.13377/
Minimum version of XenForo Enhanced Search you have installed for elasticsearch 6 is 1.1.8
cat > "/etc/yum.repos.d/elasticsearch.repo" <<EOF
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
cat /etc/yum.repos.d/elasticsearch.repo
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
yum -y install java-1.8.0-openjdk
java -version
yum -y remove elasticsearch
yum -y install elasticsearch
/usr/share/elasticsearch/bin/elasticsearch --version
chkconfig elasticsearch on
mkdir -p /etc/elasticsearch/scripts/
cp -a /path/to/xenforoinstall/library/XenES/_scripts/*.groovy /etc/elasticsearch/scripts/
chown elasticsearch: /var/log/elasticsearch/gc.log*
ls -lah /var/log/elasticsearch
service elasticsearch start
service elasticsearch status
journalctl -u elasticsearch --no-pager --since "today" --no-pager
java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
/usr/share/elasticsearch/bin/elasticsearch --version
Version: 6.6.0, Build: default/rpm/a9861f4/2019-01-24T11:27:09.439740Z, JVM: 1.8.0_191
for CentOS 7 in https://www.elastic.co/guide/en/elasticsearch/reference/6.6/setting-system-settings.html#systemd
systemctl edit elasticsearch
or edit /etc/systemd/system/elasticsearch.service.d/override.conf
mkdir -p /etc/systemd/system/elasticsearch.service.d
touch /etc/systemd/system/elasticsearch.service.d/override.conf
echo "[Service]" >> /etc/systemd/system/elasticsearch.service.d/override.conf
echo "LimitMEMLOCK=infinity" >> /etc/systemd/system/elasticsearch.service.d/override.conf
echo "LimitNOFILE=262144" >> /etc/systemd/system/elasticsearch.service.d/override.conf
echo "LimitNPROC=16384" >> /etc/systemd/system/elasticsearch.service.d/override.conf
[Service]
LimitMEMLOCK=infinity
LimitNOFILE=262144
LimitNPROC=16384
systemctl daemon-reload
in /etc/elasticsearch/elasticsearch.yml
- Adjust disk watermark levels appropriately for your system enivronment. So if on VPS with one mount / where /var/lib/elasticsearch would reside, you'd have to be careful not to set percentages or abolute disk size too high to impact other server apps competing for disk space in mount /
# custom settings
# cluster.name is same name added in XenForo field in Admin Control Panel in field Elasticsearch Index Name
cluster.name: xfes
node.name: xfnode
network.host: localhost
http.port: 9200
index.codec: best_compression
# openvz only
#bootstrap.system_call_filter: false
# https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html
cluster.routing.allocation.disk.threshold_enabled: true
# defaults to 85%
cluster.routing.allocation.disk.watermark.low: 90%
# defaults to 90%
cluster.routing.allocation.disk.watermark.high: 95%
# defaults to 95% meaning that Elasticsearch enforces a read-only index block
# (index.blocks.read_only_allow_delete) on every index that has one or more
# shards allocated on the node that has at least one disk exceeding the flood stage.
# This is a last resort to prevent nodes from running out of disk space.
# The index block must be released manually once there is enough disk space available to allow indexing operations to continue
cluster.routing.allocation.disk.watermark.flood_stage: 96%
cluster.info.update.interval: 1m
# index.blocks.read_only_allow_delete: false
# https://www.elastic.co/guide/en/elasticsearch/reference/current/indexing-buffer.html
indices.memory.index_buffer_size: 10%
indices.memory.min_index_buffer_size: 48mb
# leave unset for max
# indices.memory.max_index_buffer_size:
# https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-fielddata.html
indices.fielddata.cache.size: 20%
# https://www.elastic.co/guide/en/elasticsearch/reference/current/query-cache.html
indices.queries.cache.size: 10%
# https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-request-cache.html
indices.requests.cache.size: 2%
echo "# https://www.elastic.co/guide/en/elasticsearch/reference/current/shards-allocation.html" >> /etc/elasticsearch/elasticsearch.yml
echo "indices.recovery.max_bytes_per_sec: 40mb" >> /etc/elasticsearch/elasticsearch.yml
echo "cluster.routing.allocation.cluster_concurrent_rebalance: $(nproc)" >> /etc/elasticsearch/elasticsearch.yml
echo "cluster.routing.allocation.node_concurrent_incoming_recoveries: $(nproc)" >> /etc/elasticsearch/elasticsearch.yml
echo "cluster.routing.allocation.node_concurrent_outgoing_recoveries: $(nproc)" >> /etc/elasticsearch/elasticsearch.yml
echo "cluster.routing.allocation.node_initial_primaries_recoveries: $(echo "$(nproc)*2" | bc)" >> /etc/elasticsearch/elasticsearch.yml
force merge after changing compression methods
curl -sX POST "localhost:9200/xfes/_forcemerge?pretty"
{
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
}
}
set to zero replicas
curl -XDELETE 'http://localhost:9200/_template/default'
curl -XPUT 'http://localhost:9200/_template/default' -H 'Content-Type: application/json' -d'{
"template": "*",
"settings": {
"number_of_replicas": "0"
}
}'
curl -XPUT 'http://localhost:9200/_settings' -H 'Content-Type: application/json' -d '{
"index" : { "number_of_replicas" : "0" }
}'
enabling sharded request cache on index
curl -sX PUT "localhost:9200/xfes/_settings" -H 'Content-Type: application/json' -d' { "index.requests.cache.enable": true }'
curl -sX GET "localhost:9200/_settings?pretty"
{
"xfes" : {
"settings" : {
"index" : {
"number_of_shards" : "5",
"provided_name" : "xfes",
"creation_date" : "1550395180529",
"requests" : {
"cache" : {
"enable" : "true"
}
},
"analysis" : {
"filter" : {
"xf_stop" : {
"type" : "stop",
"stopwords" : "_english_"
},
"xf_stemmer" : {
"type" : "stemmer",
"language" : "porter2"
}
},
"analyzer" : {
"default" : {
"filter" : [
"lowercase",
"xf_stop",
"xf_stemmer"
],
"type" : "custom",
"tokenizer" : "standard"
}
}
},
"number_of_replicas" : "0",
"uuid" : "EwT19YdxQ5W1QM8xk9QuIA",
"version" : {
"created" : "6060099"
}
}
}
}
}
curl -sX GET "localhost:9200/_nodes/stats/indices/request_cache?human" | jq
{
"_nodes": {
"total": 1,
"successful": 1,
"failed": 0
},
"cluster_name": "xfes",
"nodes": {
"9ur7hG-hQ4O5HrgoRayYXg": {
"timestamp": 1550406323660,
"name": "xfnode",
"transport_address": "127.0.0.1:9300",
"host": "localhost",
"ip": "127.0.0.1:9300",
"roles": [
"master",
"data",
"ingest"
],
"attributes": {
"ml.machine_memory": "2147483648",
"xpack.installed": "true",
"ml.max_open_jobs": "20",
"ml.enabled": "true"
},
"indices": {
"request_cache": {
"memory_size": "0b",
"memory_size_in_bytes": 0,
"evictions": 0,
"hit_count": 0,
"miss_count": 0
}
}
}
}
}
thread_pool size
curl -sX GET "localhost:9200/_cat/thread_pool?h=host,name,min,max,queue,queue_size&v&s=name"
host name min max queue queue_size
localhost analyze 1 1 0 16
localhost ccr 32 32 0 100
localhost fetch_shard_started 1 4 0 -1
localhost fetch_shard_store 1 4 0 -1
localhost flush 1 1 0 -1
localhost force_merge 1 1 0 -1
localhost generic 4 128 0 -1
localhost get 2 2 0 1000
localhost index 2 2 0 200
localhost listener 1 1 0 -1
localhost management 1 5 0 -1
localhost ml_autodetect 80 80 0 80
localhost ml_datafeed 20 20 0 200
localhost ml_utility 80 80 0 500
localhost refresh 1 1 0 -1
localhost rollup_indexing 4 4 0 4
localhost search 4 4 0 1000
localhost search_throttled 1 1 0 100
localhost security-token-key 1 1 0 1000
localhost snapshot 1 1 0 -1
localhost warmer 1 1 0 -1
localhost watcher 10 10 0 1000
localhost write 2 2 0 200
tail -100 /var/log/elasticsearch/xfes.log
from log
[xfnode] JVM arguments [-Xms512m, -Xmx512m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/lib/elasticsearch/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm]
and /etc/sysconfig/elasticsearch optional enable bootstrap checks https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
change from
# Additional Java OPTS
#ES_JAVA_OPTS=
to
# Additional Java OPTS
ES_JAVA_OPTS="-Des.enforce.bootstrap.checks=true"
however, if bootstrap checks fail, elasticsearch will fail to start or crash
[2019-02-19T10:06:00,317][INFO ][o.e.b.BootstrapChecks ] [xfnode] explicitly enforcing bootstrap checks
[2019-02-19T10:06:00,355][ERROR][o.e.b.Bootstrap ] [xfnode] node validation exception
[1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-02-19T10:06:00,358][INFO ][o.e.n.Node ] [xfnode] stopping ...
[2019-02-19T10:06:00,382][INFO ][o.e.n.Node ] [xfnode] stopped
[2019-02-19T10:06:00,383][INFO ][o.e.n.Node ] [xfnode] closing ...
[2019-02-19T10:06:00,412][INFO ][o.e.n.Node ] [xfnode] closed
[2019-02-19T10:06:00,419][INFO ][o.e.x.m.p.NativeController] [xfnode] Native controller process has stopped - no new native processes can be started
and /etc/elasticsearch/jvm.options
-Xms512m
-Xmx512m
optional experimental change from
## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
to
## GC configuration
-XX:-UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseG1GC
-XX:+UseStringDeduplication
-XX:MaxGCPauseMillis=300
-XX:InitiatingHeapOccupancyPercent=75
curl -sX GET "localhost:9200/?pretty"
curl -sX GET "localhost:9200/_all/_settings?pretty"
curl -sX GET "localhost:9200/_cat/health?pretty"
curl -sX GET "localhost:9200/_cat/recovery?pretty"
curl -sX GET "localhost:9200/_nodes/stats?pretty"
curl -sX GET "localhost:9200/_nodes/stats/os?pretty"
curl -sX GET "localhost:9200/_nodes/stats/fs?pretty"
curl -sX GET "localhost:9200/_stats/request_cache?human" | jq
curl -sX GET http://localhost:9200/_cat/indices?v
curl -sX GET http://localhost:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open xfes 2pm6c0apT72xP3JoaYBtsw 5 0 52 0 63.8kb 63.8kb
curl -sX GET "localhost:9200/?pretty"
{
"name" : "xfnode",
"cluster_name" : "xfes",
"cluster_uuid" : "xf8iXzQOReioMeq0p1akdA",
"version" : {
"number" : "6.6.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "a9861f4",
"build_date" : "2019-01-24T11:27:09.439740Z",
"build_snapshot" : false,
"lucene_version" : "7.6.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
curl -sX GET "localhost:9200/_nodes/stats/os?pretty"
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "xfes",
"nodes" : {
"9ur7hG-hQ4O5HrgoRayYXg" : {
"timestamp" : 1550398414182,
"name" : "xfnode",
"transport_address" : "127.0.0.1:9300",
"host" : "localhost",
"ip" : "127.0.0.1:9300",
"roles" : [
"master",
"data",
"ingest"
],
"attributes" : {
"ml.machine_memory" : "2147483648",
"xpack.installed" : "true",
"ml.max_open_jobs" : "20",
"ml.enabled" : "true"
},
"os" : {
"timestamp" : 1550398414182,
"cpu" : {
"percent" : 26,
"load_average" : {
"1m" : 0.83,
"5m" : 0.67,
"15m" : 0.41
}
},
"mem" : {
"total_in_bytes" : 2147483648,
"free_in_bytes" : 298053632,
"used_in_bytes" : 1849430016,
"free_percent" : 14,
"used_percent" : 86
},
"swap" : {
"total_in_bytes" : 2147483648,
"free_in_bytes" : 2102550528,
"used_in_bytes" : 44933120
}
}
}
}
}
curl -sX GET "localhost:9200/_nodes/stats/fs?pretty"
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "xfes",
"nodes" : {
"9ur7hG-hQ4O5HrgoRayYXg" : {
"timestamp" : 1550398611425,
"name" : "xfnode",
"transport_address" : "127.0.0.1:9300",
"host" : "localhost",
"ip" : "127.0.0.1:9300",
"roles" : [
"master",
"data",
"ingest"
],
"attributes" : {
"ml.machine_memory" : "2147483648",
"xpack.installed" : "true",
"ml.max_open_jobs" : "20",
"ml.enabled" : "true"
},
"fs" : {
"timestamp" : 1550398611425,
"total" : {
"total_in_bytes" : 52708372480,
"free_in_bytes" : 45308841984,
"available_in_bytes" : 42624593920
},
"least_usage_estimate" : {
"path" : "/var/lib/elasticsearch/nodes/0",
"total_in_bytes" : 52708372480,
"available_in_bytes" : 42624602112,
"used_disk_percent" : 19.131249730441297
},
"most_usage_estimate" : {
"path" : "/var/lib/elasticsearch/nodes/0",
"total_in_bytes" : 52708372480,
"available_in_bytes" : 42624602112,
"used_disk_percent" : 19.131249730441297
},
"data" : [
{
"path" : "/var/lib/elasticsearch/nodes/0",
"mount" : "/ (/dev/ploop25571p1)",
"type" : "ext4",
"total_in_bytes" : 52708372480,
"free_in_bytes" : 45308841984,
"available_in_bytes" : 42624593920
}
]
}
}
}
}
curl -sX GET "localhost:9200/_stats/request_cache?human" | jq
{
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"_all": {
"primaries": {
"request_cache": {
"memory_size": "0b",
"memory_size_in_bytes": 0,
"evictions": 0,
"hit_count": 0,
"miss_count": 0
}
},
"total": {
"request_cache": {
"memory_size": "0b",
"memory_size_in_bytes": 0,
"evictions": 0,
"hit_count": 0,
"miss_count": 0
}
}
},
"indices": {
"xfes": {
"uuid": "EwT19YdxQ5W1QM8xk9QuIA",
"primaries": {
"request_cache": {
"memory_size": "0b",
"memory_size_in_bytes": 0,
"evictions": 0,
"hit_count": 0,
"miss_count": 0
}
},
"total": {
"request_cache": {
"memory_size": "0b",
"memory_size_in_bytes": 0,
"evictions": 0,
"hit_count": 0,
"miss_count": 0
}
}
}
}
}
curl -sX GET "localhost:9200/_cat/health?pretty"
1550386546 06:55:46 xfes green 1 1 0 0 0 0 0 0 - 100.0%
systemctl daemon-reload; systemctl restart elasticsearch; systemctl status elasticsearch; tail -100 /var/log/elasticsearch/xfes.log | egrep 'NodeEnvironment|JVM arguments'
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─override.conf
Active: active (running) since Sun 2019-02-17 06:30:22 UTC; 77ms ago
Docs: http://www.elastic.co
Main PID: 32227 (elasticsearch)
CGroup: /system.slice/elasticsearch.service
├─32227 /bin/bash /usr/share/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
└─32237 /bin/java -cp /usr/share/elasticsearch/lib/* org.elasticsearch.tools.java_version_checker.JavaVersionChecker
Feb 17 06:30:22 test.com systemd[1]: Started Elasticsearch.
[2019-02-17T06:29:14,055][INFO ][o.e.e.NodeEnvironment ] [C7zCcS1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [43.1gb], net total_space [59gb], types [rootfs]
[2019-02-17T06:29:14,059][INFO ][o.e.e.NodeEnvironment ] [C7zCcS1] heap size [495.3mb], compressed ordinary object pointers [true]
systemctl daemon-reload; systemctl restart elasticsearch; systemctl status elasticsearch; tail -100 /var/log/elasticsearch/xfes.log | egrep 'NodeEnvironment|JVM arguments'
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─override.conf
Active: active (running) since Sun 2019-02-17 08:08:55 UTC; 78ms ago
Docs: http://www.elastic.co
Main PID: 6817 (elasticsearch)
CGroup: /system.slice/elasticsearch.service
├─6817 /bin/bash /usr/share/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
└─6827 /bin/java -cp /usr/share/elasticsearch/lib/* org.elasticsearch.tools.java_version_checker.JavaVersionChecker
Feb 17 08:08:55 test.com systemd[1]: Started Elasticsearch.
[2019-02-17T08:08:05,834][INFO ][o.e.e.NodeEnvironment ] [xfnode] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [43.1gb], net total_space [59gb], types [rootfs]
[2019-02-17T08:08:05,837][INFO ][o.e.e.NodeEnvironment ] [xfnode] heap size [512mb], compressed ordinary object pointers [true]
[2019-02-17T08:08:05,840][INFO ][o.e.n.Node ] [xfnode] JVM arguments [-Xms512m, -Xmx512m, -XX:-UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseG1GC, -XX:+UseStringDeduplication, -XX:MaxGCPauseMillis=300, -XX:InitiatingHeapOccupancyPercent=75, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/lib/elasticsearch/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm]
df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 ext4 60G 13G 44G 24% /
devtmpfs devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs tmpfs 1.9G 17M 1.9G 1% /run
tmpfs tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/loop0 ext4 5.8G 25M 5.5G 1% /tmp
tmpfs tmpfs 379M 0 379M 0% /run/user/0
cd /path/to/xenforo/install
php cmd.php xf-rebuild:search
php cmd.php xf-rebuild:search
Rebuilding... Search index
Rebuilding... Search index (Post 46)
Rebuilding... Search index
Rebuilding... Search index
Rebuilding... Search index
Rebuilding... Search index (Thread 43)
Rebuilding... Search index
curl -sX GET "localhost:9200/_settings/?pretty"
{
"xfes" : {
"settings" : {
"index" : {
"number_of_shards" : "5",
"provided_name" : "xfes",
"creation_date" : "1550395180529",
"analysis" : {
"filter" : {
"xf_stemmer" : {
"type" : "stemmer",
"language" : "porter2"
}
},
"analyzer" : {
"default" : {
"filter" : [
"lowercase",
"xf_stemmer"
],
"type" : "custom",
"tokenizer" : "standard"
}
}
},
"number_of_replicas" : "0",
"uuid" : "EwT19YdxQ5W1QM8xk9QuIA",
"version" : {
"created" : "6060099"
}
}
}
}
}
curl -sX GET "localhost:9200/_stats/?pretty"
{
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 46,
"deleted" : 2
},
"store" : {
"size_in_bytes" : 61350
},
"indexing" : {
"index_total" : 89,
"index_time_in_millis" : 778,
"index_current" : 0,
"index_failed" : 0,
"delete_total" : 0,
"delete_time_in_millis" : 0,
"delete_current" : 0,
"noop_update_total" : 0,
"is_throttled" : false,
"throttle_time_in_millis" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 5,
"query_time_in_millis" : 247,
"query_current" : 0,
"fetch_total" : 2,
"fetch_time_in_millis" : 130,
"fetch_current" : 0,
"scroll_total" : 0,
"scroll_time_in_millis" : 0,
"scroll_current" : 0,
"suggest_total" : 0,
"suggest_time_in_millis" : 0,
"suggest_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 3,
"total_time_in_millis" : 855,
"total_docs" : 47,
"total_size_in_bytes" : 65475,
"total_stopped_time_in_millis" : 0,
"total_throttled_time_in_millis" : 0,
"total_auto_throttle_in_bytes" : 104857600
},
"refresh" : {
"total" : 24,
"total_time_in_millis" : 851,
"listeners" : 0
},
"flush" : {
"total" : 0,
"periodic" : 0,
"total_time_in_millis" : 0
},
"warmer" : {
"current" : 0,
"total" : 19,
"total_time_in_millis" : 8
},
"query_cache" : {
"memory_size_in_bytes" : 0,
"total_count" : 0,
"hit_count" : 0,
"miss_count" : 0,
"cache_size" : 0,
"cache_count" : 0,
"evictions" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 6,
"memory_in_bytes" : 10249,
"terms_memory_in_bytes" : 5841,
"stored_fields_memory_in_bytes" : 1872,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 768,
"points_memory_in_bytes" : 64,
"doc_values_memory_in_bytes" : 1704,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 0,
"max_unsafe_auto_id_timestamp" : -1,
"file_sizes" : { }
},
"translog" : {
"operations" : 89,
"size_in_bytes" : 78112,
"uncommitted_operations" : 89,
"uncommitted_size_in_bytes" : 78112,
"earliest_last_modified_age" : 0
},
"request_cache" : {
"memory_size_in_bytes" : 0,
"evictions" : 0,
"hit_count" : 0,
"miss_count" : 0
},
"recovery" : {
"current_as_source" : 0,
"current_as_target" : 0,
"throttle_time_in_millis" : 0
}
},
"total" : {
"docs" : {
"count" : 46,
"deleted" : 2
},
"store" : {
"size_in_bytes" : 61350
},
"indexing" : {
"index_total" : 89,
"index_time_in_millis" : 778,
"index_current" : 0,
"index_failed" : 0,
"delete_total" : 0,
"delete_time_in_millis" : 0,
"delete_current" : 0,
"noop_update_total" : 0,
"is_throttled" : false,
"throttle_time_in_millis" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 5,
"query_time_in_millis" : 247,
"query_current" : 0,
"fetch_total" : 2,
"fetch_time_in_millis" : 130,
"fetch_current" : 0,
"scroll_total" : 0,
"scroll_time_in_millis" : 0,
"scroll_current" : 0,
"suggest_total" : 0,
"suggest_time_in_millis" : 0,
"suggest_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 3,
"total_time_in_millis" : 855,
"total_docs" : 47,
"total_size_in_bytes" : 65475,
"total_stopped_time_in_millis" : 0,
"total_throttled_time_in_millis" : 0,
"total_auto_throttle_in_bytes" : 104857600
},
"refresh" : {
"total" : 24,
"total_time_in_millis" : 851,
"listeners" : 0
},
"flush" : {
"total" : 0,
"periodic" : 0,
"total_time_in_millis" : 0
},
"warmer" : {
"current" : 0,
"total" : 19,
"total_time_in_millis" : 8
},
"query_cache" : {
"memory_size_in_bytes" : 0,
"total_count" : 0,
"hit_count" : 0,
"miss_count" : 0,
"cache_size" : 0,
"cache_count" : 0,
"evictions" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 6,
"memory_in_bytes" : 10249,
"terms_memory_in_bytes" : 5841,
"stored_fields_memory_in_bytes" : 1872,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 768,
"points_memory_in_bytes" : 64,
"doc_values_memory_in_bytes" : 1704,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 0,
"max_unsafe_auto_id_timestamp" : -1,
"file_sizes" : { }
},
"translog" : {
"operations" : 89,
"size_in_bytes" : 78112,
"uncommitted_operations" : 89,
"uncommitted_size_in_bytes" : 78112,
"earliest_last_modified_age" : 0
},
"request_cache" : {
"memory_size_in_bytes" : 0,
"evictions" : 0,
"hit_count" : 0,
"miss_count" : 0
},
"recovery" : {
"current_as_source" : 0,
"current_as_target" : 0,
"throttle_time_in_millis" : 0
}
}
},
"indices" : {
"xfes" : {
"uuid" : "EwT19YdxQ5W1QM8xk9QuIA",
"primaries" : {
"docs" : {
"count" : 46,
"deleted" : 2
},
"store" : {
"size_in_bytes" : 61350
},
"indexing" : {
"index_total" : 89,
"index_time_in_millis" : 778,
"index_current" : 0,
"index_failed" : 0,
"delete_total" : 0,
"delete_time_in_millis" : 0,
"delete_current" : 0,
"noop_update_total" : 0,
"is_throttled" : false,
"throttle_time_in_millis" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 5,
"query_time_in_millis" : 247,
"query_current" : 0,
"fetch_total" : 2,
"fetch_time_in_millis" : 130,
"fetch_current" : 0,
"scroll_total" : 0,
"scroll_time_in_millis" : 0,
"scroll_current" : 0,
"suggest_total" : 0,
"suggest_time_in_millis" : 0,
"suggest_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 3,
"total_time_in_millis" : 855,
"total_docs" : 47,
"total_size_in_bytes" : 65475,
"total_stopped_time_in_millis" : 0,
"total_throttled_time_in_millis" : 0,
"total_auto_throttle_in_bytes" : 104857600
},
"refresh" : {
"total" : 24,
"total_time_in_millis" : 851,
"listeners" : 0
},
"flush" : {
"total" : 0,
"periodic" : 0,
"total_time_in_millis" : 0
},
"warmer" : {
"current" : 0,
"total" : 19,
"total_time_in_millis" : 8
},
"query_cache" : {
"memory_size_in_bytes" : 0,
"total_count" : 0,
"hit_count" : 0,
"miss_count" : 0,
"cache_size" : 0,
"cache_count" : 0,
"evictions" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 6,
"memory_in_bytes" : 10249,
"terms_memory_in_bytes" : 5841,
"stored_fields_memory_in_bytes" : 1872,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 768,
"points_memory_in_bytes" : 64,
"doc_values_memory_in_bytes" : 1704,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 0,
"max_unsafe_auto_id_timestamp" : -1,
"file_sizes" : { }
},
"translog" : {
"operations" : 89,
"size_in_bytes" : 78112,
"uncommitted_operations" : 89,
"uncommitted_size_in_bytes" : 78112,
"earliest_last_modified_age" : 0
},
"request_cache" : {
"memory_size_in_bytes" : 0,
"evictions" : 0,
"hit_count" : 0,
"miss_count" : 0
},
"recovery" : {
"current_as_source" : 0,
"current_as_target" : 0,
"throttle_time_in_millis" : 0
}
},
"total" : {
"docs" : {
"count" : 46,
"deleted" : 2
},
"store" : {
"size_in_bytes" : 61350
},
"indexing" : {
"index_total" : 89,
"index_time_in_millis" : 778,
"index_current" : 0,
"index_failed" : 0,
"delete_total" : 0,
"delete_time_in_millis" : 0,
"delete_current" : 0,
"noop_update_total" : 0,
"is_throttled" : false,
"throttle_time_in_millis" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 0,
"query_total" : 5,
"query_time_in_millis" : 247,
"query_current" : 0,
"fetch_total" : 2,
"fetch_time_in_millis" : 130,
"fetch_current" : 0,
"scroll_total" : 0,
"scroll_time_in_millis" : 0,
"scroll_current" : 0,
"suggest_total" : 0,
"suggest_time_in_millis" : 0,
"suggest_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 3,
"total_time_in_millis" : 855,
"total_docs" : 47,
"total_size_in_bytes" : 65475,
"total_stopped_time_in_millis" : 0,
"total_throttled_time_in_millis" : 0,
"total_auto_throttle_in_bytes" : 104857600
},
"refresh" : {
"total" : 24,
"total_time_in_millis" : 851,
"listeners" : 0
},
"flush" : {
"total" : 0,
"periodic" : 0,
"total_time_in_millis" : 0
},
"warmer" : {
"current" : 0,
"total" : 19,
"total_time_in_millis" : 8
},
"query_cache" : {
"memory_size_in_bytes" : 0,
"total_count" : 0,
"hit_count" : 0,
"miss_count" : 0,
"cache_size" : 0,
"cache_count" : 0,
"evictions" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 6,
"memory_in_bytes" : 10249,
"terms_memory_in_bytes" : 5841,
"stored_fields_memory_in_bytes" : 1872,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 768,
"points_memory_in_bytes" : 64,
"doc_values_memory_in_bytes" : 1704,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 0,
"max_unsafe_auto_id_timestamp" : -1,
"file_sizes" : { }
},
"translog" : {
"operations" : 89,
"size_in_bytes" : 78112,
"uncommitted_operations" : 89,
"uncommitted_size_in_bytes" : 78112,
"earliest_last_modified_age" : 0
},
"request_cache" : {
"memory_size_in_bytes" : 0,
"evictions" : 0,
"hit_count" : 0,
"miss_count" : 0
},
"recovery" : {
"current_as_source" : 0,
"current_as_target" : 0,
"throttle_time_in_millis" : 0
}
}
}
}
}
Centmin Mod mounts /tmp securely with noexec which may be problematic so below workaround tells elasticsearch to use a different tmp directory at /var/lib/elasticsearch/tmp
cat /etc/fstab | grep tmp
/home/usertmp_donotdelete /tmp ext4 loop,rw,noexec,nosuid 0 0
Find case insensitive matches to work tmpdir in directory /etc/elasticsearch and found the elasticsearch java config file reference on line 72 of /etc/elasticsearch/jvm.options
grep -rin tmpdir /etc/elasticsearch
/etc/elasticsearch/jvm.options:72:-Djava.io.tmpdir=${ES_TMPDIR}
/etc/elasticsearch/jvm.options.rpmnew:89:-Djava.io.tmpdir=${ES_TMPDIR}
so change location for elasticsearch's /tmp to /var/lib/elasticsearch/tmp by first
creating /var/lib/elasticsearch/tmp
mkdir -p /var/lib/elasticsearch/tmp
chown elasticsearch: /var/lib/elasticsearch/tmp
chmod +x /var/lib/elasticsearch/tmp
ls -lah /var/lib/elasticsearch
then edit line 72 of /etc/elasticsearch/jvm.options
from
-Djava.io.tmpdir=${ES_TMPDIR}
to
-Djava.io.tmpdir=/var/lib/elasticsearch/tmp
and restart elasticsearch
service elasticsearch restart
/etc/elasticsearch/jvm.options
## JVM configuration
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
#-Xms1g
#-Xmx1g
-Xms512m
-Xmx512m
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################
## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
## G1GC Configuration
# NOTE: G1GC is only supported on JDK version 10 or later.
# To use G1GC uncomment the lines below.
# 10-:-XX:-UseConcMarkSweepGC
# 10-:-XX:-UseCMSInitiatingOccupancyOnly
# 10-:-XX:+UseG1GC
# 10-:-XX:InitiatingHeapOccupancyPercent=75
## DNS cache policy
# cache ttl in seconds for positive DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.ttl; set to -1 to cache forever
-Des.networkaddress.cache.ttl=60
# cache ttl in seconds for negative DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.negative ttl; set to -1 to cache
# forever
-Des.networkaddress.cache.negative.ttl=10
## optimizations
# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch
## basic
# explicitly set the stack size
-Xss1m
# set to headless, just in case
-Djava.awt.headless=true
# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8
# use our provided JNA always versus the system one
-Djna.nosys=true
# turn off a JDK optimization that throws away stack traces for common
# exceptions because stack traces are important for debugging
-XX:-OmitStackTraceInFastThrow
# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
#-Djava.io.tmpdir=${ES_TMPDIR}
-Djava.io.tmpdir=/var/lib/elasticsearch/tmp
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch
# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log
## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
# due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
# time/date parsing will break in an incompatible way for some date patterns and locals
9-:-Djava.locale.providers=COMPAT
# temporary workaround for C2 bug with JDK 10 on hardware with AVX-512
10-:-XX:UseAVX=2
openvz systems
[2019-02-17T09:09:30,706][WARN ][o.e.b.JNANatives ] [xfnode] unable to install syscall filter:
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:342) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:617) ~[elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:260) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:108) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.6.0.jar:6.6.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.6.0.jar:6.6.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.6.0.jar:6.6.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.6.0.jar:6.6.0]
openvz systems https://stackoverflow.com/questions/50314586/elasticsearch-fails-to-start-config-seccomp-and-config-seccomp-filter-are-neede
bootstrap.system_call_filter: false
openvz systems don't have Kernel level permissions so you get error in logs as there is no way to set it
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
sysctl -w vm.max_map_count=262144
sysctl: permission denied on key 'vm.max_map_count'