multi-node elasticsearch cluster

docker-compose.yml

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic
  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.12.0
    container_name: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200","http://es03:9200"]'
    networks:
      - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge

This sample Docker Compose file brings up a three-node Elasticsearch cluster with Kibana along. Node es01 listens on localhost:9200 and es02 and es03 talk to es01 over a Docker network.

Please note that this configuration exposes port 9200 on all network interfaces, and given how Docker manipulates iptables on Linux, this means that your Elasticsearch cluster is publically accessible, potentially ignoring any firewall settings. If you don’t want to expose port 9200 and instead use a reverse proxy, replace 9200:9200 with 127.0.0.1:9200:9200 in the docker-compose.yml file. Elasticsearch will then only be accessible from the host machine itself.

The Docker named volumes data01, data02, and data03 store the node data directories so the data persists across restarts. If they don’t already exist, docker-compose creates them when you bring up the cluster.

After running $ docker-compose up to initiate the bootstrapping of the containers (es01, es02, es03 and kib01)

Access kibana through https://localhost:5601

Try to checking the health from the cluster by running: $ curl -XGET http://elasticsearch:9200/_cluster/health?pretty

Important: when running this on your machine and you plan on mimicking certain load, you might want to set the following variable: vm.max_map_count. Setting this variable depends on the host machine.

Linux
Setting the variable temporarily: sudo sysctl -w vm.max_map_count=262144
Setting the variable permanently in /etc/sysctl.conf:

grep vm.max_map_count /etc/sysctl.conf
vm.max_map_count=262144

macOS with Docker for Mac
The vm.max_map_count setting must be set within the xhyve virtual machine:

From the command line, run:

screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty

Press enter and use sysctl to configure vm.max_map_count:

sysctl -w vm.max_map_count=262144

To exit the screen session, type: Ctrl a d

Windows and macOS with Docker Desktop
The vm.max_map_count setting must be set via docker-machine:

docker-machine ssh
sudo sysctl -w vm.max_map_count=262144

Windows with Docker Desktop WSL 2 backend
The vm.max_map_count setting must be set in the docker-desktop container:

wsl -d docker-desktop
sysctl -w vm.max_map_count=262144

if I lost data with a restart or something happened in the container how i can retrieve the data ?