Develop locally with Elasticsearch on OSX using Docker

elasticsearch-local-development-with-docker-osx.md

Develop locally with Elasticsearch on OSX using Docker

Docker

Docker does not run natively on OSX, only Linux. Docker Machine was created to add a Linux VM environment to run Docker containers on OSX. Install using Homebrew:

brew install docker
brew install docker-machine
docker-machine create

There are a few more steps to using Docker Machine, read the Getting Started guide.

Installing Elasticsearch

There is an "official" Docker image for Elasticsearch that gives us the basics to get us started. Since this image will be used for development, I recommend installing a couple useful plugins as well.

First, set up a working directory to mount as a data volume that you can access from OSX.

mkdir ~/Projects/es-docker

We're then going to create temporary containers to mount the volume and install our Elasticsearch plugins. There are a few other ways to handle this, but this seems to be the cleanest to me.

# Install Head (http://mobz.github.io/elasticsearch-head/)
docker run -v /Users/ben/Projects/es-docker/data:/data elasticsearch /usr/share/elasticsearch/bin/plugin -i mobz/elasticsearch-head

# Install Marvel (http://www.elasticsearch.org/overview/marvel/)
docker run -v /Users/ben/Projects/es-docker/data:/data elasticsearch /usr/share/elasticsearch/bin/plugin -i elasticsearch/marvel/latest

Finally, run the ES docker image, pointing to our data volume. This forwards ports 9200 and 9300, mounts our volume (containing our plugins), and provides a convenient name to reference the instance, es.

docker run \
    --name=es -d \
    -p 9200:9200 -p 9300:9300 \
    -v /Users/ben/Projects/es-docker/data:/data \
    elasticsearch

Alternatively, if using Docker Compose (highly recommended), the following docker-compose.yml file will get you going:

es:
  image: elasticsearch
  ports:
   - "9200:9200"
   - "9300:9300"
  volumes:
   - ./data:/usr/share/elasticsearch/data

Run docker-compose up -d to launch the ES instance.

Accessing

Since we're running our Docker container on a VM on OSX, we cannot access the server at localhost. Instead, we need to find the IP of our VM.

docker-machine ip

In my case, my VM's IP is 192.168.99.100, making Elasticsearch accessible at,

• http://192.168.99.100:9200/
• http://192.168.99.100:9200/_plugin/marvel
• http://192.168.99.100:9200/_plugin/head

Security

I shut my ES node down when I'm not actively using it. Any script can issue commands to your local instance, which makes you vulnerable. Since we are running in a VM, the potential impact is more limited, depending on what you've got in your mounted data volume(s).

docker stop es

From the tutorial you posted, might be helpful to use a alias for the docker IP:

echo $(docker-machine ip) dockerhost | sudo tee -a /etc/hosts

or

echo $(docker-machine ip) dockerhost | sudo tee -a /etc/hosts

if you use the bash script he mentioned in that article.

I can't get it works with your compose example :

elk_logstash_data_1 exited with code 0
es_1            | [2016-02-27 23:41:12,025][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: your kernel is buggy and you should upgrade
es_1            | Exception in thread "main" java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data/elasticsearch)

Any idea why ?

The same issue with @rivetmichael, but I use the command:

docker run -p 9200:9200 -v `pwd`/es_data:/usr/share/elasticsearch/data elasticsearch:2.1.1

[2016-05-24 15:09:44,310][WARN ][bootstrap] unable to install syscall filter: seccomp unavailable: your kernel is buggy and you should upgrade

macOS El'capitan

For folks that have access issues, there is a lot of info in the comments section of the ES image page on Docker Hub. Issues stem from the ES user running in the container does not have the same access permissions as your OSX user on the host. There are some workarounds, I've had success with the PID trickery mentioned in the comments.

To get around this temporarily, you can run ES w/o any shared data volumes, but then you will lose your ES data if the container volume is destroyed.