Skip to content

Instantly share code, notes, and snippets.

@arnisoph
Last active August 29, 2015 14:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save arnisoph/a0401e4cabec474970a3 to your computer and use it in GitHub Desktop.
Save arnisoph/a0401e4cabec474970a3 to your computer and use it in GitHub Desktop.
packer-vagrant-to-boostrap-a-test-infrastructure-pt3
$ mkdir demo-elasticsearch-vagrant
$ cd demo-elasticsearch-vagrant
$ git init
$ git submodule add https://github.com/bechtoldt/iac-vagrant vagrant
Cloning into 'vagrant'...
remote: Counting objects: 403, done.
remote: Total 403 (delta 0), reused 0 (delta 0), pack-reused 403
Receiving objects: 100% (403/403), 52.14 KiB | 0 bytes/s, done.
Resolving deltas: 100% (202/202), done.
Checking connectivity... done.
$ ls -las vagrant/
total 128
0 drwxr-xr-x 18 arnoldbechtoldt staff 612 Jul 26 20:20 .
0 drwxr-xr-x 5 arnoldbechtoldt staff 170 Jul 26 20:20 ..
8 -rw-r--r-- 1 arnoldbechtoldt staff 32 Jul 26 20:20 .git
8 -rw-r--r-- 1 arnoldbechtoldt staff 642 Jul 26 20:20 .gitignore
8 -rw-r--r-- 1 arnoldbechtoldt staff 91 Jul 26 20:20 .gitmodules
8 -rw-r--r-- 1 arnoldbechtoldt staff 44 Jul 26 20:20 AUTHORS
8 -rw-r--r-- 1 arnoldbechtoldt staff 702 Jul 26 20:20 CHANGELOG.rst
8 -rw-r--r-- 1 arnoldbechtoldt staff 588 Jul 26 20:20 LICENSE
8 -rw-r--r-- 1 arnoldbechtoldt staff 1215 Jul 26 20:20 README.rst
16 -rw-r--r-- 1 arnoldbechtoldt staff 6085 Jul 26 20:20 Vagrantfile
0 drwxr-xr-x 12 arnoldbechtoldt staff 408 Jul 26 20:21 assets
8 -rw-r--r-- 1 arnoldbechtoldt staff 1436 Jul 26 20:20 nodes.yaml.cloud.dist
8 -rw-r--r-- 1 arnoldbechtoldt staff 1131 Jul 26 20:20 nodes.yaml.common.dist
8 -rw-r--r-- 1 arnoldbechtoldt staff 344 Jul 26 20:20 nodes.yaml.common_atlas.dist
8 -rw-r--r-- 1 arnoldbechtoldt staff 396 Jul 26 20:20 nodes.yaml.puppet.dist
8 -rw-r--r-- 1 arnoldbechtoldt staff 360 Jul 26 20:20 nodes.yaml.rex.dist
8 -rw-r--r-- 1 arnoldbechtoldt staff 616 Jul 26 20:20 nodes.yaml.salt.dist
8 -rwxr-xr-x 1 arnoldbechtoldt staff 62 Jul 26 20:20 update-docs
$ wget https://raw.githubusercontent.com/bechtoldt/saltstack-skeleton-formula/master/.gitignore -O .gitignore
$ mkdir -p shared/{misc/,salt} shared/salt/{assets,devenv/elastic1/file_roots/} shared/salt/devenv/elastic1/file_roots/{pillar/,states/}
$ wget --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u51-b16/jdk-8u51-linux-x64.tar.gz -O shared/misc/jdk-8u51-linux-x64.tar.gz
$ wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz -O shared/misc/kibana-4.1.1-linux-x64.tar.gz
# wget https://github.com/bechtoldt/demo-elasticsearch-vagrant/blob/master/shared/misc/MOCK_DATA_bulk_modified.json -O /vagrant/shared/misc/MOCK_DATA_bulk.json
# curl -vs -XPOST http://127.0.0.1:9200/_bulk --data-binary @/vagrant/shared/misc/MOCK_DATA_bulk_modified.json
About to connect() to 127.0.0.1 port 9200 (#0)
* Trying 127.0.0.1...
* connected
* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)
> POST /_bulk HTTP/1.1
> User-Agent: curl/7.26.0
> Host: 127.0.0.1:9200
> Accept: */*
> Content-Length: 192343
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 100 Continue
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
< Content-Length: 108038
<
{"took":1536,"errors":false,"items":[{"create":{"_index":"sampledata","_type":"default","_id":"AU7MUe5bV-fzUB3rRNnm","_version":1,"status":201}},{"create":{"_index":"sampledata","_type":"default","_id":"AU7MUe5bV-fzUB3rRNnn","_version":1,"status":201}},{"create":{"_index":"sampledata","_type":"default","_id":"AU7MUe5bV-fzUB3rRNno","_version":1,"status":201}},{"create":{"_index":"sampledata","_type":"default","_id":"AU7MUe5bV-fzUB3rRNnp","_version":1,"status":201}},{"create":{"_index":"sampledata","_type":"default","_id":"AU7MUe5bV-fzUB3r
# /var/www/standard/current/bin/kibana
{"name":"Kibana","hostname":"elastic1","pid":13171,"level":30,"msg":"No existing kibana index found","time":"2015-07-26T21:11:03.786Z","v":0}
{"name":"Kibana","hostname":"elastic1","pid":13171,"level":30,"msg":"Listening on 0.0.0.0:5601","time":"2015-07-26T21:11:04.022Z","v":0}
^Z
[1]+ Stopped /var/www/standard/current/bin/kibana
root@elastic1:~# bg
[1]+ /var/www/standard/current/bin/kibana &
$ for repo in salt-modules saltstack-elasticsearch-formula saltstack-git-formula saltstack-java-formula saltstack-kibana-formula saltstack-repos-formula saltstack-salt-formula saltstack-sysctl-formula saltstack-tools-formula; do git submodule add "https://github.com/bechtoldt/${repo}.git" "shared/salt/assets/${repo}/"; done
$ ls -las shared/salt/assets
total 0
0 drwxr-xr-x 11 arnoldbechtoldt staff 374 Jul 26 20:39 .
0 drwxr-xr-x 4 arnoldbechtoldt staff 136 Jul 26 20:27 ..
0 drwxr-xr-x 9 arnoldbechtoldt staff 306 Jul 26 20:37 salt-modules
0 drwxr-xr-x 13 arnoldbechtoldt staff 442 Jul 26 20:38 saltstack-elasticsearch-formula
0 drwxr-xr-x 10 arnoldbechtoldt staff 340 Jul 26 20:38 saltstack-git-formula
0 drwxr-xr-x 13 arnoldbechtoldt staff 442 Jul 26 20:38 saltstack-java-formula
0 drwxr-xr-x 13 arnoldbechtoldt staff 442 Jul 26 20:39 saltstack-kibana-formula
0 drwxr-xr-x 12 arnoldbechtoldt staff 408 Jul 26 20:39 saltstack-repos-formula
0 drwxr-xr-x 14 arnoldbechtoldt staff 476 Jul 26 20:39 saltstack-salt-formula
0 drwxr-xr-x 13 arnoldbechtoldt staff 442 Jul 26 20:39 saltstack-sysctl-formula
0 drwxr-xr-x 14 arnoldbechtoldt staff 476 Jul 26 20:39 saltstack-tools-formula
$ git submodule update --init --recursive .
Submodule 'test/vagrant' (https://github.com/bechtoldt/iac-vagrant.git) registered for path 'test/vagrant'
Cloning into 'test/vagrant'...
remote: Counting objects: 403, done.
remote: Total 403 (delta 0), reused 0 (delta 0), pack-reused 403
Receiving objects: 100% (403/403), 52.14 KiB | 0 bytes/s, done.
Resolving deltas: 100% (202/202), done.
Checking connectivity... done.
Submodule path 'shared/salt/assets/saltstack-elasticsearch-formula/test/vagrant': checked out '80af717eef9ced4e6e719f21229839c125be3f39'
Submodule 'assets' (https://github.com/bechtoldt/vagrant-assets.git) registered for path 'assets'
Cloning into 'assets'...
remote: Counting objects: 697, done.
remote: Total 697 (delta 0), reused 0 (delta 0), pack-reused 697
Receiving objects: 100% (697/697), 64.11 KiB | 0 bytes/s, done.
Resolving deltas: 100% (299/299), done.
Checking connectivity... done.
Submodule path 'shared/salt/assets/saltstack-elasticsearch-formula/test/vagrant/assets': checked out '9d4e3331b2a1e7abced19c45f9aca6230a906b25'
Submodule 'test/vagrant' (https://github.com/bechtoldt/iac-vagrant.git) registered for path 'test/vagrant'
Cloning into 'test/vagrant'...
remote: Counting objects: 403, done.
remote: Total 403 (delta 0), reused 0 (delta 0), pack-reused 403
Receiving objects: 100% (403/403), 52.14 KiB | 0 bytes/s, done.
Resolving deltas: 100% (202/202), done.
...
$ cat <<EOF > shared/salt/devenv/elastic1/file_roots/pillar/top.sls
base:
'*':
- common
- elasticsearch
- kibana
EOF
$ cat <<EOF > shared/salt/devenv/elastic1/file_roots/pillar/common.sls
tools:
manage:
bzip: {}
curl: {}
gzip: {}
haveged: {}
tar: {}
unzip: {}
zip: {}
java:
lookup:
manage:
jdk:
current_ver: 8u51
versions:
8u51:
source: file:///vagrant/shared/misc/jdk-8u51-linux-x64.tar.gz
source_hash: md5=b34ff02c5d98b6f372288c17e96c51cf
version: jdk1.8.0_51
EOF
$ cat <<EOF > shared/salt/devenv/elastic1/file_roots/pillar/elasticsearch.sls
tools:
manage:
bzip: {}
curl: {}
gzip: {}
haveged: {}
tar: {}
unzip: {}
zip: {}
EOF
$ cat <<EOF > shared/salt/devenv/elastic1/file_roots/pillar/elasticsearch.sls
repos:
lookup:
repos:
elasticsearch:
url: http://packages.elasticsearch.org/elasticsearch/1.5/debian
dist: stable
comps:
- main
keyurl: http://packages.elasticsearch.org/GPG-KEY-elasticsearch
elasticsearch:
lookup:
defaults:
JAVA_HOME: /opt/java/jdk/current/src
ES_HEAP_SIZE: 512m
settings:
cluster:
name: cluster1
node:
name: {{ salt['grains.get']('fqdn') }}
master: true
data: true
index:
number_of_replicas: 0
gateway:
expected_nodes: 1
discovery:
zen:
ping:
multicast:
enabled: false
plugins:
- name: lmenezes/elasticsearch-kopf
installed_name: kopf
sysctl:
lookup:
params:
- name: vm.swappiness
value: 1
- name: vm.max_map_count
value: 262144
EOF
$ cat <<EOF > shared/salt/devenv/elastic1/file_roots/pillar/kibana.sls
kibana:
lookup:
instances:
standard:
current_ver: kibana-4.1.0-linux-x64
versions:
kibana-4.1.0-linux-x64:
source: https://download.elastic.co/kibana/kibana/kibana-4.1.0-linux-x64.tar.gz
source_checksum: md5=77b65888f0ab6b8fa403732b3bf9fbe5
EOF
$ cat <<EOF > shared/salt/devenv/elastic1/file_roots/states/top.sls
base:
'*':
- repos
- git
#- salt.minion
- sysctl
- tools
- sysctl
- java
- elasticsearch
- kibana
EOF
$ cat <<EOF > nodes.yaml
---
defaults:
domain: elk-demo.local.arnoldbechtoldt.com
base_box: chef/debian-7.8
osfam: debian
synced_folders:
# Common folders
- src: assets/scripts/provision
dst: /vagrant/scripts
- src: ../shared/misc
dst: /vagrant/shared/misc
# Salt related folders
- src: ../shared/salt/devenv
dst: /vagrant/shared/salt/devenv
- src: ../shared/salt/assets/salt-modules/_modules
dst: /vagrant/salt/_modules
- src: ../shared/salt/assets/salt-modules/_states
dst: /vagrant/salt/_states
provision:
- name: basic
- name: saltstack_install_bootstrap
- name: saltstack_formulas
formulas:
- name: elasticsearch
base_dir: ../shared/salt/assets/saltstack-elasticsearch-formula
- name: git
base_dir: ../shared/salt/assets/saltstack-git-formula
- name: java
base_dir: ../shared/salt/assets/saltstack-java-formula
- name: kibana
base_dir: ../shared/salt/assets/saltstack-kibana-formula
- name: repos
base_dir: ../shared/salt/assets/saltstack-repos-formula
- name: salt
base_dir: ../shared/salt/assets/saltstack-salt-formula
- name: sysctl
base_dir: ../shared/salt/assets/saltstack-sysctl-formula
- name: tools
base_dir: ../shared/salt/assets/saltstack-tools-formula
- name: saltstack_config
- name: saltstack_services
nodes:
elastic1: {}
EOF
$ cd vagrant
$ vagrant up
Bringing machine 'elastic1' up with 'virtualbox' provider...
==> elastic1: Box 'chef/debian-7.8' could not be found. Attempting to find and install...
elastic1: Box Provider: virtualbox
elastic1: Box Version: >= 0
==> elastic1: Loading metadata for box 'chef/debian-7.8'
elastic1: URL: https://atlas.hashicorp.com/chef/debian-7.8
==> elastic1: Adding box 'chef/chef/debian-7.8' (v1.0.0) for provider: virtualbox
elastic1: Downloading: https://atlas.hashicorp.com/chef/boxes/chef/debian-7.8/versions/1.0.0/providers/virtualbox.box
==> elastic1: Successfully added box 'chef/debian-7.8' (v1.0.0) for 'virtualbox'!
==> elastic1: Importing base box 'chef/debian-7.8'...
==> elastic1: Matching MAC address for NAT networking...
==> elastic1: Checking if box 'chef/debian-7.8' is up to date...
==> elastic1: Setting the name of the VM: vagrant_elastic1_1437941607653_5687
==> elastic1: Fixed port collision for 22 => 2222. Now on port 2200.
==> elastic1: Clearing any previously set network interfaces...
==> elastic1: Preparing network interfaces based on configuration...
elastic1: Adapter 1: nat
elastic1: Adapter 2: hostonly
==> elastic1: Forwarding ports...
elastic1: 22 => 2200 (adapter 1)
==> elastic1: Running 'pre-boot' VM customizations...
==> elastic1: Booting VM...
==> elastic1: Waiting for machine to boot. This may take a few minutes...
elastic1: SSH address: 127.0.0.1:2200
elastic1: SSH username: vagrant
elastic1: SSH auth method: private key
elastic1: Warning: Connection timeout. Retrying...
elastic1:
elastic1: Vagrant insecure key detected. Vagrant will automatically replace
elastic1: this with a newly generated keypair for better security.
elastic1:
elastic1: Inserting generated public key within guest...
elastic1: Removing insecure key from the guest if its present...
elastic1: Key inserted! Disconnecting and reconnecting using new SSH key...
==> elastic1: Machine booted and ready!
==> elastic1: Checking for guest additions in VM...
==> elastic1: Setting hostname...
==> elastic1: Configuring and enabling network interfaces...
==> elastic1: Mounting shared folders...
elastic1: /vagrant/scripts => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/vagrant/assets/scripts/provision/rhel
elastic1: /vagrant/shared/misc => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/misc
elastic1: /vagrant/salt/_states => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/salt/assets/salt-modules/_states
elastic1: /vagrant/salt/_modules => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/salt/assets/salt-modules/_modules
elastic1: /vagrant/salt/formulas/git => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/salt/assets/saltstack-git-formula
elastic1: /vagrant/salt/formulas/salt => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/salt/assets/saltstack-salt-formula
elastic1: /vagrant/shared/salt/devenv => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/salt/devenv
elastic1: /vagrant/salt/formulas/java => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/salt/assets/saltstack-java-formula
elastic1: /vagrant/salt/formulas/repos => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/salt/assets/saltstack-repos-formula
elastic1: /vagrant/salt/formulas/tools => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/salt/assets/saltstack-tools-formula
elastic1: /vagrant/salt/formulas/sysctl => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/salt/assets/saltstack-sysctl-formula
elastic1: /vagrant/salt/formulas/kibana => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/salt/assets/saltstack-kibana-formula
elastic1: /vagrant/salt/formulas/elasticsearch => /Users/arnoldbechtoldt/Documents/dev/github/demo-elasticsearch-vagrant/shared/salt/assets/saltstack-elasticsearch-formula
==> elastic1: Configuring cache buckets...
==> elastic1: Running provisioner: shell...
elastic1: Running: /var/folders/m4/k0nsnv3d3lq8r0n852g5g22c0000gn/T/vagrant-shell20150726-6862-1sluzbg.sh
==> elastic1: Sourcing scripts that set environment variables for provision scripts..
==> elastic1: Making all shell scripts executable..
==> elastic1: Executing shell scripts..
==> elastic1: ##############################################
==> elastic1: Starting /vagrant/scripts/custom/basic/A_base.sh..
==> elastic1: + timezone=
==> elastic1: + [[ -z '' ]]
==> elastic1: + timezone=Europe/Berlin
==> elastic1: ++ grep -E '^[A-Za-z ]*7\.' /etc/redhat-release
...
...
==> elastic1: which: no salt-syndic in (/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/bin/)
==> elastic1: + [[ -n '' ]]
==> elastic1: + sleep 5
==> elastic1: ++ which salt-minion
==> elastic1: + [[ -n /bin/salt-minion ]]
==> elastic1: + service salt-minion restart
==> elastic1: Redirecting to /bin/systemctl restart salt-minion.service
==> elastic1: Finishing /vagrant/scripts/custom/saltstack_services/A_base.sh..
==> elastic1: + echo 'Finishing /vagrant/scripts/custom/saltstack_services/A_base.sh..'
==> elastic1: Finishing /tmp/vagrant-shell..
==> elastic1: Configuring cache buckets...
$ vagrant ssh
$ sudo -i
# hostname -f
elastic1.elk-demo.local.arnoldbechtoldt.com
# salt-call -l debug state.highstate test=True
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: elastic1.elk-demo.local.arnoldbechtoldt.com
[DEBUG ] Configuration file path: /etc/salt/minion
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] LazyLoaded cmd.run
[DEBUG ] Jinja search path: ['/srv/salt/pillar/examples', '/srv/salt/pillar/shared']
[DEBUG ] Rendered data from file: /srv/salt/pillar/shared/top.sls:
base:
'*':
- common
...
# salt-call -l debug state.highstate
# service elasticsearch status
[ ok ] elasticsearch is running.
# curl -vs http://127.0.0.1:9200/
* About to connect() to 127.0.0.1 port 9200 (#0)
* Trying 127.0.0.1...
* connected
* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.26.0
> Host: 127.0.0.1:9200
> Accept: */*
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
< Content-Length: 366
<
{
"status" : 200,
"name" : "elastic1.elk-demo.local.arnoldbechtoldt.com",
"cluster_name" : "cluster1",
"version" : {
"number" : "1.5.2",
"build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
"build_timestamp" : "2015-04-27T09:21:06Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment