Skip to content

Instantly share code, notes, and snippets.

@gamename
Last active August 29, 2015 14:15
Show Gist options
  • Save gamename/d8c83914076c12f18df5 to your computer and use it in GitHub Desktop.
Save gamename/d8c83914076c12f18df5 to your computer and use it in GitHub Desktop.
Vagrant+Ansible+SSH Connection Timeouts With Multiple Hosts
> vup
Bringing machine 'pgdb-1' up with 'virtualbox' provider...
Bringing machine 'pgdb-2' up with 'virtualbox' provider...
Bringing machine 'consul-1' up with 'virtualbox' provider...
Bringing machine 'consul-2' up with 'virtualbox' provider...
Bringing machine 'consul-3' up with 'virtualbox' provider...
==> pgdb-1: Importing base box 'nextgxdx/centos-70'...
==> pgdb-1: Matching MAC address for NAT networking...
==> pgdb-1: Checking if box 'nextgxdx/centos-70' is up to date...
==> pgdb-1: Setting the name of the VM: pgdb-1
==> pgdb-1: Fixed port collision for 22 => 2222. Now on port 2201.
==> pgdb-1: Clearing any previously set network interfaces...
==> pgdb-1: Preparing network interfaces based on configuration...
pgdb-1: Adapter 1: nat
pgdb-1: Adapter 2: intnet
==> pgdb-1: Forwarding ports...
pgdb-1: 22 => 2201 (adapter 1)
==> pgdb-1: Running 'pre-boot' VM customizations...
==> pgdb-1: Booting VM...
==> pgdb-1: Waiting for machine to boot. This may take a few minutes...
pgdb-1: SSH address: 127.0.0.1:2201
pgdb-1: SSH username: ngd
pgdb-1: SSH auth method: private key
pgdb-1: Warning: Connection timeout. Retrying...
==> pgdb-1: Machine booted and ready!
GuestAdditions versions on your host (4.3.22) and guest (4.3.14) do not match.
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.mia.host-engine.com
* elrepo: mirror.symnds.com
* epel: mirror.us.leaseweb.net
* extras: mirror.fdcservers.net
* updates: mirror.cogentco.com
Package gcc-4.8.2-16.2.el7_0.x86_64 already installed and latest version
Package 1:make-3.82-21.el7.x86_64 already installed and latest version
Package 4:perl-5.16.3-283.el7.x86_64 already installed and latest version
Nothing to do
Copy iso file /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso into the box /tmp/VBoxGuestAdditions.iso
mount: /dev/loop0 is write-protected, mounting read-only
Installing Virtualbox Guest Additions 4.3.22 - guest version is 4.3.14
Verifying archive integrity... All good.
Uncompressing VirtualBox 4.3.22 Guest Additions for Linux............
VirtualBox Guest Additions installer
Removing installed version 4.3.22 of VirtualBox Guest Additions...
Copying additional installer modules ...
Installing additional modules ...
Removing existing VirtualBox non-DKMS kernel modules[ OK ]
Building the VirtualBox Guest Additions kernel modules
Building the main Guest Additions module[ OK ]
Building the shared folder support module[ OK ]
Building the OpenGL support module[ OK ]
Doing non-kernel setup of the Guest Additions[ OK ]
Starting the VirtualBox Guest Additions [ OK ]
Installing the Window System drivers
Could not find the X.Org or XFree86 Window System, skipping.
An error occurred during installation of VirtualBox Guest Additions 4.3.22. Some functionality may not work as intended.
In most cases it is OK that the "Window System drivers" installation failed.
==> pgdb-1: Checking for guest additions in VM...
==> pgdb-1: Setting hostname...
==> pgdb-1: Configuring and enabling network interfaces...
==> pgdb-1: Running provisioner: ansible...
PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false ANSIBLE_SSH_ARGS='-o UserKnownHostsFile=/dev/null -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --private-key=/Users/tennis/.vagrant.d/insecure_private_key --user=ngd --connection=ssh --limit='pgdb-1' --inventory-file=/Users/tennis/Documents/inf-catalog/vagrant/all/.vagrant/provisioners/ansible/inventory -vvvv ./provision.yml
PLAY [all] ********************************************************************
GATHERING FACTS ***************************************************************
<127.0.0.1> ESTABLISH CONNECTION FOR USER: ngd
<127.0.0.1> REMOTE_MODULE setup
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/tennis/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2201 -o IdentityFile="/Users/tennis/.vagrant.d/insecure_private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ngd -o ConnectTimeout=10 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1424707605.43-180494441707909 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1424707605.43-180494441707909 && echo $HOME/.ansible/tmp/ansible-tmp-1424707605.43-180494441707909'
<127.0.0.1> PUT /var/folders/0v/wkwzypbd25v619fqs75gm41r0000gp/T/tmp4_qoVt TO /home/ngd/.ansible/tmp/ansible-tmp-1424707605.43-180494441707909/setup
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/tennis/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2201 -o IdentityFile="/Users/tennis/.vagrant.d/insecure_private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ngd -o ConnectTimeout=10 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=kxudsobsnlipmjquqhlwrxavikvajxgj] password: " -u root /bin/sh -c '"'"'echo SUDO-SUCCESS-kxudsobsnlipmjquqhlwrxavikvajxgj; LANG=C LC_CTYPE=C /usr/bin/python /home/ngd/.ansible/tmp/ansible-tmp-1424707605.43-180494441707909/setup; rm -rf /home/ngd/.ansible/tmp/ansible-tmp-1424707605.43-180494441707909/ >/dev/null 2>&1'"'"''
ok: [pgdb-1]
TASK: [Set our hostname] ******************************************************
skipping: [pgdb-1]
TASK: [Set to UTF8] ***********************************************************
<127.0.0.1> ESTABLISH CONNECTION FOR USER: ngd
<127.0.0.1> REMOTE_MODULE command localectl set-locale LANG=en_US.UTF-8 #USE_SHELL
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/tennis/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2201 -o IdentityFile="/Users/tennis/.vagrant.d/insecure_private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ngd -o ConnectTimeout=10 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1424707607.02-7374834535325 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1424707607.02-7374834535325 && echo $HOME/.ansible/tmp/ansible-tmp-1424707607.02-7374834535325'
<127.0.0.1> PUT /var/folders/0v/wkwzypbd25v619fqs75gm41r0000gp/T/tmpHB7L4F TO /home/ngd/.ansible/tmp/ansible-tmp-1424707607.02-7374834535325/command
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/tennis/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2201 -o IdentityFile="/Users/tennis/.vagrant.d/insecure_private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ngd -o ConnectTimeout=10 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=umxtotcofasnhocglemztaoubupqglbd] password: " -u root /bin/sh -c '"'"'echo SUDO-SUCCESS-umxtotcofasnhocglemztaoubupqglbd; LANG=C LC_CTYPE=C /usr/bin/python /home/ngd/.ansible/tmp/ansible-tmp-1424707607.02-7374834535325/command; rm -rf /home/ngd/.ansible/tmp/ansible-tmp-1424707607.02-7374834535325/ >/dev/null 2>&1'"'"''
changed: [pgdb-1] => {"changed": true, "cmd": "localectl set-locale LANG=en_US.UTF-8", "delta": "0:00:00.048206", "end": "2015-02-23 16:06:47.260450", "rc": 0, "start": "2015-02-23 16:06:47.212244", "stderr": "", "stdout": "", "warnings": []}
PLAY [postgresdb_servers] *****************************************************
GATHERING FACTS ***************************************************************
<127.0.0.1> ESTABLISH CONNECTION FOR USER: ngd
<127.0.0.1> REMOTE_MODULE setup
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/tennis/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2201 -o IdentityFile="/Users/tennis/.vagrant.d/insecure_private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ngd -o ConnectTimeout=10 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1424707607.28-105594680019742 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1424707607.28-105594680019742 && echo $HOME/.ansible/tmp/ansible-tmp-1424707607.28-105594680019742'
<127.0.0.1> PUT /var/folders/0v/wkwzypbd25v619fqs75gm41r0000gp/T/tmpVS8SAm TO /home/ngd/.ansible/tmp/ansible-tmp-1424707607.28-105594680019742/setup
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/tennis/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2201 -o IdentityFile="/Users/tennis/.vagrant.d/insecure_private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ngd -o ConnectTimeout=10 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=mdtubuokpjvdgvvzxtoiwyoeltkqhbew] password: " -u root /bin/sh -c '"'"'echo SUDO-SUCCESS-mdtubuokpjvdgvvzxtoiwyoeltkqhbew; LANG=C LC_CTYPE=C /usr/bin/python /home/ngd/.ansible/tmp/ansible-tmp-1424707607.28-105594680019742/setup; rm -rf /home/ngd/.ansible/tmp/ansible-tmp-1424707607.28-105594680019742/ >/dev/null 2>&1'"'"''
ok: [pgdb-1]
TASK: [Create fake block device for /dev/xvdf, simulating an EBS volume] ******
<127.0.0.1> ESTABLISH CONNECTION FOR USER: ngd
<127.0.0.1> REMOTE_MODULE file src=/dev/sdb path=/dev/xvdf mode=0660 owner=root group=root state=link
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/tennis/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2201 -o IdentityFile="/Users/tennis/.vagrant.d/insecure_private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ngd -o ConnectTimeout=10 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1424707607.65-122324043968724 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1424707607.65-122324043968724 && echo $HOME/.ansible/tmp/ansible-tmp-1424707607.65-122324043968724'
<127.0.0.1> PUT /var/folders/0v/wkwzypbd25v619fqs75gm41r0000gp/T/tmpEjY1jx TO /home/ngd/.ansible/tmp/ansible-tmp-1424707607.65-122324043968724/file
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/tennis/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2201 -o IdentityFile="/Users/tennis/.vagrant.d/insecure_private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ngd -o ConnectTimeout=10 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=ojhctxdrkaxajrythzztjgceprdydxhn] password: " -u root /bin/sh -c '"'"'echo SUDO-SUCCESS-ojhctxdrkaxajrythzztjgceprdydxhn; LANG=C LC_CTYPE=C /usr/bin/python /home/ngd/.ansible/tmp/ansible-tmp-1424707607.65-122324043968724/file; rm -rf /home/ngd/.ansible/tmp/ansible-tmp-1424707607.65-122324043968724/ >/dev/null 2>&1'"'"''
changed: [pgdb-1] => {"changed": true, "dest": "/dev/xvdf", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:device_t:s0", "size": 8, "src": "/dev/sdb", "state": "link", "uid": 0}
PLAY RECAP ********************************************************************
pgdb-1 : ok=4 changed=2 unreachable=0 failed=0
==> pgdb-2: Importing base box 'nextgxdx/centos-70'...
==> pgdb-2: Matching MAC address for NAT networking...
==> pgdb-2: Checking if box 'nextgxdx/centos-70' is up to date...
==> pgdb-2: Setting the name of the VM: pgdb-2
==> pgdb-2: Fixed port collision for 22 => 2222. Now on port 2202.
==> pgdb-2: Clearing any previously set network interfaces...
==> pgdb-2: Preparing network interfaces based on configuration...
pgdb-2: Adapter 1: nat
pgdb-2: Adapter 2: hostonly
pgdb-2: Adapter 3: intnet
==> pgdb-2: Forwarding ports...
pgdb-2: 22 => 2202 (adapter 1)
==> pgdb-2: Running 'pre-boot' VM customizations...
==> pgdb-2: Booting VM...
==> pgdb-2: Waiting for machine to boot. This may take a few minutes...
pgdb-2: SSH address: 127.0.0.1:2202
pgdb-2: SSH username: ngd
pgdb-2: SSH auth method: private key
pgdb-2: Warning: Connection timeout. Retrying...
pgdb-2: Warning: Remote connection disconnect. Retrying...
Text will be echoed in the clear. Please install the HighLine or Termios libraries to suppress echoed text.
ngd@127.0.0.1's password: pgdb-2: Warning: Connection timeout. Retrying...
Text will be echoed in the clear. Please install the HighLine or Termios libraries to suppress echoed text.
ngd@127.0.0.1's password: pgdb-2: Warning: Connection timeout. Retrying...
Text will be echoed in the clear. Please install the HighLine or Termios libraries to suppress echoed text.
ngd@127.0.0.1's password: pgdb-2: Warning: Connection timeout. Retrying...
Text will be echoed in the clear. Please install the HighLine or Termios libraries to suppress echoed text.
ngd@127.0.0.1's password: pgdb-2: Warning: Connection timeout. Retrying...
Text will be echoed in the clear. Please install the HighLine or Termios libraries to suppress echoed text.
ngd@127.0.0.1's password:Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.
If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.
If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.
If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.
- hosts: all
sudo: yes
tasks:
- name: Set our hostname
hostname: name={{ inventory_hostname_short }}
when: inventory_hostname_short != ansible_hostname
- name: Set to UTF8
shell: localectl set-locale LANG=en_US.UTF-8
- hosts: postgresdb_servers
sudo: yes
tasks:
- name: Create fake block device for /dev/xvdf, simulating an EBS volume
file: >
src=/dev/sdb
path=/dev/xvdf
mode=0660
owner=root
group=root
state=link
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
require "ipaddr"
variable_path = "variables.json"
jvars = (JSON.parse(File.read("#{variable_path}")))
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = jvars['vm_box']
config.ssh.username = jvars['ssh_user_name']
config.ssh.forward_agent = jvars['ssh_forward_agent']
config.ssh.insert_key = jvars['ssh_insert_key']
disable_share_folder = jvars['vb_disable_share_folder']
my_share_folder = jvars['vb_my_share_folder']
guest_share_folder = jvars['vb_guest_share_folder']
my_share_readme = "#{my_share_folder}/README.txt"
ansible_provisioner_file = jvars['ansible_provisioner_file']
# ansible_hosts_file = jvars['ansible_hosts_file']
# do NOT want to check for the correct additions version when booting
config.vbguest.auto_update = jvars['vb_auto_update']
# do NOT download the iso file from a webserver
config.vbguest.no_remote = jvars['vb_no_remote']
if !disable_share_folder
unless Dir.exist?(my_share_folder)
Dir.mkdir(my_share_folder)
f = open(my_share_readme, 'w')
readme = "This is a special folder to share data with vm guests.\n"\
"Do not save anything valuable here.\n"
f.write("#{readme}")
f.close
end
end
config.vm.synced_folder "#{my_share_folder}", "#{guest_share_folder}",
disabled: disable_share_folder
config.vm.provision "ansible" do |ansible|
ansible.groups = {
"consul_servers" => ["consul-1", "consul-2", "consul-3"],
"load_balancers" => ["lb-1"],
"nat" => ["nat-1"],
"rethinkdb_servers" => ["rtdb-1", "rtdb-2"],
"postgresdb_servers" => ["pgdb-1", "pgdb-2"]
}
ansible.playbook = "#{ansible_provisioner_file}"
ansible.verbose = "vvvv"
# ansible.inventory_path = "#{ansible_hosts_file}"
end
jvars['hosts'].each do | h |
(1..h['count']).each do | i |
# Use the ip address stub as a starting point
ip = IPAddr.new(h['ip_address_stub'])
# Increment the address to match the current machine count
i.times{ ip = ip.succ}
#puts " counter #{i} ip #{ip}"
hostname = "%s-%s" % [h['name_stub'],i]
#puts "==> creating #{hostname}"
# Get the autostart value from the individual host def or globally,
# whichever is first
start_flag = h['autostart'] ? h['autostart'] : jvars['autostart']
config.vm.define "#{hostname}", autostart: start_flag do | box |
box.vm.hostname = hostname
box.vm.network "private_network",
ip: "#{ip}",
virtualbox__intnet: jvars['vbox_net']
box.vm.provider "virtualbox" do |v|
v.name = "#{hostname}"
v.gui = h['gui'] ? h['gui'] : jvars['gui']
v.memory = h['ram'] ? h['ram'] : jvars['ram']
v.cpus = h['cpu'] ? h['cpu'] : jvars['cpu']
v.customize ["modifyvm", :id, "--natnet1", "172.16/16"]
if h['attach_disk_size'] then
disk_name = "/tmp/#{hostname}.vdi"
disk_size = h['attach_disk_size']
unless File.exist?(disk_name)
v.customize ['createhd',
'--filename', disk_name,
'--size', disk_size,
'--variant', 'Fixed' ]
end
v.customize ['storageattach', :id,
'--storagectl', 'IDE Controller',
'--port', 1,
'--device', 0,
'--type', 'hdd',
'--medium', disk_name]
end
end
end
end
end
end
{
"vm_box": "nextgxdx/centos-70",
"ssh_user_name": "ngd",
"ssh_forward_agent": true,
"ssh_insert_key": false,
"vb_auto_update": true,
"vb_no_remote": false,
"vb_disable_share_folder": true,
"vb_my_share_folder": "./share",
"vb_guest_share_folder": "/vagrant",
"ansible_provisioner_file": "./provision.yml",
"autostart": false,
"gui": false,
"ram": 256,
"cpu": 1,
"count": 1,
"vbox_net": "vboxnet0",
"hosts": [
{
"name_stub": "rtdb",
"count": 2,
"ip_address_stub": "10.0.0.20"
},
{
"name_stub": "pgdb",
"autostart": true,
"count": 2,
"ram": 2048,
"attach_disk_size": 512,
"db_device_path": "/dev/xvdf",
"ip_address_stub": "10.0.0.30"
},
{
"name_stub": "nat",
"count": 1,
"ip_address_stub": "10.0.0.50"
},
{
"name_stub": "lb",
"count": 1,
"ip_address_stub": "10.0.0.40"
},
{
"name_stub": "consul",
"count": 3,
"autostart": true,
"ip_address_stub": "10.0.0.10"
}
]
}
tennis@tennisMBP: /usr/local (master|✔)
> VBoxManage --version
4.3.22r98236
tennis@tennisMBP: /usr/local (master|✔)
> ansible --version
ansible 1.8.2
configured module search path = None
tennis@tennisMBP: /usr/local (master|✔)
> vagrant version
Installed Version: 1.7.2
Latest Version: 1.7.2
You're running an up-to-date version of Vagrant!
tennis@tennisMBP: /usr/local (master|✔)
> ssh -V
OpenSSH_6.7p1, OpenSSL 1.0.2 22 Jan 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment