Skip to content

Instantly share code, notes, and snippets.

@navidshaikh

navidshaikh/CDK+TLS-enabled-Docker-deamon+Eclipse.md

Last active September 23, 2015 11:41
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save navidshaikh/d97a938c62df3b156149 to your computer and use it in GitHub Desktop.
Save navidshaikh/d97a938c62df3b156149 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
CDK+TLS-enabled-Docker-deamon+Eclipse.md

Setup TLS-Enabled-Daemon from CDK and Connect to Eclipse

NEW VERSION OF THIS DOC https://github.com/navidshaikh/testing-adb

Bex has done a better job in building up the context and outlining the idea [here] (https://gist.github.com/bexelbie/35f421ee247c7621c0d0) . Goal here is to configure a TLS enabled Docker docker daemon running over TCP from inside CDK and configure client at Eclipse to be able to connect to it.

In order to get configure CDK + Eclipse

  1. docker daemon inside CDK has to run over TCP (as well, if running on local unix socket) - This need required certs to be generated inside CDK and updating the docker daemon config and restarting service.
  2. Required client side certs has to be copied over to host (from guest) for clients to connect
  3. Port at which docker inside CDK running need to be mapped at host machine port for client to easily connect

Step -1 above is taken care by incorporating steps to generate certs and updating config in Vagrantfile

Step -3 is again taken care by specifying port mapping in Vagrantfile with auto_correct option incase of port collision at host - There are few complexities (to be covered later) for libvirt provider however for Virutalbox it works as expected.

For solving problem stated in Step-2 there are following two options

  1. Use ssh and scp to copy over the certs from inside the box to host
  • Need for Windows platform: ssh and scp availability on host
  • vagrant-triggers plugin for triggering the scp operation as box is "provisioned"
    • Knowledge about the mapped ssh port at host machine for the scp command (working on making this automated)
  1. Setup VirtualBox guest additions inside guest and use vagrant synced folders feature. this will need
  • a line in Vagrantfile for specifying the folder to be synced
  • for atomic guest - vbox guest additions configured in a (SPC) container
  • for non atomic guest - vbox guest additions configured
Note: plugin [vagrant-vbguest] (https://github.com/dotless-de/vagrant-vbguest) is another option which can set vbox guest additions for community non-atomic platforms (CentOS and Fedora)

##Hands-on NEW VERSION OF THIS DOC https://github.com/navidshaikh/testing-adb ###1. CDK + Eclipse with vbox guest additions for VirtualBox provider

  • vagrant plugin install vagrant-vbguest
  • Use following Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  
  config.vm.box = "atomicapp/dev"

  config.vm.network "forwarded_port", guest: 2376, host: 2379, auto_correct: true

  if !Vagrant.has_plugin?("vagrant-vbguest")
    puts "'vagrant-vbguest' plugin is required"
    puts "This can be installed by running:"
    puts
    puts " vagrant plugin install vagrant-vbguest"
    puts
    exit
  end


  config.vm.provision "shell", inline: <<-SHELL
    # =========================================================                                                                                             
    # Generate Certs for running TLS enabled docker daemon
    
    # Generate Certificates to use with the docker daemon
    # Instructions sourced from http://docs.docker.com/articles/https/
    
    # Get the certificate location, i.e. setting the DOCKER_CERT_PATH variable
    . /etc/sysconfig/docker
    
    # randomString from http://utdream.org/post.cfm/bash-generate-a-random-string
    # modified to echo value
    
    function randomString {
            # if a param was passed, it's the length of the string we want
            if [[ -n $1 ]] && [[ "$1" -lt 20 ]]; then
                    local myStrLength=$1;
            else
                    # otherwise set to default
                    local myStrLength=8;
            fi
    
            local mySeedNumber=$$`date +%N`; # seed will be the pid + nanoseconds
            local myRandomString=$( echo $mySeedNumber | md5sum | md5sum );
            # create our actual random string
            #myRandomResult="${myRandomString:2:myStrLength}"
            echo "${myRandomString:2:myStrLength}"
    }
    
    # Get a temporary workspace
    dir=`mktemp -d`
    cd $dir
    
    # Get a random password for the CA and save it
    passfile=tmp.pass
    password=$(randomString 10)
    echo $password > $passfile
    
    # Generate the CA
    openssl genrsa -aes256 -passout file:$passfile -out ca-key.pem 2048
    openssl req -new -x509 -passin file:$passfile -days 365 -key ca-key.pem -sha256 -out ca.pem -subj "/C=/ST=/L=/O=/OU=/CN=example.com"
    
    # Generate Server Key and Sign it
    openssl genrsa -out server-key.pem 2048
    openssl req -subj "/CN=example.com" -new -key server-key.pem -out server.csr
    # Allow from 127.0.0.1
    extipfile=extfile.cnf
    echo subjectAltName = IP:127.0.0.1 > $extipfile
    openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -passin file:$passfile -extfile $extipfile
    
    # Generate the Client Key and Sign it
    openssl genrsa -out key.pem 2048
    openssl req -subj '/CN=client' -new -key key.pem -out client.csr
    extfile=tmp.ext
    echo extendedKeyUsage = clientAuth > $extfile
    openssl x509 -req -days 365 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cert.pem -extfile $extfile -passin file:$passfile
    
    # Clean up
    
    # set the cert path as configured in /etc/sysconfig/docker
    
    ## Move files into place
    mv ca.pem $DOCKER_CERT_PATH
    mv server-cert.pem $DOCKER_CERT_PATH
    mv server-key.pem $DOCKER_CERT_PATH
    
    # since the default user is vagrant and it can run docker without sudo
    CLIENT_SIDE_CERT_PATH=/home/vagrant/.docker
    
    mkdir -p $CLIENT_SIDE_CERT_PATH
    cp $DOCKER_CERT_PATH/ca.pem $CLIENT_SIDE_CERT_PATH
    mv cert.pem key.pem $CLIENT_SIDE_CERT_PATH
    
    chown vagrant:vagrant $CLIENT_SIDE_CERT_PATH
    
    chmod 0444 $CLIENT_SIDE_CERT_PATH/ca.pem
    chmod 0444 $CLIENT_SIDE_CERT_PATH/cert.pem
    chmod 0444 $CLIENT_SIDE_CERT_PATH/key.pem
    chown vagrant:vagrant $CLIENT_SIDE_CERT_PATH/ca.pem
    chown vagrant:vagrant $CLIENT_SIDE_CERT_PATH/cert.pem
    chown vagrant:vagrant $CLIENT_SIDE_CERT_PATH/key.pem
    
    chmod -v 0400 $DOCKER_CERT_PATH/ca.pem $DOCKER_CERT_PATH/server-cert.pem $DOCKER_CERT_PATH/server-key.pem
    
    ## Remove remaining files
    cd
    echo rm -rf $dir
    
    # ============= end of script for generating the certs for TLS enabled docker daemon===
 
    sed -i.back '/OPTIONS=*/c\OPTIONS="--selinux-enabled -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server-cert.pem --tlskey=/etc/docker/server-key.pem --tlsverify"' /etc/sysconfig/docker
    sudo systemctl restart docker
  SHELL
  
  # setup the synced folders
  config.vm.synced_folder ".", "/home/vagrant/.docker/"
  config.vbguest.no_remote = true

end

  • Run vagrant up this will install the vbox guest additions inside the guest, (be patient)

  • you should have the certs in your current directory

  • clients can make connection using 127.0.0.1:2379 and using the certs present in local directory

  • for testing with docker do (on host)

    cp -r *.pem  ~/.docker
docker -H 127.0.0.1:2379 --tlsverify images

###2. CDK + Eclipse with scp and ssh for VirtualBox provider NEW VERSION OF THIS DOC https://github.com/navidshaikh/testing-adb

  • vagrant plugin install vagrant-triggers
  • Export following into a Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  config.vm.box = "atomicapp/dev"

  config.vm.network "forwarded_port", guest: 2376, host: 2379, auto_correct: true
    if !Vagrant.has_plugin?("vagrant-triggers")
      puts "'vagrant-triggers' plugin is required"
      puts "This can be installed by running:"
      puts
      puts " vagrant plugin install vagrant-triggers"
      puts
      exit
  end

  config.vm.provision "shell", inline: <<-SHELL
    # =========================================================                                                                                             
    # Generate Certs for running TLS enabled docker daemon
    #!/bin/bash
    
    # Todo: move the files into place
    
    # Generate Certificates to use with the docker daemon
    # Instructions sourced from http://docs.docker.com/articles/https/
    
    # Get the certificate location, i.e. setting the DOCKER_CERT_PATH variable
    . /etc/sysconfig/docker
    
    # randomString from http://utdream.org/post.cfm/bash-generate-a-random-string
    # modified to echo value
    
    function randomString {
            # if a param was passed, it's the length of the string we want
            if [[ -n $1 ]] && [[ "$1" -lt 20 ]]; then
                    local myStrLength=$1;
            else
                    # otherwise set to default
                    local myStrLength=8;
            fi
    
            local mySeedNumber=$$`date +%N`; # seed will be the pid + nanoseconds
            local myRandomString=$( echo $mySeedNumber | md5sum | md5sum );
            # create our actual random string
            #myRandomResult="${myRandomString:2:myStrLength}"
            echo "${myRandomString:2:myStrLength}"
    }
    
    # Get a temporary workspace
    dir=`mktemp -d`
    cd $dir
    
    # Get a random password for the CA and save it
    passfile=tmp.pass
    password=$(randomString 10)
    echo $password > $passfile
    
    # Generate the CA
    openssl genrsa -aes256 -passout file:$passfile -out ca-key.pem 2048
    openssl req -new -x509 -passin file:$passfile -days 365 -key ca-key.pem -sha256 -out ca.pem -subj "/C=/ST=/L=/O=/OU=/CN=example.com"
    
    # Generate Server Key and Sign it
    openssl genrsa -out server-key.pem 2048
    openssl req -subj "/CN=example.com" -new -key server-key.pem -out server.csr
    # Allow from 127.0.0.1
    extipfile=extfile.cnf
    echo subjectAltName = IP:127.0.0.1 > $extipfile
    openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -passin file:$passfile -extfile $extipfile
    
    # Generate the Client Key and Sign it
    openssl genrsa -out key.pem 2048
    openssl req -subj '/CN=client' -new -key key.pem -out client.csr
    extfile=tmp.ext
    echo extendedKeyUsage = clientAuth > $extfile
    openssl x509 -req -days 365 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cert.pem -extfile $extfile -passin file:$passfile
    
    # Clean up
    
    # set the cert path as configured in /etc/sysconfig/docker
    
    ## Move files into place
    mv ca.pem $DOCKER_CERT_PATH
    mv server-cert.pem $DOCKER_CERT_PATH
    mv server-key.pem $DOCKER_CERT_PATH
    
    # since the default user is vagrant and it can run docker without sudo
    CLIENT_SIDE_CERT_PATH=/home/vagrant/.docker
    
    mkdir -p $CLIENT_SIDE_CERT_PATH
    cp $DOCKER_CERT_PATH/ca.pem $CLIENT_SIDE_CERT_PATH
    mv cert.pem key.pem $CLIENT_SIDE_CERT_PATH
    
    chown vagrant:vagrant $CLIENT_SIDE_CERT_PATH
    
    chmod 0444 $CLIENT_SIDE_CERT_PATH/ca.pem
    chmod 0444 $CLIENT_SIDE_CERT_PATH/cert.pem
    chmod 0444 $CLIENT_SIDE_CERT_PATH/key.pem
    chown vagrant:vagrant $CLIENT_SIDE_CERT_PATH/ca.pem
    chown vagrant:vagrant $CLIENT_SIDE_CERT_PATH/cert.pem
    chown vagrant:vagrant $CLIENT_SIDE_CERT_PATH/key.pem
    
    chmod -v 0400 $DOCKER_CERT_PATH/ca.pem $DOCKER_CERT_PATH/server-cert.pem $DOCKER_CERT_PATH/server-key.pem
    
    ## Remove remaining files
    cd
    echo rm -rf $dir
    
    # ============= end of script for generating the certs for TLS enabled docker daemon===
 
    sed -i.back '/OPTIONS=*/c\OPTIONS="--selinux-enabled -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server-cert.pem --tlskey=/etc/docker/server-key.pem --tlsverify"' /etc/sysconfig/docker
    sudo systemctl restart docker
  SHELL

  # triggers
  config.trigger.after :provision do
    hport = `vagrant ssh-config`.split("\n  ").find{|e| e.start_with?("Port") }.split(" ")[1] 
    run "scp -r -P #{hport} -o StrictHostKeyChecking=no -i .vagrant/machines/default/virtualbox/private_key vagrant@127.0.0.1:/home/vagrant/.docker/* ."
  end

end

  • Run vagrant up --provider virtualbox (in the same directory where Vagrantfile exist)

  • Run vagrant provision

  • You should have the certs in your current directory which is created (or updated) as part of above step

  • To test the client connection to docker daemon inside CDK: In the Vagrantfile at line no: 7, we have mapped host port 2379 to docker daemon port (2376) inside CDK, which means that you can access the daemon at (host machine) 127.0.0.1:2379. Following is an example of connecting via docker CLI to daemon over TLS enabled TCP connection

    # copy the certs (generated above) to proper place where `docker` (by default) look up
cp -r *.pem  ~/.docker
docker -H 127.0.0.1:2379 --tlsverify images

    Note: On host machine, if you have port 2379 in use, vagrant will try to auto_correct it and assign a different port. You need to keep an eye on the corrected port while vagrant is brigning the machine up. Note that this vagrant auto_correct feature works as expected for Virtualbox provider but for libvirt provider it does not auto_correct and does not even fail!

  • To test out the connection with Eclipse kindly check https://www.eclipse.org/community/eclipse_newsletter/2015/june/article3.php and this Video by Xavier Coulon https://www.youtube.com/watch?v=RUgEgtLux8Q. More Eclipse Docker Tooling documentations are at <https://wiki.eclipse.org/Linux_Tools_Project/Docker_Tooling/User_Guide>

@LalatenduMohanty
Copy link

As per the readme in https://github.com/dotless-de/vagrant-vbguest , it tries to auto install vbox additions if possible. So in OSX thats what is happening when I am running the 2nd Vagrant file.

@maxandersen
Copy link

I would not expect this to run on 127.0.0.1 but on some specific separate IP.

Anyways doing the above and running the docker command gives me:

docker -H 127.0.0.1:2379 --tlsverify images
An error occurred trying to connect: Get https://127.0.0.1:2379/v1.20/images/json: x509: certificate signed by unknown authority

@maxandersen
Copy link

It does work with eclipse when using https://127.0.0.1:2379 (at least I can connect).

But search registry does not seem to work - it can find images but cannot find their tags so I cannot pull anything in easily.

@navidshaikh
Copy link
Author

@maxandersen: Firstly, thank you for trying it out! :-)

I would not expect this to run on 127.0.0.1 but on some specific separate IP.

any specific reason?
And you meant you would like to run in on specific separate IP - that is the IP allocated to vagrant box , right?

docker -H 127.0.0.1:2379 --tlsverify images
An error occurred trying to connect: Get https://127.0.0.1:2379/v1.20/images/json: x509: certificate signed by unknown authority

Regarding the certificates error - can you please add a bit detail about the operation:

  • docker version
  • host machine platform from where you are trying to connect via docker client

@navidshaikh
Copy link
Author

@maxandersen:

But search registry does not seem to work - it can find images but cannot find their tags so I cannot pull anything in easily.

I think it's more related to Eclipse Docker Tooling plugin - @xcoulon do you want to chime in here?
Apart from the search issue, I hope you can pull images (and run containers as well? ).

@navidshaikh
Copy link
Author

@LalatenduMohanty: Give it a try by removing the plugin ?

vagrant plugin uninstall vagrant-vbguest

@maxandersen
Copy link

expecting something else than 127.0.0.1 because it is running in a virtual machine and I would prefer it would not block out other common ports. By using the vagrant box ip this could also just run on the standard port.

docker --version says 1.8.1, i'm running OSX

And I cannot verify pull images work due to the search bug and creating timesout - but those are eclipse issues.

@xcoulon
Copy link

xcoulon commented Sep 2, 2015

Yes, I hope we can fix the Eclipse tooling issue soon. I should have a patch ready for that problem today.
With the code I have on my machine, I was able to pull an image and run it.

@LalatenduMohanty
Copy link

I could pull a docker image! works for me on a macbook :)

@vpavlin
Copy link

vpavlin commented Sep 3, 2015

This was my first experience with OS X and like second with Eclipse... but still I was able to install all dependencies (with a little help:) ), run a box, connect Eclipse to it and pull fedora image: http://imgur.com/qOT1qbV

I am not able to run a container - run button does nothing:(

@navidshaikh
Copy link
Author

@vpavlin noted following warning on OS X

$ vagrant ssh
bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

@maxandersen
Copy link

I still can't get OSX docker cli to connect to this. Any info on how would be appreciated.

I think the only reason eclipse works is that it does not seem to check certificates which is another issue.

@navidshaikh
Copy link
Author

NEW VERSION OF THIS DOC https://github.com/navidshaikh/testing-adb

@navidshaikh
Copy link
Author

@maxandersen:
docker daemon inside vagrant box will not allow the clients to be connected without certs. Requesting to please file any further issues on the new version of this doc at repository https://github.com/navidshaikh/testing-adb

@navidshaikh
Copy link
Author

@maxandersen: filed the OS X issue at new repo navidshaikh/testing-adb#1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment