Summer is not over yet, but coding part of Google summer of code yes. This is my final report.
I was working with fedora organization on conu - python API for your containers. Before GSOC, you could use library just to work with standalone containers. Goal for GSOC was to implement support also for orchestration tools like Kubernetes and OpenShift. For more information see my proposal.
At first I started with implementation of container and image metadata classes in conu. Conu is python API for containers so we needed to figure it out how we can work with containers and images in general without dependency on any container engine. Right now we have Docker and Nspawn (not stable) support, but in the future we want to implement support for other engines. This work takes almost three weeks to be completed and was not planned during application period.
Next step was to implement Kubernetes backend. Kubernetes has official python client implementation. Problem is that project is not very well maintained and doesn't have a lot of active contributors. I started with basic implementation of pod primitive. Next was service and deployment. Then I implemented K8sBackend class that allows you to use Kubernetes backend like:
with K8sBackend() as k8s_backend:
# do all k8s magic here!In the end if you want to verify that your image is running correctly inside of Kubernetes cluster you can easily do it in few lines of python code like this:
with K8sBackend() as k8s_backend:
# create new namespace with random name
namespace = k8s_backend.create_namespace()
with DockerBackend() as backend:
image = backend.ImageClass('nginx')
pod = image.run_in_pod(namespace=namespace)
try:
pod.wait(200)
assert pod.get_phase() == PodPhase.RUNNING
finally:
pod.delete()
assert pod.get_phase() == PodPhase.TERMINATING
k8s_backend.delete_namespace(namespace)For more examples check github repository and examples section.
Hardest part of my GSOC journey was to setup enviroment for testing. I need to figure it out way how to run tests in our CI. There was more options. First approach that we tried, was using oc cluster up for Kubernetes and for OpenShift in future. This implementation didn't work so I decided to use minikube with option --vm-driver=None which allows us to run Kubernetes tests inside virtual machine.
This solution works for all our Kubernetes tests and we get finally green CI.
As last part I implemented Openshift backend. Main functionality here was to implement oc new-app command. This command can be used in many ways. Using source to image or using OpenShift templates. This still needs improvements because we still have problems with API calls to Openshift cluster. This code is not merged in upstream repository yet. Example is taken from this WIP PR.
If you want to verify your image is running in OpenShift you can do it like this:
import logging
from conu.backend.origin.backend import OpenshiftBackend
from conu.backend.docker.backend import DockerBackend
# insert your API key - oc whoami -t
API_KEY = '<API_KEY>'
with OpenshiftBackend(API_KEY, logging_level=logging.DEBUG) as openshift_backend:
with DockerBackend(logging_level=logging.DEBUG) as backend:
# builder image
python_image = backend.ImageClass("centos/python-36-centos7")
# docker login inside OpenShift internal registry
openshift_backend.login_to_registry('developer')
# create new app from remote source in OpenShift cluster
app_name = openshift_backend.new_app(python_image,
source="https://github.com/openshift/django-ex.git",
project='myproject')
try:
# wait until service is ready to accept requests
openshift_backend.wait_for_service(
app_name=app_name,
expected_output='Welcome to your Django application on OpenShift',
timeout=300)
finally:
openshift_backend.clean_project(app_name)After running this code you can see output:
07:39:42.185 backend.py INFO conu has initiated, welcome to the party!
07:39:42.185 backend.py DEBUG conu version: 0.4.0
07:39:42.296 backend.py INFO conu has initiated, welcome to the party!
07:39:42.377 __init__.py INFO docker environment info: 'Client:\n Version: 1.13.1\n API version: 1.26\n Package version: docker-1.13.1-59.gitaf6b32b.fc27.x86_64\n Go version: go1.9.6\n Git commit: 30c1ca9-unsupported\n Built: Tue Jun 12 19:29:50 2018\n OS/Arch: linux/amd64\n\nServer:\n Version: 1.13.1\n API version: 1.26 (minimum version 1.12)\n Package version: docker-1.13.1-59.gitaf6b32b.fc27.x86_64\n Go version: go1.9.6\n Git commit: 30c1ca9-unsupported\n Built: Tue Jun 12 19:29:50 2018\n OS/Arch: linux/amd64\n Experimental: false\n'
07:39:42.412 backend.py INFO conu has initiated, welcome to the party!
07:39:42.791 backend.py INFO Login to 172.30.1.1:5000 succeed
07:39:43.080 image.py INFO The push refers to a repository [172.30.1.1:5000/myproject/python-36-centos7]
07:39:43.086 image.py INFO Preparing
07:39:43.086 image.py INFO Preparing
07:39:43.086 image.py INFO Preparing
07:39:43.086 image.py INFO Preparing
07:39:43.086 image.py INFO Preparing
07:39:43.087 image.py INFO Preparing
07:39:43.087 image.py INFO Preparing
07:39:43.087 image.py INFO Preparing
07:39:43.087 image.py INFO Preparing
07:39:43.117 image.py INFO Waiting
07:39:43.120 image.py INFO Waiting
07:39:43.121 image.py INFO Waiting
07:39:43.121 image.py INFO Waiting
07:39:43.261 image.py INFO Layer already exists
07:39:43.361 image.py INFO Layer already exists
07:39:43.365 image.py INFO Layer already exists
07:39:43.366 image.py INFO Layer already exists
07:39:43.412 image.py INFO Layer already exists
07:39:43.413 image.py INFO Layer already exists
07:39:43.472 image.py INFO Layer already exists
07:39:43.515 image.py INFO Layer already exists
07:39:43.522 image.py INFO Layer already exists
07:39:43.747 image.py INFO latest: digest: sha256:7ab4d97b2f68a261dc32fe75dfa9aa8990e258f6f911f13a2c165b228eff45d2 size: 2211
07:39:43.749 backend.py INFO Creating new app in project myproject
07:39:44.305 backend.py INFO Waiting for service to get ready
07:40:32.534 backend.py INFO Connection to service established and return expected output!
07:40:33.487 backend.py INFO Deleting app
07:40:35.872 backend.py INFO buildconfig "app-45yb" deleted
07:40:35.872 backend.py INFO imagestream "app-45yb" deleted
07:40:35.872 backend.py INFO deploymentconfig "app-45yb" deleted
07:40:35.872 backend.py INFO service "app-45yb" deleted
07:40:35.872 backend.py INFO pod "app-45yb-1-nscg0" deleted
Both backends implemented during Google summer of code needs more improvements. Orchestration support is too complicated to be fully covered during 3 months period. I am planning to continue to maintain this project and pushing forward development of orchestration support. If you would be interested in contributing to this project just contact me on telegram @rpitonak and we can figure out how you can help.
Every year Fedora project organizing their community conference named Flock. This year all GSOC students and mentors were presenting about our projects and experience with GSOC. This was great opportunity for us to get feedback and practice also our presenting skills.Thank you fedora!
Application period
- minor update of .gitignore
- Image cleanup policy upgrade
- real examples linked from README.md
- docker backend cleanup containers on exit
- don't create .pyc files in conu container
- run_via_binary accept command as a list of strings
- don't create pyc files in test container
Coding period
- Metadata classes for image and container
- Documentation of metadata class
- First shot at CONTRIBUTING.md
- get container metadata from docker metadata
- docker run parameters
- drop enum34 requirment for python3
- get_metadata for DockerImage
- name of image inside Makefile
- Kubernetes Pod support
- k8s - Service and Deployment
- K8s backend improvements
- create k8s deployment from template
- Minikube for CI
- k8s cleanup policy enum and cleanup methods
- WIP OpenShift origin backend
You can see graph of my contributions.
I am really happy that I could be part of fedora community during this summer. Once again I want to thanks my mentors for all support and help that they provided to me. I am definitely going to be part of fedora community also after GSOC, working on containers in new esthablished Container SIG and contributing to conu as well.
See you somewhere around!