Skip to content

Instantly share code, notes, and snippets.

@maxstr

maxstr/proposal.md

Last active August 29, 2015 13:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save maxstr/9579808 to your computer and use it in GitHub Desktop.
Save maxstr/9579808 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
proposal.md

Proposer: Max Stritzinger

University of California, Santa Barbara
mstritzing@gmail.com
CARAM on freenode, synirc, gamesurge

Synopsis

This is a proposal for TimVideos for GSoC 2014. This proposal is centered around the 'Dockerization' of Flumotion allowing for quick and portable deployments of the TimVideos streaming system. Following completion of Dockerizing the streaming system, work will begin on an automated build tool based on the newly Docker'd components.

Dockerization

Docker is an open-source framework allowing for the creation of portable application containers. Docker allows for an application and all of its dependencies to be wrapped in one of these containers and deployed with ease to any host with Docker installed. My proposal is to wrap the streaming system up in a few Docker packages to make it extremely easy and quick to deploy. For example, following 'Dockerization', setting up the website portion of the streaming system might be as simple as:

docker pull streaming-system/website
docker run -name website -p 80:80 -d streaming-system/website python streamingsystem/tools/server_start.py some_conf_file

Automated Build System

Dockerization would be the meat of the project, however my belief is that time will allow for development of a script/tool to generate most of the streaming system automatically assisted by Docker. Following delivery of the Docker portion of the proposal, I would plan to continue working on an automated build/deployment tool. Docker fits in really well with an automated deployment tool. Docker supports running in daemon mode, so you can imagine PXE/AWS booting an image running a Docker daemon. Then, deployment and configuration for our sample website component becomes as simple as:

docker -H remote_host:4243 pull streaming-system/website
docker -H remote_host:4243 run -name website -p 80:80 -d streaming-system/website python streamingsystem/tools/server_start.py some_conf_file

You can imagine the ease at which this can be scripted out!

My Proposed Work

  1. Starting with a base Ubuntu Docker image, I will build a streaming-system/base Docker image with all dependencies (including watchdog, register) installed. This image will be capable of functioning as any of the three components, given some additional configuration

  2. Building off of the streaming-system/base package, I will build separate images for the Encoder, Collector, and Website components of streaming system. These images will come preconfigured for their specific component function and will be able to accept configuration arguments on startup for easy setup. This will involve generation of custom scripts/config files for each of the different components.

  3. streaming-system/(base, website, encoder, collector) will published to the docker.io index under the streaming-system or similar username

  4. Following the completion of the Docker portion of the project, I will then begin work on the automated build and deployment system.

  5. The automated build system will be integrated around a Cobbler daemon and the AWS Python API.

  6. The build tool will, based on the configuration file/utility, boot any PXE images it needs via cobbler and start EC2 instances via AWS, push the correct component(s) to each host (through a docker remote pull), and then start each docker instance based on the configuration options specified.

Timeline

  • Now - May 18th : Prior to the official start of the program I'll be active on IRC and poking around the streaming system components to gain familiarity with the different parts of the streaming system.

  • May 19th - May 25th: Development environment setup, setup of the streaming-system/base container.

  • May 26th - June 1st: Setup / Configuration of the streaming-system/website container. By the end of this week we will be able to 'docker pull' a website container and start the web service with a simple configuration file and start script.

  • June 2nd - June 11th: Setup / Configuration of the streaming-system/collector container. By the end of this week we will be able to 'docker pull' a collector container and start the encoding service with a simple configuration file and start script

  • June 12th - June 22nd: Setup / Configuration of the streaming-system/encoder container. By the end of this week we will be able to 'docker pull' an encoder container and start the collecting service with a simple configuration file and start script

  • June 23rd - June 29th: Dockerization should be fully complete! Work begins on the automated build system. By the end of this week we will have a tool that will take a component type, some parameters regarding it's associated components, and will output a configuration file.

  • June 30th - July 13th: Extend build tool (or possibly create a separate 'boot' tool) to incorporate host booting. We will be able to take a list of hosts and their types(EC2, PXE) and deploy a docker-enabled image to them.

  • July 14th - August 10th: Add support to our tool for specifying entire systems of components with the hosts they will run on. Tool will boot hosts, deploy docker containers to them, and start the components. This should correctly start the streaming system

  • August 11th - August 18th: Clean up documentation, final testing/debugging and debrief to TimVideos team.

Accountability

I am very much interested in completing my project successfully as I am sure the TimVideos team is. Some accountability is a great way to make sure that happens! I am (and will be) almost always available on IRC for interrogation and would propose bi or tri-weekly updates/progress meetings with my mentor(s) to ensure that I'm on the right track. In addition, I would plan on keeping a blog tracking my work throughout the project on my website.

About Me

  • Second year student at UC Santa Barbara, studying Biochemistry and Computer Science. I currently hold a 3.89 GPA
  • Student Web/App developer for Associated Students at UCSB.
  • Very available this summer (the second week of June I have one or two finals, but I'm free otherwise)
  • Familiar with many languages and web technologies including, but not limited to:
  • AWS
  • Python
    • Flask
  • Haskell
  • SQL (experienced with MySQL specifically)
  • PHP
    • Wordpress
  • General Bash/Shell scripting
  • (Somewhat rusty) DevOps/SysAdmin experience with:
  • CentOS/RHEL
  • Ubuntu Server
  • MySQL Server
  • I am, of course, available for contact almost anytime on IRC or through e-mail as listed above.

Comments, Concerns?

If there's anything you'd like to share, feel free to post it below or give me a ping on IRC.

@mithro
Copy link

mithro commented Mar 16, 2014

I'm not 100% convinced that you understand all the components inside the streaming system and how they fit together. Can you please expand that section further. The second diagram at https://github.com/timvideos/streaming-system might help (and would be good to reference).

@mithro
Copy link

mithro commented Mar 16, 2014

I feel like it is unclear exactly what the outcome of this proposal will be. Can restructure your proposal to explain in terms of the final results.

@mithro
Copy link

mithro commented Mar 16, 2014

How are you going to test this stuff?

@mithro
Copy link

mithro commented Mar 16, 2014

Can you reuse any existing docker images as a base?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment