Skip to content

Instantly share code, notes, and snippets.

@andrewseidl
Last active September 9, 2016 16:59
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save andrewseidl/4a8ffec4b75de41c6a28 to your computer and use it in GitHub Desktop.
Save andrewseidl/4a8ffec4b75de41c6a28 to your computer and use it in GitHub Desktop.

                 ##            .          
           ## ## ##           ==          
        ## ## ## ## ##       ===
     /""""""""""""""""""\___/ ===-
~~~ {~~ ~~~~ ~~~ ~~~ ~~ ~~ ~ /  ===- ~~~
     \       o            __/           
      \____/           __/
       \______________/

        |          |
     __ |  __   __ | _  __   _
    /  \| /  \ /   |/  / _\ | 
    \__/| \__/ \__ |\_ \__  |
     for scientific computing

A Modest Proposal

  In addition to your regular packaging,
build containers for your simulation code

   This will help 'driveby downloads':
   start quickly, just to play around

Why?

      `""==,,__  
        `"==..__"=..__ _    _..-==""_
   -----     .-,`"=/ /\ \""/_)==""``
  / I'm \   ( (    | | | \/ |
  \lazy /    \ '.  |  \;  \ /
      \       |  \ |   |   ||
       \ ,-._.'  |_|   |   ||
        .\_/\     -'   ;   Y
       |  `  |        /    |-.
       '. __/_    _.-'     /'
   jgs        `'-.._____.-'

Installing dependencies suc^H^H^H is a pain

apt-get install cmake make libboost-all-dev
libxml2-dev libxml++2.6-dev libsqlite3-dev pip install -r requirements.txt # in a venv cmake ../src && make # hope it's all good

curl Miniconda.sh | bash conda install packages

  ___________________
 < that's not so bad >
  -------------------
         \   ^__^
          \  (oo)\_______
             (__)\       )\/\
                 ||----w |
                 ||     ||

But now repeat on multiple machines

    .--.          .--.          .--.     
   |o_o |        |o_o |        |o_o |    
   |:_/ |        |:_/ |        |:_/ |    
  //   \ \      //   \ \      //   \ \   
 (|     | )    (|     | )    (|     | )  
/'\_   _/`\   /'\_   _/`\   /'\_   _/`\  
\___)=(___/   \___)=(___/   \___)=(___/  

    .--.          .--.          .--.     
   |o_o |        |o_o |        |o_o |    
   |:_/ |        |:_/ |        |:_/ |    
  //   \ \      //   \ \      //   \ \   
 (|     | )    (|     | )    (|     | )  
/'\_   _/`\   /'\_   _/`\   /'\_   _/`\  
\___)=(___/   \___)=(___/   \___)=(___/  

There's no guarantee that your users' or 
 clusters' machines will have a working
      combination of dependencies.


     (unless they're running NixOS)

            Enter Docker

    _________
   < whales! >
    ---------
           \   ^__^
            \  (oo)\_______
               (__)\       )\/\
                   ||----w |
                   ||     ||

       Nope, containers.

Containers

Wikipedia:

LXC (Linux Containers) is an operating-system-level virtualization environment for running multiple isolated Linux systems (containers) on a single Linux control host.

    _____
   < wat >
    -----
           \   ^__^
            \  (oo)\_______
               (__)\       )\/\
                   ||----w |
                   ||     ||

What are containers?

Run multiple copies of 'Linux' at once.

Each app/container has its own:

  • file system
    • libraries, binaries, etc
  • PIDs
  • network

    ___________
   < like VMs! >
    -----------
           \   ^__^
            \  (oo)\_______
               (__)\       )\/\
                   ||----w |
                   ||     ||

      sort of...

Virtual Machines

 ---------   ---------   ---------
 |  App  |   |  App  |   |  App  |
 |-------|   |-------|   |-------|
 | bins/ |   | bins/ |   | bins/ |
 | libs  |   | libs  |   | libs  |
 |-------|   |-------|   |-------|
 | Guest |   | Guest |   | Guest |
 |  OS   |   |  OS   |   |  OS   |
 |-------------------------------|
 |          Hypervisor           |
 |-------------------------------|
 |           Host OS             |
 |-------------------------------|
 |            Server             |
 ---------------------------------

Containers

 ---------   ---------   ---------
 |  App  |   |  App  |   |  App  |
 |-------|   |-------|   |-------|
 | bins/ |   | bins/ |   | bins/ |
 | libs  |   | libs  |   | libs  |
 |-------------------------------|
 |            Docker             |
 |-------------------------------|
 |           Host OS             |
 |-------------------------------|
 |            Server             |
 ---------------------------------

Many Containers

 ---------   ---------   ---------
 | A | A |   | A | A |   | A | A |
 | p | p |   | p | p |   | p | p |
 | p | p |   | p | p |   | p | p |
 |-------|   |-------|   |-------|
 | bins/ |   | bins/ |   | bins/ |
 | libs  |   | libs  |   | libs  |
 |-------------------------------|
 |            Docker             |
 |-------------------------------|
 |           Host OS             |
 |-------------------------------|
 |            Server             |
 ---------------------------------

Actually...

(Docker manages the containers, but otherwise isn't involved)

 ---------   ---------   ---------
 | A | A |   | A | A |   | A | A |
 | p | p |   | p | p |   | p | p |
 | p | p |   | p | p |   | p | p |
 |-------|   |-------|   |-------|
 | bins/ |   | bins/ |   | bins/ |
 | libs  |   | libs  |   | libs  |
 |-------------------------------|
 |           Host OS             |
 |-------------------------------|
 |            Server             |
 ---------------------------------

  _______________________________
 < so what do you propose again? >
  -------------------------------
         \   ^__^
          \  (oo)\_______
             (__)\       )\/\
                 ||----w |
                 ||     ||

A Modest Proposal

  In addition to your regular packaging,
build containers for your simulation code

    Running the code becomes a simple:
      docker run my-super-simulation

  _______________
 < what do I do? >
  ---------------
         \   ^__^
          \  (oo)\_______
             (__)\       )\/\
                 ||----w |
                 ||     ||

Building images

Simply...

  vim Dockerfile
  docker build -t .... TIME'S UP

Learn more

             docker.com

        Email me: 🐋 @♥☢.ws
(whales@andrewseidl.com also works)

Docker and Containers Madison Meetup

A Modest Proposal, Jonathan Swift

Your precious code is a burden
    on its maintainers and
        the publick.
           Eat it.
    Rewrite in Go or Rust.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment