Instantly share code, notes, and snippets.

@Filirom1 /README.md
Last active Aug 29, 2015

Embed
What would you like to do?
What happens when you create an application on OpenShift.

What happens when you create an application on OpenShift.

             __________                __________
            |          |     mco      |          |__
rhc ----->  |  Broker  |  --------->  |   Node   |  |__
            |__________| \            |__________|  |  |
                  /\     \  --------->   |__________|  |
                  |       \ --------->      |__________|
             _____|____ 
            |          |
            |  Console |
            |__________|

Rhc and Console contact the Broker

The user create an application from the CLI rhc or via the web-console. In both case, rhc and the web-console will contact the broker (the REST API) to create the application.

Broker contact Nodes via Mcollective

On each Node, there is an mcollectiver server running on it. The broker is an mcollective client, and will query the mcollective cluster to find nodes.

For exemple if you want to retrieve all nodes of the mcollective cluster, you can query this command on the broker:

$ oo-mco ping
node4.example.com              time=109.14 ms
node3.example.com              time=148.76 ms
node1.example.com              time=149.25 ms
node2.example.com              time=149.92 ms
node5.example.com              time=151.02 ms
node6.example.com              time=151.99 ms

If you only want the nodes with the profile small, you will query :

oo-mco ping --with-fact node_profile=small
node6.example.com              time=79.70 ms
node2.example.com              time=118.87 ms
node3.example.com              time=119.21 ms

OpenShift extend mcollective facts with some precious values. The openshift facts definition are declared here openshift_facts.rb

Here is a set of facts that is very helpful when you want to query specific Nodes.

cat /opt/rh/ruby193/root/etc/mcollective/facts.yaml
---
architecture: x86_64
kernel: Linux
domain: example.com
macaddress: 00:50:56:99:33:B4
osfamily: RedHat
operatingsystem: CentOS
lsbdistid: CentOS
fqdn: node6.example.com
hardwareisa: x86_64
hardwaremodel: x86_64
hostname: node6
id: root
...
memorysize: 24.58 GB
memoryfree: 16.98 GB
swapsize: 12.30 GB
swapfree: 11.23 GB
swapsize_mb: '12592.00'
swapfree_mb: '11495.39'
...
district_uuid: 51a4c5174b44050d57000001
district_active: true
district_first_uid: '1000'
district_max_uid: '6999'
public_ip: 10.34.56.63
public_hostname: node6.example.com
...
node_profile: small
max_active_gears: '100'
no_overcommit_active: false
quota_blocks: '1048576'
quota_files: '80000'
gears_active_count: 44
gears_total_count: 79
gears_idle_count: 34
gears_stopped_count: 1
gears_started_count: 43
gears_deploying_count: 1
gears_unknown_count: 0
gears_usage_pct: 79.0
gears_active_usage_pct: 44.0
git_repos: 0
capacity: '79.0'
active_capacity: '44.0'
...

When you ask OpenShift to create an application on a small Node, it will do an mcollective query to find the best node with node_profile=small. If you want more informations about gear placement, check this link Gear_Placement_Algorithm

If you only want the nodes with the openshift agent, you have to query this

$ oo-mco ping --with-agent openshift
node5.example.com              time=131.18 ms
node1.example.com              time=131.83 ms
node3.example.com              time=132.49 ms
node4.example.com              time=132.99 ms
node6.example.com              time=133.60 ms
node2.example.com              time=134.07 ms

In my case, every node in the cluster are used for openshift, so the same result is returned. OpenShift extend mcollective with an agent, to execute actions like stopping/starting/creating... applications. The agent is defined here openshift.ddl and here openshift.rb

To sum up, the role of mcollective is to find the best Node with the facts and run an action on it with the agent. Now let's take a look on what happen on the Node when mcollective ask it to create an app.

Node

The best way to understand what happens in the node is to take a look at the log file /var/log/openshift/node/platform.log. I will explain it line by line.

User/Group creation

OpenShift create a group and a linux user. The group and user id is the Gear UUID : 53cf7165e133e5d4fc00006f. The command oo-trap-user will be called when the user connect via ssh or git, to trap the user under an SELinux context.

July 23 10:25:15 INFO Shell command 'groupadd -g 1012 53cf7165e133e5d4fc00006f' ran. rc=0 out=
July 23 10:25:16 INFO Shell command 'useradd -u 1012 -d /var/lib/openshift/53cf7165e133e5d4fc00006f -s /usr/bin/oo-trap-user -g 1012 -c 'OpenShift guest' -m -k /etc/openshift/skel 53cf7165e133e5d4fc00006f -G ""' ran. rc=0 out=

Configure traffic control

tc (traffic control) is used to share the bandwidth fairly amoung applications, and to limit the bandwidth for some protocols, like SMTP to avoid spammers.

July 23 10:25:16 INFO Shell command 'ip link show dev eth0' ran. rc=0 out=2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
	link/ether 00:16:3e:5a:d7:1f brd ff:ff:ff:ff:ff:ff

July 23 10:25:16 INFO Shell command 'tc -s class show dev eth0 classid 1:3f4' ran. rc=0 out=
July 23 10:25:16 INFO Shell command 'cat /tmp/tc-batch20140723-7858-1ds0ls3 | tc -force -batch' ran. rc=0 out=

Set linux Quota

Quota are used to limit disk usage and the number of files allowed for an application.

July 23 10:25:17 INFO Shell command 'quota --always-resolve -w 53cf7165e133e5d4fc00006f' ran. rc=0 out=Disk quotas for user 53cf7165e133e5d4fc00006f (uid 1012): none

July 23 10:25:17 INFO Shell command 'setquota --always-resolve -u 53cf7165e133e5d4fc00006f 0 1048576 0 80000 -a /var' ran. rc=0 out=
July 23 10:25:17 INFO Shell command 'quota --always-resolve -w 53cf7165e133e5d4fc00006f' ran. rc=0 out=Disk quotas for user 53cf7165e133e5d4fc00006f (uid 1012):
	 Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
	 /dev/xvda3     172       0 1048576              25       0   80000

Cartridge initilization

Every cartridges installed on openshift is present in /var/lib/openshift/.cartridge_repository/<cartridge_name>. OpenShift will simply copy this cartidge folder inside the gear ~/<cartridge_name>.

July 23 10:25:18 INFO Creating cartridge directory 53cf7165e133e5d4fc00006f/php
July 23 10:25:18 INFO Shell command '/bin/cp -ad /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/README.md /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/env /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/COPYRIGHT /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/bin /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/metadata /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/LICENSE /var/lib/openshift/53cf7165e133e5d4fc00006f/php' ran. rc=0 out=
July 23 10:25:18 INFO Cartridge php recorded as primary within gear 53cf7165e133e5d4fc00006f
July 23 10:25:18 INFO Created cartridge directory 53cf7165e133e5d4fc00006f/php

Cartridge endpoints

OpenShift will allocate an IP and ports for every endpoints declared in the manifest.yml file. Here OPENSHIFT_PHP_IP=127.1.250.1, OPENSHIFT_PHP_PORT=8080

July 23 10:25:18 INFO Creating 1 private endpoints for 53cf7165e133e5d4fc00006f/php
July 23 10:25:18 INFO IPs already allocated for 8080 in gear 53cf7165e133e5d4fc00006f: []
July 23 10:25:18 INFO Created private endpoint for cart php in gear 53cf7165e133e5d4fc00006f: [OPENSHIFT_PHP_IP=127.1.250.1, OPENSHIFT_PHP_PORT=8080]
July 23 10:25:18 INFO Shell command '/usr/sbin/lsof -i @127.1.250.1:8080' ran. rc=1 out=

Cartridge scripts

A cartridge contains a set a bash scripts that are called in a specific order. At each steps, several template files are populated.

cartridge env

Environment variables definied in env/*.erb are populated

July 23 10:25:19 INFO Running setup for 53cf7165e133e5d4fc00006f/php
July 23 10:25:19 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/PHP_INI_SCAN_DIR.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/PHP_INI_SCAN_DIR\""' ran. rc=0 out=
July 23 10:25:19 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/PHPRC.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/PHPRC\""' ran. rc=0 out=
July 23 10:25:19 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/OPENSHIFT_PHP_LOG_DIR.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/OPENSHIFT_PHP_LOG_DIR\""' ran. rc=0 out=

cartridge bin/setup

The setup script is called

July 23 10:25:19 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/var/lib/openshift/53cf7165e133e5d4fc00006f/php/bin/setup --version 5.4\""' ran. rc=0 out=
July 23 10:25:19 INFO Ran /var/lib/openshift/53cf7165e133e5d4fc00006f/php/bin/setup --version 5.4 for 53cf7165e133e5d4fc00006f/php

cartridge template

The template, that will be use to create the default web application is populated

July 23 10:25:19 INFO Processing ERB templates for php
July 23 10:25:20 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/conf/performance.conf.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/conf/performance.conf\""' ran. rc=0 out=
July 23 10:25:20 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/conf.d/openshift.conf.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/conf.d/openshift.conf\""' ran. rc=0 out=
July 23 10:25:20 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/conf.d/performance.conf.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/conf.d/performance.conf\""' ran. rc=0 out=
July 23 10:25:20 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/php.ini.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/php.ini\""' ran. rc=0 out=

cartridge bin/install

The install script is called

July 23 10:25:20 INFO Running install for 53cf7165e133e5d4fc00006f/php
July 23 10:25:22 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/var/lib/openshift/53cf7165e133e5d4fc00006f/php/bin/install --version 5.4\""' ran. rc=0 out=CONFIGURATION (CHANNEL PEAR.PHP.NET):
=====================================
Auto-discover new Channels     auto_discover    <not set>
...

July 23 10:25:22 INFO Ran /var/lib/openshift/53cf7165e133e5d4fc00006f/php/bin/install --version 5.4 for 53cf7165e133e5d4fc00006f/php
CONFIGURATION (CHANNEL PEAR.PHP.NET):
=====================================
Auto-discover new Channels     auto_discover    <not set>
...

git repo

OpenShift pushes are based on git. The user push to a repo defined like this : ssh://<LINUX_USER_GEAR_UUID>@<NODE_IP>/~/git/<APP_NAME>.git/ Now OpenShift will create a git repo.

It creates the first commit based on the cartridge template.

July 23 10:25:22 INFO Creating gear repo for 53cf7165e133e5d4fc00006f/php from ``
July 23 10:25:22 INFO Using '/var/lib/openshift/53cf7165e133e5d4fc00006f/php/usr/template' to populate git repository for 53cf7165e133e5d4fc00006f
July 23 10:25:22 INFO Shell command '/bin/cp -ad /var/lib/openshift/53cf7165e133e5d4fc00006f/php/usr/template /var/lib/openshift/53cf7165e133e5d4fc00006f/git' ran. rc=0 out=
July 23 10:25:22 INFO Shell command '\
set -xe;
git init;
git config user.email "builder@example.com";
git config user.name "Template builder";
git config core.logAllRefUpdates true;
git add -f .;
git commit -a -m "Creating template";
' ran. rc=0 out=Initialized empty Git repository in /var/lib/openshift/53cf7165e133e5d4fc00006f/git/template/.git/
[master (root-commit) 34738b3] Creating template
10 files changed, 352 insertions(+), 0 deletions(-)
create mode 100644 .openshift/README.md
create mode 100644 .openshift/action_hooks/README.md
create mode 100644 .openshift/cron/README.cron
create mode 100644 .openshift/cron/daily/.gitignore
create mode 100644 .openshift/cron/hourly/.gitignore
create mode 100644 .openshift/cron/minutely/.gitignore
create mode 100644 .openshift/cron/monthly/.gitignore
create mode 100644 .openshift/cron/weekly/README
create mode 100644 .openshift/cron/weekly/chrono.dat
create mode 100755 .openshift/cron/weekly/chronograph
create mode 100644 .openshift/cron/weekly/jobs.allow
create mode 100644 .openshift/cron/weekly/jobs.deny
create mode 100644 .openshift/markers/README.md
create mode 100644 .openshift/pear.txt
create mode 100644 index.php

It creates a git bare repo. Bare repo are git repo without workspace. It is only used for servers where you don't commit directly on the repo.

July 23 10:25:22 INFO Shell command '\
set -xe;
git clone --bare --no-hardlinks template php.git;
GIT_DIR=./php.git git config core.logAllRefUpdates true;
GIT_DIR=./php.git git config pack.windowMemory "200m";
GIT_DIR=./php.git git repack;
' ran. rc=0 out=Initialized empty Git repository in /var/lib/openshift/53cf7165e133e5d4fc00006f/git/php.git/

app-root/repo

The app-root/runtime/repo or app-root/repo contains the source code of your application. It will be extracted from git.

July 23 10:25:23 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"
set -xe;
shopt -s dotglob;
if [ "$(find objects -type f 2>/dev/null | wc -l)" -eq "0" ]; then
  exit 0;
fi
git archive --format=tar master | (cd /var/lib/openshift/53cf7165e133e5d4fc00006f/app-root/runtime/repo && tar --warning=no-timestamp -xf -);
\""' ran. rc=0 out=

cartridge endpoint mapping

The OpenShift Node contains a httpd server listening on the port 80 and 443. When you contact you application, it first go the Apache Httpd, then with a rewrite rule it is send to the real web server: 127.1.250.1:8080.

Httpd is configured with a particular configuration file openshift_route.include, that will read rewrite rules from /etc/httpd/conf.d/openshift/nodes.txt.

July 23 10:25:23 INFO Connecting frontend mapping for 53cf7165e133e5d4fc00006f/php: [] => [127.1.250.1:8080] with options: {"websocket"=>true, "protocols"=>["http", "ws"]}
July 23 10:25:23 INFO Shell command '/usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20140723-7858-buro17/new.db' ran. rc=0 out=
July 23 10:25:23 INFO Connecting frontend mapping for 53cf7165e133e5d4fc00006f/php: [/health] => [127.1.250.1:8080] with options: {"health"=>true, "protocols"=>["http", "ws"]}
July 23 10:25:23 INFO Shell command '/usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20140723-7858-1j9hnb4/new.db' ran. rc=0 out=

Note that there is an other reverse proxy for the websocket protocol, listening on the port 8000 and 8443.

app-deployments

OpenShift backup your deployed application into the app-deployments dir so you can rollback to this deployement version latter.

The deployment will save the app-root/repo.

July 23 10:25:24 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/rsync -av --delete /var/lib/openshift/53cf7165e133e5d4fc00006f/app-root/runtime/repo/ /var/lib/openshift/53cf7165e133e5d4fc00006f/app-deployments/2014-07-23_10-25-16.759/repo/\""' ran. rc=0 out=sending incremental file list
./
index.php
.openshift/
.openshift/README.md
.openshift/pear.txt
.openshift/action_hooks/
.openshift/action_hooks/README.md
.openshift/cron/
.openshift/cron/README.cron
.openshift/cron/daily/
.openshift/cron/daily/.gitignore
.openshift/cron/hourly/
.openshift/cron/hourly/.gitignore
.openshift/cron/minutely/
.openshift/cron/minutely/.gitignore
.openshift/cron/monthly/
.openshift/cron/monthly/.gitignore
.openshift/cron/weekly/
.openshift/cron/weekly/README
.openshift/cron/weekly/chrono.dat
.openshift/cron/weekly/chronograph
.openshift/cron/weekly/jobs.allow
.openshift/cron/weekly/jobs.deny
.openshift/markers/
.openshift/markers/README.md

sent 44302 bytes  received 336 bytes  89276.00 bytes/sec
total size is 43060  speedup is 0.96

The deployment will save the app-root/runtime/dependencies.

July 23 10:25:24 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/rsync -av --delete /var/lib/openshift/53cf7165e133e5d4fc00006f/app-root/runtime/dependencies/ /var/lib/openshift/53cf7165e133e5d4fc00006f/app-deployments/2014-07-23_10-25-16.759/dependencies/\""' ran. rc=0 out=sending incremental file list
./
php/
php/phplib/
php/phplib/pear/
php/phplib/pear/cache/
php/phplib/pear/cfg/
php/phplib/pear/data/
php/phplib/pear/docs/
php/phplib/pear/download/
php/phplib/pear/ext/
php/phplib/pear/pear/
php/phplib/pear/pear/php/
php/phplib/pear/pear/php/.filemap
php/phplib/pear/pear/php/.lock
php/phplib/pear/pear/php/.channels/
php/phplib/pear/pear/php/.channels/__uri.reg
php/phplib/pear/pear/php/.channels/doc.php.net.reg
php/phplib/pear/pear/php/.channels/pear.php.net.reg
php/phplib/pear/pear/php/.channels/pecl.php.net.reg
php/phplib/pear/pear/php/.channels/.alias/
php/phplib/pear/pear/php/.channels/.alias/pear.txt
php/phplib/pear/pear/php/.channels/.alias/pecl.txt
php/phplib/pear/pear/php/.channels/.alias/phpdocs.txt
php/phplib/pear/pear/php/.registry/
php/phplib/pear/pear/php/.registry/.channel.__uri/
php/phplib/pear/pear/php/.registry/.channel.doc.php.net/
php/phplib/pear/pear/php/.registry/.channel.pecl.php.net/
php/phplib/pear/php/
php/phplib/pear/temp/
php/phplib/pear/tests/
php/phplib/pear/www/

sent 3116 bytes  received 270 bytes  6772.00 bytes/sec
total size is 2052  speedup is 0.61

The deployment will save the app-root/runtime/build-dependencies.

July 23 10:25:24 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/rsync -av --delete /var/lib/openshift/53cf7165e133e5d4fc00006f/app-root/runtime/build-dependencies/ /var/lib/openshift/53cf7165e133e5d4fc00006f/app-deployments/2014-07-23_10-25-16.759/build-dependencies/\""' ran. rc=0 out=sending incremental file list

sent 79 bytes  received 12 bytes  182.00 bytes/sec
total size is 0  speedup is 0.00

The whole deployment is then archived

July 23 10:25:24 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"tar -c --exclude metadata.json . | tar -xO | sha1sum | cut -f 1 -d ' '\""' ran. rc=0 out=374ecdb4c8596ca3018f471eb6ca878a41d10b3b

And tagged with the git version

July 23 10:25:25 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"
set -xe;
git rev-parse --short master
\""' ran. rc=0 out=34738b3

bin/control start

Now that everything is in place, the app is started with the bin/control start script

July 23 10:25:25 INFO 53cf7165e133e5d4fc00006f start against 'php'
July 23 10:25:26 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"set -e; /var/lib/openshift/53cf7165e133e5d4fc00006f/php/bin/control start \""' ran. rc=0 out=Starting PHP 5.4 cartridge (Apache+mod_php)
Application directory "/" selected as DocumentRoot

bin/post_install

And at the end, the post install script is called.

July 23 10:25:26 INFO Running post_install for 53cf7165e133e5d4fc00006f/php
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment