__________ __________
| | mco | |__
rhc -----> | Broker | ---------> | Node | |__
|__________| \ |__________| | |
/\ \ ---------> |__________| |
| \ ---------> |__________|
_____|____
| |
| Console |
|__________|
The user create an application from the CLI rhc or via the web-console. In both case, rhc and the web-console will contact the broker (the REST API) to create the application.
On each Node, there is an mcollectiver server running on it. The broker is an mcollective client, and will query the mcollective cluster to find nodes.
For exemple if you want to retrieve all nodes of the mcollective cluster, you can query this command on the broker:
$ oo-mco ping
node4.example.com time=109.14 ms
node3.example.com time=148.76 ms
node1.example.com time=149.25 ms
node2.example.com time=149.92 ms
node5.example.com time=151.02 ms
node6.example.com time=151.99 ms
If you only want the nodes with the profile small, you will query :
oo-mco ping --with-fact node_profile=small
node6.example.com time=79.70 ms
node2.example.com time=118.87 ms
node3.example.com time=119.21 ms
OpenShift extend mcollective facts with some precious values. The openshift facts definition are declared here openshift_facts.rb
Here is a set of facts that is very helpful when you want to query specific Nodes.
cat /opt/rh/ruby193/root/etc/mcollective/facts.yaml
---
architecture: x86_64
kernel: Linux
domain: example.com
macaddress: 00:50:56:99:33:B4
osfamily: RedHat
operatingsystem: CentOS
lsbdistid: CentOS
fqdn: node6.example.com
hardwareisa: x86_64
hardwaremodel: x86_64
hostname: node6
id: root
...
memorysize: 24.58 GB
memoryfree: 16.98 GB
swapsize: 12.30 GB
swapfree: 11.23 GB
swapsize_mb: '12592.00'
swapfree_mb: '11495.39'
...
district_uuid: 51a4c5174b44050d57000001
district_active: true
district_first_uid: '1000'
district_max_uid: '6999'
public_ip: 10.34.56.63
public_hostname: node6.example.com
...
node_profile: small
max_active_gears: '100'
no_overcommit_active: false
quota_blocks: '1048576'
quota_files: '80000'
gears_active_count: 44
gears_total_count: 79
gears_idle_count: 34
gears_stopped_count: 1
gears_started_count: 43
gears_deploying_count: 1
gears_unknown_count: 0
gears_usage_pct: 79.0
gears_active_usage_pct: 44.0
git_repos: 0
capacity: '79.0'
active_capacity: '44.0'
...
When you ask OpenShift to create an application on a small Node, it will do an mcollective query to find the best node with node_profile=small. If you want more informations about gear placement, check this link Gear_Placement_Algorithm
If you only want the nodes with the openshift agent, you have to query this
$ oo-mco ping --with-agent openshift
node5.example.com time=131.18 ms
node1.example.com time=131.83 ms
node3.example.com time=132.49 ms
node4.example.com time=132.99 ms
node6.example.com time=133.60 ms
node2.example.com time=134.07 ms
In my case, every node in the cluster are used for openshift, so the same result is returned. OpenShift extend mcollective with an agent, to execute actions like stopping/starting/creating... applications. The agent is defined here openshift.ddl and here openshift.rb
To sum up, the role of mcollective is to find the best Node with the facts and run an action on it with the agent. Now let's take a look on what happen on the Node when mcollective ask it to create an app.
The best way to understand what happens in the node is to take a look at the log file /var/log/openshift/node/platform.log.
I will explain it line by line.
OpenShift create a group and a linux user. The group and user id is the Gear UUID : 53cf7165e133e5d4fc00006f.
The command oo-trap-user will be called when the user connect via ssh or git, to trap the user under an SELinux context.
July 23 10:25:15 INFO Shell command 'groupadd -g 1012 53cf7165e133e5d4fc00006f' ran. rc=0 out=
July 23 10:25:16 INFO Shell command 'useradd -u 1012 -d /var/lib/openshift/53cf7165e133e5d4fc00006f -s /usr/bin/oo-trap-user -g 1012 -c 'OpenShift guest' -m -k /etc/openshift/skel 53cf7165e133e5d4fc00006f -G ""' ran. rc=0 out=
tc (traffic control) is used to share the bandwidth fairly amoung applications, and to limit the bandwidth for some protocols, like SMTP to avoid spammers.
July 23 10:25:16 INFO Shell command 'ip link show dev eth0' ran. rc=0 out=2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
link/ether 00:16:3e:5a:d7:1f brd ff:ff:ff:ff:ff:ff
July 23 10:25:16 INFO Shell command 'tc -s class show dev eth0 classid 1:3f4' ran. rc=0 out=
July 23 10:25:16 INFO Shell command 'cat /tmp/tc-batch20140723-7858-1ds0ls3 | tc -force -batch' ran. rc=0 out=
Quota are used to limit disk usage and the number of files allowed for an application.
July 23 10:25:17 INFO Shell command 'quota --always-resolve -w 53cf7165e133e5d4fc00006f' ran. rc=0 out=Disk quotas for user 53cf7165e133e5d4fc00006f (uid 1012): none
July 23 10:25:17 INFO Shell command 'setquota --always-resolve -u 53cf7165e133e5d4fc00006f 0 1048576 0 80000 -a /var' ran. rc=0 out=
July 23 10:25:17 INFO Shell command 'quota --always-resolve -w 53cf7165e133e5d4fc00006f' ran. rc=0 out=Disk quotas for user 53cf7165e133e5d4fc00006f (uid 1012):
Filesystem blocks quota limit grace files quota limit grace
/dev/xvda3 172 0 1048576 25 0 80000
Every cartridges installed on openshift is present in /var/lib/openshift/.cartridge_repository/<cartridge_name>.
OpenShift will simply copy this cartidge folder inside the gear ~/<cartridge_name>.
July 23 10:25:18 INFO Creating cartridge directory 53cf7165e133e5d4fc00006f/php
July 23 10:25:18 INFO Shell command '/bin/cp -ad /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/README.md /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/env /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/COPYRIGHT /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/bin /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/metadata /var/lib/openshift/.cartridge_repository/redhat-php/0.0.20/LICENSE /var/lib/openshift/53cf7165e133e5d4fc00006f/php' ran. rc=0 out=
July 23 10:25:18 INFO Cartridge php recorded as primary within gear 53cf7165e133e5d4fc00006f
July 23 10:25:18 INFO Created cartridge directory 53cf7165e133e5d4fc00006f/php
OpenShift will allocate an IP and ports for every endpoints declared in the manifest.yml file. Here OPENSHIFT_PHP_IP=127.1.250.1, OPENSHIFT_PHP_PORT=8080
July 23 10:25:18 INFO Creating 1 private endpoints for 53cf7165e133e5d4fc00006f/php
July 23 10:25:18 INFO IPs already allocated for 8080 in gear 53cf7165e133e5d4fc00006f: []
July 23 10:25:18 INFO Created private endpoint for cart php in gear 53cf7165e133e5d4fc00006f: [OPENSHIFT_PHP_IP=127.1.250.1, OPENSHIFT_PHP_PORT=8080]
July 23 10:25:18 INFO Shell command '/usr/sbin/lsof -i @127.1.250.1:8080' ran. rc=1 out=
A cartridge contains a set a bash scripts that are called in a specific order. At each steps, several template files are populated.
Environment variables definied in env/*.erb are populated
July 23 10:25:19 INFO Running setup for 53cf7165e133e5d4fc00006f/php
July 23 10:25:19 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/PHP_INI_SCAN_DIR.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/PHP_INI_SCAN_DIR\""' ran. rc=0 out=
July 23 10:25:19 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/PHPRC.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/PHPRC\""' ran. rc=0 out=
July 23 10:25:19 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/OPENSHIFT_PHP_LOG_DIR.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/env/OPENSHIFT_PHP_LOG_DIR\""' ran. rc=0 out=
The setup script is called
July 23 10:25:19 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/var/lib/openshift/53cf7165e133e5d4fc00006f/php/bin/setup --version 5.4\""' ran. rc=0 out=
July 23 10:25:19 INFO Ran /var/lib/openshift/53cf7165e133e5d4fc00006f/php/bin/setup --version 5.4 for 53cf7165e133e5d4fc00006f/php
The template, that will be use to create the default web application is populated
July 23 10:25:19 INFO Processing ERB templates for php
July 23 10:25:20 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/conf/performance.conf.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/conf/performance.conf\""' ran. rc=0 out=
July 23 10:25:20 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/conf.d/openshift.conf.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/conf.d/openshift.conf\""' ran. rc=0 out=
July 23 10:25:20 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/conf.d/performance.conf.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/conf.d/performance.conf\""' ran. rc=0 out=
July 23 10:25:20 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/oo-erb -S 2 -- /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/php.ini.erb > /var/lib/openshift/53cf7165e133e5d4fc00006f/php/configuration/etc/php.ini\""' ran. rc=0 out=
The install script is called
July 23 10:25:20 INFO Running install for 53cf7165e133e5d4fc00006f/php
July 23 10:25:22 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/var/lib/openshift/53cf7165e133e5d4fc00006f/php/bin/install --version 5.4\""' ran. rc=0 out=CONFIGURATION (CHANNEL PEAR.PHP.NET):
=====================================
Auto-discover new Channels auto_discover <not set>
...
July 23 10:25:22 INFO Ran /var/lib/openshift/53cf7165e133e5d4fc00006f/php/bin/install --version 5.4 for 53cf7165e133e5d4fc00006f/php
CONFIGURATION (CHANNEL PEAR.PHP.NET):
=====================================
Auto-discover new Channels auto_discover <not set>
...
OpenShift pushes are based on git. The user push to a repo defined like this : ssh://<LINUX_USER_GEAR_UUID>@<NODE_IP>/~/git/<APP_NAME>.git/ Now OpenShift will create a git repo.
It creates the first commit based on the cartridge template.
July 23 10:25:22 INFO Creating gear repo for 53cf7165e133e5d4fc00006f/php from ``
July 23 10:25:22 INFO Using '/var/lib/openshift/53cf7165e133e5d4fc00006f/php/usr/template' to populate git repository for 53cf7165e133e5d4fc00006f
July 23 10:25:22 INFO Shell command '/bin/cp -ad /var/lib/openshift/53cf7165e133e5d4fc00006f/php/usr/template /var/lib/openshift/53cf7165e133e5d4fc00006f/git' ran. rc=0 out=
July 23 10:25:22 INFO Shell command '\
set -xe;
git init;
git config user.email "builder@example.com";
git config user.name "Template builder";
git config core.logAllRefUpdates true;
git add -f .;
git commit -a -m "Creating template";
' ran. rc=0 out=Initialized empty Git repository in /var/lib/openshift/53cf7165e133e5d4fc00006f/git/template/.git/
[master (root-commit) 34738b3] Creating template
10 files changed, 352 insertions(+), 0 deletions(-)
create mode 100644 .openshift/README.md
create mode 100644 .openshift/action_hooks/README.md
create mode 100644 .openshift/cron/README.cron
create mode 100644 .openshift/cron/daily/.gitignore
create mode 100644 .openshift/cron/hourly/.gitignore
create mode 100644 .openshift/cron/minutely/.gitignore
create mode 100644 .openshift/cron/monthly/.gitignore
create mode 100644 .openshift/cron/weekly/README
create mode 100644 .openshift/cron/weekly/chrono.dat
create mode 100755 .openshift/cron/weekly/chronograph
create mode 100644 .openshift/cron/weekly/jobs.allow
create mode 100644 .openshift/cron/weekly/jobs.deny
create mode 100644 .openshift/markers/README.md
create mode 100644 .openshift/pear.txt
create mode 100644 index.php
It creates a git bare repo. Bare repo are git repo without workspace. It is only used for servers where you don't commit directly on the repo.
July 23 10:25:22 INFO Shell command '\
set -xe;
git clone --bare --no-hardlinks template php.git;
GIT_DIR=./php.git git config core.logAllRefUpdates true;
GIT_DIR=./php.git git config pack.windowMemory "200m";
GIT_DIR=./php.git git repack;
' ran. rc=0 out=Initialized empty Git repository in /var/lib/openshift/53cf7165e133e5d4fc00006f/git/php.git/
The app-root/runtime/repo or app-root/repo contains the source code of your application.
It will be extracted from git.
July 23 10:25:23 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"
set -xe;
shopt -s dotglob;
if [ "$(find objects -type f 2>/dev/null | wc -l)" -eq "0" ]; then
exit 0;
fi
git archive --format=tar master | (cd /var/lib/openshift/53cf7165e133e5d4fc00006f/app-root/runtime/repo && tar --warning=no-timestamp -xf -);
\""' ran. rc=0 out=
The OpenShift Node contains a httpd server listening on the port 80 and 443. When you contact you application, it first go the Apache Httpd, then with a rewrite rule it is send to the real web server: 127.1.250.1:8080.
Httpd is configured with a particular configuration file openshift_route.include, that will read rewrite rules from /etc/httpd/conf.d/openshift/nodes.txt.
July 23 10:25:23 INFO Connecting frontend mapping for 53cf7165e133e5d4fc00006f/php: [] => [127.1.250.1:8080] with options: {"websocket"=>true, "protocols"=>["http", "ws"]}
July 23 10:25:23 INFO Shell command '/usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20140723-7858-buro17/new.db' ran. rc=0 out=
July 23 10:25:23 INFO Connecting frontend mapping for 53cf7165e133e5d4fc00006f/php: [/health] => [127.1.250.1:8080] with options: {"health"=>true, "protocols"=>["http", "ws"]}
July 23 10:25:23 INFO Shell command '/usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20140723-7858-1j9hnb4/new.db' ran. rc=0 out=
Note that there is an other reverse proxy for the websocket protocol, listening on the port 8000 and 8443.
OpenShift backup your deployed application into the app-deployments dir so you can rollback to this deployement version latter.
The deployment will save the app-root/repo.
July 23 10:25:24 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/rsync -av --delete /var/lib/openshift/53cf7165e133e5d4fc00006f/app-root/runtime/repo/ /var/lib/openshift/53cf7165e133e5d4fc00006f/app-deployments/2014-07-23_10-25-16.759/repo/\""' ran. rc=0 out=sending incremental file list
./
index.php
.openshift/
.openshift/README.md
.openshift/pear.txt
.openshift/action_hooks/
.openshift/action_hooks/README.md
.openshift/cron/
.openshift/cron/README.cron
.openshift/cron/daily/
.openshift/cron/daily/.gitignore
.openshift/cron/hourly/
.openshift/cron/hourly/.gitignore
.openshift/cron/minutely/
.openshift/cron/minutely/.gitignore
.openshift/cron/monthly/
.openshift/cron/monthly/.gitignore
.openshift/cron/weekly/
.openshift/cron/weekly/README
.openshift/cron/weekly/chrono.dat
.openshift/cron/weekly/chronograph
.openshift/cron/weekly/jobs.allow
.openshift/cron/weekly/jobs.deny
.openshift/markers/
.openshift/markers/README.md
sent 44302 bytes received 336 bytes 89276.00 bytes/sec
total size is 43060 speedup is 0.96
The deployment will save the app-root/runtime/dependencies.
July 23 10:25:24 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/rsync -av --delete /var/lib/openshift/53cf7165e133e5d4fc00006f/app-root/runtime/dependencies/ /var/lib/openshift/53cf7165e133e5d4fc00006f/app-deployments/2014-07-23_10-25-16.759/dependencies/\""' ran. rc=0 out=sending incremental file list
./
php/
php/phplib/
php/phplib/pear/
php/phplib/pear/cache/
php/phplib/pear/cfg/
php/phplib/pear/data/
php/phplib/pear/docs/
php/phplib/pear/download/
php/phplib/pear/ext/
php/phplib/pear/pear/
php/phplib/pear/pear/php/
php/phplib/pear/pear/php/.filemap
php/phplib/pear/pear/php/.lock
php/phplib/pear/pear/php/.channels/
php/phplib/pear/pear/php/.channels/__uri.reg
php/phplib/pear/pear/php/.channels/doc.php.net.reg
php/phplib/pear/pear/php/.channels/pear.php.net.reg
php/phplib/pear/pear/php/.channels/pecl.php.net.reg
php/phplib/pear/pear/php/.channels/.alias/
php/phplib/pear/pear/php/.channels/.alias/pear.txt
php/phplib/pear/pear/php/.channels/.alias/pecl.txt
php/phplib/pear/pear/php/.channels/.alias/phpdocs.txt
php/phplib/pear/pear/php/.registry/
php/phplib/pear/pear/php/.registry/.channel.__uri/
php/phplib/pear/pear/php/.registry/.channel.doc.php.net/
php/phplib/pear/pear/php/.registry/.channel.pecl.php.net/
php/phplib/pear/php/
php/phplib/pear/temp/
php/phplib/pear/tests/
php/phplib/pear/www/
sent 3116 bytes received 270 bytes 6772.00 bytes/sec
total size is 2052 speedup is 0.61
The deployment will save the app-root/runtime/build-dependencies.
July 23 10:25:24 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"/usr/bin/rsync -av --delete /var/lib/openshift/53cf7165e133e5d4fc00006f/app-root/runtime/build-dependencies/ /var/lib/openshift/53cf7165e133e5d4fc00006f/app-deployments/2014-07-23_10-25-16.759/build-dependencies/\""' ran. rc=0 out=sending incremental file list
sent 79 bytes received 12 bytes 182.00 bytes/sec
total size is 0 speedup is 0.00
The whole deployment is then archived
July 23 10:25:24 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"tar -c --exclude metadata.json . | tar -xO | sha1sum | cut -f 1 -d ' '\""' ran. rc=0 out=374ecdb4c8596ca3018f471eb6ca878a41d10b3b
And tagged with the git version
July 23 10:25:25 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"
set -xe;
git rev-parse --short master
\""' ran. rc=0 out=34738b3
Now that everything is in place, the app is started with the bin/control start script
July 23 10:25:25 INFO 53cf7165e133e5d4fc00006f start against 'php'
July 23 10:25:26 INFO Shell command '/sbin/runuser -s /bin/sh 53cf7165e133e5d4fc00006f -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1012' /bin/sh -c \"set -e; /var/lib/openshift/53cf7165e133e5d4fc00006f/php/bin/control start \""' ran. rc=0 out=Starting PHP 5.4 cartridge (Apache+mod_php)
Application directory "/" selected as DocumentRoot
And at the end, the post install script is called.
July 23 10:25:26 INFO Running post_install for 53cf7165e133e5d4fc00006f/php