Self Hosted Kadira APM - Docker Compose - Ubuntu

_Self-Hosted-Kadira-APM-Docker-Compose -Ubuntu.md

1 - Get an Ubuntu machine up and running (make sure ports 11011 and 4000 are open)

2 - Install docker + docker-compose (full_install_docker_compose.sh)

3 - Get a docker-compose.yml with mongodb, kadira-ui, kadira-engine and kadira-rma (docker-compose.yml)

3 - Run "sudo docker-compose up" (-d if you want it running on background)

4 - Access http://machineIp:4000 and login as "admin@gmail.com" and "admin" password

5 - Create your APP, get the ID and Secret

6 - Setup kadira in your app by code or settings.json (Config)

7 - Update your app from "free" plan to "paid" to get full access at Kadira (Update App Plan)

8 - Enjoy!

changePlan.md

Install MongoDB client if not already

$ sudo apt install mongodb-clients

Access MongoDB (make sure port 27018 is accessible)

$ mongo apm --host michineIP --port 27018

Execute update command (free, paid, startup, business, pro)

>  db.apps.update({},{$set:{plan:'pro'}},{multi:true})

Done!

config.md

Add Kadira package to your project

$ meteor add meteorhacks:kadira

Config Kadira in your code or settings.json

# code
Kadira.connect('xxxxxxxxxxx', 'xxxxxxxxxxxxxxx', {"endpoint":"http://machineIp:11011"});

#settings.json    
{  
  "kadira": {
    "appId": "xxxxxxxxxxx",
    "appSecret": "xxxxxxxxxxxxxxx",
    "options": {
        "endpoint":"http://machineIp:11011"
    }
  } 
}

You can find more advanced options here:

Link

docker-compose.yml

# Source: https://github.com/mdemri/kadira-docker-compose/blob/bece9ca502f0770ce9af7abda729d081b5147368/docker-compose.yml

version: '2'

services:
  kadira-engine:
    image: vladgolubev/kadira-engine
    environment:
      - MONGO_URL=mongodb://mongodb/apm
      - MONGO_SHARD_URL_one=mongodb://mongodb/apm
      - PORT=11011
    ports:
      - "11011:11011"
    restart: always
    depends_on:
        - mongodb

  kadira-rma:
    image: dkoo761/kadira-rma
    environment:
      - MONGO_URL=mongodb://mongodb/apm
    restart: always
    depends_on:
        - mongodb

  kadira-ui:
    image: vladgolubev/kadira-ui
    environment:
      - MONGO_URL=mongodb://mongodb/apm
      - MONGO_SHARD_URL_one=mongodb://mongodb/apm
    ports:
      - "4000:4000"
    restart: always
    depends_on:
        - mongodb
  mongodb:
    image: mongo
    ports:
    - "27018:27017"
    restart: always

full_install_docker_compose.sh

# Source: https://gist.github.com/wdullaer/f1af16bd7e970389bad3

sudo true
wget -qO- https://get.docker.com/ | sh
COMPOSE_VERSION=`git ls-remote https://github.com/docker/compose | grep refs/tags | grep -oP "[0-9]+\.[0-9][0-9]+\.[0-9]+$" | tail -n 1`
sudo sh -c "curl -L https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose"
sudo chmod +x /usr/local/bin/docker-compose
sudo sh -c "curl -L https://raw.githubusercontent.com/docker/compose/${COMPOSE_VERSION}/contrib/completion/bash/docker-compose > /etc/bash_completion.d/docker-compose"
cd /tmp
git clone https://gist.github.com/76b450a0c986e576e98b.git
cd 76b450a0c986e576e98b
sudo mv docker-cleanup /usr/local/bin/docker-cleanup
sudo chmod +x /usr/local/bin/docker-cleanup

This works pretty amazingly. Would be great if #1 was flushed out a bit more -- I had a bit of trouble there in initial setup, and now that I've rebooted my server, I can't get that to work at all. Using nginx -- port 80 is fine, but struggling to open 4000 and 11011 easily. I also imagine this is going to take a fair bit of customization to get it SSL compliant?

The command you want to execute to get the "paid" plan should be:
db.apps.update({},{$set:{plan:'business'}},{multi:true})

Thanks! I changed "paid" to "pro", it works like "business". I also changed the port from 11011 to 80, but that was because my VPC had that port already open. As I'm using all inside a VPC I did not feel the need to use SSL, let me know if you figure out how to do it!

Got an nginx config working, but... out of the blue (after a couple hours of successful tracking), it seems like my docker mongo image has messed up. Error below, with IP address redacted.

root@kadira:~# mongo apm --host <IP> --port 27018
MongoDB shell version: 2.6.10
connecting to: <IP>:27018/apm
2017-07-12T21:31:06.403+0000 warning: Failed to connect to <IP>:27018, reason: errno:111 Connection refused
2017-07-12T21:31:06.404+0000 Error: couldn't connect to server <IP>:27018 (<IP>), connection attempt failed at src/mongo/shell/mongo.js:148
exception: connect failed

Struggling through this now -- will hopefully figure it out soon :)

Update: The docker image for mongodb was stuck. Did a sudo reboot and it worked properly. Wondering if my $10 Digital Ocean droplet maybe had a heart attack. If it happens again, will attempt with a $20 drop.

Update v2: It went down again, so I upgraded to a $20 droplette, and it seems to be working fine now. So must be a minimum memory/CPU threshold for it to work well.

@PeterMerkert Can you specify your server info, i was using the minimum Digital Ocean droplet and it seem like this crash all the time.

@phucphung I run it on a 1 GB Memory / 20 GB Disk droplet and it works perfectly

@6uimorais Thanks so much for putting this together, it works like a charm. One question: How can we create additional accounts with this setup?

Anyone had any luck with changing the admin password through the UI? Always telling me that the old password must be different to the new one when I try change it

this kinda slow loading, like 1 mins to refresh the data in kadira. Can i change the refresh time to make it real time ? or just refresh every 30secs ?

Thank you for this awesome tutorial !

im failed at this step

https://gist.githubusercontent.com/wdullaer/76b450a0c986e576e98b/raw/4753eee44c81eb70fe58d9e2a56341587ac63e30/docker-cleanup

Removing containers:
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.34/containers/json?all=1: dial unix /var/run/docker.sock: connect: permission denied
Removing images:
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.34/images/json: dial unix /var/run/docker.sock: connect: permission denied
Removing volumes:
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.34/volumes/prune: dial unix /var/run/docker.sock: connect: permission denied

its working after add self to docker group

sudo usermod -aG docker ${USER}
and login back

do we have to clean old aggreation data by manually? will it getting slow after month?

this is another approach that i've used old days, but its getting slow day by day. then down. i had to stop mongo rs mode and drop db and oplog collection.

https://medium.com/@foodfitnesscode/setting-up-kadira-on-a-ubuntu-16-04-server-part-1-a821cdddd1e6

Mongodb image needs to be mongo:3.4, otherwise it will pull the latest version of mongo (3.6), which is not compatible with this setup.
Thank you very much for sharing this!

Hello everyone,
I got it running. however, I could not see the data.
I used 'settings.json' option at the root of my project

{
"kadira": {
"appId": "HdSwWdDJfzXn*****",
"appSecret": "c9efac10-5e7c-4ee0-9b3e-*****",
"options": {
"endpoint":"http://192.168.1.105:11011"
}
}
}

My app said: [ Kadira: completed instrumenting the app]

on Kadira web: localhost:4000, still showing "Configure Your App" page, the expected data not showing.

whats wrong?
Thanks a lot!

I got it working and connecting, but all the charts just show "Loading ..." in red and never get anywhere. Any suggestions?

@thumptech did you resolve your issue?

@thumptech did you resolve your issue?

Tried on another two machines and still have the same problem unfortunately.

Use mongodb 3.4 for this Kadira setup, this will resolve a lot of problems.
If someone is experiencing problem, just edit docker-compose.yml, and rerun docker-compose up -d

mongodb:
    image: mongo:3.4
    ports:
    - "27018:27017"
    restart: always

Also if you want to change password, connect to mongodb, open users collection, and change services.password.bcrypt field directly. You can use online tools like https://bcrypt-generator.com to hash your password first, before editing services.password.bcrypt field directly.

I have noticed that Kadira's mongo process is causing very high CPU usage even when meteor app is idle.
Meteor's mongo process is consuming less than 1% of cpu.
Kadira's mongo version is 3.4.9, and Meteor's mongo is 3.4.1.

I noticed that it was because my apm collection size was 10GB and as there are no indexes configured so it was causing lot of CPU usage.
After creating indexes it is better now. I will cleanup the old data as well.

https://stackoverflow.com/questions/23111631/cannot-download-docker-images-behind-a-proxy