This installation guide is an alternative to Docker Compose definition provided within the root of project.
It guides you with installation of all neccessary dependencies to install REMP tool.
The guide is split to the following sections:
- General System Installation
- PHP 7.2
- Nginx
- Percona
- Redis
- Composer (PHP dependency manager)
- Yarn (JS dependency manager)
- SSO
- Web admin
- Mailer
- Web admin
- Crontab
- Systemd services
- Web admin
- Campaign
- Web admin
- Crontab
- Systemd services
- Web admin
- Beam
- Application dependencies
- Elasticsearch
- Systemd services
- Kafka
- Systemd services
- Elasticsearch
- Web admin
- Crontab
- Systemd services
- Telegraf
- Systemd services
- Tracker API
- Systemd services
- Segments API
- Systemd services
- Application dependencies
All steps are tested against clean minimal Ubuntu 18.04 installation as of February 2019. The user used for installation is uremp - you'll see the references across the tutorial, change this to the user you use for installation.
If you feel the guide is outdated, please create an issue for us and we'll make necessary changes.
All of the following dependencies are shared for all the REMP tools. Some of the tools might require additional installations. Those are mentioned in corresponding section of the REMP tool.
This tutorial uses systemd for managing services and crontab for scheduling jobs. If you plan to use different tools, you can skip the corresponding sections during installation of REMP services.
Even when it's not really possible to say what the configuration should be, we can recommend configuration for testing and running the system based on our usage.
- 2 CPU
- 4GB RAM
- 40GB HDD
- 8 CPU: PHP processes can be hit a lot of times at peaks
- 8GB RAM: Kafka and Elasticsearch can be resource-heavy
- 120GB SSD: This really depends on the amount of events tracked and the amount of time you're willing to keep the events. We archive our events to cold storage (for later analysis) after 90 days.
As we expect this is a clean installation of the system, it's always good to refresh the repository cache and start with all the system upgrades.
sudo apt update
sudo apt upgrade
These applications are not installed in Ubuntu by default, however they're used later in the guide. If you use their alternatives, you can install them now and skip this section.
sudo apt install -y git curl
sudo apt install -y vim
# you can set vim as a default editor
sudo update-alternatives --set editor /usr/bin/vim.basic
sudo apt install php
sudo apt install php-fpm php-dev php-zip php-curl php-bcmath php-iconv php-mysql php-mbstring php-intl php-sockets php-cli php-pear
php -v
The result should be similar to this:
PHP 7.2.15-0ubuntu0.18.04.1 (cli) (built: Feb 8 2019 14:54:22) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.2.15-0ubuntu0.18.04.1, Copyright (c) 1999-2018, by Zend Technologies
PHP installed Apache2 as a webserver in the previous step. We prefer Nginx and our Docker configs are created for Nginx web server. Let's remove Apache2 first so it doesn't interfere with Nginx.
sudo systemctl stop apache2
sudo systemctl disable apache2
sudo apt remove apache2
Install Nginx and download prepared configuration files for REMP vhosts.
sudo apt install nginx
curl https://gist.githubusercontent.com/rootpd/860308c47fea584884fdcc879d6c4301/raw/605eede77872c72ab67578533b4d3bd9690518d1/nginx.conf | sudo tee /etc/nginx/sites-available/remp.press
sudo ln -s /etc/nginx/sites-available/remp.press /etc/nginx/sites-enabled/
At this point please review the vhost configuration. All default hosts are running under *.remp.press domains. As you might want to install the application to different domains, you can change the vhosts now:
sudo vim /etc/nginx/sites-available/remp.press
Please bear in mind that all .env.example files provided by repository use *.remp.press hosts. Due to that please verify all .env and config files after installation and change hosts to your desired ones. Eventhough the applications could communicate via *.remp.press hosts internally, there are cases where frontend application uses these hosts (SSO redirects, AJAX API calls) and they wouldn't work. It's therefore mandatory to change the hosts everywhere.
After you install all REMP services, you can run following command to check whether all hosts were changed properly - command will list all .remp.press usage across configuration files:
grep ".remp.press" -r . | grep -Ev "vendor|log|README|example|framework|docker|Docker"
If you plan to run the services with the default *.remp.press domains, you need to update your hosts file so they're resolvable:
sudo vim /etc/hosts
And append the following block to the end of file:
127.0.0.1 campaign.remp.press mailer.remp.press beam.remp.press sso.remp.press tracker.beam.remp.press segments.beam.remp.press
When you're done with the changes, test and reload the nginx service.
sudo service nginx configtest # to validate config
sudo service nginx reload # to apply config changes
If you stick with the default configuration - using provided vhosts for PHP applications and also for Go applications as a reverse proxy - all services will be accessible via ports 80/443 and no other ports will need to be accessible for the outside world (from the point of application).
HTTPS configuration is not part of this manual. We can only recommend very nice Let's Encrypt tutorial created by DigitalOcean.
sudo systemctl status nginx
The result should be similar to this (trimmed):
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-02-13 13:28:24 CET; 12min ago
Docs: man:nginx(8)
Main PID: 30766 (nginx)
Tasks: 2 (limit: 4663)
CGroup: /system.slice/nginx.service
├─30766 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
└─30767 nginx: worker process
feb 13 13:28:24 rempbox systemd[1]: Starting A high performance web server and a reverse proxy server...
feb 13 13:28:24 rempbox systemd[1]: nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument
feb 13 13:28:24 rempbox systemd[1]: Started A high performance web server and a reverse proxy server.
Percona is a fork of MySQL with some performance improvements and it's fully compatible with MySQL/MariaDB. If you prefer to use them, feel free to install them and skip this step.
Percona/MySQL is our default persistent storage used by web administration applications.
wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
sudo apt update
sudo apt install percona-server-server-5.7
You'll be asked to enter password for root user during the installation. Please save this password for later, as you'll need it when creating databases for applications.
It's recommended to create separate user for accessing REMP databases. For reference we create dremp user with password secret. Please change it to something more secure. We use this default password for reference later in the tutorial and also in the provided configuration files.
If you use different user, please change the commands using dremp user accordingly.
mysql -u root -p -e "CREATE USER 'dremp'@'localhost' IDENTIFIED BY 'secret'"
# you'll be asked to enter password for mysql user "root"
mysql -u dremp -p -e "SHOW DATABASES"
# you'll be asked to enter password for mysql user "dremp"
The result should be similar to this:
+--------------------+
| Database |
+--------------------+
| information_schema |
+--------------------+
Redis is used as a cache layer in all of our applications and as a semi-persistent highly available storage used for background processing.
sudo apt install redis-server
sudo sed -i 's/supervised no/supervised systemd/g' /etc/redis/redis.conf
sudo systemctl restart redis.service
redis-cli
The result should be similar to this (exit via CTRL+D):
127.0.0.1:6379>
curl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/local/bin --filename=composer
composer --version
The result should be similar to this:
Composer version 1.8.0 2018-12-03 10:31:16
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt update
sudo apt install yarn
yarn --version
The result should be similar to this:
1.13.0
All the base dependencies are installed now, we can checkout the project. This tutorial uses two paths for installation, that are then referenced in various configuration files:
~/workspaceas a folder where the project is checked out~/libas a folder where all the pre-built tools are downloaded/installed
If you plan to use different paths, please make sure you also update those paths in all the configuration files and snippets used later in this tutorial.
cd
mkdir ~/lib # we'll need this later
mkdir ~/workspace && cd ~/workspace
git clone https://github.com/remp2020/remp
SSO is used for authentication of users and API clients across all the REMP services.
Usage of SSO is mandatory unless you provide your own authorization layer. Usage of the default configuration is seamless and by default allows everyone to log in with their Google account.
You can whitelist emails with the access into the system in the .env file once the SSO is running.
If you follow the manual and won't change any other configuration settings, the application will be installed to sso.remp.press host. As this is only used for the reference, please update the host usage in snippets and configs to your desired host.
mysql -u root -p -e "CREATE DATABASE IF NOT EXISTS sso"
# you'll be asked to enter password for mysql user "root"
mysql -u root -p -e "GRANT ALL ON sso.* TO 'dremp'@'localhost'"
# you'll be asked to enter password for mysql user "root"
The default contents of .env file uses values that are meant for the Docker Compose appliance. As this tutorial expects everything to be run locally, we added two sed commands to replace host paths to MySQL database and Redis.
It's always recommended to edit the .env file and validate the values before running the application.
cd ~/workspace/remp/Sso
cp -p .env.example .env
sudo sed -i 's/DB_HOST=mysql/DB_HOST=localhost/g' .env # override default docker mysql host
sudo sed -i 's/DB_USERNAME=root/DB_USERNAME=dremp/g' .env # override default docker mysql user
sudo sed -i 's/REDIS_HOST=redis/REDIS_HOST=localhost/g' .env # override default docker redis host
The .env file also contains reference credentials to MySQL database: user dremp with password secret. If you used different password for dremp user, please edit DB_PASSWORD option in the .env file.
In the default configuration .env provides credentials of our testing Google account. It's configured to work on *.remp.press domains and serves mainly for our development purpose and for out-of-the-box working experience when using this tutorial or Docker.
It's up to you to generate your own credentials. You can do so by creating Google project and creating new credentials. Once you're ready, you can set these environment variables:
GOOGLE_CLIENT_IDGOOGLE_CLIENT_SECRETGOOGLE_CALLBACK_URL
We don't guarantee correctness of the default credentials in the future. Therefore we highly recommend to generate your own set of credentials.
composer install --verbose --prefer-dist --no-progress --no-interaction --no-dev --optimize-autoloader
yarn install --production
yarn upgrade remp --force # local package is not updated during install
yarn production
php artisan migrate
php artisan db:seed
php artisan key:generate
php artisan jwt:secret
chmod -R 777 storage
SSO is now ready to be used by other REMP tools. To test the installation, please visit sso.remp.press to see the (very simple) web administration of SSO.
- Visit
sso.remp.press(change the hostname to one you configured in nginx) - Login via Google account (everyone is whitelisted to be logged in by default)
- See the (empty) list of available API keys
If you see the list of API keys, the installation was successful.
Mailer is used to:
- Manage system emails (by creating layouts and templates)
- Manage newsletters and people subscribed to them
- Manage jobs to send newsletters to specific groups (segments) of people
If you follow the manual and won't change any other configuration settings, the application will be installed to mailer.remp.press host. As this is only used for the reference, please update the host usage in snippets and configs to your desired host.
mysql -u root -p -e "CREATE DATABASE IF NOT EXISTS mailer"
mysql -u root -p -e "GRANT ALL ON mailer.* TO 'dremp'@'localhost'"
The default contents of .env file uses values that are meant for the Docker Compose appliance. As this tutorial expects everything to be run locally, we added two sed commands to replace host paths to MySQL database and Redis.
It's always recommended to edit the .env file and validate the values before running the application.
cd ~/workspace/remp/Mailer
cp -p .env.example .env
sudo sed -i 's/DB_HOST=mysql/DB_HOST=localhost/g' .env # override default docker mysql host
sudo sed -i 's/DB_USER=root/DB_USER=dremp/g' .env # override default docker mysql user
sudo sed -i 's/REDIS_HOST=redis/REDIS_HOST=localhost/g' .env # override default docker redis host
The .env file also contains reference credentials to MySQL database: user dremp with password secret. If you used different password for dremp user, please edit DB_PASSWORD option in the .env file.
As a complement to .env file, application uses also config.local.neon file which extends the default config.neon.
It consists of all registered services in the DI (Dependency Injection) container and serves as a place to define overrides of the default application behavior.
It's always recommended to read and understand the default config.local.neon. The examples there (and classes they refer to) should help you understand how to change the config to enable your integrations.
cp app/config/config.local.neon.example app/config/config.local.neon
composer install --verbose --prefer-dist --no-progress --no-interaction --no-dev --optimize-autoloader
yarn install --production
yarn upgrade remp --force # local package is not updated during install
yarn production
php bin/command.php migrate:migrate
php bin/command.php db:seed
php bin/command.php demo:seed
sudo chmod -R 777 temp log
There are two mailer implememtations by default: SMTP, Mailgun. Both of them require to be configured before they can be used. You can visit the configuration page directly or by pressing user icon in the top right corner and seleging settings.
Two commands are required to be placed to the scheduler. They're responsible for generating mail job data when sending the emails (deciding who will get the email, A/B test split, ...) and for stats aggregations
crontab -e
* * * * * php /home/uremp/workspace/remp/Mailer/bin/command.php mail:process-job
* * * * * php /home/uremp/workspace/remp/Mailer/bin/command.php mail:job-stats
- Visit
mailer.remp.press(change the hostname to one you configured in nginx)- If you already logged in via SSO, you'll be logged in automatically
- If you haven't logged in via SSO, you'll be redirected to the authentication flow (Google login)
- See the empty dashboard
If you're able to see empty dashboard with sending stats, the installation was successful. You can start using mailer features by creating new "Layouts" and "Emails".
Hermes is responsible for asynchronous (background) processing of events emitted by the Web administration application. Following is a way how to enable it as a systemd service.
Create a definition file:
sudo vim /etc/systemd/system/remp-mailer-hermes-worker.service
Paste the following snippet:
[Unit]
Description="REMP Mailer Hermes worker"
After=network.target
[Service]
Type=simple
UMask=0022
LimitNOFILE=1024
ExecStart=/usr/bin/php /home/uremp/workspace/remp/Mailer/bin/command.php worker:hermes
User=uremp
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Start and enable the service to be started automatically in the future:
sudo systemctl enable remp-mailer-hermes-worker
sudo systemctl start remp-mailer-hermes-worker
Mail worker is responsible for actual sending of emails. It's responsible for handling newsletter priorities and processing the queue generating by processing job from the scheduled command defined earlier.
Create a definition file:
sudo vim /etc/systemd/system/remp-mailer-mail-worker.service
Paste the following snippet:
[Unit]
Description="REMP Mailer Mail worker"
After=network.target
[Service]
Type=simple
UMask=0022
LimitNOFILE=1024
ExecStart=/usr/bin/php /home/uremp/workspace/remp/Mailer/bin/command.php worker:mail
User=uremp
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Start and enable the service to be started automatically in the future:
sudo systemctl enable remp-mailer-mail-worker
sudo systemctl start remp-mailer-mail-worker
Systemd service should now be active and running. You can test verify that by running:
sudo systemctl status remp-mailer-hermes-worker
sudo systemctl status remp-mailer-mail-worker
The output should be similar to this (using first service as a reference) :
● remp-mailer-hermes-worker.service - "REMP Mailer Hermes worker"
Loaded: loaded (/etc/systemd/system/remp-mailer-hermes-worker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-02-13 13:39:35 CET; 2s ago
Main PID: 890 (php)
Tasks: 1 (limit: 4663)
CGroup: /system.slice/remp-mailer-hermes-worker.service
└─890 /usr/bin/php /home/uremp/workspace/remp/Mailer/bin/command.php worker:hermes
feb 13 13:39:35 rempbox systemd[1]: Started "REMP Mailer Hermes worker".
feb 13 13:39:35 rempbox php[890]: ***** REMP MAILER HERMES WORKER *****
feb 13 13:39:35 rempbox php[890]: Listening to:
feb 13 13:39:35 rempbox php[890]: - list-created: Remp\MailerModule\Hermes\ListCreatedHandler
feb 13 13:39:35 rempbox php[890]: - mailgun-event: Remp\MailerModule\Hermes\MailgunEventHandler
feb 13 13:39:35 rempbox php[890]: - heartbeat: Remp\MailerModule\Hermes\PingMysql
Campaign is used to:
- Define how displayed banners should look like
- Configure campaigns to display the banners based on different rules
- Display the stats
Note: All the events related to statistics are tracked to REMP Beam. REMP Campaign is able to function on its own, however the statistics are now linked to the Segments API belonging to the REMP Beam.
It is possible to track the statistics to other systems by definining specific JS functions - for instance to track all the events to Google Analytics. However you would have to use Google Analytics to display the statistics too and you would loose nice features of Campaign, such as A/B test resolution. If you're interested in such usage, please get in touch with us.
If you follow the manual and won't change any other configuration settings, the application will be installed to campaign.remp.press host. As this is only used for the reference, please update the host usage in snippets and configs to your desired host.
mysql -u root -p -e "CREATE DATABASE IF NOT EXISTS campaign"
mysql -u root -p -e "GRANT ALL ON campaign.* TO 'dremp'@'localhost'"
The default contents of .env file uses values that are meant for the Docker Compose appliance. As this tutorial expects everything to be run locally, we added two sed commands to replace host paths to MySQL database and Redis.
It's always recommended to edit the .env file and validate the values before running the application.
cd ~/workspace/remp/Campaign
cp -p .env.example .env
sudo sed -i 's/DB_HOST=mysql/DB_HOST=localhost/g' .env # override default docker mysql host
sudo sed -i 's/DB_USERNAME=root/DB_USERNAME=dremp/g' .env # override default docker mysql user
sudo sed -i 's/REDIS_HOST=redis/REDIS_HOST=localhost/g' .env # override default docker redis host
The .env file also contains reference credentials to MySQL database: user dremp with password secret. If you used different password for dremp user, please edit DB_PASSWORD option in the .env file.
composer install --verbose --prefer-dist --no-progress --no-interaction --no-dev --optimize-autoloader
yarn install --production
yarn upgrade remp --force # local package is not updated during install
yarn production
php artisan migrate
php artisan db:seed
php artisan key:generate
chmod -R 777 storage
Laravel defines its schedule tasks in the \App\Console\Kernel class. To run these tasks, scheduler needs to be placed to the crontab.
crontab -e
Paste the following snippet and save.
* * * * * php /home/uremp/workspace/remp/Campaign/artisan schedule:run
Laravel provides internal asynchronous queue processing through the command running as daemon. We recommend to have this command run by systemd.
Create a definition file:
sudo vim /etc/systemd/system/remp-campaign-queue-worker.service
Paste the following snippet:
[Unit]
Description="REMP Campaign Queue Worker"
After=network.target
[Service]
Type=simple
UMask=0022
LimitNOFILE=1024
ExecStart=/usr/bin/php /home/uremp/workspace/remp/Campaign/artisan queue:work --tries=1 --timeout=1200
User=uremp
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Start and enable the service to be started automatically in the future:
sudo systemctl enable remp-campaign-queue-worker
sudo systemctl start remp-campaign-queue-worker
Systemd service should now be active and running. You can test verify that by running:
sudo systemctl status remp-campaign-queue-worker
The output should be similar to this*
● remp-campaign-queue-worker.service - "REMP Campaign Queue Worker"
Loaded: loaded (/etc/systemd/system/remp-campaign-queue-worker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-02-13 13:49:50 CET; 2s ago
Main PID: 3998 (php)
Tasks: 1 (limit: 4663)
CGroup: /system.slice/remp-campaign-queue-worker.service
└─3998 /usr/bin/php /home/uremp/workspace/remp/Campaign/artisan queue:work --tries=1 --timeout=1200
feb 13 13:49:50 rempbox systemd[1]: Started "REMP Campaign Queue Worker".
Zookeeper is a service discovery tool used by Kafka.
sudo apt install zookeeperd
sudo systemctl enable zookeeper
Systemd service should now be active and running. You can test verify that by running:
sudo systemctl status zookeeper
The output should be similar to this*
● zookeeper.service - LSB: centralized coordination service
Loaded: loaded (/etc/init.d/zookeeper; generated)
Active: active (running) since Wed 2019-02-13 13:50:44 CET; 30s ago
Docs: man:systemd-sysv-generator(8)
Tasks: 17 (limit: 4663)
CGroup: /system.slice/zookeeper.service
└─6471 /usr/bin/java -cp /etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/
feb 13 13:50:44 rempbox systemd[1]: Starting LSB: centralized coordination service...
feb 13 13:50:44 rempbox systemd[1]: Started LSB: centralized coordination service.
Kafka is used as a queue and preliminary storage of all events tracked by Tracker API. It's being read by Telegraf and pushed to Elasticsearch.
cd
sudo adduser --system --no-create-home --disabled-password --disabled-login kafka
wget https://www-eu.apache.org/dist/kafka/2.3.0/kafka_2.12-2.3.0.tgz
tar xvzf kafka_2.12-2.3.0.tgz
ln -s kafka_2.12-2.3.0 kafka
sudo chown -R kafka kafka/
Create a definition file:
sudo vim /etc/systemd/system/kafka.service
Paste the following snippet:
[Unit]
Requires=zookeeper.service
After=zookeeper.service
[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/uremp/kafka/bin/kafka-server-start.sh /home/uremp/kafka/config/server.properties > /home/uremp/kafka/kafka.log 2>&1'
ExecStop=/home/uremp/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
Start and enable the service to be started automatically in the future:
sudo systemctl enable kafka
sudo systemctl start kafka
Systemd service should now be active and running. You can test verify that by running:
sudo systemctl status kafka
The output should be similar to this:
● kafka.service
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-02-13 13:53:06 CET; 2s ago
Main PID: 7447 (sh)
Tasks: 19 (limit: 4663)
CGroup: /system.slice/kafka.service
├─7447 /bin/sh -c /home/uremp/kafka/bin/kafka-server-start.sh /home/uremp/kafka/config/server.properties > /home/uremp/kafka/kafka.log 2>&1
└─7448 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xlog:gc*:file=/home/uremp
feb 13 13:53:06 rempbox systemd[1]: Started kafka.service.
Elasticsearch is used as a storage of all tracked events and it's utilized for various aggregations within the REMP Beam. We highly recommend to use cluster of Elasticsearch nodes in production and to configure Elasticsearch based on your server specification.
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt update
sudo apt install elasticsearch
Start and enable the service to be started automatically in the future:
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch
Systemd service should now be active and running. You can test verify that by running:
sudo systemctl status elasticsearch
The output should be similar to this:
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-02-13 13:53:56 CET; 7s ago
Docs: http://www.elastic.co
Main PID: 8797 (java)
Tasks: 18 (limit: 4663)
CGroup: /system.slice/elasticsearch.service
├─8797 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.ne
└─8941 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
feb 13 13:53:56 rempbox systemd[1]: Started Elasticsearch.
feb 13 13:53:57 rempbox elasticsearch[8797]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Telegraf is a tool developed by Influxdata to read events/metrict from configured data source and write to configured data target.
Eventhough Telegraf provides Kafka as an input source and Elasticsearch as an output target, we needed extra functionallity in the Elasticsearch plugin which resulted in a maintained fork of the application. The single purpose of the fork is maintanance of remp_elasticsearch output plugin which is used to properly process events tracked to the Tracker API.
cd ~/lib
wget https://github.com/remp2020/telegraf/releases/download/1.10.4/telegraf_linux_amd64.tar.gz
mkdir telegraf
tar xvzf telegraf_linux_amd64.tar.gz -C telegraf
The configuration used here is the one provided to our Docker Compose appliance. We replace here path to hosts to reflect local installation.
We recommend to get familiar with the Telegraf configuration options in case of changes in your production environment.
cd ~/lib
wget https://raw.githubusercontent.com/remp2020/remp/master/Docker/telegraf/telegraf.conf
sed -i 's/elasticsearch:9200/localhost:9200/g' telegraf.conf
sed -i 's/kafka:9092/localhost:9092/g' telegraf.conf
Create a definition file:
sudo vim /etc/systemd/system/remp-telegraf.service
Paste the following snippet:
[Unit]
Description="The plugin-driven server agent for reporting metrics into InfluxDB"
After=network.target
[Service]
Type=simple
UMask=0022
LimitNOFILE=1024
ExecStart=/home/uremp/lib/telegraf/telegraf -config /home/uremp/lib/telegraf.conf $TELEGRAF_OPTS
User=uremp
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Start and enable the service to be started automatically in the future:
sudo systemctl start remp-telegraf
sudo systemctl enable remp-telegraf
Systemd service should now be active and running. You can test verify that by running:
sudo systemctl status remp-telegraf
The output should be similar to this:
● remp-telegraf.service - "The plugin-driven server agent for reporting metrics into InfluxDB"
Loaded: loaded (/etc/systemd/system/remp-telegraf.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-02-13 14:09:10 CET; 5s ago
Main PID: 12483 (telegraf)
Tasks: 8 (limit: 4663)
CGroup: /system.slice/remp-telegraf.service
└─12483 /home/uremp/lib/telegraf/telegraf -config /home/uremp/lib/telegraf.conf
feb 13 14:09:10 rempbox telegraf[12483]: 2019-02-13T13:09:10Z I! Tags enabled: host=rempbox
feb 13 14:09:10 rempbox telegraf[12483]: 2019-02-13T13:09:10Z I! [agent] Config: Interval:1s, Quiet:false, Hostname:"rempbox", Flush Interval:1s
feb 13 14:09:10 rempbox telegraf[12483]: 2019-02-13T13:09:10Z I! Elasticsearch version: 6.6.0
feb 13 14:09:10 rempbox telegraf[12483]: 2019-02-13T13:09:10Z I! Elasticsearch version: 6.6.0
feb 13 14:09:10 rempbox telegraf[12483]: 2019-02-13T13:09:10Z I! Elasticsearch version: 6.6.0
feb 13 14:09:10 rempbox telegraf[12483]: 2019-02-13T13:09:10Z I! Elasticsearch version: 6.6.0
feb 13 14:09:10 rempbox telegraf[12483]: 2019-02-13T13:09:10Z I! Elasticsearch version: 6.6.0
feb 13 14:09:10 rempbox telegraf[12483]: 2019-02-13T13:09:10Z I! Elasticsearch version: 6.6.0
feb 13 14:09:10 rempbox telegraf[12483]: 2019-02-13T13:09:10Z I! Elasticsearch version: 6.6.0
feb 13 14:09:14 rempbox telegraf[12483]: 2019-02-13T13:09:14Z I! Started the kafka consumer service, brokers: [localhost:9092], topics: [beam_events]
If you follow the manual and won't change any other configuration settings, the application will be installed to beam.remp.press host. As this is only used for the reference, please update the host usage in snippets and configs to your desired host.
mysql -u root -p -e "CREATE DATABASE IF NOT EXISTS beam"
mysql -u root -p -e "GRANT ALL ON beam.* TO 'dremp'@'localhost'"
cd ~/workspace/remp/Beam
cp -p .env.example .env
sudo sed -i 's/DB_HOST=mysql/DB_HOST=localhost/g' .env # override default docker mysql host
sudo sed -i 's/DB_USERNAME=root/DB_USERNAME=dremp/g' .env # override default docker mysql user
sudo sed -i 's/REDIS_HOST=redis/REDIS_HOST=localhost/g' .env # override default docker redis host
composer install --verbose --prefer-dist --no-progress --no-interaction --no-dev --optimize-autoloader
yarn install --production
yarn upgrade remp --force # local package is not updated during install
yarn production
php artisan migrate
php artisan db:seed
php artisan key:generate
chmod -R 777 storage
Laravel defines its schedule tasks in the \App\Console\Kernel class. To run these tasks, scheduler needs to be placed to the crontab.
crontab -e
Paste the following snippet and save.
* * * * * php /home/uremp/workspace/remp/Beam/artisan schedule:run
- Visit
beam.remp.press/accounts(change the hostname to one you configured in nginx)- If you already logged in via SSO, you'll be logged in automatically
- If you haven't logged in via SSO, you'll be redirected to the authentication flow (Google login)
- See the list of Beam accounts
If you're able to see the list of accounts, the installation was successful. You can start using Beam features by creating new "Account" and "Property" to get Property token required for event tracking.
You might also try to visit the main page of Beam (beam.remp.press), however you'll see a JS error caused by Segment API not being available yet. We'll install it in the next steps.
Laravel provides internal asynchronous queue processing through the command running as daemon. We recommend to have this command run by systemd.
Create a definition file:
sudo vim /etc/systemd/system/remp-beam-queue-worker.service
Paste the following snippet:
[Unit]
Description="REMP Beam Queue Worker"
After=network.target
[Service]
Type=simple
UMask=0022
LimitNOFILE=1024
ExecStart=/usr/bin/php /home/uremp/workspace/remp/Beam/artisan queue:work --tries=1 --timeout=1200
User=uremp
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Start and enable the service to be started automatically in the future:
sudo systemctl enable remp-beam-queue-worker
sudo systemctl start remp-beam-queue-worker
Systemd service should now be active and running. You can test verify that by running:
sudo systemctl status remp-beam-queue-worker
The output should be similar to this:
● remp-beam-queue-worker.service - "REMP Beam Queue Worker"
Loaded: loaded (/etc/systemd/system/remp-beam-queue-worker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-02-13 14:04:32 CET; 2s ago
Main PID: 12139 (php)
Tasks: 1 (limit: 4663)
CGroup: /system.slice/remp-beam-queue-worker.service
└─12139 /usr/bin/php /home/uremp/workspace/remp/Beam/artisan queue:work --tries=1 --timeout=1200
feb 13 14:04:32 rempbox systemd[1]: Started "REMP Beam Queue Worker".
Tracker API is a very simple pre-built Go application that validates the requests and pushes messages with events to Kafka for later processing.
As mentioned in the beginning of this tutorial, we install it into the ~/lib folder. If you change the installation path, make the changes in the systemd configs as well.
cd ~/lib
wget https://github.com/remp2020/remp/releases/download/0.8.2/tracker_linux_amd64.tar.gz
mkdir tracker
tar xvzf tracker_linux_amd64.tar.gz -C tracker
Even Tracker API contains its own .env file. As before, the default contents of .env file uses values that are meant for the Docker Compose appliance. This tutorial expects everything to be run locally, so we added two sed commands to replace host paths to MySQL database and Redis.
It's always recommended to edit the .env file and validate the values before running the application.
cp tracker/.env.example tracker/.env
sed -i 's/TRACKER_BROKER_ADDRS=kafka/TRACKER_BROKER_ADDRS=localhost/g' tracker/.env
sed -i 's/TRACKER_MYSQL_ADDR=mysql/TRACKER_MYSQL_ADDR=localhost/g' tracker/.env
sed -i 's/TRACKER_MYSQL_USER=root/TRACKER_MYSQL_USER=dremp/g' tracker/.env
The .env file also contains reference credentials to MySQL database: user dremp with password secret. If you used different password for dremp user, please edit TRACKER_MYSQL_PASSWD option in the .env file.
Create a definition file:
sudo vim /etc/systemd/system/remp-tracker.service
Paste the following snippet:
[Unit]
Description="REMP Beam Tracker"
After=network.target
[Service]
Type=simple
UMask=0022
LimitNOFILE=65535
ExecStart=/home/uremp/lib/tracker/tracker
WorkingDirectory=/home/uremp/lib/tracker/
User=uremp
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Start and enable the service to be started automatically in the future:
sudo systemctl start remp-tracker
sudo systemctl enable remp-tracker
Systemd service should now be active and running. You can test verify that by running:
sudo systemctl status remp-tracker
The output should be similar to this:
● remp-tracker.service - "REMP Beam Tracker"
Loaded: loaded (/etc/systemd/system/remp-tracker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-02-13 14:06:46 CET; 6s ago
Main PID: 12233 (tracker)
Tasks: 7 (limit: 4663)
CGroup: /system.slice/remp-tracker.service
└─12233 /home/uremp/lib/tracker/tracker
feb 13 14:06:46 rempbox tracker[12233]: 2019/02/13 14:06:46 property cache reloaded
feb 13 14:06:46 rempbox tracker[12233]: 2019/02/13 14:06:46 [INFO] starting property caching
feb 13 14:06:46 rempbox tracker[12233]: 2019/02/13 14:06:46 entity schema cache reloaded
feb 13 14:06:46 rempbox tracker[12233]: 2019/02/13 14:06:46 [INFO] mount ctrl=Swagger files=swagger/swagger.json route=GET /swagger.json
feb 13 14:06:46 rempbox tracker[12233]: 2019/02/13 14:06:46 [INFO] mount ctrl=Track action=Commerce route=POST /track/commerce
feb 13 14:06:46 rempbox tracker[12233]: 2019/02/13 14:06:46 [INFO] mount ctrl=Track action=Entity route=POST /track/entity
feb 13 14:06:46 rempbox tracker[12233]: 2019/02/13 14:06:46 [INFO] mount ctrl=Track action=Event route=POST /track/event
feb 13 14:06:46 rempbox tracker[12233]: 2019/02/13 14:06:46 [INFO] mount ctrl=Track action=Pageview route=POST /track/pageview
feb 13 14:06:46 rempbox tracker[12233]: 2019/02/13 14:06:46 [INFO] starting server bind=:8081
feb 13 14:06:46 rempbox tracker[12233]: 2019/02/13 14:06:46 [INFO] starting entity schemas caching
You can also test the connection and see the API definitions by visiting http://tracker.beam.remp.press/swagger.json. The host might be different depending on whether you used different domain in nginx configuration. If so, please change the hostname in URL accordingly.
Segments API is Go application providing interfaces to read, filter and aggregate data from Elasticsearch.
As mentioned in the beginning of this tutorial, we install it into the ~/lib folder. If you change the installation path, make the changes in the systemd configs as well.
cd ~/lib
wget https://github.com/remp2020/remp/releases/download/0.8.2/segments_linux_amd64.tar.gz
mkdir segments
tar xvzf segments_linux_amd64.tar.gz -C segments
Even Segments API contains its own .env file. As before, the default contents of .env file uses values that are meant for the Docker Compose appliance. This tutorial expects everything to be run locally, so we added two sed commands to replace host paths to MySQL database and Redis.
cp segments/.env.example segments/.env
sed -i 's/SEGMENTS_ELASTIC_ADDRS=http:\/\/elasticsearch/SEGMENTS_ELASTIC_ADDRS=http:\/\/localhost/g' segments/.env
sed -i 's/SEGMENTS_MYSQL_ADDR=mysql/SEGMENTS_MYSQL_ADDR=localhost/g' segments/.env
sed -i 's/SEGMENTS_MYSQL_USER=root/SEGMENTS_MYSQL_USER=dremp/g' segments/.env
The .env file also contains reference credentials to MySQL database: user dremp with password secret. If you used different password for dremp user, please edit SEGMENTS_MYSQL_PASSWD option in the .env file.
Create a definition file:
sudo vim /etc/systemd/system/remp-segments.service
Paste the following snippet:
[Unit]
Description="REMP Beam Segments"
After=network.target
[Service]
Type=simple
UMask=0022
LimitNOFILE=65535
ExecStart=/home/uremp/lib/segments/segments
WorkingDirectory=/home/uremp/lib/segments/
User=uremp
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Start and enable the service to be started automatically in the future:
sudo systemctl start remp-segments
sudo systemctl enable remp-segments
Systemd service should now be active and running. You can test verify that by running:
sudo systemctl status remp-segments
The output should be similar to this:
● remp-segments.service - "REMP Beam Segments"
Loaded: loaded (/etc/systemd/system/remp-segments.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-02-13 14:07:53 CET; 5s ago
Main PID: 12303 (segments)
Tasks: 7 (limit: 4663)
CGroup: /system.slice/remp-segments.service
└─12303 /home/uremp/lib/segments/segments
feb 13 14:07:53 rempbox segments[12303]: 2019/02/13 14:07:53 [INFO] mount ctrl=Pageviews action=List route=POST /journal/pageviews/list
feb 13 14:07:53 rempbox segments[12303]: 2019/02/13 14:07:53 [INFO] mount ctrl=Pageviews action=Sum route=POST /journal/pageviews/actions/:action/sum
feb 13 14:07:53 rempbox segments[12303]: 2019/02/13 14:07:53 [INFO] mount ctrl=Pageviews action=Unique route=POST /journal/pageviews/actions/:action/unique/:item
feb 13 14:07:53 rempbox segments[12303]: 2019/02/13 14:07:53 [INFO] mount ctrl=Segments action=CheckBrowser route=GET /segments/:segment_code/browsers/check/:browser_id
feb 13 14:07:53 rempbox segments[12303]: 2019/02/13 14:07:53 [INFO] mount ctrl=Segments action=CheckUser route=GET /segments/:segment_code/users/check/:user_id
feb 13 14:07:53 rempbox segments[12303]: 2019/02/13 14:07:53 [INFO] mount ctrl=Segments action=List route=GET /segments
feb 13 14:07:53 rempbox segments[12303]: 2019/02/13 14:07:53 [INFO] mount ctrl=Segments action=Users route=GET /segments/:segment_code/users
feb 13 14:07:53 rempbox segments[12303]: 2019/02/13 14:07:53 [INFO] mount ctrl=Concurrents action=Count route=POST /journal/concurrents/count
feb 13 14:07:53 rempbox segments[12303]: 2019/02/13 14:07:53 [INFO] starting server bind=:8082
feb 13 14:07:53 rempbox segments[12303]: 2019/02/13 14:07:53 [INFO] starting property caching
You can also test the connection and see the API definitions by visiting http://segments.beam.remp.press/swagger.json. The host might be different depending on whether you used different domain in nginx configuration. If so, please change the hostname in URL accordingly.
This sections serves mainly as a set of reminders and pointers to important steps of the tutorial.
- If you want to run the services at non
*.remp.pressdomains, verify that you've gone through Virtual hosts section - You should generate your own set of Google credentials for SSO, see Google API Credentials section.