rmoff/OGG_KafkaConnect_Elasticsearch_setup.md

## OGG_KafkaConnect_Elasticsearch_setup.md

      
    Raw
  

              OGG_KafkaConnect_Elasticsearch_setup.md
            
          
    This is a set of instructions for use with the blog article Streaming data from Oracle using Oracle GoldenGate and Kafka Connect.
@rmoff / September 15, 2016

First up, download the BigDataLite VM, unpack it and import it to VirtualBox. You'll need internet access from the VM for the downloads, so make sure you include a NAT network adaptor, or bridged onto a network with internet access. Login to the machine as oracle/welcome1. All the work done in this article is from the command line, so you can either work in Terminal, or you can run ip a to determine the IP address of the VM and then SSH into it from your host machine.
Install Confluent Platform

Now install the Confluent Platform by running the following commands.
# Gimme root
sudo su -

# Add the Confluent yum repository
rpm --import http://packages.confluent.io/rpm/3.0/archive.key

cat >> /etc/yum.repos.d/confluent.repo <<EOF
[Confluent.dist]
name=Confluent repository (dist)
baseurl=http://packages.confluent.io/rpm/3.0/6
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/3.0/archive.key
enabled=1

[Confluent]
name=Confluent repository
baseurl=http://packages.confluent.io/rpm/3.0
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/3.0/archive.key
enabled=1
EOF

# Remove the existing v0.8.2 kafka
yum remove -y kafka

# Install confluent platform via yum
yum clean all
yum install -y confluent-platform-2.11

# Don't need root now, so exit the root shell back to oracle
logout

Install Oracle GoldenGate Adapter for Kafka Connect

This connector is published by Oracle on the java.net site.
# Download it
cd ~/Downloads
wget https://java.net/projects/oracledi/downloads/download/GoldenGate/Oracle%20GoldenGate%20Adapter%20for%20Kafka%20Connect/OGG_KafkaConnectHandlerFormatter1.0.tar
# Unpack
mkdir OGG_KafkaConnectHandlerFormatter
mv OGG_KafkaConnectHandlerFormatter1.0.tar OGG_KafkaConnectHandlerFormatter
cd OGG_KafkaConnectHandlerFormatter/
tar -xvf OGG_KafkaConnectHandlerFormatter1.0.tar
# Copy to OGG-BD location
cp kafka-connect/bin/ogg-kafka-connect-1.0.jar /u01/ogg-bd/ggjava/resources/lib/
cp kafka-connect/dirprm/conf* /u01/ogg-bd/dirprm/

Install and Build Elasticsearch Kafka Connect connector

The Elasticsearch connector currently has to be built from source. You can do this as follows.
Install Apache Maven
sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
sudo yum install -y apache-maven

Install Gradle
cd ~/Downloads
wget https://downloads.gradle.org/distributions/gradle-3.0-bin.zip
unzip gradle-3.0-bin.zip

Clone the connector github repository, as well as Kafka repository itself
cd ~/Downloads
git clone https://github.com/confluentinc/kafka-connect-elasticsearch.git
git clone https://github.com/confluentinc/kafka.git

Build Kafka and install in the local Maven repository - this will take 5-10 minutes to run.
export GRADLE_HOME=/home/oracle/Downloads/gradle-3.0
export PATH=$PATH:$GRADLE_HOME/bin
cd ~/Downloads/kafka
gradle wrapper
gradle install

It should complete succesfully, with a few warnings scattered along the way
[...]
:streams:examples:javadoc SKIPPED
:streams:examples:javadocJar
:streams:examples:srcJar
:streams:examples:signArchives SKIPPED
:streams:examples:install

BUILD SUCCESSFUL

Now build the connector itself (takes a few minutes to run):
cd ~/Downloads/kafka-connect-elasticsearch
mvn clean package -DskipTests

You should get a success message
[...]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:18 min
[INFO] Finished at: 2016-09-02T13:49:43+01:00
[INFO] Final Memory: 37M/438M
[INFO] ------------------------------------------------------------------------

and a jar file for the connector plus dependent libraries:
$ ls -l ~/Downloads/kafka-connect-elasticsearch/target/kafka-connect-elasticsearch-3.1.0-SNAPSHOT-package/share/java/kafka-connect-elasticsearch/
total 5016
-rw-r--r--. 1 oracle oinstall  263965 Sep  2 13:38 commons-codec-1.9.jar
-rw-r--r--. 1 oracle oinstall  434678 Sep  2 13:38 commons-lang3-3.4.jar
-rw-r--r--. 1 oracle oinstall   61829 Sep  2 13:38 commons-logging-1.2.jar
-rw-r--r--. 1 oracle oinstall  212164 Sep  2 13:38 gson-2.4.jar
-rw-r--r--. 1 oracle oinstall 2256213 Sep  2 13:38 guava-18.0.jar
-rw-r--r--. 1 oracle oinstall  177013 Sep  2 13:38 httpasyncclient-4.1.1.jar
-rw-r--r--. 1 oracle oinstall  732765 Sep  2 13:38 httpclient-4.5.1.jar
-rw-r--r--. 1 oracle oinstall  326724 Sep  2 13:38 httpcore-4.4.4.jar
-rw-r--r--. 1 oracle oinstall  356091 Sep  2 13:38 httpcore-nio-4.4.4.jar
-rw-r--r--. 1 oracle oinstall   18398 Sep  2 13:38 jest-2.0.0.jar
-rw-r--r--. 1 oracle oinstall  216228 Sep  2 13:38 jest-common-2.0.0.jar
-rw-r--r--. 1 oracle oinstall   48947 Sep  2 13:49 kafka-connect-elasticsearch-3.1.0-SNAPSHOT.jar
-rw-r--r--. 1 oracle oinstall   10680 Sep  2 13:38 slf4j-simple-1.7.5.jar

Move this to the /opt folder:
$ sudo cp -r /home/oracle/Downloads/kafka-connect-elasticsearch/target/kafka-connect-elasticsearch-3.1.0-SNAPSHOT-package/share/java/kafka-connect-elasticsearch/ /opt/kafka-connect-elasticsearch/

$ ls -l /opt/kafka-connect-elasticsearch/
total 5016
-rw-r--r--. 1 root root  263965 Sep  5 15:33 commons-codec-1.9.jar
-rw-r--r--. 1 root root  434678 Sep  5 15:33 commons-lang3-3.4.jar
-rw-r--r--. 1 root root   61829 Sep  5 15:33 commons-logging-1.2.jar
-rw-r--r--. 1 root root  212164 Sep  5 15:33 gson-2.4.jar
-rw-r--r--. 1 root root 2256213 Sep  5 15:33 guava-18.0.jar
-rw-r--r--. 1 root root  177013 Sep  5 15:33 httpasyncclient-4.1.1.jar
-rw-r--r--. 1 root root  732765 Sep  5 15:33 httpclient-4.5.1.jar
-rw-r--r--. 1 root root  326724 Sep  5 15:33 httpcore-4.4.4.jar
-rw-r--r--. 1 root root  356091 Sep  5 15:33 httpcore-nio-4.4.4.jar
-rw-r--r--. 1 root root   18398 Sep  5 15:33 jest-2.0.0.jar
-rw-r--r--. 1 root root  216228 Sep  5 15:33 jest-common-2.0.0.jar
-rw-r--r--. 1 root root   48947 Sep  5 15:33 kafka-connect-elasticsearch-3.1.0-SNAPSHOT.jar
-rw-r--r--. 1 root root   10680 Sep  5 15:33 slf4j-simple-1.7.5.jar

Install Elasticsearch

To install Elasticsearch and Kibana it's just a case of downloading the files and unzipping them. As part of the install I'm including the Kopf and Sense plugins which make working with Elasticsearch easier.
cd ~/Downloads/

# Download the files
wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/zip/elasticsearch/2.4.0/elasticsearch-2.4.0.zip
wget https://download.elastic.co/kibana/kibana/kibana-4.6.0-linux-x86_64.tar.gz

# Unpack
unzip elasticsearch-2.4.0.zip
tar -xf kibana-4.6.0-linux-x86_64.tar.gz

# Move to /opt
sudo mv elasticsearch-2.4.0 /opt/
sudo mv kibana-4.6.0-linux-x86_64 /opt

# Install plugins
/opt/kibana-4.6.0-linux-x86_64/bin/kibana plugin --install elastic/sense
/opt/elasticsearch-2.4.0/bin/plugin install lmenezes/elasticsearch-kopf

Add configuration to Elasticsearch to bind to all available network addresses (not just loopback):
cat >> /opt/elasticsearch-2.4.0/config/elasticsearch.yml <<EOF
network.host: 0.0.0.0
EOF

Install Swingbench

Install swingbench:
cd ~/Downloads
# Download
wget http://dominicgiles.com/swingbench/swingbench25971.zip
# Unzip
unzip swingbench25971.zip
# Move to /opt
sudo mv swingbench /opt

Create the Swingbench schema

Having installed Swingbench, we'll now run the installation of the schema and initial data.
Make sure that the database is running
sudo service dbora start

Run Swingbench's initialisation, which will create the table and data necessary to run Swingbench.
/opt/swingbench/bin/oewizard -cs localhost:1521/orcl -df /u01/app/oracle/oradata/cdb/orcl/soe.dbf -cl -scale 0.1 -dbap welcome1 -create -u soe -p soe -ts soe -v

On my VM this took ~25 minutes to complete.