Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

This is a set of instructions for use with the blog article Streaming data from Oracle using Oracle GoldenGate and Kafka Connect.

@rmoff / September 15, 2016


First up, download the BigDataLite VM, unpack it and import it to VirtualBox. You'll need internet access from the VM for the downloads, so make sure you include a NAT network adaptor, or bridged onto a network with internet access. Login to the machine as oracle/welcome1. All the work done in this article is from the command line, so you can either work in Terminal, or you can run ip a to determine the IP address of the VM and then SSH into it from your host machine.

Install Confluent Platform

Now install the Confluent Platform by running the following commands.

# Gimme root
sudo su -

# Add the Confluent yum repository
rpm --import http://packages.confluent.io/rpm/3.0/archive.key

cat >> /etc/yum.repos.d/confluent.repo <<EOF
[Confluent.dist]
name=Confluent repository (dist)
baseurl=http://packages.confluent.io/rpm/3.0/6
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/3.0/archive.key
enabled=1

[Confluent]
name=Confluent repository
baseurl=http://packages.confluent.io/rpm/3.0
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/3.0/archive.key
enabled=1
EOF

# Remove the existing v0.8.2 kafka
yum remove -y kafka

# Install confluent platform via yum
yum clean all
yum install -y confluent-platform-2.11

# Don't need root now, so exit the root shell back to oracle
logout

Install Oracle GoldenGate Adapter for Kafka Connect

This connector is published by Oracle on the java.net site.

# Download it
cd ~/Downloads
wget https://java.net/projects/oracledi/downloads/download/GoldenGate/Oracle%20GoldenGate%20Adapter%20for%20Kafka%20Connect/OGG_KafkaConnectHandlerFormatter1.0.tar
# Unpack
mkdir OGG_KafkaConnectHandlerFormatter
mv OGG_KafkaConnectHandlerFormatter1.0.tar OGG_KafkaConnectHandlerFormatter
cd OGG_KafkaConnectHandlerFormatter/
tar -xvf OGG_KafkaConnectHandlerFormatter1.0.tar
# Copy to OGG-BD location
cp kafka-connect/bin/ogg-kafka-connect-1.0.jar /u01/ogg-bd/ggjava/resources/lib/
cp kafka-connect/dirprm/conf* /u01/ogg-bd/dirprm/

Install and Build Elasticsearch Kafka Connect connector

The Elasticsearch connector currently has to be built from source. You can do this as follows.

Install Apache Maven

sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
sudo yum install -y apache-maven

Install Gradle

cd ~/Downloads
wget https://downloads.gradle.org/distributions/gradle-3.0-bin.zip
unzip gradle-3.0-bin.zip

Clone the connector github repository, as well as Kafka repository itself

cd ~/Downloads
git clone https://github.com/confluentinc/kafka-connect-elasticsearch.git
git clone https://github.com/confluentinc/kafka.git

Build Kafka and install in the local Maven repository - this will take 5-10 minutes to run.

export GRADLE_HOME=/home/oracle/Downloads/gradle-3.0
export PATH=$PATH:$GRADLE_HOME/bin
cd ~/Downloads/kafka
gradle wrapper
gradle install

It should complete succesfully, with a few warnings scattered along the way

[...]
:streams:examples:javadoc SKIPPED
:streams:examples:javadocJar
:streams:examples:srcJar
:streams:examples:signArchives SKIPPED
:streams:examples:install

BUILD SUCCESSFUL

Now build the connector itself (takes a few minutes to run):

cd ~/Downloads/kafka-connect-elasticsearch
mvn clean package -DskipTests

You should get a success message

[...]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:18 min
[INFO] Finished at: 2016-09-02T13:49:43+01:00
[INFO] Final Memory: 37M/438M
[INFO] ------------------------------------------------------------------------

and a jar file for the connector plus dependent libraries:

$ ls -l ~/Downloads/kafka-connect-elasticsearch/target/kafka-connect-elasticsearch-3.1.0-SNAPSHOT-package/share/java/kafka-connect-elasticsearch/
total 5016
-rw-r--r--. 1 oracle oinstall  263965 Sep  2 13:38 commons-codec-1.9.jar
-rw-r--r--. 1 oracle oinstall  434678 Sep  2 13:38 commons-lang3-3.4.jar
-rw-r--r--. 1 oracle oinstall   61829 Sep  2 13:38 commons-logging-1.2.jar
-rw-r--r--. 1 oracle oinstall  212164 Sep  2 13:38 gson-2.4.jar
-rw-r--r--. 1 oracle oinstall 2256213 Sep  2 13:38 guava-18.0.jar
-rw-r--r--. 1 oracle oinstall  177013 Sep  2 13:38 httpasyncclient-4.1.1.jar
-rw-r--r--. 1 oracle oinstall  732765 Sep  2 13:38 httpclient-4.5.1.jar
-rw-r--r--. 1 oracle oinstall  326724 Sep  2 13:38 httpcore-4.4.4.jar
-rw-r--r--. 1 oracle oinstall  356091 Sep  2 13:38 httpcore-nio-4.4.4.jar
-rw-r--r--. 1 oracle oinstall   18398 Sep  2 13:38 jest-2.0.0.jar
-rw-r--r--. 1 oracle oinstall  216228 Sep  2 13:38 jest-common-2.0.0.jar
-rw-r--r--. 1 oracle oinstall   48947 Sep  2 13:49 kafka-connect-elasticsearch-3.1.0-SNAPSHOT.jar
-rw-r--r--. 1 oracle oinstall   10680 Sep  2 13:38 slf4j-simple-1.7.5.jar

Move this to the /opt folder:

$ sudo cp -r /home/oracle/Downloads/kafka-connect-elasticsearch/target/kafka-connect-elasticsearch-3.1.0-SNAPSHOT-package/share/java/kafka-connect-elasticsearch/ /opt/kafka-connect-elasticsearch/

$ ls -l /opt/kafka-connect-elasticsearch/
total 5016
-rw-r--r--. 1 root root  263965 Sep  5 15:33 commons-codec-1.9.jar
-rw-r--r--. 1 root root  434678 Sep  5 15:33 commons-lang3-3.4.jar
-rw-r--r--. 1 root root   61829 Sep  5 15:33 commons-logging-1.2.jar
-rw-r--r--. 1 root root  212164 Sep  5 15:33 gson-2.4.jar
-rw-r--r--. 1 root root 2256213 Sep  5 15:33 guava-18.0.jar
-rw-r--r--. 1 root root  177013 Sep  5 15:33 httpasyncclient-4.1.1.jar
-rw-r--r--. 1 root root  732765 Sep  5 15:33 httpclient-4.5.1.jar
-rw-r--r--. 1 root root  326724 Sep  5 15:33 httpcore-4.4.4.jar
-rw-r--r--. 1 root root  356091 Sep  5 15:33 httpcore-nio-4.4.4.jar
-rw-r--r--. 1 root root   18398 Sep  5 15:33 jest-2.0.0.jar
-rw-r--r--. 1 root root  216228 Sep  5 15:33 jest-common-2.0.0.jar
-rw-r--r--. 1 root root   48947 Sep  5 15:33 kafka-connect-elasticsearch-3.1.0-SNAPSHOT.jar
-rw-r--r--. 1 root root   10680 Sep  5 15:33 slf4j-simple-1.7.5.jar

Install Elasticsearch

To install Elasticsearch and Kibana it's just a case of downloading the files and unzipping them. As part of the install I'm including the Kopf and Sense plugins which make working with Elasticsearch easier.

cd ~/Downloads/

# Download the files
wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/zip/elasticsearch/2.4.0/elasticsearch-2.4.0.zip
wget https://download.elastic.co/kibana/kibana/kibana-4.6.0-linux-x86_64.tar.gz

# Unpack
unzip elasticsearch-2.4.0.zip
tar -xf kibana-4.6.0-linux-x86_64.tar.gz

# Move to /opt
sudo mv elasticsearch-2.4.0 /opt/
sudo mv kibana-4.6.0-linux-x86_64 /opt

# Install plugins
/opt/kibana-4.6.0-linux-x86_64/bin/kibana plugin --install elastic/sense
/opt/elasticsearch-2.4.0/bin/plugin install lmenezes/elasticsearch-kopf

Add configuration to Elasticsearch to bind to all available network addresses (not just loopback):

cat >> /opt/elasticsearch-2.4.0/config/elasticsearch.yml <<EOF
network.host: 0.0.0.0
EOF

Install Swingbench

Install swingbench:

cd ~/Downloads
# Download
wget http://dominicgiles.com/swingbench/swingbench25971.zip
# Unzip
unzip swingbench25971.zip
# Move to /opt
sudo mv swingbench /opt

Create the Swingbench schema

Having installed Swingbench, we'll now run the installation of the schema and initial data.

Make sure that the database is running

sudo service dbora start

Run Swingbench's initialisation, which will create the table and data necessary to run Swingbench.

/opt/swingbench/bin/oewizard -cs localhost:1521/orcl -df /u01/app/oracle/oradata/cdb/orcl/soe.dbf -cl -scale 0.1 -dbap welcome1 -create -u soe -p soe -ts soe -v

On my VM this took ~25 minutes to complete.

@JV084

This comment has been minimized.

Copy link

commented Nov 21, 2017

Hi @rmoff

I'm getting this error when I try to build elastic search connector, any clue?

Thanks!

$ mvn clean package -DskipTests

[INFO] Scanning for projects...
[WARNING] Could not transfer metadata io.confluent:common:4.1.0-SNAPSHOT/maven-metadata.xml from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for io.confluent:kafka-connect-elasticsearch:[unknown-version]: Could not transfer artifact io.confluent:common:pom:4.1.0-SNAPSHOT from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 7, column 13
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project io.confluent:kafka-connect-elasticsearch:[unknown-version] (/u01/userhome/oracle/Downloads/kafka-connect-elasticsearch/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for io.confluent:kafka-connect-elasticsearch:[unknown-version]: Could not transfer artifact io.confluent:common:pom:4.1.0-SNAPSHOT from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 7, column 13: Cannot access ${confluent.maven.repo} using the registered transporter factories: WagonTransporterFactory: Unsupported transport protocol -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
[oracle@localhost kafka-connect-elasticsearch]$

@hadooping

This comment has been minimized.

Copy link

commented Mar 16, 2018

I get the same error message

@benpague

This comment has been minimized.

Copy link

commented Apr 16, 2018

same error here

@deepuak01

This comment has been minimized.

Copy link

commented Nov 3, 2018

Following steps helped me solve the error:

  1. clone https://github.com/confluentinc/common and build with mvn install
  2. comment the repositories section of pom.xml file for kafka-connect-elasticsearch:
  3. run mvn clean package -DskipTests
@deepuak01

This comment has been minimized.

Copy link

commented Nov 3, 2018

When executing the steps: cd ~/Downloads/kafka , gradle wrapper , gradle install, there was below error:
A problem occurred evaluating root project 'kafka' + Failed to apply plugin [id 'org.owasp.dependencycheck'] + Could not create task of type 'Analyze'
To solve the error, follow the below steps:

  1. Make sure to use higher version of gradle preferably gradle-4.10.2
  2. set GRADLE_HOME to point to gradle-4.10.2 and make sure that gradle -v gives version 4.10.2
  3. Now run gradle wrapper & gradle install
@prgore

This comment has been minimized.

Copy link

commented Jun 14, 2019

Hi ,

I am getting below error while download the OGG_KafkaConnectHandlerFormatter , any solution.

[oracle@bigdatalite Downloads]$ wget https://java.net/projects/oracledi/downloads/download/GoldenGate/Oracle%20GoldenGate%20Adapter%20for%20Kafka%20Connect/OGG_KafkaConnectHandlerFormatter1.0.tar
--2019-06-14 09:43:55-- https://java.net/projects/oracledi/downloads/download/GoldenGate/Oracle%20GoldenGate%20Adapter%20for%20Kafka%20Connect/OGG_KafkaConnectHandlerFormatter1.0.tar
Resolving java.net... 137.254.56.25
Connecting to java.net|137.254.56.25|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.oracle.com/splash/java.net/maintenance/index.html [following]
--2019-06-14 09:43:56-- http://www.oracle.com/splash/java.net/maintenance/index.html
Resolving www.oracle.com... 23.9.246.36, 2600:140f:3000:180::366, 2600:140f:3000:181::366
Connecting to www.oracle.com|23.9.246.36|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2019-06-14 09:43:57 ERROR 503: Service Unavailable.

@rmoff

This comment has been minimized.

Copy link
Owner Author

commented Jun 14, 2019

@prgore This is 2.5 years old - probably URLs are since moved/dead.

@prgore

This comment has been minimized.

Copy link

commented Jun 14, 2019

Issue got resolved by downloading the Oracle GoldenGate Kafka Connect Adapter from below active link.

https://www.oracle.com/technetwork/middleware/goldengate/oracle-goldengate-exchange-3805527.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.