Skip to content

Instantly share code, notes, and snippets.

What would you like to do?

This is a set of instructions for use with the blog article Streaming data from Oracle using Oracle GoldenGate and Kafka Connect.

@rmoff / September 15, 2016

First up, download the BigDataLite VM, unpack it and import it to VirtualBox. You'll need internet access from the VM for the downloads, so make sure you include a NAT network adaptor, or bridged onto a network with internet access. Login to the machine as oracle/welcome1. All the work done in this article is from the command line, so you can either work in Terminal, or you can run ip a to determine the IP address of the VM and then SSH into it from your host machine.

Install Confluent Platform

Now install the Confluent Platform by running the following commands.

# Gimme root
sudo su -

# Add the Confluent yum repository
rpm --import

cat >> /etc/yum.repos.d/confluent.repo <<EOF
name=Confluent repository (dist)

name=Confluent repository

# Remove the existing v0.8.2 kafka
yum remove -y kafka

# Install confluent platform via yum
yum clean all
yum install -y confluent-platform-2.11

# Don't need root now, so exit the root shell back to oracle

Install Oracle GoldenGate Adapter for Kafka Connect

This connector is published by Oracle on the site.

# Download it
cd ~/Downloads
# Unpack
mkdir OGG_KafkaConnectHandlerFormatter
mv OGG_KafkaConnectHandlerFormatter1.0.tar OGG_KafkaConnectHandlerFormatter
cd OGG_KafkaConnectHandlerFormatter/
tar -xvf OGG_KafkaConnectHandlerFormatter1.0.tar
# Copy to OGG-BD location
cp kafka-connect/bin/ogg-kafka-connect-1.0.jar /u01/ogg-bd/ggjava/resources/lib/
cp kafka-connect/dirprm/conf* /u01/ogg-bd/dirprm/

Install and Build Elasticsearch Kafka Connect connector

The Elasticsearch connector currently has to be built from source. You can do this as follows.

Install Apache Maven

sudo wget -O /etc/yum.repos.d/epel-apache-maven.repo
sudo yum install -y apache-maven

Install Gradle

cd ~/Downloads

Clone the connector github repository, as well as Kafka repository itself

cd ~/Downloads
git clone
git clone

Build Kafka and install in the local Maven repository - this will take 5-10 minutes to run.

export GRADLE_HOME=/home/oracle/Downloads/gradle-3.0
cd ~/Downloads/kafka
gradle wrapper
gradle install

It should complete succesfully, with a few warnings scattered along the way

:streams:examples:javadoc SKIPPED
:streams:examples:signArchives SKIPPED


Now build the connector itself (takes a few minutes to run):

cd ~/Downloads/kafka-connect-elasticsearch
mvn clean package -DskipTests

You should get a success message

[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:18 min
[INFO] Finished at: 2016-09-02T13:49:43+01:00
[INFO] Final Memory: 37M/438M
[INFO] ------------------------------------------------------------------------

and a jar file for the connector plus dependent libraries:

$ ls -l ~/Downloads/kafka-connect-elasticsearch/target/kafka-connect-elasticsearch-3.1.0-SNAPSHOT-package/share/java/kafka-connect-elasticsearch/
total 5016
-rw-r--r--. 1 oracle oinstall  263965 Sep  2 13:38 commons-codec-1.9.jar
-rw-r--r--. 1 oracle oinstall  434678 Sep  2 13:38 commons-lang3-3.4.jar
-rw-r--r--. 1 oracle oinstall   61829 Sep  2 13:38 commons-logging-1.2.jar
-rw-r--r--. 1 oracle oinstall  212164 Sep  2 13:38 gson-2.4.jar
-rw-r--r--. 1 oracle oinstall 2256213 Sep  2 13:38 guava-18.0.jar
-rw-r--r--. 1 oracle oinstall  177013 Sep  2 13:38 httpasyncclient-4.1.1.jar
-rw-r--r--. 1 oracle oinstall  732765 Sep  2 13:38 httpclient-4.5.1.jar
-rw-r--r--. 1 oracle oinstall  326724 Sep  2 13:38 httpcore-4.4.4.jar
-rw-r--r--. 1 oracle oinstall  356091 Sep  2 13:38 httpcore-nio-4.4.4.jar
-rw-r--r--. 1 oracle oinstall   18398 Sep  2 13:38 jest-2.0.0.jar
-rw-r--r--. 1 oracle oinstall  216228 Sep  2 13:38 jest-common-2.0.0.jar
-rw-r--r--. 1 oracle oinstall   48947 Sep  2 13:49 kafka-connect-elasticsearch-3.1.0-SNAPSHOT.jar
-rw-r--r--. 1 oracle oinstall   10680 Sep  2 13:38 slf4j-simple-1.7.5.jar

Move this to the /opt folder:

$ sudo cp -r /home/oracle/Downloads/kafka-connect-elasticsearch/target/kafka-connect-elasticsearch-3.1.0-SNAPSHOT-package/share/java/kafka-connect-elasticsearch/ /opt/kafka-connect-elasticsearch/

$ ls -l /opt/kafka-connect-elasticsearch/
total 5016
-rw-r--r--. 1 root root  263965 Sep  5 15:33 commons-codec-1.9.jar
-rw-r--r--. 1 root root  434678 Sep  5 15:33 commons-lang3-3.4.jar
-rw-r--r--. 1 root root   61829 Sep  5 15:33 commons-logging-1.2.jar
-rw-r--r--. 1 root root  212164 Sep  5 15:33 gson-2.4.jar
-rw-r--r--. 1 root root 2256213 Sep  5 15:33 guava-18.0.jar
-rw-r--r--. 1 root root  177013 Sep  5 15:33 httpasyncclient-4.1.1.jar
-rw-r--r--. 1 root root  732765 Sep  5 15:33 httpclient-4.5.1.jar
-rw-r--r--. 1 root root  326724 Sep  5 15:33 httpcore-4.4.4.jar
-rw-r--r--. 1 root root  356091 Sep  5 15:33 httpcore-nio-4.4.4.jar
-rw-r--r--. 1 root root   18398 Sep  5 15:33 jest-2.0.0.jar
-rw-r--r--. 1 root root  216228 Sep  5 15:33 jest-common-2.0.0.jar
-rw-r--r--. 1 root root   48947 Sep  5 15:33 kafka-connect-elasticsearch-3.1.0-SNAPSHOT.jar
-rw-r--r--. 1 root root   10680 Sep  5 15:33 slf4j-simple-1.7.5.jar

Install Elasticsearch

To install Elasticsearch and Kibana it's just a case of downloading the files and unzipping them. As part of the install I'm including the Kopf and Sense plugins which make working with Elasticsearch easier.

cd ~/Downloads/

# Download the files

# Unpack
tar -xf kibana-4.6.0-linux-x86_64.tar.gz

# Move to /opt
sudo mv elasticsearch-2.4.0 /opt/
sudo mv kibana-4.6.0-linux-x86_64 /opt

# Install plugins
/opt/kibana-4.6.0-linux-x86_64/bin/kibana plugin --install elastic/sense
/opt/elasticsearch-2.4.0/bin/plugin install lmenezes/elasticsearch-kopf

Add configuration to Elasticsearch to bind to all available network addresses (not just loopback):

cat >> /opt/elasticsearch-2.4.0/config/elasticsearch.yml <<EOF

Install Swingbench

Install swingbench:

cd ~/Downloads
# Download
# Unzip
# Move to /opt
sudo mv swingbench /opt

Create the Swingbench schema

Having installed Swingbench, we'll now run the installation of the schema and initial data.

Make sure that the database is running

sudo service dbora start

Run Swingbench's initialisation, which will create the table and data necessary to run Swingbench.

/opt/swingbench/bin/oewizard -cs localhost:1521/orcl -df /u01/app/oracle/oradata/cdb/orcl/soe.dbf -cl -scale 0.1 -dbap welcome1 -create -u soe -p soe -ts soe -v

On my VM this took ~25 minutes to complete.

Copy link

JV084 commented Nov 21, 2017

Hi @rmoff

I'm getting this error when I try to build elastic search connector, any clue?


$ mvn clean package -DskipTests

[INFO] Scanning for projects...
[WARNING] Could not transfer metadata io.confluent:common:4.1.0-SNAPSHOT/maven-metadata.xml from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for io.confluent:kafka-connect-elasticsearch:[unknown-version]: Could not transfer artifact io.confluent:common:pom:4.1.0-SNAPSHOT from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 7, column 13
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR] The project io.confluent:kafka-connect-elasticsearch:[unknown-version] (/u01/userhome/oracle/Downloads/kafka-connect-elasticsearch/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for io.confluent:kafka-connect-elasticsearch:[unknown-version]: Could not transfer artifact io.confluent:common:pom:4.1.0-SNAPSHOT from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 7, column 13: Cannot access ${confluent.maven.repo} using the registered transporter factories: WagonTransporterFactory: Unsupported transport protocol -> [Help 2]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1]
[ERROR] [Help 2]
[oracle@localhost kafka-connect-elasticsearch]$

Copy link

hadooping commented Mar 16, 2018

I get the same error message

Copy link

benpague commented Apr 16, 2018

same error here

Copy link

deepuak01 commented Nov 3, 2018

Following steps helped me solve the error:

  1. clone and build with mvn install
  2. comment the repositories section of pom.xml file for kafka-connect-elasticsearch:
  3. run mvn clean package -DskipTests

Copy link

deepuak01 commented Nov 3, 2018

When executing the steps: cd ~/Downloads/kafka , gradle wrapper , gradle install, there was below error:
A problem occurred evaluating root project 'kafka' + Failed to apply plugin [id 'org.owasp.dependencycheck'] + Could not create task of type 'Analyze'
To solve the error, follow the below steps:

  1. Make sure to use higher version of gradle preferably gradle-4.10.2
  2. set GRADLE_HOME to point to gradle-4.10.2 and make sure that gradle -v gives version 4.10.2
  3. Now run gradle wrapper & gradle install

Copy link

prgore commented Jun 14, 2019

Hi ,

I am getting below error while download the OGG_KafkaConnectHandlerFormatter , any solution.

[oracle@bigdatalite Downloads]$ wget
--2019-06-14 09:43:55--
Connecting to||:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: [following]
--2019-06-14 09:43:56--
Resolving, 2600:140f:3000:180::366, 2600:140f:3000:181::366
Connecting to||:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2019-06-14 09:43:57 ERROR 503: Service Unavailable.

Copy link

rmoff commented Jun 14, 2019

@prgore This is 2.5 years old - probably URLs are since moved/dead.

Copy link

prgore commented Jun 14, 2019

Issue got resolved by downloading the Oracle GoldenGate Kafka Connect Adapter from below active link.

Copy link

venkatkumbha commented Nov 30, 2019

i am getting below exception when pushing message to kafka

2019-11-30 06:19:56 ERROR OGG-15051 Oracle GoldenGate Delivery, rconf.prm: Java or JNI exception:
java.lang.AbstractMethodError: oracle.goldengate.kafkaconnect.formatter.KafkaConnectFormatter.metaDataChanged(Loracle/goldengate/datasource/DsEvent;Loracle/goldengate/datasource/meta/DsMetaData;)V.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment