Build Confluent Platform manually

testc3.md

1. Clone ce-kafka checkout 5.3.x and build

   gradle && ./gradlew jar

2. Start ZK in ce-kafka repo

   ./bin/zookeeper-server-start.sh config/zookeeper.properties

3. Start Kafka (change logs to /tmp) in ce-kafka repo

   ./bin/kafka-server-start.sh config/server.properties

4. Start Connect in ce-kafka repo

   ./bin/connect-distributed.sh config/connect-distributed.properties

5. Clone schema-registry checkout 5.3.x and build

   mvn clean package -DskipTests

6. Start schema registry in schema-registry repo

   ./bin/schema-registry-start config/schema-registry.properties

7. For KSQL, clone the KSQL repo checkout 5.3.x and build:

   mvn clean package -DskipTests

   • Run KSQL Server and the CLI:

     ./bin/ksql-server-start -daemon config/ksql-server.properties
     ./bin/ksql
     

8. Clone blueway checkout 5.3.x and build

   mvn clean package -DskipTests

9. Start control center from blueway repo

   ./bin/control-center-start config/control-center-dev.properties

10. Clone frontend-vault checkout 5.3.x and build

    yarn in && yarn workspace @confluent/control-center start

Before step 9, run yarn in to pull in all dependencies.

As developers commit frontend fixes, do a git pull origin --rebase in the frontend-vault cp-5.3.x branch for example and run step 9 again.

Super helpful, thanks!

For KSQL, clone the KSQL repo and build:

mvn clean package -DskipTests

Run KSQL Server and the CLI:

./bin/ksql-server-start -daemon config/ksql-server.properties
./bin/ksql

Works on Ubuntu 19.04 Disco Dingo! I found that need to delete the Kafka log files and the Zookeeper metadata between sessions:

sudo rm -rf /var/lib/kafka/
sudo rm -rf /var/lib/zookeeper/

This path is set by the log.dir property in the server.properties file.

I hacked together a couple of quick scripts to start and stop the stack:

sudo ./start-hand-rolled-cp.sh
sudo ./stop-hand-rolled-cp.sh

TODO: start/stop C3.

The start-hand-rolled-cp.sh script starts the cluster:

#!/bin/bash
# Change CP_ROOT to your Git root directory.
CP_ROOT="~/repos"
KAFKA_BIN=$CP_ROOT"/ce-kafka/bin"
SR_BIN=$CP_ROOT"/schema-registry/bin"
KSQL_BIN=$CP_ROOT"/ksql/bin"

KAFKA_ETC=$CP_ROOT"/ce-kafka/config"
SR_ETC=$CP_ROOT"/schema-registry/config"
KSQL_ETC=$CP_ROOT"/ksql/config"

echo $CP_ROOT
echo $KAFKA_BIN
echo $SR_BIN
echo $KSQL_BIN

echo "Removing old state"
# This directory is set by log.dir in the server.properties file and may vary by OS.
sudo rm -rf /var/lib/kafka/
sudo rm -rf /var/lib/zookeeper/

echo "Starting Zookeeper"
eval $KAFKA_BIN"/zookeeper-server-start "$KAFKA_ETC"/zookeeper.properties &"

echo "Starting Kafka"
eval $KAFKA_BIN"/kafka-server-start "$KAFKA_ETC"/server.properties &"

echo "Starting connect"
eval $KAFKA_BIN"/connect-distributed "$KAFKA_ETC"/connect-distributed.properties &"

echo "Starting Schema Registry"
eval $SR_BIN"/schema-registry-start "$SR_ETC"/schema-registry.properties &"

echo "KSQL Server"
eval $KSQL_BIN"/ksql-server-start "$KSQL_ETC"/ksql-server.properties &"
#eval $KSQL_BIN"/ksql"
~                                                                                                                                                                                                          ~                                                                                                         

The stop-hand-rolled-cp.sh script stops the cluster:

#!/bin/bash
# Change CP_ROOT to your Git root directory.
CP_ROOT="~/repos"
KAFKA_BIN=$CP_ROOT"/ce-kafka/bin"
SR_BIN=$CP_ROOT"/schema-registry/bin"
KSQL_BIN=$CP_ROOT"/ksql/bin"

KAFKA_ETC=$CP_ROOT"/ce-kafka/config"
SR_ETC=$CP_ROOT"/schema-registry/config"
KSQL_ETC=$CP_ROOT"/ksql/config"

echo $CP_ROOT
echo $KAFKA_BIN
echo $SR_BIN
echo $KSQL_BIN

eval $KAFKA_BIN"/zookeeper-server-stop"
eval $KAFKA_BIN"/kafka-server-stop"
eval $SR_BIN"/schema-registry-stop"
eval $KSQL_BIN"/ksql-server-stop"
# TODO: stop connect-distributed