redpanda_data_transforms_docker.md

This is directly based on the Redpanda docs for Data Transforms link.

You should probably start from there to get a rough idea of where data transforms fit in.
The following steps are for if you want to try out transforms within a single-node docker container, e.g. as part of a testing pipeline. Some if not most of the steps can be moved into the image building stage.

Start the RedPanda container:

IMAGE_TAG_REDPANDA:='docker.redpanda.com/vectorized/redpanda:latest'
CONTAINER_TAG_REDPANDA:=redpanda-1
docker run -d --pull=always --name=redpanda-1 \
	-p 8081:8081 -p 8082:8082 -p 9092:9092 -p 9644:9644 \
	$(IMAGE_TAG_REDPANDA) \
	redpanda start \
	--overprovisioned \
	--smp 1  \
	--memory 1G \
	--reserve-memory 0M \
	--node-id 0 \
	--check=false

Keep in mind rpk is aliased to docker container exec redpanda-1 rpk throughout the rest of this document unless specified otherwise:

alias rpk="docker container exec redpanda-1 rpk"

My current version redpanda version:

rpk version
# v22.1.4 (rev 491e569)

Configure as needed:

rpk cluster config set enable_coproc true
# Successfully updated configuration. New configuration version is 3.

rpk cluster config set enable_idempotence false
# Successfully updated configuration. New configuration version is 4.

Further config, (I don't know how to do this part without getting 'inside the container'):

docker exec -it redpanda-1 bash
vi /etc/redpanda/redpanda.yaml

Add the following lines under redpanda then save and exit:

enable_coproc: true
developer_mode: true
enable_idempotence : false

If you check the config status, you'll see that you need to restart:

rpk cluster config status
NODE  CONFIG-VERSION  NEEDS-RESTART  INVALID  UNKNOWN
0     4               true           []       []

Restart the container:

rpk redpanda stop
docker container restart redpanda-1

Check that the config has been applied:

rpk cluster config status
# NODE  CONFIG-VERSION  NEEDS-RESTART  INVALID  UNKNOWN
# 0     4               false          []       []

Create directory to hold transforms:

docker container exec --user root redpanda-1 mkdir /transforms
docker container exec --user root redpanda-1 chown redpanda:redpanda /transforms

Generate the transform starter code:

docker container exec -w /transforms redpanda-1 rpk wasm generate uppercase

Copy out the code to your host environment:

docker container cp redpanda-1:/transforms/uppercase .

Your host environment must have node and npm installed. Install the dependencies. Note that some of the dependencies are deprecated but it will have to do for now.

From there, make the necessary changes as you see fit. Also make sure to create the source topic. By default the source topic is test-topic.

rpk topic create test-topic

Build the transform:

cd uppercase
npm run build

I got the following error when running npm run build, my node version was v18.4.0:

...
Error: error:0308010C:digital envelope routines::unsupported
...

To fix it, I ran build with the following environment variable:

NODE_OPTIONS='--openssl-legacy-provider' npm run build

Copy the compiled bundle back into the container:

docker container cp dist/main.js redpanda-1:/transforms/uppercase.js
docker container exec --user root redpanda-1 chown redpanda:redpanda /transforms/uppercase.js

Finally, deploy the transform:

docker container exec redpanda-1 rpk wasm deploy /transforms/uppercase.js --name uppercase