Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dfparker2002/5d4b7e2932af1122ee0def8b0acb4b32 to your computer and use it in GitHub Desktop.
Save dfparker2002/5d4b7e2932af1122ee0def8b0acb4b32 to your computer and use it in GitHub Desktop.
current status: not using docker-compose. need to run nifi, registry and jupyter notebook under once container

sOME sETUP:

ubuntu@ip-10-201-97-84:~/nifi-persisted/nouser$ ls
getnifi.sh  nifi-1.8.0  nifi-1.8.0-bin.zip  nifi-toolkit-1.8.0  nifi-toolkit-1.8.0-bin.zip  scripts

ubuntu@ip-10-201-97-84:~/nifi-persisted/nouser$ ls scripts/
common.sh  start.sh           toolkit.sh                          update_login_providers.sh
secure.sh  update_cluster_state_management.sh

ubuntu@ip-10-201-97-84:~/nifi-persisted/nouser$ cat getnifi.sh

#!/bin/sh
export nifiversion=1.8.0
wget https://archive.apache.org/dist/nifi/${nifiversion}/nifi-toolkit-${nifiversion}-bin.zip
wget https://archive.apache.org/dist/nifi/${nifiversion}/nifi-${nifiversion}-bin.zip
unzip nifi-toolkit-${nifiversion}-bin.zip
unzip nifi-${nifiversion}-bin.zip

useful commands: jupyter notebook --ip=127.0.0.1 --port=19999 --allow-root

version: "3"
services:
# zookeeper:
# hostname: zookeeper
# container_name: zookeeper-${NIFIUSER}
# image: 'bitnami/zookeeper:latest'
# restart: always
# environment:
# - ALLOW_ANONYMOUS_LOGIN=yes
nifi:
image: mannharleen/nifi:1.8.0
build: .
restart: always
# container_name: mynifi-${NIFIUSER} # cannot give container name in cluser mode
volumes:
- "/home/ubuntu/nifi-persisted/${NIFIUSER}/:/opt/nifi/" # mounts both nifi and nifi-toolkit
ports:
# - "8080:8080"
- "3838:3838" # for testing
- "80:80"
- "10001-10010:10001-10010" # ports provided to users
environment:
- NIFI_WEB_HTTP_PORT=3838 # 8080
# - NIFI_CLUSTER_IS_NODE=true
# - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
# - NIFI_ZK_CONNECT_STRING=zookeeper:2181
# - NIFI_ELECTION_MAX_WAIT=1 min
registry-030:
image: apache/nifi-registry:0.3.0
restart: always
container_name: nifiregistry-${NIFIUSER}
ports:
- "18080:18080"
mysql-57:
image: mysql
restart: always
container_name: mysql-${NIFIUSER}
command: --default-authentication-plugin=mysql_native_password
environment:
MYSQL_ROOT_PASSWORD: mysql
ports:
- "3306:3306"
# Dockerfile v0.2
# v0.2 WIP , with jupyter notebook??
# 0.2: use of entry point instead of command to run the serices
# 0.1: runs nifi standalone
#
# use command:
# --------------
# # sudo docker build -t mannharleen/nifiwithregistry:1.8.0 .
# # export NIFIUSER=ubuntu
# # sudo docker run --rm --name mynifi-${NIFIUSER} -p 3838:3838 -v /home/ubuntu/nifi-persisted/${NIFIUSER}/:/opt/nifi/ -e NIFIUSER=${NIFIUSER} mannharleen/nifiwithregistry:1.8.0
# --------------
#
FROM openjdk:8-jre
LABEL maintainer="Harleen <mannharleen@gmail.com>"
LABEL site="https://nifi.apache.org"
ARG UID=1000
ARG GID=1000
ARG NIFI_VERSION=1.8.0
ARG NIFI_TOOLKIT_VERSION=1.8.0
ARG NIFI_REGISTRY_VERSION=0.3.0
ARG BASE_URL=https://archive.apache.org/dist
ARG MIRROR_BASE_URL=${MIRROR_BASE_URL:-${BASE_URL}}
ARG NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
ARG NIFI_TOOLKIT_BINARY_PATH=${NIFI_TOOLKIT_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-toolkit-${NIFI_VERSION}-bin.zip}
#ARG NIFIUSER=${NIFIUSER:-ubuntu}
ARG NIFI_WEB_HTTP_PORT=${NIFI_WEB_HTTP_PORT:-3838}
# ARG USERNAME
ENV NIFI_BASE_DIR=/opt/nifi
ENV NIFI_HOME=${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}
ENV NIFI_TOOLKIT_HOME ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_TOOLKIT_VERSION}
ENV NIFI_REGISTRY_HOME=${NIFI_BASE_DIR}/nifi-registry-${NIFI_REGISTRY_VERSION}
#ENV NIFI_HOME ${NIFI_BASE_DIR}/nifi-current
#ENV NIFI_TOOLKIT_HOME ${NIFI_BASE_DIR}/nifi-toolkit-current
ENV NIFI_PID_DIR=${NIFI_HOME}/run
ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ENV NIFI_WEB_HTTP_PORT=${NIFI_WEB_HTTP_PORT}
#ENV NIFIUSER=${NIFIUSER:-ubuntu}
# Apache archives:
# https://archive.apache.org/dist/nifi/1.8.0/
# nifi-1.8.0-bin.zip
# nifi-toolkit-1.8.0-bin.zip
#ADD sh/ ${NIFI_BASE_DIR}/scripts/
# Setup NiFi user and create necessary directories
#RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
# && useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
# && mkdir -p ${NIFI_BASE_DIR} \
# && chown -R nifi:nifi ${NIFI_BASE_DIR} \
# && apt-get update \
# && apt-get install -y jq xmlstarlet procps
#USER nifi
USER ${NIFIUSER}
RUN apt-get update && \
apt-get -y upgrade && \
apt-get install -y python3-pip && \
echo 'python 3 installed' && \
python3 -m pip install --upgrade pip && \
python3 -m pip install jupyter
# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
#RUN echo "#!/bin/sh\n" > $NIFI_HOME/bin/nifi-env.sh
# Web HTTP(s) & Socket Site-to-Site Ports
EXPOSE 8080 8443 10000
#VOLUME "/home/ubuntu/nifi-persisted/${fi_pidNIFIUSER}/:/opt/nifi/"
WORKDIR ${NIFI_HOME}
# Apply configuration and start NiFi
#
# We need to use the exec form to avoid running our command in a subshell and omitting signals,
# thus being unable to shut down gracefully:
# https://docs.docker.com/engine/reference/builder/#entrypoint
#
# Also we need to use relative path, because the exec form does not invoke a command shell,
# thus normal shell processing does not happen:
# https://docs.docker.com/engine/reference/builder/#exec-form-entrypoint-example
# ENTRYPOINT ["../scripts/start.sh"]
# CMD bash
# CMD "../scripts/start.sh"
#CMD "echo"
ENTRYPOINT ["../scripts/start.sh"]
#ENTRYPOINT ["../scripts/start-all.sh"]
#
#
#
# USAGE:
#
# # docker run --rm --name mynifi-${NIFIUSER} -v /home/ubuntu/nifi-persisted/${NIFIUSER}:/opt/nifi/ -p 8080:8080 mynifi
#
# # sudo docker build -t mannharleen/nifiwithregistry:1.8.0 .
# # export NIFIUSER=ubuntu
# # sudo docker run --rm --name mynifi-${NIFIUSER} -p 3838:3838 -v /home/ubuntu/nifi-persisted/${NIFIUSER}/:/opt/nifi/ -e NIFIUSER=${NIFIUSER} mannharleen/nifiwithregistry:1.8.0 ../scripts/start.sh
# Dockerfile
# v0.1
# runs nifi standalone without registry
# use command:
# --------------
# # sudo docker build -t mannharleen/nifiwithregistry:1.8.0 .
# # export NIFIUSER=ubuntu
# # sudo docker run --rm --name mynifi-${NIFIUSER} -p 3838:3838 -v /home/ubuntu/nifi-persisted/${NIFIUSER}/:/opt/nifi/ -e NIFIUSER=${NIFIUSER} mannharleen/nifiwithregistry:1.8.0 ../scripts/start.sh
# --------------
#
FROM openjdk:8-jre
LABEL maintainer="Harleen <mannharleen@gmail.com>"
LABEL site="https://nifi.apache.org"
ARG UID=1000
ARG GID=1000
ARG NIFI_VERSION=1.8.0
ARG NIFI_TOOLKIT_VERSION=1.8.0
ARG NIFI_REGISTRY_VERSION=0.3.0
ARG BASE_URL=https://archive.apache.org/dist
ARG MIRROR_BASE_URL=${MIRROR_BASE_URL:-${BASE_URL}}
ARG NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
ARG NIFI_TOOLKIT_BINARY_PATH=${NIFI_TOOLKIT_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-toolkit-${NIFI_VERSION}-bin.zip}
#ARG NIFIUSER=${NIFIUSER:-ubuntu}
ARG NIFI_WEB_HTTP_PORT=${NIFI_WEB_HTTP_PORT:-3838}
# ARG USERNAME
ENV NIFI_BASE_DIR=/opt/nifi
ENV NIFI_HOME=${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}
ENV NIFI_TOOLKIT_HOME ${NIFI_BASE_DIR}/nifi-toolkit-${NIFI_TOOLKIT_VERSION}
ENV NIFI_REGISTRY_HOME=${NIFI_BASE_DIR}/nifi-registry-${NIFI_REGISTRY_VERSION}
#ENV NIFI_HOME ${NIFI_BASE_DIR}/nifi-current
#ENV NIFI_TOOLKIT_HOME ${NIFI_BASE_DIR}/nifi-toolkit-current
ENV NIFI_PID_DIR=${NIFI_HOME}/run
ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ENV NIFI_WEB_HTTP_PORT=${NIFI_WEB_HTTP_PORT}
ENV NIFIUSER=${NIFIUSER:-ubuntu}
# Apache archives:
# https://archive.apache.org/dist/nifi/1.8.0/
# nifi-1.8.0-bin.zip
# nifi-toolkit-1.8.0-bin.zip
#ADD sh/ ${NIFI_BASE_DIR}/scripts/
# Setup NiFi user and create necessary directories
#RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d: -f1` \
# && useradd --shell /bin/bash -u ${UID} -g ${GID} -m nifi \
# && mkdir -p ${NIFI_BASE_DIR} \
# && chown -R nifi:nifi ${NIFI_BASE_DIR} \
# && apt-get update \
# && apt-get install -y jq xmlstarlet procps
#USER nifi
#USER ${NIFIUSER}
# Clear nifi-env.sh in favour of configuring all environment variables in the Dockerfile
#RUN echo "#!/bin/sh\n" > $NIFI_HOME/bin/nifi-env.sh
# Web HTTP(s) & Socket Site-to-Site Ports
EXPOSE 8080 8443 10000
#VOLUME "/home/ubuntu/nifi-persisted/${NIFIUSER}/:/opt/nifi/"
WORKDIR ${NIFI_HOME}
# Apply configuration and start NiFi
#
# We need to use the exec form to avoid running our command in a subshell and omitting signals,
# thus being unable to shut down gracefully:
# https://docs.docker.com/engine/reference/builder/#entrypoint
#
# Also we need to use relative path, because the exec form does not invoke a command shell,
# thus normal shell processing does not happen:
# https://docs.docker.com/engine/reference/builder/#exec-form-entrypoint-example
# ENTRYPOINT ["../scripts/start.sh"]
# CMD bash
# CMD "../scripts/start.sh"
CMD "echo"
#
# USAGE:
#
# # docker run --rm --name mynifi-${NIFIUSER} -v /home/ubuntu/nifi-persisted/${NIFIUSER}:/opt/nifi/ -p 8080:8080 mynifi
#
# # sudo docker build -t mannharleen/nifiwithregistry:1.8.0 .
# # export NIFIUSER=ubuntu
# # sudo docker run --rm --name mynifi-${NIFIUSER} -p 3838:3838 -v /home/ubuntu/nifi-persisted/${NIFIUSER}/:/opt/nifi/ -e NIFIUSER=${NIFIUSER} mannharleen/nifiwithregistry:1.8.0 ../scripts/start.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment