This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FROM apache/nifi-minifi:latest | |
USER root | |
ENV MINIFI_HOME /opt/minifi/minifi-0.5.0 | |
ADD config.yml $MINIFI_HOME/conf/config.yml | |
ADD *.nar $MINIFI_HOME/lib/ | |
RUN chown -R minifi:minifi $MINIFI_HOME |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
git clone git@github.com:pvillard31/nifi-cloudrun-example.git | |
cd nifi-cloudrun-example | |
gcloud auth configure-docker | |
# docker build -t [HOSTNAME]/[PROJECT-ID]/[IMAGE]:[TAG] . | |
docker build -t eu.gcr.io/my-gcp-project/minifi-csvjson:0.0.1 . | |
# docker push [HOSTNAME]/[PROJECT-ID]/[IMAGE]:[TAG] . | |
docker push eu.gcr.io/my-gcp-project/minifi-csvjson:0.0.1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
$ time curl --data-binary "@test.csv" -H "Content-Type: text/csv" -X POST https://minifi-csvjson-t546z2l6aq-uc.a.run.app | |
[ { | |
"name" : "Pierre Villard", | |
"company" : "Google" | |
} ] | |
real 0m0.725s | |
user 0m0.036s | |
sys 0m0.026s |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
$ ssh-keygen -t rsa -b 4096 -m PEM -C "NiFi Registry" | |
Generating public/private rsa key pair. | |
Enter file in which to save the key (~/.ssh/id_rsa): ~/temp/id_rsa | |
Enter passphrase (empty for no passphrase): | |
Enter same passphrase again: | |
Your identification has been saved in ~/temp/id_rsa. | |
Your public key has been saved in ~/temp/id_rsa.pub. | |
The key fingerprint is: | |
SHA256:/hu6FZLvcvDigP4ixPFPEGzfpY1RHOoTRytjqYOoYHo NiFi Registry |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gcloud beta compute instances create-with-container my-nifi-instance --tags=nifi --container-image=apache/nifi | |
gcloud compute firewall-rules create allow-nifi-unsecured --action=ALLOW --rules=tcp:8080 --target-tags=nifi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
git clone https://github.com/pvillard31/nifi-gcp-terraform.git | |
cd nifi-gcp-terraform/gcp-single-secured-nifi-oidc/ | |
terraform init | |
terraform apply |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
git clone https://github.com/pvillard31/nifi-gcp-terraform.git | |
cd nifi-gcp-terraform/gcp-cluster-secured-nifi-oidc | |
/bin/sh deploy.sh <gcp-project-id> <gcs-bucket> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pvillard@cloudshell:~/nifi-gcp-terraform/gcp-cluster-secured-nifi-oidc (nifi-dev-project)$ /bin/sh deploy.sh nifi-dev-project nifi_bin | |
Updated property [core/project]. | |
Initializing the backend... | |
Initializing provider plugins... | |
The following providers do not have any version constraints in configuration, | |
so the latest version was installed. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@Test | |
public void testNoMatchingRecords() throws InitializationException { | |
TestRunner runner = getRunner(); | |
final JsonTreeReader jsonReader = new JsonTreeReader(); | |
runner.addControllerService("reader", jsonReader); | |
runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaInferenceUtil.INFER_SCHEMA); | |
final JsonRecordSetWriter jsonWriter = new JsonRecordSetWriter(); | |
runner.addControllerService("writer", jsonWriter); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import org.apache.commons.io.IOUtils | |
import java.nio.charset.* | |
import org.json.JSONObject; | |
import org.json.XML; | |
def flowFile = session.get() | |
if (!flowFile) return | |
flowFile = session.write(flowFile, | |
{ inputStream, outputStream -> |