Igor De Souza igfasouza
igfasouza
/ oci-serviceconnector-streaming-json-to-csv-python
Created
April 12, 2021 09:18
Oracle Function example top convert JSON to CSV using Service Connector Hub
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import io | |
| import oci | |
| import json | |
| import requests | |
| import logging | |
| import base64 | |
| import pandas as pd | |
| from fdk import response | |
igfasouza
/ oci-serviceconnector-streaming-json-to-parquet-python
Created
April 12, 2021 09:16
Oracle function example to convert JSON to Parquet using Service Connector Hub
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import io | |
| import os | |
| import oci | |
| import json | |
| import requests | |
| import logging | |
| import base64 | |
| import gzip | |
| import time | |
| import pandas as pd |
igfasouza
/ kafka-console-consumer.sh
Created
August 19, 2020 12:07
example to run kafka-console-consumer.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ./bin/kafka-console-consumer.sh --consumer.config consumer.properties --bootstrap-server streaming.{region}.oci.oraclecloud.com:9092 --topic demo-console --from-beginning |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| KafkaClient { | |
| org.apache.kafka.common.security.plain.PlainLoginModule required | |
| username="{tenancyName}/{username}/{stream pool OCID}\" password="{authToken}"; | |
| }; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import java.util.Properties; | |
| import java.util.Arrays; | |
| import org.apache.kafka.clients.consumer.KafkaConsumer; | |
| import org.apache.kafka.clients.consumer.ConsumerRecords; | |
| import org.apache.kafka.clients.consumer.ConsumerRecord; | |
| public class Consumer { | |
| public static void main(String[] args) throws Exception { | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import java.util.Properties; | |
| import org.apache.kafka.clients.producer.Producer; | |
| import org.apache.kafka.clients.producer.KafkaProducer; | |
| import org.apache.kafka.clients.producer.ProducerRecord; | |
| public class Producer { | |
| public static void main(String[] args) throws Exception{ | |
| String topicName = “Your Stream name”; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Properties properties = new Properties(); | |
| properties.put("bootstrap.servers", "streaming.{region}.oci.oraclecloud.com:9092"); | |
| properties.put("security.protocol", "SASL_SSL"); | |
| properties.put("sasl.mechanism", "PLAIN"); | |
| properties.put("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"{tenancyName}/{username}/{stream pool OCID}\" password=\"{authToken}\";"); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| oci streaming admin stream list --compartment-id $Compartment_OCID --lifecycle-state active | |
| { | |
| "data": [ | |
| { | |
| "compartment-id": "ocid1.compartment.oc1..aaaaaaaa2siikqmkgvtya3yoawa6jq7ibekyootsqqeczpao3kkqrapi5dgq", | |
| "defined-tags": {}, | |
| "freeform-tags": {}, | |
| "id": "ocid1.stream.oc1.phx.aaaaaaaah7yn4zysb5iftdf2laj5utena2wzqpoo3e65yfgpbxeg7f6iajaq", | |
| "lifecycle-state": "ACTIVE", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| export PATH=$OCI_FOLDER:$PATH |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [DEFAULT] | |
| user=ocid1.user.oc1..<unique_ID> | |
| fingerprint=<your_fingerprint> | |
| key_file=~/.oci/oci_api_key.pem | |
| tenancy=ocid1.tenancy.oc1..<unique_ID> | |
| region=us-ashburn-1 |
NewerOlder