import logging
import os
import pandas
from impala.dbapi import connect
from impala.util import as_pandas
logging.basicConfig(level=logging.DEBUG)
HIVE_HS2_HOST='HIVE_HOST_ENDPOINT'
conn = connect(host=HIVE_HS2_HOST,port=443,
View Impyla_CDW_connection.md
View AWS_CHEAT_SHEET.md
IAM
List users
aws iam list-users
Current user
aws iam get-user
List access key
View Client.md
package org.example;
import java.net.URI;
import java.net.URLEncoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
View HiveServer2-Remote-Debug-on-Kubernetes.md
Following steps can be used to debug java application running on Remote Cluster, for sake of similicity I am using bare minimum pod config to spin the HS2 pod on the cluster.
-
Build a docker image as per https://gist.github.com/rajkrrsingh/e2a5d02a3a895c7f7e8f1d6d0e71e0e9, The image push the debug config -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n to the JVM opts, the same can be achieved through the ConfigMap descriptor.
-
Create HS2 Deployments
apiVersion: apps/v1
View Helm-Cheat-Sheet.md
Installation and setup
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
helm version
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm search and install app
View Hive_Merge_Quick_Start.md
Create Non-ACID table
create external table nonacid_n1 (key int, a1 string, value string) stored as orc;
insert into nonacid_n1 values(1, 'a11', 'val1');
insert into nonacid_n1 values(2, 'a12', 'val2');
+-----------------+----------------+-------------------+
| nonacid_n1.key | nonacid_n1.a1 | nonacid_n1.value |
+-----------------+----------------+-------------------+
| 1 | a11 | val1 |
View Kafka_Cheat_Sheet.md
CDP-DC kafka useful kafka commands
-- list topics
kafka-topics --list --bootstrap-server `hostname -f`:9092
-- create topic
kafka-topics --create --bootstrap-server `hostname -f`:9092 --replication-factor 1 --partitions 1 --topic kafkatopic
-- produce messages
kafka-console-producer --broker-list `hostname -f`:9092 --topic kafkatopic
View golang-web-app.md
Golang App
mkdir golang-web-app
cd golang-web-app
vim main.go
package main
import (
"fmt"
"log"
View Hive_Client_using_Go_Lang.md
for more information on golang hive driver, please refer https://github.com/beltran/gohive
Sample Table
create table test (id int);
insert into table test values (1),(2),(3),(4),(5);
select * from test;
1
2
3
4
View SettableFutureTest
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package listenablefuture; | |
import com.google.common.util.concurrent.FutureCallback; | |
import com.google.common.util.concurrent.Futures; | |
import com.google.common.util.concurrent.ListenableFuture; | |
import com.google.common.util.concurrent.ListeningExecutorService; | |
import com.google.common.util.concurrent.MoreExecutors; | |
import com.google.common.util.concurrent.SettableFuture; | |
import com.google.common.util.concurrent.ThreadFactoryBuilder; |
NewerOlder