Skip to content

Instantly share code, notes, and snippets.

Avatar

Rajkumar Singh rajkrrsingh

View GitHub Profile
@rajkrrsingh
rajkrrsingh / Impyla_CDW_connection.md
Created May 28, 2021
Configuring Impyla for CDW Hive LLAP Cluster
View Impyla_CDW_connection.md
import logging
import os
import pandas
from impala.dbapi import connect
from impala.util import as_pandas
logging.basicConfig(level=logging.DEBUG)

HIVE_HS2_HOST='HIVE_HOST_ENDPOINT'
conn = connect(host=HIVE_HS2_HOST,port=443,
@rajkrrsingh
rajkrrsingh / AWS_CHEAT_SHEET.md
Last active Mar 3, 2021
aws cli commands for day to day work
View AWS_CHEAT_SHEET.md

IAM

List users

aws iam list-users

Current user

aws iam get-user

List access key

@rajkrrsingh
rajkrrsingh / Client.md
Created Aug 2, 2020
Java11 HttpClient to test the jwt authentication
View Client.md
package org.example;


import java.net.URI;
import java.net.URLEncoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
@rajkrrsingh
rajkrrsingh / HiveServer2-Remote-Debug-on-Kubernetes.md
Created Jun 15, 2020
Remote Debug Java Application running on Kubernetes Cluster
View HiveServer2-Remote-Debug-on-Kubernetes.md

Following steps can be used to debug java application running on Remote Cluster, for sake of similicity I am using bare minimum pod config to spin the HS2 pod on the cluster.

  1. Build a docker image as per https://gist.github.com/rajkrrsingh/e2a5d02a3a895c7f7e8f1d6d0e71e0e9, The image push the debug config -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n to the JVM opts, the same can be achieved through the ConfigMap descriptor.

  2. Create HS2 Deployments

apiVersion: apps/v1
@rajkrrsingh
rajkrrsingh / Helm-Cheat-Sheet.md
Last active May 25, 2020
Helm cheat sheet to the help on helm commands
View Helm-Cheat-Sheet.md

Installation and setup

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
helm version
helm repo add stable https://kubernetes-charts.storage.googleapis.com/

helm search and install app

@rajkrrsingh
rajkrrsingh / Hive_Merge_Quick_Start.md
Last active May 13, 2020
quick start guide to test Hive Merge features
View Hive_Merge_Quick_Start.md

Create Non-ACID table

create external table nonacid_n1 (key int, a1 string, value string) stored as orc;
insert into nonacid_n1 values(1, 'a11', 'val1');
insert into nonacid_n1 values(2, 'a12', 'val2');

+-----------------+----------------+-------------------+
| nonacid_n1.key  | nonacid_n1.a1  | nonacid_n1.value  |
+-----------------+----------------+-------------------+
| 1               | a11            | val1              |
@rajkrrsingh
rajkrrsingh / Kafka_Cheat_Sheet.md
Last active Jun 23, 2020
reference guide to run kafka command on CDP-DC cluster
View Kafka_Cheat_Sheet.md

CDP-DC kafka useful kafka commands

-- list topics
kafka-topics --list --bootstrap-server `hostname -f`:9092

-- create topic 
kafka-topics --create --bootstrap-server `hostname -f`:9092 --replication-factor 1 --partitions 1 --topic kafkatopic

-- produce messages
kafka-console-producer --broker-list `hostname -f`:9092 --topic kafkatopic
@rajkrrsingh
rajkrrsingh / golang-web-app.md
Created Dec 29, 2019
simple golang webapp on docker to quick test
View golang-web-app.md

Golang App

mkdir golang-web-app
cd golang-web-app
vim main.go
package main

import (
	"fmt"
	"log"
@rajkrrsingh
rajkrrsingh / Hive_Client_using_Go_Lang.md
Last active Sep 27, 2020
sample Hive Client implemented in GoLang to connect to hiveserver2
View Hive_Client_using_Go_Lang.md

for more information on golang hive driver, please refer https://github.com/beltran/gohive

Sample Table
create table test (id int);
insert into table test values (1),(2),(3),(4),(5);
select * from test;
1
2
3
4
@rajkrrsingh
rajkrrsingh / SettableFutureTest
Created Aug 3, 2019
Quick test of google guava Settable Future
View SettableFutureTest
package listenablefuture;
import com.google.common.util.concurrent.FutureCallback;
import com.google.common.util.concurrent.Futures;
import com.google.common.util.concurrent.ListenableFuture;
import com.google.common.util.concurrent.ListeningExecutorService;
import com.google.common.util.concurrent.MoreExecutors;
import com.google.common.util.concurrent.SettableFuture;
import com.google.common.util.concurrent.ThreadFactoryBuilder;