Skip to content

Instantly share code, notes, and snippets.

Avatar

Rajkumar Singh rajkrrsingh

View GitHub Profile
@rajkrrsingh
rajkrrsingh / Hive Replication V2 startup guide.md
Last active Nov 4, 2021
Jump start guide for Hive Replication V2 - to know more about hive replication please refer https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development
View Hive Replication V2 startup guide.md

Prerequisite hive settings:

set hive.server2.logging.operation.level=execution;
set hive.metastore.transactional.event.listeners=org.apache.hive.hcatalog.listener.DbNotificationListener;
set hive.metastore.dml.events=true;

Setup database and tables

@rajkrrsingh
rajkrrsingh / Google protobuf installation on Mac
Created Nov 27, 2016
Steps to Install google protobuf on Mac
View Google protobuf installation on Mac
$wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.bz2
$tar xvf protobuf-2.5.0.tar.bz2
$cd protobuf-2.5.0
$./configure CC=clang CXX=clang++ CXXFLAGS='-std=c++11 -stdlib=libc++ -O3 -g' LDFLAGS='-stdlib=libc++' LIBS="-lc++ -lc++abi"
$make -j 4
$sudo make install
$protoc --version
@rajkrrsingh
rajkrrsingh / Hive Thrift Client in Java
Last active Jul 6, 2021
sample java code to connect hive server using thrift client
View Hive Thrift Client in Java
package com.rajkrrsingh.thrift.client;
import org.apache.hive.service.auth.HiveAuthFactory;
import org.apache.hive.service.auth.PlainSaslHelper;
import org.apache.hive.service.cli.thrift.*;
import org.apache.thrift.TException;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.transport.TTransport;
import javax.security.sasl.SaslException;
@rajkrrsingh
rajkrrsingh / Impyla_CDW_connection.md
Created May 28, 2021
Configuring Impyla for CDW Hive LLAP Cluster
View Impyla_CDW_connection.md
import logging
import os
import pandas
from impala.dbapi import connect
from impala.util import as_pandas
logging.basicConfig(level=logging.DEBUG)

HIVE_HS2_HOST='HIVE_HOST_ENDPOINT'
conn = connect(host=HIVE_HS2_HOST,port=443,
@rajkrrsingh
rajkrrsingh / Interaction with zookeeper using java code
Last active May 5, 2021
Zookeeper : create,list,update zknodes using java
View Interaction with zookeeper using java code
package com.rajkrrsingh.zk;
import java.io.IOException;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
@rajkrrsingh
rajkrrsingh / Kafka-Metrics.md
Last active Mar 9, 2021
Monitoring Kafka Broker JMX using jolokia JVM Agent
View Kafka-Metrics.md
Download Jolokia JVM Agent from following location
https://jolokia.org/download.html
wget http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.3.7/jolokia-jvm-1.3.7-agent.jar

mv jolokia-jvm-1.3.7-agent.jar agent.jar
here is the small shell script to get metrics you are intersted in
@rajkrrsingh
rajkrrsingh / AWS_CHEAT_SHEET.md
Last active Mar 3, 2021
aws cli commands for day to day work
View AWS_CHEAT_SHEET.md

IAM

List users

aws iam list-users

Current user

aws iam get-user

List access key

View oozie_spark_shell_action.md

workflow dir @hdfs

 hadoop fs -ls /tmp/sparkOozieShellAction/
Found 4 items
-rw-r--r--   3 oozie hdfs        178 2017-05-08 07:00 /tmp/sparkOozieShellAction/job.properties
drwxr-xr-x   - oozie hdfs          0 2017-05-08 07:01 /tmp/sparkOozieShellAction/lib
-rw-r--r--   3 oozie hdfs        279 2017-05-08 07:12 /tmp/sparkOozieShellAction/spark-pi-job.sh
-rw-r--r--   3 oozie hdfs        712 2017-05-08 07:34 /tmp/sparkOozieShellAction/workflow.xml
@rajkrrsingh
rajkrrsingh / Hive_Client_using_Go_Lang.md
Last active Sep 27, 2020
sample Hive Client implemented in GoLang to connect to hiveserver2
View Hive_Client_using_Go_Lang.md

for more information on golang hive driver, please refer https://github.com/beltran/gohive

Sample Table
create table test (id int);
insert into table test values (1),(2),(3),(4),(5);
select * from test;
1
2
3
4
@rajkrrsingh
rajkrrsingh / Client.md
Created Aug 2, 2020
Java11 HttpClient to test the jwt authentication
View Client.md
package org.example;


import java.net.URI;
import java.net.URLEncoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;