Skip to content

Instantly share code, notes, and snippets.

Rajkumar Singh rajkrrsingh

Block or report user

Report or block rajkrrsingh

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@rajkrrsingh
rajkrrsingh / Hive_Client_using_Go_Lang.md
Last active Aug 21, 2019
sample Hive Client implemented in GoLang to connect to hiveserver2
View Hive_Client_using_Go_Lang.md

for more information on golang hive driver, please refer https://github.com/beltran/gohive

Sample Table
create table test (id int);
insert into table test values (1),(2),(3),(4),(5);
select * from test;
1
2
3
4
@rajkrrsingh
rajkrrsingh / SettableFutureTest
Created Aug 3, 2019
Quick test of google guava Settable Future
View SettableFutureTest
package listenablefuture;
import com.google.common.util.concurrent.FutureCallback;
import com.google.common.util.concurrent.Futures;
import com.google.common.util.concurrent.ListenableFuture;
import com.google.common.util.concurrent.ListeningExecutorService;
import com.google.common.util.concurrent.MoreExecutors;
import com.google.common.util.concurrent.SettableFuture;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
@rajkrrsingh
rajkrrsingh / Docker_HIVESERVER_Remote_Debugging.md
Last active Jul 25, 2019
Debugging HiveServer2 Docker container Remotely
View Docker_HIVESERVER_Remote_Debugging.md
HiveServer2 Dockerfile

watch out the JAVA_TOOL_OPTIONS which are having remote debugging options.

FROM centos

# Basic hygene
RUN yum upgrade -y && \
    yum update -y && \
@rajkrrsingh
rajkrrsingh / Hive_Docker.md
Last active Jul 24, 2019
Running HiveServer2 and Beeline as docker containers
View Hive_Docker.md

Create and run hiveserver2 docker container

mkdir hive-3-image
cd hive-3-image/
wget https://raw.githubusercontent.com/alanfgates/sqltest/master/dbs/hive/v3_1/Dockerfile
docker build . -t hive3-image
// Run it
docker run -it --net=myNetwork -p 10000:10000 hive3-image

You will see following logs on stdout if everything goes well

@rajkrrsingh
rajkrrsingh / Hive Replication V2 startup guide.md
Last active Jul 16, 2019
Jump start guide for Hive Replication V2 - to know more about hive replication please refer https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development
View Hive Replication V2 startup guide.md

Prerequisite hive settings:

set hive.server2.logging.operation.level=execution;
set hive.metastore.transactional.event.listeners=org.apache.hive.hcatalog.listener.DbNotificationListener;
set hive.metastore.dml.events=true;

Setup database and tables

@rajkrrsingh
rajkrrsingh / Create_Bulk_Topic_In_Kafka.md
Last active Jul 20, 2019
create kafka bulk topic from admin client
View Create_Bulk_Topic_In_Kafka.md

import kafka.admin.AdminUtils;
import kafka.admin.RackAwareMode;
import kafka.utils.ZKStringSerializer$;
import kafka.utils.ZkUtils;
import org.I0Itec.zkclient.ZkClient;
import org.I0Itec.zkclient.ZkConnection;

import java.io.BufferedReader;
@rajkrrsingh
rajkrrsingh / Hive LLAP Workload Management.md
Last active Jul 20, 2019
hive llap workload management commands
View Hive LLAP Workload Management.md

Create RESOURCE PLAN testrp;

create RESOURCE PLAN testrp;
CREATE POOL testrp.default.c1 WITH ALLOC_FRACTION=0.3, QUERY_PARALLELISM=3, SCHEDULING_POLICY='fair';
CREATE POOL testrp.default.c2 WITH ALLOC_FRACTION=0.5, QUERY_PARALLELISM=1, SCHEDULING_POLICY='fair';
SELECT * FROM SYS.WM_POOLS;
ALTER RESOURCE PLAN testrp VALIDATE;
ALTER RESOURCE PLAN testrp ENABLE ACTIVATE;
@rajkrrsingh
rajkrrsingh / Hive Kafka Integration
Last active Jul 20, 2019
a quick start guide to query kafka topic from hive table
View Hive Kafka Integration
#### ENV: HDP-3.1
#### Data setup:
```
cat sample-data.json
{"name": "Raj","address": {"a": "b","c": "d","e": "f"}}
{"name": "Raj1","address": {"a": "bb","c": "dd","e": "ff"}}
```
#### Create topic in Kafka and Ingest data into it.
@rajkrrsingh
rajkrrsingh / docker-help.md
Created Feb 2, 2019 — forked from bradtraversy/docker-help.md
Docker Commands, Help & Tips
View docker-help.md

Docker Commands, Help & Tips

Show commands & management commands

$ docker

Docker version info

View Druid_Supervisor_Rest_API.md

Supervisor REST API

Get Supervisor Spec

curl    http://host215-node2:8090/druid/indexer/v1/supervisor/metrics-kafka
{"type":"kafka","dataSchema":{"dataSource":"metrics-kafka","parser":{"type":"string","parseSpec":{"format":"json","timestampSpec":{"column":"timestamp","format":"auto"},"dimensionsSpec":{"dimensions":[],"dimensionExclusions":["timestamp","value"]}}},"metricsSpec":[{"type":"count","name":"count"},{"type":"doubleSum","name":"value_sum","fieldName":"value","expression":null},{"type":"doubleMin","name":"value_min","fieldName":"value","expression":null},{"type":"doubleMax","name":"value_max","fieldName":"value","expression":null}],"granularitySpec":{"type":"uniform","segmentGranularity":"HOUR","queryGranularity":{"type":"none"},"rollup":true,"intervals":null}},"tuningConfig":{"type":"kafka","maxRowsInMemory":75000,"maxRowsPerSegment":5000000,"intermediatePersistPeriod":"PT10M","basePersistDirectory":"/tmp/1530551810169-0","maxPendingPersists":0,"indexSpec":{"bitmap":{"type":"concise"},"
You can’t perform that action at this time.