Skip to content

Instantly share code, notes, and snippets.

View dgadiraju's full-sized avatar

Durga Gadiraju dgadiraju

2.3k followers · 2 following
View GitHub Profile
@dgadiraju
dgadiraju / 01_nyse_create_table_quickstart.sql
Last active February 19, 2023 12:46
CREATE DATABASE nyse;
CREATE USER 'nyse_user' IDENTIFIED BY 'itversity';
GRANT ALL ON nyse.* TO nyse_user;
GRANT FILE ON *.* TO nyse_user;
GRANT SUPER ON *.* TO nyse_user;
FLUSH PRIVILEGES;
@dgadiraju
dgadiraju / kafka-getting-started-on-itversity-labs.sh
Last active November 11, 2022 14:08
# Create topics
kafka-topics.sh --create \
--zookeeper m01.itversity.com:2181,m02.itversity.com:2181,w01.itversity.com:2181 \
--replication-factor 1 \
--partitions 1 \
--topic kafkadg
# List all topics
kafka-topics.sh --list \
--zookeeper m01.itversity.com:2181,m02.itversity.com:2181,w01.itversity.com:2181
@dgadiraju
dgadiraju / cloudera-dbsetup-01-install-mysql.sh
Last active September 9, 2022 21:08
sudo yum -y install mariadb-server
sudo systemctl enable mariadb.service
sudo systemctl start mariadb
sudo /usr/bin/mysql_secure_installation
@dgadiraju
dgadiraju / sqoop-import-auto-reset-to-one-mapper.sh
Created November 13, 2017 03:57
sqoop import \
--connect jdbc:mysql://ms.itversity.com:3306/retail_db \
--username retail_user \
--password itversity \
--table order_items_nopk \
--warehouse-dir /user/dgadiraju/sqoop_import/retail_db \
--autoreset-to-one-mapper
@dgadiraju
dgadiraju / sfleads_transform.py
Last active August 7, 2022 21:25
import pandas as pd
import json
df = pd.read_csv(
's3://airetail/bronze/sfleads/addresses/Addresses.csv'
)
df.address = df['address'].apply(json.loads)
df.phone_numbers = df['phone_numbers']. \
apply(lambda pn: json.loads(pn) if not pd.isnull(pn) else None)
@dgadiraju
dgadiraju / cloudera-repo-01-cm.sh
Last active April 7, 2022 11:29
cd /etc/yum.repos.d/
sudo wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo
cd /var/www/html
sudo mkdir -p cm5/redhat/7/x86_64/cm/5
sudo reposync -r cloudera-manager
sudo mv /var/www/html/cloudera-manager/RPMS /var/www/html/cm5/redhat/7/x86_64/cm/5
cd /var/www/html/cm5/redhat/7/x86_64/cm/5
sudo createrepo .
@dgadiraju
dgadiraju / itversity_bloggers.md
Last active March 12, 2022 00:49
@dgadiraju
dgadiraju / kafka-structured-streaming-01-GetStreamingDepartmentTrafficHBase.scala
Created February 4, 2019 00:50
import java.sql.Timestamp
import java.text.SimpleDateFormat
import java.util.Date
import com.typesafe.config.ConfigFactory
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.client.{Connection, ConnectionFactory, Get, Put}
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.{HBaseConfiguration, TableName}
import org.apache.spark.sql.functions._
@dgadiraju
dgadiraju / streaming-pipelines-01-build.sbt
Created October 11, 2018 15:15
name := "bcstructuredstreamingdemo"
version := "1.0"
scalaVersion := "2.11.12"
libraryDependencies += "com.typesafe" % "config" % "1.3.2"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.0"
libraryDependencies += "org.apache.hbase" % "hbase-client" % "1.1.8"
@dgadiraju
dgadiraju / 06_csv_to_pandas_data_frame.ipynb
Created December 17, 2020 12:11
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.