Skip to content

Instantly share code, notes, and snippets.

Avatar
💤
living the data dream

Raki mdrakiburrahman

💤
living the data dream
View GitHub Profile
@mdrakiburrahman
mdrakiburrahman / fluentbit_otlp_json_kafka_export.json
Created May 10, 2022
A sample JSON export from the Open Telemetry Kafka Exporter as OTLP_JSON
View fluentbit_otlp_json_kafka_export.json
{
"resourceLogs": [
{
"resource": {},
"scopeLogs": [
{
"scope": {},
"logRecords": [
{
"timeUnixNano": "1652102891470000028",
@mdrakiburrahman
mdrakiburrahman / arc-scc.yaml
Last active May 6, 2022
SCCs for Bootstrapper 1.3.0_2022-01-27
View arc-scc.yaml
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
name: arc-data-scc
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
@mdrakiburrahman
mdrakiburrahman / Synapse-Logical-DWH.sql
Last active May 5, 2022
Creating a simple External on top of Parquet file that exposes a subset of the columns to end user
View Synapse-Logical-DWH.sql
CREATE DATABASE Ldw
COLLATE Latin1_General_100_BIN2_UTF8;
USE Ldw;
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'MyRand0mPa33W0rd1!';
CREATE DATABASE SCOPED CREDENTIAL WorkspaceIdentity
WITH IDENTITY = 'Managed Identity';
GO
@mdrakiburrahman
mdrakiburrahman / Arc-onboarding-ansible-mimic.sh
Last active Apr 21, 2022
Arc Onboarding Script - working version 1.2.3
View Arc-onboarding-ansible-mimic.sh
# Login Into Azure Using Service Principal
az_cli_cmd login --service-principal -u az_client_id -p az_client_secret --tenant az_tenant_id
# Set Azure Subscription
az_cli_cmd account set --subscription az_subscription_id
# Create AZ Resource Group: Arc K8s
az_cli_cmd group create -l az_location -n az_resource_group
# OCP Login
@mdrakiburrahman
mdrakiburrahman / get-schema-purview.py
Last active Apr 13, 2022
Extracting metadata from Azure Purview with Synapse Spark Pools
View get-schema-purview.py
# Reusable Functions
def azuread_auth(tenant_id: str, client_id: str, client_secret: str, resource_url: str):
"""
Authenticates Service Principal to the provided Resource URL, and returns the OAuth Access Token
"""
url = f"https://login.microsoftonline.com/{tenant_id}/oauth2/token"
payload= f'grant_type=client_credentials&client_id={client_id}&client_secret={client_secret}&resource={resource_url}'
headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
@mdrakiburrahman
mdrakiburrahman / arc-april-PITR.sh
Created Apr 12, 2022
Testing Arc PITR with incorrect dates
View arc-april-PITR.sh
# Correct date
cat <<EOF | kubectl create -f -
apiVersion: tasks.sql.arcdata.microsoft.com/v1
kind: SqlManagedInstanceRestoreTask
metadata:
name: sql-restore-raki-correct
namespace: arc
spec:
source:
name: sql-gp-1
@mdrakiburrahman
mdrakiburrahman / tina-onboarder-rbac.yaml
Last active Apr 7, 2022
ClusterRoles needed to deploy Arc Data Services in Indirect Mode
View tina-onboarder-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: arc-data-deployer-cluster-role
rules:
# CRDs in general
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ['create', 'update', 'patch', 'delete', 'list', 'get', 'watch'] # Can figure out lesser scope if needed
# All Arc Data apiGroups, I think some of these may be redundant - K8s doesn't give an easy way to get custom apiGroups so I put everything in
@mdrakiburrahman
mdrakiburrahman / create-kubeconfig.sh
Created Apr 7, 2022
Create Kubeconfig from Service Account
View create-kubeconfig.sh
# the Namespace and ServiceAccount name that is used for the config
namespace=arc
serviceAccount=arcOnboard
######################
# actual script starts
set -o errexit
secretName=$(kubectl --namespace $namespace get serviceAccount $serviceAccount -o jsonpath='{.secrets[0].name}')
ca=$(kubectl --namespace $namespace get secret/$secretName -o jsonpath='{.data.ca\.crt}')
View SQL_MI_Restore_issue.md
-- Create DB
CREATE DATABASE raki_pitr_test

CREATE TABLE table1 (ID int, value nvarchar(10))
GO

INSERT INTO table1 VALUES (1, 'demo1')
INSERT INTO table1 VALUES (2, 'demo2')
@mdrakiburrahman
mdrakiburrahman / list_synapse_files_recursively.py
Created Dec 24, 2020
Recursively listing Data Lake files with `display` implemented
View list_synapse_files_recursively.py
def deep_ls(path: str, max_depth=1):
"""
List all files and folders in specified path and
subfolders within maximum recursion depth.
"""
# List all files in path and apply sorting rules
li = mssparkutils.fs.ls(path)
# Return all files