Skip to content

Instantly share code, notes, and snippets.

View mdrakiburrahman's full-sized avatar
💤
living the data dream

Raki mdrakiburrahman

💤
living the data dream
View GitHub Profile

Pictures and stuff.

@mdrakiburrahman
mdrakiburrahman / get-schema-purview.py
Last active March 26, 2024 17:51
Extracting metadata from Azure Purview with Synapse Spark Pools
# Reusable Functions
def azuread_auth(tenant_id: str, client_id: str, client_secret: str, resource_url: str):
"""
Authenticates Service Principal to the provided Resource URL, and returns the OAuth Access Token
"""
url = f"https://login.microsoftonline.com/{tenant_id}/oauth2/token"
payload= f'grant_type=client_credentials&client_id={client_id}&client_secret={client_secret}&resource={resource_url}'
headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
@mdrakiburrahman
mdrakiburrahman / SHIRInstall.ps1
Created July 9, 2021 16:44
Install Purview Integration Runtime on Windows Machine with Authentication Key
param(
[string]
$gatewayKey
)
# init log setting
$logPath = "$PWD\tracelog.log"
"Start to excute SHIRInstall.ps1. `n" | Out-File $logPath
function Now-Value()
@mdrakiburrahman
mdrakiburrahman / list_synapse_files_recursively.py
Created December 24, 2020 21:16
Recursively listing Data Lake files with `display` implemented
def deep_ls(path: str, max_depth=1):
"""
List all files and folders in specified path and
subfolders within maximum recursion depth.
"""
# List all files in path and apply sorting rules
li = mssparkutils.fs.ls(path)
# Return all files
graph TD
subgraph SqlServerInstancePrimary1
createEndpoint1
grantConnectOnEndpoint1
createAvailabilityGroup1
end
subgraph SqlServerInstanceSecondary2
createEndpoint2
@mdrakiburrahman
mdrakiburrahman / azure-cdn.json
Last active June 18, 2023 23:24
Website ARM templates
{
"id": "/subscriptions/d4aa0040-6651-443d-a078-728ce72a87ab/resourcegroups/rakirahman.me/providers/Microsoft.Cdn/profiles/rakirahman/endpoints/rakirahman",
"type": "Microsoft.Cdn/profiles/endpoints",
"name": "rakirahman",
"location": "Global",
"tags": {},
"properties": {
"hostName": "rakirahman.azureedge.net",
"originHostHeader": "rakirahman.z9.web.core.windows.net",
"originPath": null,
#!/bin/bash
# Continue on error
set +e
# Params
resource_group="arcdata-ci-direct"
days_to_keep=7
# Get current date and calculate 7 days ago in seconds
@mdrakiburrahman
mdrakiburrahman / calculate-kubernetes-eviction-signals.sh
Last active January 7, 2023 03:34
Calculates eviction signals used by kubelet to trigger eviction manager
# Calculate Eviction Signals - Current State vs Threshold:
#
# https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#eviction-signals
# https://medium.com/kubernetes-tutorials/efficient-node-out-of-resource-management-in-kubernetes-67f158da6e59
#
NODES=($(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'))
NODE_COUNT=${#NODES[@]}
while true;
do
@mdrakiburrahman
mdrakiburrahman / Chaos-mesh-demo.sh
Last active December 5, 2022 15:27
Script to run to apply and clean up chaos Mesh workflow
##########################################################
# APPLY
##########################################################
# Apply workflow
#
kubectl apply -f sqlinstance-ha.yaml
# Get primary
#
kubectl get pods -l=role.ag.mssql.microsoft.com/chaos-bc-chaos-bc=primary -n ns1669830131581917320
@mdrakiburrahman
mdrakiburrahman / sqlinstance-ha.yaml.tmpl
Created December 4, 2022 21:37
Chaos Mesh Workflow Jinja template for YAML generation
# ==========================================================================
# SQLINSTANCE-HA
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# WORKFLOW DURATION: 18m
# --------------------------------------------------------------------------
# CHAOS DESCRIPTION
# --------------------------------------------------------------------------
# In this chaos simulation we target a 2 or 3 replica MIAA:
#
# 1. Setup a User DB