Skip to content

Instantly share code, notes, and snippets.

Avatar
💤
living the data dream

Raki mdrakiburrahman

💤
living the data dream
View GitHub Profile
View 50-cloud-init.yaml
network:
ethernets:
eth0:
dhcp4: true
dhcp4-overrides:
route-metric: 100
dhcp6: false
addresses: [172.20.0.0/21]
nameservers:
addresses: [172.20.0.4, 168.63.129.16]
View Purview Asset Definition.json
{
"entities": [
{
"attributes": {
"name": "aemigration.database.windows.net",
"qualifiedName": "mssql://aemigration.database.windows.net"
},
"collectionId": "aia-purview-new",
"status": "ACTIVE",
"typeName": "azure_sql_server"
View create-collections-root.py
import requests
import json
from termcolor import colored
def aadtoken(client_id, client_secret, client_tenant):
url = "https://login.microsoftonline.com/{}/oauth2/token".format(client_tenant)
payload='grant_type=client_credentials&client_id={}&client_secret={}&resource=https%3A%2F%2Fpurview.azure.net'.format(client_id, client_secret)
response = requests.request("POST", url, data=payload)
return json.loads(response.text)['access_token']
View create-collections.py
import requests
import json
from termcolor import colored
def aadtoken(client_id, client_secret, client_tenant):
url = "https://login.microsoftonline.com/{}/oauth2/token".format(client_tenant)
payload='grant_type=client_credentials&client_id={}&client_secret={}&resource=https%3A%2F%2Fpurview.azure.net'.format(client_id, client_secret)
response = requests.request("POST", url, data=payload)
return json.loads(response.text)['access_token']
@mdrakiburrahman
mdrakiburrahman / SHIRInstall.ps1
Created Jul 9, 2021
Install Purview Integration Runtime on Windows Machine with Authentication Key
View SHIRInstall.ps1
param(
[string]
$gatewayKey
)
# init log setting
$logPath = "$PWD\tracelog.log"
"Start to excute SHIRInstall.ps1. `n" | Out-File $logPath
function Now-Value()
@mdrakiburrahman
mdrakiburrahman / ssis-ir-creation-vnet-integrated.ps1
Created Apr 28, 2021
Create VNet integrated SSIS IR Integration Runtime on an existing ADF
View ssis-ir-creation-vnet-integrated.ps1
### Azure Data Factory info
$SubscriptionName = "Azure Subscription Name"
$ResourceGroupName = "Resource Group Name containing Data Factory"
$DataFactoryName = "Data Factory Name"
$AzureSSISName = "SSIS IR Name that will appear in Portal"
$AzureSSISDescription = "E.g. Testing SSIS IR Deployment through PowerShell"
$AzureSSISLocation = "CanadaCentral"
$AzureSSISNodeSize = "Standard_D8_v3"
$AzureSSISNodeNumber = 4
$AzureSSISEdition = "Standard"
@mdrakiburrahman
mdrakiburrahman / query-purview-insights.py
Last active Dec 7, 2021
Querying various Azure Purview REST APIs to gather Insights data into Pandas DataFrames
View query-purview-insights.py
import os
import requests
import json
import jmespath
import pandas as pd
from pprint import pprint
def azuread_auth(tenant_id: str, client_id: str, client_secret: str, resource_url: str):
"""
Authenticates Service Principal to the provided Resource URL, and returns the OAuth Access Token
@mdrakiburrahman
mdrakiburrahman / consume-eh-capture-with-autoloader.py
Created Jan 4, 2021
Ingest Event Hub Capture data with Autoloader
View consume-eh-capture-with-autoloader.py
# Import widgets for Event Hub (i.e. topic) name
# Set default
dbutils.widgets.text("topic", "your--topic--name")
topic = dbutils.widgets.get("topic")
# Import required modules
from pyspark.sql import *
from pyspark.sql.functions import *
from pyspark.sql.types import *
@mdrakiburrahman
mdrakiburrahman / get-schema-purview.py
Last active Sep 28, 2021
Extracting metadata from Azure Purview with Synapse Spark Pools
View get-schema-purview.py
# Reusable Functions
def azuread_auth(tenant_id: str, client_id: str, client_secret: str, resource_url: str):
"""
Authenticates Service Principal to the provided Resource URL, and returns the OAuth Access Token
"""
url = f"https://login.microsoftonline.com/{tenant_id}/oauth2/token"
payload= f'grant_type=client_credentials&client_id={client_id}&client_secret={client_secret}&resource={resource_url}'
headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
@mdrakiburrahman
mdrakiburrahman / list_synapse_files_recursively.py
Created Dec 24, 2020
Recursively listing Data Lake files with `display` implemented
View list_synapse_files_recursively.py
def deep_ls(path: str, max_depth=1):
"""
List all files and folders in specified path and
subfolders within maximum recursion depth.
"""
# List all files in path and apply sorting rules
li = mssparkutils.fs.ls(path)
# Return all files