This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Python code to mount and access Azure Data Lake Storage Gen2 Account to Azure Databricks with Service Principal and OAuth | |
# Author: Dhyanendra Singh Rathore | |
# Define the variables used for creating connection strings | |
adlsAccountName = "dlscsvdataproject" | |
adlsContainerName = "csv-data-store" | |
adlsFolderName = "covid19-data" | |
mountPoint = "/mnt/csvFiles" | |
# Application (Client) ID |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Python code to connect to Azure SQL Databases from Azure Databricks with Screts Scope | |
# Author: Dhyanendra Singh Rathore | |
# Declare variables for creating JDBC URL | |
jdbcHostname = "sql-csv-data-server.database.windows.net" # Replace with your SQL Server name | |
jdbcPort = 1433 # Replace with your SQL Server port number | |
jdbcDatabase = "syn-csv-data-dw" # Replace with your database name | |
# Connection secrets from vault | |
jdbcUsername = dbutils.secrets.get(scope="CSVProjectKeyVault",key="SQLAdmin") # Replace the scope and key accordingly |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Python/PySpark code for cleansing and transforming schema drifted csv files into relational data with incremental loads in Azure Databricks | |
# Author: Dhyanendra Singh Rathore | |
# Define the variables used for creating connection strings | |
adlsAccountName = "dlscsvdataproject" | |
adlsContainerName = "csv-data-store" | |
adlsFolderName = "covid19-data" | |
mountPoint = "/mnt/csvFiles" | |
# Application (Client) ID |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"name": "pl_autoCopyCsvFiles", | |
"properties": { | |
"activities": [ | |
{ | |
"name": "ac_checkAllAvailableFiles", | |
"type": "GetMetadata", | |
"dependsOn": [], | |
"policy": { | |
"timeout": "7.00:00:00", |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// Azure Function: Node.js code to read PostgreSQL data and return results as JSON | |
// Author: Dhyanendra Singh Rathore | |
// Import the pg (node-postgres) library | |
const pg = require('pg'); | |
// Entry point of the function | |
module.exports = async function(context, req) { | |
// Define variables to store connection details and credentials |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// Azure Function: Node.js code to read PostgreSQL data with query parameter and return results as JSON | |
// Author: Dhyanendra Singh Rathore | |
// Import the pg (node-postgres) library | |
const pg = require('pg'); | |
// Entry point of the function | |
module.exports = async function(context, req) { | |
// Define variables to store connection details and credentials |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// Azure Function: Node.js code to read Environment Variables during function execution and return variables as JSON | |
// Author: Dhyanendra Singh Rathore | |
// Entry point of the function | |
module.exports = async function(context, req) { | |
// Fetch environment variables during execution | |
const postgresServerName = process.env["POSTGRES_SERVER_NAME"]; | |
const postgresUserName = process.env["POSTGRES_USER"]; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// Azure Function: Node.js code to read data from Azure Synapse Analytics with query parameter and return results as JSON | |
// Author: Dhyanendra Singh Rathore | |
// Import the tedious library | |
const Connection = require('tedious').Connection; | |
const Request = require('tedious').Request; | |
const TYPES = require('tedious').TYPES; | |
// Entry point of the function | |
module.exports = function(context, req) { |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using Microsoft.AspNetCore.Http; | |
using Microsoft.AspNetCore.Mvc; | |
using Microsoft.Azure.WebJobs; | |
using Microsoft.Azure.WebJobs.Extensions.Http; | |
using Microsoft.Extensions.Logging; | |
using System.Security.Claims; | |
using System.IdentityModel.Tokens.Jwt; | |
// DEMO: Get user Identity and Claims from the Token Headers | |
namespace ClaimsDemo.Function |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using Microsoft.AspNetCore.Http; | |
using Microsoft.AspNetCore.Mvc; | |
using Microsoft.Azure.WebJobs; | |
using Microsoft.Azure.WebJobs.Extensions.Http; | |
using Microsoft.Extensions.Logging; | |
using System.Security.Claims; | |
// DEMO: Get ClaimsPrincipal as a binding parameter | |
namespace ClaimsDemo.Function | |
{ |
OlderNewer