Skip to content

Instantly share code, notes, and snippets.

View amalgjose's full-sized avatar
🎯
Focusing

Amal G Jose amalgjose

🎯
Focusing
7 followers · 1 following
View GitHub Profile
@amalgjose
amalgjose / list_ec2_instances.py
Last active February 22, 2024 13:06
Python program to list all ec2 instances in a region using boto3
import boto3
access_key = "XXXXXXXXXXXXXXXXXX"
secret_key = "XXXXXXXXXXXXXXXXXX"
region = 'us-east-1'
client = boto3.client('ec2', aws_access_key_id=access_key, aws_secret_access_key=secret_key,
region_name=region)
conn = boto3.resource('ec2', aws_access_key_id=access_key, aws_secret_access_key=secret_key, region_name=region)
@amalgjose
amalgjose / chrony.conf
Created May 26, 2021 18:45
Chrony configuration file to sync the system timestamp with host server's timestamp. For more details visit https://amalgjose.com
apt-get update
apt-get install chrony -y
cp -pr /etc/chrony/chrony.conf /etc/chrony/chrony.conf.bk
vim /etc/chrony/chrony.conf
=========================chrony.conf==============================
# Welcome to the chrony configuration file. See chrony.conf(5) for more
# information about usuable directives.
@amalgjose
amalgjose / print_pprint_examples.py
Created May 9, 2021 09:13
from pprint import pprint
sample_json = {
"id": "0001",
"type": "donut",
"name": "Cake",
"ppu": 0.55,
"batters":
{
"batter":
@amalgjose
amalgjose / python_adls_connect_service_principle.py
Created April 22, 2021 17:07
A Python program to connect to Azure ADLS Gen2 (Storage Account) using azure service principle instead of the connection string.
from azure.identity import ClientSecretCredential
from azure.storage.blob import BlobServiceClient
# Tenant ID for your Azure Subscription
TENANT_ID = "85XXX93e-XXXX-XXXX-XXXXX-96150XXX893e"
# Your Service Principal App ID (Client ID)
CLIENT_ID = "a3XXX40d-xxxxxxx-0ff72XXXX66a"
# Your Service Principal Password (Client Secret)
@amalgjose
amalgjose / gunicorn_hooks_config.py
Created March 23, 2021 10:17
A sample gunicorn hooks configuration
def on_starting(server):
"""
Do something on server start
"""
print("Server has started")
def on_reload(server):
"""
@amalgjose
amalgjose / fast_api_sample.py
Created February 28, 2021 05:33
A sample GET API implementation using Python FastAPI. For more details, refer to https://amalgjose.com
from fastapi import FastAPI
app = FastAPI()
@app.get("/")
def my_first_get_api():
"""
Sample GET method implementation using FastAPI
:return:
@amalgjose
amalgjose / slack_send_notification.py
Created December 20, 2020 04:05
Sample program to send slack notification using webhook
import json
import sys
import requests
URL = ""
message = ""
def send_notification(url, message):
title = (f"DMP Notification :zap:")
@amalgjose
amalgjose / split_large_csv_file.py
Created October 10, 2020 12:59
Python program to split a large csv or delimited file into smaller file. For more details, refer to https://amalgjose.com
import os
import json
import pandas as pd
def data_extractor(file_path, delimiter, required_fields=[]):
"""
:param file_path:
:param delimiter:
:param required_fields:
@amalgjose
amalgjose / python_git_to_adls_sync.py
Last active February 5, 2023 02:42
Python program to clone or copy a git repository to Azure Data Lake Storage ( ADLS Gen 2). This program is helpful for people who uses spark and hive script in Azure Data Factory. Azure Data Factory needs the hive and spark scripts on ADLS. The developers can commit the code in the git. The git repository can be synced to ADLS using this program…
import os
# Modify this path as per the execution environment
os.environ['GIT_PYTHON_GIT_EXECUTABLE'] = r"C:\Program Files\Git\bin\git.exe"
import uuid
import git
import shutil
from git import RemoteProgress
from azure.storage.blob import BlobServiceClient
@amalgjose
amalgjose / download_adls_directory.py
Last active December 4, 2023 16:15
Python program to download a complete directory or file from Microsoft Azure ADLS. This program is capable of recursively download a complete directory from Azure Data Lake Storage. This uses Azure Blob Storage API to iterate over the directories, files and download the data. This is tested as of 05-October-2020. For more details, refer to https…
# coding: utf-8
import os
from azure.storage.blob import BlobServiceClient
class DownloadADLS:
def __init__(self, connection_string, container_name):
service_client = BlobServiceClient.from_connection_string(connection_string)
self.client = service_client.get_container_client(container_name)