Skip to content

Instantly share code, notes, and snippets.

View yike5460's full-sized avatar

Aaron Yi yike5460

  • Amazon Web Service
View GitHub Profile
@yike5460
yike5460 / gist:8c95b145d144dd233e58efc6286d63a0
Created April 25, 2024 09:51
writing tool to check if file contain Chinese character
#!/bin/bash
# Check if a file path was provided
if [ "$#" -ne 1 ]; then
echo "Usage: \$0 <path-to-python-file>"
exit 1
fi
# The file path is the first argument to the script
FILE_PATH="$1"
@yike5460
yike5460 / gist:813969b0ee8e99053b2a7fd16b94dca6
Created April 16, 2024 10:17
process lifecycle management
"""
Process lifecycle management controller, the main purpose of such controller will accept requests from the client, transform incoming batch requests to multiple parallel process inside, and execute those processes to initialize associated job and distributed to multiple backend server considering the workload even & request reliable, the controller will keep monitoring the job result, respond to client, handle the error exception and recycle the process resource.
Sample below only demonstrates the basic concept of how to manage the process lifecycle, it's not a complete solution, the real-world scenario will be more complex and need to consider more factors like the process resource limitation, the process priority, the process dependency, the process retry etc. it mainly focus on the process state transition and managament in parallel processing.
Below are Mermaid state diagram illustrating the transitions between these Linux process states:
stateDiagram-v2
[*] --> Running
Running --> Interrup
@yike5460
yike5460 / gist:ee178f12496c720079a3c767ce39a656
Created May 30, 2023 01:00
stable diffusion bootstrap shell in ec2
# disable the restart dialogue and install several packages
sudo sed -i "/#\$nrconf{restart} = 'i';/s/.*/\$nrconf{restart} = 'a';/" /etc/needrestart/needrestart.conf
sudo apt-get update
sudo apt install wget git python3 python3-venv build-essential net-tools awscli -y
# install CUDA (from https://developer.nvidia.com/cuda-downloads)
wget https://developer.download.nvidia.com/compute/cuda/12.0.0/local_installers/cuda_12.0.0_525.60.13_linux.run
sudo sh cuda_12.0.0_525.60.13_linux.run --silent
# install git-lfs
@yike5460
yike5460 / gist:c9a1b6b75320efc6e707a526fc0b34f7
Created April 6, 2023 14:39
update python version in AWS EC2 (3.7->3.9)
# install python 3.9 in AWS EC2
cd /opt
wget https://www.python.org/ftp/python/3.9.16/Python-3.9.16.tgz
tar -xvf Python-3.9.16.tgz
cd Python-3.9.16
./configure --enable-optimizations
make altinstall
python3.9 -V
python3.9 -m pip install --upgrade pip
python3.9 -m pip install --upgrade setuptools
@yike5460
yike5460 / gist:7d7598287a73849923f10016a542a370
Created November 14, 2022 07:16
daemon sample with python
# -*- coding: utf-8 -*-
"""
Python daemon implementation from:
http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
"""
import atexit
import os
import sys
import time
@yike5460
yike5460 / shell-template.sh
Created October 31, 2022 14:54
best practice template of shell
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
if [[ "${TRACE-0}" == "1" ]]; then
set -o xtrace
fi
if [[ "${1-}" =~ ^-*h(elp)?$ ]]; then
import * as cdk from "aws-cdk-lib";
import {CfnUserPoolUserToGroupAttachment, IUserPool} from "@aws-cdk/aws-cognito";
import {AwsCustomResource, AwsCustomResourcePolicy, PhysicalResourceId} from "@aws-cdk/custom-resources";
export class UserPoolUser extends cdk.Stack {
constructor(scope: cdk.App, id: string, props: {
userPool: IUserPool,
username: string,
password: string,
@yike5460
yike5460 / auto generate ssl certs
Last active August 15, 2022 10:16
auto generate ssl certs for dns managed in godaddy/aliyun/route53
# script to auto generate ssl certs and upload to s3 bucket
import os
import boto3
import logging
import subprocess
import sys
import time
from multiprocessing import Pool
# set logging to stdout
@yike5460
yike5460 / gist:a2818c0bd80e43067ce5f2c8ac90ace2
Created August 4, 2022 02:32
txt records operation in GoDaddy for certificate validation
from cmath import log
from godaddypy import Client, Account
import configparser
import os
import logging
import sys
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
handlers=[
@yike5460
yike5460 / gist:bbecd850744daf20f05578a3f59808c5
Last active June 17, 2022 03:17
Preflight scan with given domain
# scan given domain with dig command
#
# Usage:
# ./dig_scan.py <domain>
#
# Example:
# ./dig_scan.py example.com
#
# Output:
#