Skip to content

Instantly share code, notes, and snippets.

@gitgithan
gitgithan / big_data_platform_docker_swarm_diagrams.txt
Created May 7, 2025 18:11
Big Data Platform Docker Swarm Diagrams
########## Overall Architecture ##########
flowchart TB
%% Define service clusters
subgraph Storage ["Storage Layer"]
direction LR
postgres["PostgreSQL<br>(Hive Metastore DB)"]
minio["MinIO<br>S3-compatible<br>Object Storage"]
@gitgithan
gitgithan / mslq_highlight.py
Last active April 27, 2025 14:35
Read MSLQ.pdf and add highlights and summary page to create MSLQ_highlighted.pdf
import pymupdf # Using pymupdf directly as requested
import os
import pandas as pd
from collections import defaultdict
def process_mslq_pdf(input_pdf_path, output_pdf_path):
"""
Process the MSLQ manual PDF to precisely highlight:
1. Question text
@gitgithan
gitgithan / mslq_excel_template.py
Created April 27, 2025 11:12
MSLQ_excel_template to create excel after OCR on MSLQ.pdf
import pandas as pd
import numpy as np
from openpyxl import Workbook
from openpyxl.styles import Font, PatternFill, Alignment, Border, Side
from openpyxl.utils import get_column_letter
import os
# Define the MSLQ questions and their corresponding scales
mslq_questions = [
# Format: [question_number, question_text, scale, is_reversed]
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
show TIMEZONE;
set timezone = 'UTC'; --utc0
set timezone = 'US/Pacific'; --utc-8
set timezone = 'Asia/Ho_Chi_Minh'; --utc+7
set timezone = 'Asia/Singapore'; --utc+8
set timezone = 'Asia/Tokyo'; --utc+9
select '2021-02-16 09:00+07'::timestamp AT local;
@gitgithan
gitgithan / data.csv
Created November 1, 2024 11:27
PolynomialFeatures interactions
x y z
120.02546181352442 290.1890638016333 a
176.11287493681954 365.9309254666542 a
209.1320297230522 342.8618005735121 a
181.0624796366929 345.8792392596705 a
178.57888332963995 331.34269723696787 a
163.03258710130567 300.9397808973796 a
206.17581421916165 339.20035732970064 a
167.6984963532772 275.34225131900916 a
192.94897784491891 369.5082303658907 a

FWIW: I'm not the author of the content presented here (which is an outline from Edmond Lau's book). I've just copy-pasted it from somewhere over the Internet, but I cannot remember what exactly the original source is. I was also not able to find the author's name, so I cannot give him/her the proper credits.


Effective Engineer - Notes

What's an Effective Engineer?

@gitgithan
gitgithan / modern_sql_style_guide.md
Created February 8, 2022 10:34 — forked from mattmc3/modern_sql_style_guide.md
Modern SQL Style Guide
layout author title revision version description
default
mattmc3
Modern SQL Style Guide
2019-01-17
1.0.1
A guide to writing clean, clear, and consistent SQL.

Modern SQL Style Guide

@gitgithan
gitgithan / SETUP_DAY.md
Created May 3, 2021 10:52
Starting my coding journey...
@gitgithan
gitgithan / SETUP_DAY.md
Created April 7, 2021 04:33
Starting my coding journey...