Skip to content

Instantly share code, notes, and snippets.

View avisionh's full-sized avatar
🧽

A Ho avisionh

🧽
View GitHub Profile
@avisionh
avisionh / .gitconfig
Last active November 15, 2023 12:12
Create default settings, including nice git log and overcoming organisational proxies, for git
# set global .gitconfig file details
# to edit global .gitconfig
#git config --edit --global
# to see where global .gitconfig file is enter following commands on BASH
#cd~
#pwd
[user]
@avisionh
avisionh / delete_tables_by_schema.sql
Created January 7, 2020 21:08
Deleting multiple tables based on schema and name
USE [<database_name>]
/* Deleting Tables by Schema */
-- Usage: This script generates dynamic SQL that writes a set of SQL queries that delete tables in a specified database
-- by the schema those tables belong to.
DECLARE @query NVARCHAR(MAX)
SELECT @query =
COALESCE(@query, N'') + N'DROP TABLE [<schema_name>].' + QUOTENAME(TABLE_NAME) + N';' + CHAR(13)
@avisionh
avisionh / find_columns_to_join.sql
Created January 7, 2020 21:10
Find common columns across two tables to join
USE [<schema_name>]
GO
/* Joining via intermediary join */
-- DESC: Purpose of this script is to find a way to join [REGBODY1] and [REGBODY2]
-- columns from [usr].[Y17_18_Student_Course] onto [usr].[Y17_18_Student_Core].
-- Q. IS THERE A COMMON UNIQUE IDENTIFIER COLUMN BETWEEN THE TWO TABLES?
-- A. Yes, [UKPRN], [COURSEID], [COURSEAIM]
SELECT
@avisionh
avisionh / common_columns_for_similar_tablesnames_in_database.sql
Created January 7, 2020 21:12
Find common columns for tables named similarly in a database. This makes use of dynamic pivoting so we don't have to explicitly specify the column names
DECLARE @cols AS NVARCHAR(MAX), @query AS NVARCHAR(MAX);
SET @cols =
(
SELECT STRING_AGG([TABLE_NAME],',')
FROM (
SELECT DISTINCT [TABLE_NAME]
FROM [INFORMATION_SCHEMA].[COLUMNS]
WHERE [TABLE_NAME] LIKE '%Y08_09_Student%'
) AS t
@avisionh
avisionh / change_column_collation.sql
Created January 7, 2020 21:14
Find and change a column's collation so joins can be performed
DECLARE @cols AS NVARCHAR(MAX), @query AS NVARCHAR(MAX);
SET @cols =
(
SELECT STRING_AGG([TABLE_NAME],',')
FROM (
SELECT DISTINCT [TABLE_NAME]
FROM [INFORMATION_SCHEMA].[COLUMNS]
WHERE [TABLE_NAME] LIKE '%Y08_09_Student%'
) AS t
@avisionh
avisionh / connect_and_query_sql.R
Created January 7, 2020 21:16
Connecting R to a SQL Server instance via the DBI package
# ----------- #
# SQL Connect #
# ----------- #
# DESCRIPTION: This script is a short tutorial in connecting R to a SQL server.
# It uses the 'DBI' package as opposed to the 'RODBC' packages due to the
# security that the DBI package can bring in preventing SQL injection attacks.
# NOTE: SQL injection attacks are destructive actions that can be taken to your database
# that are brought from providing users the ability to query from SQL via a different
# software/programme/platform. Typically, it can involve things like DROP TABLE.
@avisionh
avisionh / write_to_sql.R
Created January 7, 2020 21:17
Writing data into SQL Server environment
# ----------- #
# SQL Connect #
# ----------- #
# DESCRIPTION: This script is a short tutorial in connecting R to the EDAP SQL server.
# It uses the 'DBI' package as opposed to the 'RODBC' packages due to the
# security that the DBI package can bring in preventing SQL injection attacks.
# NOTE: SQL injection attacks are destructive actions that can be taken to your database
# that are brought from providing users the ability to query from SQL via a different
# software/programme/platform. Typically, it can involve things like DROP TABLE.
@avisionh
avisionh / dataimport_sqldata.R
Last active January 10, 2020 01:08
Mapping shapefiles in R
# ------------------ #
# dataimport_sqldata #
# ------------------ #
# DESCRIPTION: imports and wrangles the geography lookups in SQL.
# AUTHOR: Avision Ho
# ASSUMPTIONS: none
# SCRIPT DEPENDENCIES:
# 1. 'functions.R'
# PACKAGE DEPENDENCIES:
@avisionh
avisionh / gethtmltable.py
Created March 27, 2020 07:54
Example of extracting tables from HTML webpages
import pandas as pd
import requests
# get country lookups
url = "https://unstats.un.org/unsd/methodology/m49/"
html = requests.get(url).content
data_countries = pd.read_html(html)
# take first list element which is in English
data_countries = data_countries[0]
@avisionh
avisionh / getdatagithubapi.py
Created March 27, 2020 08:00
Does not work because have 2FA. Attempts to pull data from a repo on GitHub via the PyGitHub API
# doesn't work because have 2FA
# https://pygithub.readthedocs.io/en/latest/introduction.html
# https://pygithub.readthedocs.io/en/latest/examples/Repository.html#get-a-specific-content-file
from github import Github
# create Github instance
password = os.getenv("GITHUB_PASSWORD")
g = Github("avisionh", password)
repo = g.get_repo("lukes/ISO-3166-Countries-with-Regional-Codes")