Skip to content

Instantly share code, notes, and snippets.

View ronniejoshua's full-sized avatar
:octocat:
Nerding Out on Data

Ronnie Joshua ronniejoshua

:octocat:
Nerding Out on Data
View GitHub Profile
@ronniejoshua
ronniejoshua / jinfo_data.py
Created November 22, 2020 19:38
Scraping Jewish Nobel Prize Winner Data
from bs4 import BeautifulSoup
import requests
import re
# Scrapping the jinfo.org website for jewish nobel prize winners
def jewish_nobel_winners():
sub_domains = [
"Nobels_Chemistry",
"Nobels_Economics",
@ronniejoshua
ronniejoshua / nobel_api.py
Created November 22, 2020 19:42
Extract Information about Nobel Laureates
import requests
# Flattening out the json object for parsing nobel.org api endpoint response
def flatten_json(y):
out = {}
def flatten(x, name=""):
if type(x) is dict:
for a in x:
@ronniejoshua
ronniejoshua / utils.py
Created November 22, 2020 19:46
Setting up of Fuzzy Match Merge to Join the two Dataframes
from fuzzywuzzy import process
import pandas as pd
def fuzzy_merge(df_left, df_right, key_left, key_right, threshold=84, limit=1):
"""
df_1: the left table to join
df_2: the right table to join
key_left: the key column of the left table
key_right: the key column of the right table
@ronniejoshua
ronniejoshua / app.py
Created November 22, 2020 19:49
Implementation of Join - Using Key Column & Data Generation
import pandas as pd
from data_extractor.nobel_api import nobel_api_laureates
from data_extractor.jinfo_data import jewish_nobel_winners
from data_extractor.utils import nobel_laureates_dataframe
if __name__ == "__main__":
# Extract data
df_jew = pd.DataFrame(jewish_nobel_winners())
df_nobel = pd.DataFrame(nobel_api_laureates())
@ronniejoshua
ronniejoshua / connect-gcp-compute-engine.sh
Last active June 9, 2021 15:05
Automate external ip white listing and connecting to gcp compute engine using shell script
#!/bin/bash
GCLOUD="/Users/<your-username>/google-cloud-sdk/bin"
${GCLOUD}/gcloud compute firewall-rules update <gcp-firewall-rule-name> --project=<gcp-project-id> --source-ranges=$(curl ifconfig.me)
external_ip=$(${GCLOUD}/gcloud compute instances describe <gcp-compute-engine> --zone=<gcp-ce-zone> --format='get(networkInterfaces[0].accessConfigs[0].natIP)')
ssh naya@$external_ip -p 122
@ronniejoshua
ronniejoshua / download-pdf.js
Created June 4, 2021 07:13
How to download view only protected PDF from Google Drive (JS code)?
// https://codingcat.codes/2019/01/09/download-view-protected-pdf-google-drive-js-code/
// https://www.linkedin.com/in/ronnie-joshua/
// 1. Open the presentation
// 2. Click three times on zoom to enlarge the presentation
// 3. Scroll down slowly to the last page [Let the entire document load that's the idea]
// 4. Then go to the developer console and run this script
// 5. Check the results. if not okay then adjust width and length line of the code
// Landscape PDF
@ronniejoshua
ronniejoshua / connect-gcp-compute-engine.py
Created June 11, 2021 08:09
Python code to whitelist External IP for GCP Compute Engine Firewall Rules
import requests
import os
def gcp_firewall_ip_whitelisting():
my_external_ip = requests.get("https://api.ipify.org").text
print(f"My External IP Address: {my_external_ip}")
known_ip_addresses = ""
cmd_to_run = f"gcloud compute firewall-rules update nc-de-doc-sec --source-ranges {my_external_ip}, {known_ip_addresses}"
os.system(cmd_to_run)
@ronniejoshua
ronniejoshua / mp_mysql_pandas_gbq_v2.py
Last active September 18, 2021 03:46
Loading data from MySQL Server (DB) to Google BigQuery Using Pandas
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Copyright 2021 Ronnie Joshua
#
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
@ronniejoshua
ronniejoshua / pandas-for-data-engineers-part-1.py
Last active May 4, 2023 16:21
Pandas for Data Engineers
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Copyright 2021 Ronnie Joshua
# Email: ron.juden@gmail.com
# Linkedin: https://www.linkedin.com/in/ronnie-joshua/
# Vist: http://www.webanalyticsinfo.com/
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ronniejoshua
ronniejoshua / unbounce-api-connector.py
Last active August 31, 2021 07:18
Unbounce API Connector
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Copyright 2021 Ronnie Joshua
# Email: ron.juden@gmail.com
# Linkedin: https://www.linkedin.com/in/ronnie-joshua/
# Vist: http://www.webanalyticsinfo.com/
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.