Skip to content

Instantly share code, notes, and snippets.

View JoaoCarabetta's full-sized avatar
🏊
data swimming

João Carabetta JoaoCarabetta

🏊
data swimming
View GitHub Profile
@JoaoCarabetta
JoaoCarabetta / github_repos_with_string
Created September 20, 2021 13:42
All unique GitHub repos that contains a string
from github import Github
from time import sleep
g = Github(token)
search_str = 'basedosdados'
repo = []
for i in g.search_code(search_str):
sleep(0.2)
@JoaoCarabetta
JoaoCarabetta / katana.py
Last active August 3, 2021 18:25
Katana Algorithm Minimal Working Example
from shapely.geometry import box, Polygon, MultiPolygon, GeometryCollection
from shapely.wkt import loads
def threshold_func(geometry, threshold_value):
"""Compares the threshold values with the polygon area"""
return geometry.area < threshold_value
def katana(geometry, threshold_func, threshold_value, number_tiles=0, max_number_tiles=250):
"""Splits a geometry in tiles forming a grid given a threshold function and
a maximum number of tiles.
@JoaoCarabetta
JoaoCarabetta / intersect_point_to_hexagon
Created April 19, 2021 21:48
Intersect point to hexagon - SQL Base dos Dados
select id_grid_h3, hora, ST_ASTEXT(ANY_VALUE(geometria)) wkt, count(*) n_registros, ANY_VALUE(quantidade_pessoas) populacao
from `rj-smtr.br_rj_riodejaneiro_onibus_gps.registros_tratada` t1
join `basedosdados.br_ipea_acesso_oportunidades.estatisticas_2019` t2
on st_intersects(geometria, st_geogpoint(longitude, latitude))
where id_municipio in (
select id_municipio
from `basedosdados.br_bd_diretorios_brasil.municipio`
where municipio = 'Rio de Janeiro')
group by id_grid_h3, hora
@JoaoCarabetta
JoaoCarabetta / chess_highlights.js
Created November 11, 2020 20:07
chess.com highlights snippet
function coord_boundaries(coord) {
coord = coord.toString()
return coord[0] >= 1 && coord[0] <= 8 && coord[1] >= 1 && coord[1] <= 8
}
function highlight_square(coord, color) {
board = document.getElementsByClassName('layout-board')[0]
@JoaoCarabetta
JoaoCarabetta / line_polygon_intersection.py
Created September 2, 2020 20:15
Line and Polygon Intersection for Geopandas
def line_polygon_intersection(line_df, poly_df):
"""
It cuts the line if it sits between polygons.
"""
column_geom_poly = poly_df._geometry_column_name
column_geom_line = line_df._geometry_column_name
spatial_index = line_df.sindex
bbox = poly_df.geometry.apply(lambda x: x.bounds)
@JoaoCarabetta
JoaoCarabetta / Makefile
Created July 14, 2020 17:04
Makefile to setup python env. for Data Science projects
.PHONY: create-env update-env
# It creates an env. with the directory name
REPO=$(shell basename $(CURDIR))
create-env:
python3 -m venv .$(REPO);
source .$(REPO)/bin/activate; \
pip3 install --upgrade -r requirements.txt; \
@JoaoCarabetta
JoaoCarabetta / list_in_chuncks.py
Created March 26, 2020 19:05
Break list in chuncks
break_list_in_chuncks = lambda data, chunck: [data[x:x+chunck] for x in range(0, len(data), chunck)]
@JoaoCarabetta
JoaoCarabetta / safely_access_key_from_nested_dict.py
Created February 11, 2020 20:36
Safely access key from nested dict
def accessr(d, keys, default=None):
if len(keys) and d is not None:
return accessr(d.get(keys[0], default), keys[1:], default)
else:
return d
def access(d, keys, default=None):
for k in keys:
if d is not None:
d = d.get(k, default)
@JoaoCarabetta
JoaoCarabetta / safe_create_path.py
Created February 6, 2020 17:17
Safely create a path in python
def safe_create_path(path, replace=False):
try:
if replace:
if os.path.isfile(path):
shutil.rmtree(path)
os.makedirs(path)
except Exception as e:
pass
@JoaoCarabetta
JoaoCarabetta / request2pdf2string.py
Created February 2, 2020 21:45
From request to pdf to string
import requests
# Download
res = requests.get('https://www.camara.leg.br/proposicoesWeb/prop_mostrarintegra?codteor=938381&filename=PL+2699/2011')
# To PDF
with open('metadata.pdf', 'wb') as f:
f.write(res.content)
# To string