Skip to content

Instantly share code, notes, and snippets.

View makmanalp's full-sized avatar

Mehmet Ali "Mali" Akmanalp makmanalp

View GitHub Profile
@makmanalp
makmanalp / README.md
Created September 21, 2023 15:16
Why "let's do force index on every query we have" might not be helpful

TLDR: well intentioned but ultimately unhelpful IMHO. Here's why:

  1. It's easy to make a judgement about bad query plans based on an extremely biased sample: To give you a sense of the variety of queries we have: as of today there are over 180k unique query fingerprints at HubSpot. Let's ignore the trivial ones: about 18k unique query fingerprints do > 1000 queries/sec. To be sure, query planner bugs are real, and I'm currently fairly sure we've hit one here (details later) but of the total a miniscule amount is /truly/ (more on this later) query planner silliness.
  2. By contrast, humans can be quite bad at figuring out what index a query needs and will compare dismally to the above success rate if they start doing FORCE INDEX on everything manually. I mess it up often. I see smart, competent, experienced engineers mess it up quite literally every day. People have attempted to codify rules for this exhaustively - every time I scroll through that page I
@makmanalp
makmanalp / comparison.md
Last active March 14, 2023 14:58
Angular vs Backbone vs React vs Ember notes

Note: these are pretty rough notes I made for my team on the fly as I was reading through some pages. Some could be mildly inaccurate but hopefully not terribly so. I might resort to convenient fiction & simplification sometimes.

My top contenders, mostly based on popularity / community etc:

  • Angular
  • Backbone
  • React
  • Ember

Mostly about MVC (or derivatives, MVP / MVVM).

@makmanalp
makmanalp / stata_dask.py
Last active August 14, 2021 16:10
Read STATA .dta files chunk by chunk (streaming) into dask with pandas's read_stata / StataReader and some hackery
import dask.dataframe as dd
from dask.dataframe.utils import make_meta
from dask.delayed import delayed
import pandas as pd
from itertools import chain
def get_stata_dask_meta(file_name, meta_chunksize=10000, *args, **kwargs):
"""Load up first bit of the file for type metadata info. We have to resort
@makmanalp
makmanalp / sshtest-playbook.yml
Last active June 18, 2021 15:06
SSH agent forwarding tester for ansible / vagrant
---
- hosts: all
sudo: no
tasks:
- shell: echo "Client= [$SSH_CLIENT] Sock= [$SSH_AUTH_SOCK]"
register: myecho
- debug: msg="{{myecho.stdout}}"
- shell: ssh-add -l
register: myecho
- debug: msg="{{myecho.stdout}}"
@makmanalp
makmanalp / .block
Last active February 4, 2021 13:41
Multivariate radar charts with different axes
license: mit
scrolling: yes
@makmanalp
makmanalp / gist:ddffd79bdbd75fbff5126c69eb07c1bb
Created March 11, 2019 19:27
ads-0-backup-1552296000-l2zs8 backu
ads-0-backup-1552296000-l2zs8 backup INFO 2019/03/11 19:22:01 tablet prod_iad-1360915300 still has decreasing replication lag of 208.710618394 seconds, will continue waiting
ads-0-backup-1552296000-l2zs8 backup INFO 2019/03/11 19:24:01 tablet prod_iad-1360915300 has caught up on replication
ads-0-backup-1552296000-l2zs8 backup INFO 2019/03/11 19:24:01 (prod_iad-1360915300) checking health
ads-0-backup-1552296000-l2zs8 backup INFO 2019/03/11 19:24:06 (prod_iad-1360915300) succeeded 1 of 3 healthchecks
ads-0-backup-1552296000-l2zs8 backup INFO 2019/03/11 19:24:11 (prod_iad-1360915300) succeeded 2 of 3 healthchecks
ads-0-backup-1552296000-l2zs8 backup INFO 2019/03/11 19:24:16 (prod_iad-1360915300) succeeded 3 of 3 healthchecks
ads-0-backup-1552296000-l2zs8 backup INFO 2019/03/11 19:24:16 getting replication status for replicas
ads-0-backup-1552296000-l2zs8 backup INFO 2019/03/11 19:24:16 getting replication status for master
ads-0-backup-1552296000-l2zs8 backup INFO 2019/03/11 19:24:16 comparing GTIDSets for err
@makmanalp
makmanalp / README.md
Last active February 6, 2019 00:12
Orchestrator OOM investigation

Summary:

When we run a rolling restart on our orchestrator statefulset, the node that is the previous master will get stuck in a crash loop.

Findings so far:

@makmanalp
makmanalp / get_circleci_artifact.py
Last active January 1, 2019 15:50
Quick and dirty ansible module for fetching CircleCI build artifacts (latest on a branch, or by build num & git SHA)
#!/usr/bin/env python
import requests
from ansible.module_utils.basic import AnsibleModule
import traceback
try:
from urllib.parse import quote
except ImportError:
@makmanalp
makmanalp / rectangularize.py
Last active December 12, 2018 19:56
monchtangularize / rectangularize data
from collections import OrderedDict
from itertools import product
def rectangularize(data, keys):
"""Make sure there is a row in the dataset for each unique combination of
values for the given keys.
E.g. If your rows are:
[
@makmanalp
makmanalp / validate.py
Last active November 16, 2018 21:18
SQLAlchemy validator event example
from sqlalchemy import Column, Integer, String, DateTime, Boolean
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import event
import datetime
Base = declarative_base()
def validate_int(instance, value, oldvalue, initiator):
# Assigning a string to an Integer column will try to coerce it to the