Skip to content

Instantly share code, notes, and snippets.

View varunchitale's full-sized avatar

Varun Chitale varunchitale

  • Berlin
View GitHub Profile
@varunchitale
varunchitale / locks.sql
Created March 25, 2019 07:10
Get a list of queries and their details that hold locks over relations in a dB.
SELECT
blocked_locks.pid AS blocked_pid,
blocked_activity.usename AS blocked_user,
blocking_locks.pid AS blocking_pid,
blocking_activity.usename AS blocking_user,
blocked_activity.query AS blocked_statement,
blocking_activity.query AS current_statement_in_blocking_process,
blocked_activity.application_name AS blocked_application,
blocking_activity.application_name AS blocking_application
FROM pg_catalog.pg_locks blocked_locks
@varunchitale
varunchitale / dirty_updates.sql
Last active March 25, 2019 07:12
Update a table and skip the conflicting rows/take some action.
DO $$
DECLARE r RECORD;
BEGIN
FOR r IN <query that will return a record set for r to iterate over>
LOOP
BEGIN
RAISE NOTICE 'Dealing with ID: %',r.id;
UPDATE <table> as t1
SET var1 = t2.var1
from <table2> t2
@varunchitale
varunchitale / delete_duplicates.sql
Created March 25, 2019 06:56
Efficiently delete duplicates rows from a table with a set of specific constraints.
DELETE FROM <table> a USING (
SELECT MIN(ctid) as ctid, var1, var2
FROM <same_table> b
GROUP BY 2,3 HAVING COUNT(*) > 1
) b
WHERE a.var1 = b.var1
and a.var2 = b.var2
AND a.ctid <> b.ctid
@varunchitale
varunchitale / prompts_mview.sql
Last active October 11, 2018 08:42
Materialized View operations
--Create the materialized view
CREATE materialized view <schema_name>.mv_prompts AS
SELECT id, ngram, freq
FROM <schema_name>.prompts
WHERE add_conditions_here
ORDER BY freq DESC;
--Create a search index using GIN
CREATE INDEX prompts_ngram_idx
ON <schema_name>.mv_prompts
import spacy
#Load the (small) model
_nlp = spacy.load('en_core_web_sm')
#Adjust the max length of the document
#_nlp.max_length = 1000000
#text contains the data that we want to extract phrases from, in string/buffer format
all_prompts = []
@varunchitale
varunchitale / populate_prompts.py
Created October 11, 2018 08:07
Python + Spacy to generate and store key phrases
import psycopg2
from collections import Counter
query_populate = """
INSERT into <schema_name>.prompts (ngram, freq)
values (%(ngram)s, %(freq)s)
ON CONFLICT (ngram) DO UPDATE
SET freq = prompts.freq + %(freq)s
WHERE prompts.ngram = %(ngram)s
"""
@varunchitale
varunchitale / fetch_prompts.py
Last active October 11, 2018 08:27
Simple Python function to retrieve propmts
def prompts(cursor, search_string):
'''
In: cursor module, search_string a tsquery acceptable search string
Out: Prompts dict
'''
default_data = {'prompts':[{'id' : 0, 'key': 'No prompts available'}]}
prompts_data = {}
query_get_prompts = """
@varunchitale
varunchitale / prompts_incompleteSearch.csv
Created October 11, 2018 07:47
Auto complete prompts for search term 'charge'
ngram
service charges
chargeable interest
labour charges
late payment charges
the chargeable accounting periods
the additional discount charges
development charges
additional discount charges
advertisement charges
@varunchitale
varunchitale / prompts_wordSearch.csv
Last active October 11, 2018 07:53
Auto complete results for the search term 'charges'
ngram
service charges
labour charges
late payment charges
the additional discount charges
development charges
additional discount charges
advertisement charges
Process Fee Charges
metal labour charges
@varunchitale
varunchitale / prompts.csv
Last active October 11, 2018 07:26
Prompts
id ngram freq
106951 the aggregate average advances 3
109565 This Petition 2
105276 M/s Gillette India Ltd. 11
105869 that expenditure 3
107390 the trading loss 2
110562 a shareholder 3
109200 the Co-ordinate Bench 3
104688 the direction 3
106465 his business 3