Skip to content

Instantly share code, notes, and snippets.

Rob Cowie robcowie

Block or report user

Report or block robcowie

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@robcowie
robcowie / pg_vacuuming_queries.md
Last active Feb 18, 2020
PostgreSQL vacuuming monitoring queries
View pg_vacuuming_queries.md

Postgresql Vacuuming

To monitor progress of autovacuum

SELECT
    heap_blks_scanned/cast(heap_blks_total as numeric) * 100 as heap_blks_percent,
    progress.*,
    activity.query
FROM
@robcowie
robcowie / marketplace.py
Created Nov 4, 2019
Amazon marketplaces with api endpoints
View marketplace.py
"""Amazon Marketplace and Region Info.
* [MWS docs](https://docs.developer.amazonservices.com/en_US/dev_guide/DG_Endpoints.html)
* [Advertising API docs](https://advertising.amazon.com/API/docs/en-us/get-started/how-to-use-api)
"""
from dataclasses import dataclass
@dataclass
@robcowie
robcowie / circleci.yml
Created Oct 25, 2019
Example circleci config for running tests with a postgresql DB
View circleci.yml
version: 2.1
jobs:
test-cf-audit-report-data:
docker:
- image: circleci/python:3.7.1
- image: circleci/postgres:11.4-alpine-ram
environment:
POSTGRES_USER: user
POSTGRES_DB: test
@robcowie
robcowie / install_lsp_sublime.md
Last active Feb 20, 2020
Install and configure the python language server for Sublime Text 3 (using the LSP plugin)
View install_lsp_sublime.md

Install and configure Python LSP & Sublime Text 3

Install the langauge server

This installs the python server provided by Palantir.

Microsoft also provide the C# server which requires the .net runtime.

pipenv install 'python-language-server[all]'
@robcowie
robcowie / sqlalchemy_create_statement.py
Last active Aug 30, 2019
Print create SQL statement for tales and indices defined with SQLAlchemy
View sqlalchemy_create_statement.py
from sqlalchemy import create_engine, Table, MetaData
from sqlalchemy.schema import CreateIndex, CreateTable
metadata = MetaData()
engine = create_engine('')
def create_table_statement(connection, table):
stmts = []
@robcowie
robcowie / postgresql_notes.md
Created Apr 6, 2019
Notes and tips for using Postgresql
View postgresql_notes.md

PostgreSQL Notes

Column Names

All identifiers are coerced to lower-case so column names are effectively case-insensitive. To avoid this, quote the identifier with double quotes.

SELECT "myColA" FROM "camelCaseTable";
@robcowie
robcowie / config_arg_parser.py
Created Feb 18, 2019
ArgumentParser and argparse Action that can pull args from a yaml config file
View config_arg_parser.py
# -*- coding: utf-8 -*-
"""
Part of the undertime app https://gitlab.com/anarcat/undertime by Antoine Beaupré.
AGPLv3 licence (https://gitlab.com/anarcat/undertime/blob/master/LICENSE)
"""
import argparse
import os
@robcowie
robcowie / ec2_operator.py
Created Feb 11, 2019
EC2 Airflow Operator
View ec2_operator.py
# -*- coding: utf-8 -*-
"""
NOTE THIS IS UNTESTED AS IT WAS NOT REQUIRED.
See:
- https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/aws_hook.py
- https://github.com/apache/airflow/blob/master/airflow/contrib/operators/ecs_operator.py
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances
- https://stackabuse.com/automating-aws-ec2-management-with-python-and-boto3/
@robcowie
robcowie / largest_partition.sh
Created Feb 8, 2019
Mounted partition with most free space
View largest_partition.sh
df | grep / | sort -k 4 -n -r | head -n 1 | awk '{print $6}'
@robcowie
robcowie / list_s3_with_metadata.py
Created Jan 28, 2019
List S3 with pagination and metadata
View list_s3_with_metadata.py
def list_s3_with_metadata(s3_conn, prefix):
"""List all keys at `prefix` and return metadata."""
bucket, prefix = prefix.split('://')[1].split('/', 1)
paginator = s3_conn.get_paginator('list_objects_v2')
response = paginator.paginate(Bucket=bucket, Prefix=prefix)
def attrs(d):
return {'Key': 's3://{}/{}'.format(bucket, d['Key']), 'ETag': d['ETag'].replace('"', ''), 'Size': d['Size']}
You can’t perform that action at this time.