Skip to content

Instantly share code, notes, and snippets.

Rob Cowie robcowie

Block or report user

Report or block robcowie

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@robcowie
robcowie / install_lsp_sublime.md
Last active Jun 23, 2019
Install and configure the python language server for Sublime Text 3 (using the LSP plugin)
View install_lsp_sublime.md

Install and configure Python LSP & Sublime Text 3

Install the langauge server

This installs the python server provided by Palantir.

Microsoft also provide the C# server which requires the .net runtime.

pipenv install 'python-language-server[all]'
@robcowie
robcowie / sqlalchemy_create_statement.py
Last active Jul 31, 2019
Print create SQL statement for tales and indices defined with SQLAlchemy
View sqlalchemy_create_statement.py
from sqlalchemy import Table, MetaData
from sqlalchemy.schema import CreateIndex, CreateTable
def create_table_statement(connection, table):
stmts = []
stmts.append(CreateTable(table).compile(connection))
for index in table.indexes:
stmts.append(CreateIndex(index).compile(connection))
@robcowie
robcowie / postgresql_notes.md
Created Apr 6, 2019
Notes and tips for using Postgresql
View postgresql_notes.md

PostgreSQL Notes

Column Names

All identifiers are coerced to lower-case so column names are effectively case-insensitive. To avoid this, quote the identifier with double quotes.

SELECT "myColA" FROM "camelCaseTable";
@robcowie
robcowie / config_arg_parser.py
Created Feb 18, 2019
ArgumentParser and argparse Action that can pull args from a yaml config file
View config_arg_parser.py
# -*- coding: utf-8 -*-
"""
Part of the undertime app https://gitlab.com/anarcat/undertime by Antoine Beaupré.
AGPLv3 licence (https://gitlab.com/anarcat/undertime/blob/master/LICENSE)
"""
import argparse
import os
@robcowie
robcowie / ec2_operator.py
Created Feb 11, 2019
EC2 Airflow Operator
View ec2_operator.py
# -*- coding: utf-8 -*-
"""
NOTE THIS IS UNTESTED AS IT WAS NOT REQUIRED.
See:
- https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/aws_hook.py
- https://github.com/apache/airflow/blob/master/airflow/contrib/operators/ecs_operator.py
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances
- https://stackabuse.com/automating-aws-ec2-management-with-python-and-boto3/
@robcowie
robcowie / largest_partition.sh
Created Feb 8, 2019
Mounted partition with most free space
View largest_partition.sh
df | grep / | sort -k 4 -n -r | head -n 1 | awk '{print $6}'
@robcowie
robcowie / list_s3_with_metadata.py
Created Jan 28, 2019
List S3 with pagination and metadata
View list_s3_with_metadata.py
def list_s3_with_metadata(s3_conn, prefix):
"""List all keys at `prefix` and return metadata."""
bucket, prefix = prefix.split('://')[1].split('/', 1)
paginator = s3_conn.get_paginator('list_objects_v2')
response = paginator.paginate(Bucket=bucket, Prefix=prefix)
def attrs(d):
return {'Key': 's3://{}/{}'.format(bucket, d['Key']), 'ETag': d['ETag'].replace('"', ''), 'Size': d['Size']}
@robcowie
robcowie / .gitignore_global
Created Nov 27, 2018
Global gitignore for discussion
View .gitignore_global
# Compiled source #
###################
*.com
*.class
*.dll
*.exe
*.o
*.so
*.pyc
*.cache
@robcowie
robcowie / boto3_emr_cluster_definition.py
Created Nov 21, 2018
EMR cluster definition for boto3
View boto3_emr_cluster_definition.py
CLUSTER_DEFINITION = {
'Name': 'name',
'Instances': {
'InstanceGroups': [
{
'Name': 'Master',
'Market': 'SPOT',
'InstanceRole': 'MASTER',
'BidPrice': '1',
'InstanceType': 'r4.2xlarge',
You can’t perform that action at this time.