Skip to content

Instantly share code, notes, and snippets.

View pgolding's full-sized avatar
🎯
Focusing

Oberon Dude pgolding

🎯
Focusing
View GitHub Profile
# coding=utf8
# the above tag defines encoding for this document and is for Python 2.x compatibility
import re
# Find words in the csv list that begin with 'he'
regex = r"\bhe"
test_str = "shoes, footwear, protection, comfort, human, foot, activity, walking, running, high-heel, stiletto, flat, sole, heel, hole, fashion, design, style, cost, sandal, boot, leather, wood, canvas, rubber, plastic, terrain, climate, well-heeled, work, empty, throw, insole, laces, slip, athletic, sneakers, gym, ballet, tap, horseshoe, polish, factory"
@pgolding
pgolding / dynamo_expression_builder.md
Last active August 22, 2017 15:40
DynamoDB Expression builder for update_item method in Boto3 (Python)

The update_item method in Boto3 requires preparing something like this:

try:
    result = table.update_item(
        Key={
            'id': event['pathParameters']['id']
        },
        ExpressionAttributeNames=attr_names,
        ExpressionAttributeValues=expression_vals,
@pgolding
pgolding / remove_adobe_crud.py
Last active February 15, 2018 05:50
Using BeautifulSoup to remove Adobe entity tags (and width/height attributes) from SVG
'''
When rendering SVG files exported from Adobe Illustrator using Phantom JS,
the injected Adobe ENTITY tags can intefere with the render. This removes
the tags and also removes width/height attributes from the SVG tag.
'''
SVG_FILE = 'Some-adobe-illustrator-export.svg'
# By default, BeautifulSoup will ignore the <!ENTITY> array when parsing the file
@pgolding
pgolding / streamingbody.md
Last active January 30, 2024 16:28
Handling of StreamingBody Response from Invoking a Lambda Function

Handling of StreamingBody Response from Invoking a Lambda Function

When calling a Lambda Function via Boto3, the returned object is something like:

{u'Payload': <botocore.response.StreamingBody object at 0x7f8d5b62ac50>, 
'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '5bdbb3ca-3b35-11e7-9816-397b140c3bac', 'HTTPHeaders': {'x-amzn-requestid': '5bdbb3ca-3b35-11e7-9816-397b140c3bac', 'content-length': '1636', 'x-amzn-trace-id': 'root=1-591ca199-00d34d5474d16275ec2c8d10;sampled=0', 'x-amzn-remapped-content-length': '0', 'connection': 'keep-alive', 'date': 'Wed, 17 May 2017 19:16:41 GMT', 'content-type': 'application/json'}}, u'StatusCode': 200}

The Payload parameter is <botocore.response.StreamingBody> which is a data streaming object.

@pgolding
pgolding / cosine_similarity.py
Created May 27, 2017 20:26
Cosine Similarity Python Scikit Learn
# http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
# The usual creation of arrays produces wrong format (as cosine_similarity works on matrices)
x = np.array([2,3,1,0])
y = np.array([2,3,0,0])
# Need to reshape these
x = x.reshape(1,-1)
@pgolding
pgolding / scan.py
Created June 5, 2017 19:40
scan all elements from a dynamodb table using Python (boto3)
from __future__ import print_function # Python 2/3 compatibility
import boto3
import json
import decimal
from boto3.dynamodb.conditions import Key, Attr
# Helper class to convert a DynamoDB item to JSON.
class DecimalEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, decimal.Decimal):
@pgolding
pgolding / dynamodb_reserved.txt
Created June 5, 2017 19:41
DynamoDB reserved words
ABORT
ABSOLUTE
ACTION
ADD
AFTER
AGENT
AGGREGATE
ALL
ALLOCATE
ALTER
@pgolding
pgolding / dynamodb_reserved.py
Created June 5, 2017 19:51
dynamodb reserved words python list
reserved_words = ['ABORT', 'ABSOLUTE', 'ACTION', 'ADD', 'AFTER', 'AGENT', 'AGGREGATE', 'ALL', 'ALLOCATE', 'ALTER', 'ANALYZE', 'AND', 'ANY', 'ARCHIVE', 'ARE', 'ARRAY', 'AS', 'ASC', 'ASCII', 'ASENSITIVE', 'ASSERTION', 'ASYMMETRIC', 'AT', 'ATOMIC', 'ATTACH', 'ATTRIBUTE', 'AUTH', 'AUTHORIZATION', 'AUTHORIZE', 'AUTO', 'AVG', 'BACK', 'BACKUP', 'BASE', 'BATCH', 'BEFORE', 'BEGIN', 'BETWEEN', 'BIGINT', 'BINARY', 'BIT', 'BLOB', 'BLOCK', 'BOOLEAN', 'BOTH', 'BREADTH', 'BUCKET', 'BULK', 'BY', 'BYTE', 'CALL', 'CALLED', 'CALLING', 'CAPACITY', 'CASCADE', 'CASCADED', 'CASE', 'CAST', 'CATALOG', 'CHAR', 'CHARACTER', 'CHECK', 'CLASS', 'CLOB', 'CLOSE', 'CLUSTER', 'CLUSTERED', 'CLUSTERING', 'CLUSTERS', 'COALESCE', 'COLLATE', 'COLLATION', 'COLLECTION', 'COLUMN', 'COLUMNS', 'COMBINE', 'COMMENT', 'COMMIT', 'COMPACT', 'COMPILE', 'COMPRESS', 'CONDITION', 'CONFLICT', 'CONNECT', 'CONNECTION', 'CONSISTENCY', 'CONSISTENT', 'CONSTRAINT', 'CONSTRAINTS', 'CONSTRUCTOR', 'CONSUMED', 'CONTINUE', 'CONVERT', 'COPY', 'CORRESPONDING', 'COUNT', 'COUN
@pgolding
pgolding / example.md
Last active August 25, 2023 16:07
Passing AWS Lambda Function parameters using the ClientContext object in Python

Boto3 documentation for lambda_client.invoke suggests it is possible to pass data via the ClientContext object as a string.

The string is required to be base64 encoded JSON.

To encode such a string, use:

import baseb4
base64.b64encode(b'<string>').decode('utf-8')
@pgolding
pgolding / conditional_check_fail.py
Created December 12, 2017 17:13
The unclearly documented error exceptions of Boto3 (Python) DynamoDB - e.g. ConditionalCheckFailedException
from botocore import exceptions
try:
# to update table conditionally
except exceptions.ClientError as e:
if e.response['Error']['Code'] == 'ConditionalCheckFailedException':
print("ConditionalCheckFailedException detected")
# do something else, like raise a custom error based on the condition