DaisukeMiyamoto

####################################################################
# Create compute environment and job queue for AWS Batch with Boto3
# Daisuke Miyamoto
# 20200330
#

import boto3
import botocore.waiter
from botocore.exceptions import WaiterError
import time

goliatone

This TRIGGER function calls PosgreSQL's NOTIFY command with a JSON payload. You can listen for these calls and then send the JSON payload to a message queue (like AMQP/RabbitMQ) or trigger other actions.

Create the trigger with notify_trigger.sql.

When declaring the trigger, supply the column names you want the JSON payload to contain as arguments to the function (see create_triggers.sql)

The payload returns a JSON object:

uhho

def s3_to_pandas(client, bucket, key, header=None):

    # get key using boto3 client
    obj = client.get_object(Bucket=bucket, Key=key)
    gz = gzip.GzipFile(fileobj=obj['Body'])
    
    # load stream directly to DF
    return pd.read_csv(gz, header=header, dtype=str)

def s3_to_pandas_with_processing(client, bucket, key, header=None):

blaylockbk

import multiprocessing #:)

def do_this(number):
  print number
  return number*2

# Create a list to iterate over. 
# (Note: Multiprocessing only accepts one item at a time)
some_list = range(0,10)

eldondevcg

# IF YOU INCUR HUGE COSTS WITH THIS OR IT BREAKS DON'T BLAME ME License

# This is a throw-away script I wrote to pull the json events for all of the streams from a cloudwatch log
# For some reason, the naive way to do vpc network logging does logging to different streams in a cloudwatch
# log based on interface.
# Great for diagnosing lots of things, and generating verbose logs, but for the broad-stroke analysis I was doing,
# all I really wanted was the basic data. This would have been easier if I had logged to s3, but I did not see a
# way to do that in 2 clicks.

group_name = 'CHANGEME'

Zearin

NOTE: This is a question I found on StackOverflow which I’ve archived here, because the answer is so effing phenomenal.

---

Q: How can I make a chain of function decorators in Python?

---

If you are not into long explanations, see [Paolo Bergantino’s answer][2].

rgreenjr

-- show running queries (pre 9.2)
SELECT procpid, age(clock_timestamp(), query_start), usename, current_query 
FROM pg_stat_activity 
WHERE current_query != '<IDLE>' AND current_query NOT ILIKE '%pg_stat_activity%' 
ORDER BY query_start desc;

-- show running queries (9.2)
SELECT pid, age(clock_timestamp(), query_start), usename, query 
FROM pg_stat_activity 
WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%' 

josef-pkt

#Copyright (c) 2011, Josh Hemann  (hemann @ colorado . edu)
#All rights reserved.
#
#Redistribution and use in source and binary forms, with or without
#modification, are permitted provided that the following conditions are met:
#    * Redistributions of source code must retain the above copyright
#      notice, this list of conditions and the following disclaimer.
#    * Redistributions in binary form must reproduce the above copyright
#      notice, this list of conditions and the following disclaimer in the
#      documentation and/or other materials provided with the distribution.