virattt

jthandy

Running vacuum and analyze in Sinter

dbt and Sinter have the ability to run regular Redshift maintenance jobs. It's great to set these up early on in a project so that things stay clean as the project grows, and implementing these jobs in Sinter allows the same easy transparency and notifications as with your other dbt jobs.

This document will go through the specific steps necessary to configure vacuum and analyze jobs in the current version of dbt and Sinter. In the future, there will likely be a more idiomatically consistent way to express this logic using native dbt operations. Currently, this does work even if it is not elegant.

Step 1: Create the macros

macros/redshift_maintenance.sql

FilippoBovo

Decorator Example in Data Science

This example explains Python decorators in the context of data science. The example acts as a quick reminder, rather than a complete guide.

Consider a Pandas DataFrame about posts on a social media. The DataFrame, called posts, contains a column with the number of likes for each post.

post_id	...	likes	...
1	...	43	...
2	...	92	...

FilippoBovo

Great Python and General Programming Resources

Nice and entertaining talks about Python by Raymond Hettinger:

• Transforming Code into Beautiful, Idiomatic Python
• Python’s Class Development Toolkit
• Beyond PEP 8 -- Best practices for beautiful intelligible code

Advanced Python talks by James Powell:

• So you want to be a Python expert?

jakebrinkmann

import pandas as pd
import pandas.io.sql as sqlio
import psycopg2

conn = psycopg2.connect("host='{}' port={} dbname='{}' user={} password={}".format(host, port, dbname, username, pwd))
sql = "select count(*) from table;"
dat = sqlio.read_sql_query(sql, conn)
conn = None

msubel

=LOWER(CONCATENATE(DEC2HEX(RANDBETWEEN(0;POWER(16;8));8);"-";DEC2HEX(RANDBETWEEN(0;POWER(16;4));4);"-";"4";DEC2HEX(RANDBETWEEN(0;POWER(16;3));3);"-";DEC2HEX(RANDBETWEEN(8;11));DEC2HEX(RANDBETWEEN(0;POWER(16;3));3);"-";DEC2HEX(RANDBETWEEN(0;POWER(16;8));8);DEC2HEX(RANDBETWEEN(0;POWER(16;4));4)))

rogiervandenberg

<?php

/*
Journal (by Journey) extractor

Journey is a great Diary app for Android and Google Chrome. But, your files are
only accessible by Journey itself.

Do you want to move away from Journey to another diary app? Do you want to have
a readable export of your diary? This script converts Journey backup/export

wadewegner

def addBatch(instance, sessionId, jobId, objects):

  request = u"""<?xml version="1.0" encoding="UTF-8"?>
  <sObjects xmlns="http://www.force.com/2009/06/asyncapi/dataload">
     """ + objects + """
  </sObjects>"""

  encoded_request = request.encode('utf-8')
  url = "https://" + instance + ".salesforce.com/services/async/30.0/job/" + jobId + "/batch"
   

mziwisky

OAUTH2

The Problem

I’m a web app that wants to allow other web apps access to my users’ information, but I want to ensure that the user says it’s ok.

The Solution

I can’t trust the other web apps, so I must interact with my users directly. I’ll let them know that the other app is trying to get their info, and ask whether they want to grant that permission. Oauth defines a way to initiate that permission verification from the other app’s site so that the user experience is smooth. If the user grants permission, I issue an AuthToken to the other app which it can use to make requests for that user's info.

Note on encryption

Oauth2 has nothing to do with encryption -- it relies upon SSL to keep things (like the client app’s shared_secret) secure.