chrisalbon

# -*- coding: utf-8 -*-

""" Deletes all tweets below a certain retweet threshold.
"""

import tweepy
from datetime import datetime

# Constants
CONSUMER_KEY = ''

ryancollingwood

# Credit for this: Nicholas Swift
# as found at https://medium.com/@nicholas.w.swift/easy-a-star-pathfinding-7e6689c7f7b2
from warnings import warn
import heapq

class Node:
    """
    A node class for A* Pathfinding
    """

stirlingw

Introduction to Installing PySpark & Jupyter Notebooks on Mac OSX

Spark is used for large-scale distributed data processing. It has become the go to standard for a lot of companies in the technology industry. The Spark framework is capable of computing at high speeds, processing massive amounts of resilient sets of data, and it does it all while computing in a highly distributed manner.

Jupyter Notebooks, commenly called "Jupyter", has been a popular application within the Data Science community for many years.   It enables you to edit, run, and share Python code into a web view. It allows you to execute your code in a step by step process in order to share parts of your code in a very flexible way for data analysis work. This is why Jupyter is a great tool to prototype in, and should be used at all companies that are data centric.

Why use PySpark in a Jupyter Notebook?

Most data engineers argue that the Scala programming language version is more performant than Python version, and it is. Howev

nathairtras

import logging
from airflow.models import DagBag

def callback_subdag_clear(context):
    """Clears a subdag's tasks on retry."""
    dag_id = "{}.{}".format(
        context['dag'].dag_id,
        context['ti'].task_id,
    )
    execution_date = context['execution_date']

hemebond

A SaltStack AWS Auto Scaling Solution

Overview

The AWS Auto Scaling Goup, configured with a customised Cloud-Init file, sends a notification to an SNS Topic, which in turn passes it onto an SQS queue that the Salt Master is subscribed to. A Reactor watches for the auto scaling events and pre-approves the new minion based on its Auto Scaling group name and instance ID.

Salt Master Configuration

ClintLiddick

find_package(Threads REQUIRED)

ExternalProject_Add(
  googletest
  GIT_REPOSITORY https://github.com/google/googletest.git
  UPDATE_COMMAND ""
  INSTALL_COMMAND ""
  LOG_DOWNLOAD ON
  LOG_CONFIGURE ON
  LOG_BUILD ON)

jasongilman

Atom Clojure Setup

This describes how I setup Atom for an ideal Clojure development workflow. This fixes indentation on newlines, handles parentheses, etc. The keybinding settings for enter (in keymap.cson) are important to get proper newlines with indentation at the right level. There are other helpers in init.coffee and keymap.cson that are useful for cutting, copying, pasting, deleting, and indenting Lisp expressions.

Install Atom

Download Atom

The Atom documentation is excellent. It's highly worth reading the flight manual.

everilae

# A simple generator wrapper, not sure if it's good for anything at all.
# With basic python threading
from threading import Thread

try:
    from queue import Queue

except ImportError:
    from Queue import Queue
 

debasishg

General Background and Overview

1. Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
2. Models and Issues in Data Stream Systems
3. Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
4. Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
5. [Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep=rep1&t

pkmishra

import os
import random
from scrapy.conf import settings
class RandomUserAgentMiddleware(object):
    def process_request(self, request, spider):
        ua  = random.choice(settings.get('USER_AGENT_LIST'))
        if ua:
            request.headers.setdefault('User-Agent', ua)

class ProxyMiddleware(object):