Skip to content

Instantly share code, notes, and snippets.

View acslater00's full-sized avatar

Adam Cohen acslater00

  • Branch
  • Palo Alto
View GitHub Profile
@acslater00
acslater00 / bot_traffic.md
Last active August 29, 2015 14:07
Bot Traffic

First, most bot traffic doesn't use a browser, it just makes raw web requests to grab whatever content it is interested in. This means it does not request javascript or images, and as a result, most analytics packages use javascript or tiny fake images to track traffic/impressions. This is first & most obvious line of defense.

Second is something called a user agent, which identifies the source of traffic via a label. This is easy to fake -- I can make a bot request with a user agent that looks like an iphone safari browser, but much bot traffic doesn't bother.

Those two things catch ~all cases where bot is not trying to hide the fact that it is a bot. The alternative is surprisingly rare, at least in my experience. Beyond that you can do things like analyze user behavior to look for unusual patterns (e.g. hitting every article on a site exactly once) to get the rest.

Not perfect, but not something we worry about on a daily basis in the industry.

[Slurp:~]$ curl "https://xml.telecharge.com/TrafficCop/Service.asmx?WSDL"
<?xml version="1.0" encoding="utf-8"?>
<wsdl:definitions xmlns:tm="http://microsoft.com/wsdl/mime/textMatching/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/" xmlns:tns="http://tempuri.org/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:s="http://www.w3.org/2001/XMLSchema" xmlns:soap12="http://schemas.xmlsoap.org/wsdl/soap12/" xmlns:http="http://schemas.xmlsoap.org/wsdl/http/" targetNamespace="http://tempuri.org/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">
<wsdl:types>
<s:schema elementFormDefault="qualified" targetNamespace="http://tempuri.org/">
<s:element name="HelloWorld">
<s:complexType />
</s:element>
<s:element name="HelloWorldResponse">
<s:complexType>
[2015-04-29 19:06:27] listingfeed.sources.base - WARNING - Error received from ticketcity -- HTTP 599: Timeout 15200ms
[2015-04-29 19:06:30] listingfeed.sources.base - WARNING - Error received from ticketcity -- HTTP 599: Timeout 15001ms
[2015-04-29 19:06:30] listingfeed.sources.base - WARNING - Error received from ticketcity -- HTTP 599: Timeout 15001ms
[2015-04-29 19:06:30] listingfeed.sources.base - WARNING - Error received from ticketcity -- HTTP 599: Timeout 15001ms
[2015-04-29 19:06:31] listingfeed.sources.base - WARNING - Error received from ticketcity -- HTTP 599: Timeout 15005ms
[2015-04-29 19:06:31] listingfeed.sources.base - WARNING - Error received from ticketcity -- HTTP 599: Timeout 15070ms
[2015-04-29 19:06:31] listingfeed.sources.base - WARNING - Error received from ticketcity -- HTTP 599: Timeout 15000ms
[2015-04-29 19:06:32] listingfeed.sources.base - WARNING - Error received from ticketcity -- HTTP 599: Timeout 15001ms
[2015-04-29 19:06:33] listingfeed.sources.base - WARNING - Error receive

How Yahoo default waivers work

  • waivers clear tuesday morning
  • each player has a waiver priority, which is initialized to Inverse(Draft Order Position)
  • each waiver request includes [Player to Pick Up | Player To Drop]
  • if multiple managers request to pick up the same player the player goes to the manager with the highest priority
  • each time a waiver succeeds, the priority resets to 14 and all other managers move up (called 'rolling' priority list)

How waiver auctions work

from fabric.api import *
env.hosts = ['acslater@hostname.com]
def deploy(staging="/home/acslater/staging/iris", target="/home/acslater/irisforwestchester.com/"):
# pull latest code
with cd(staging):
run("git pull origin master")
@acslater00
acslater00 / example.py
Created September 15, 2011 00:27
pandas/statsmodel
import numpy as np
import scikits.statsmodels.api as sm
from pandas import *
df = DataFrame(transactions)
# easy aggregates
print np.mean(df['x'])
print np.mean(df['price'])
-----BEGIN PGP MESSAGE-----
Comment: GPGTools - https://gpgtools.org
hQIMA8P5ORc1B805ARAArlSuow6Du3bL6j/r0ULfmAftZkgf9El5RJcFx1F2B386
AjIQenzZX6rHDS1U+TPaFByfGdH7Lffv6bfXmsdKGQNfpZlgLD2Nx44FoTUM70SB
2wRUAgwJcC7fIqfvq43cjkJO5dnQjJzaD6giq5xIv9+sd6kDukaLlUNREcOqjK6P
J2m93k1M2T1i85PmFKizbgp7y/9Jkxt72NZ3xUlmzTfGcBuLmXZkOaB/lsA6Bk12
BKKUrFJ5jxYBDzjXV9nPzxsi5uWftr24PULA4OeY2AtfYuOqCRkTxDsxUq6v2xh/
lZw8jAzLUoMsAt3D9wYumRqjg8ovBqhhaZhDZu+RovzAc8zgdlnH3lu2EENGJBQG
h5HNC1F7Rjpv68obpCIDRRQx19AmALHIizAxXDJYWHc17C6UY7FFrF5CdWGf3zIv
hKRib2R5hqhkZXRhY2hlZMOpaGFzaF90eXBlCqNrZXnEIwEg6xXcg4O4PHZufEA6DOkUF8deJbkdjiiDR5ZxgP03bIkKp3BheWxvYWTFAvB7ImJvZHkiOnsia2V5Ijp7ImVsZGVzdF9raWQiOiIwMTIwZWIxNWRjODM4M2I4M2M3NjZlN2M0MDNhMGNlOTE0MTdjNzVlMjViOTFkOGUyODgzNDc5NjcxODBmZDM3NmM4OTBhIiwiaG9zdCI6ImtleWJhc2UuaW8iLCJraWQiOiIwMTIwZWIxNWRjODM4M2I4M2M3NjZlN2M0MDNhMGNlOTE0MTdjNzVlMjViOTFkOGUyODgzNDc5NjcxODBmZDM3NmM4OTBhIiwidWlkIjoiMDI1MTg5YTA0NDZjOWIzOWU5NmRmOTEyZmYxNmIwMTkiLCJ1c2VybmFtZSI6ImFjc2xhdGVyMDAifSwic2VydmljZSI6eyJuYW1lIjoiZ2l0aHViIiwidXNlcm5hbWUiOiJhY3NsYXRlcjAwIn0sInR5cGUiOiJ3ZWJfc2VydmljZV9iaW5kaW5nIiwidmVyc2lvbiI6MX0sImNsaWVudCI6eyJuYW1lIjoia2V5YmFzZS5pbyBnbyBjbGllbnQiLCJ2ZXJzaW9uIjoiMS4wLjE4In0sImN0aW1lIjoxNDg3ODY5MzE0LCJleHBpcmVfaW4iOjUwNDU3NjAwMCwibWVya2xlX3Jvb3QiOnsiY3RpbWUiOjE0ODc4NjkyOTMsImhhc2giOiIwZGUyYzIwZTFjMmVlYTkxYjg2MGZiMDQxZmU5OWRmYTI0NDhiMWQ4MTlmMmVmMTYxMGVkZWRlY2ZhYzY3MmZmNGY0MjU2YmI3MDk5MTY2YTU0OTM0ZDg1NDI3ZDRiMjYzYWMxMjgwOGVhYzQ4MjQ5MTc3MDBjYWFmMzBmZGEwYSIsInNlcW5vIjo5MTc0ODV9LCJwcmV2IjoiM2M5NTY4Y2NlYzgxNDA1OTU0N2UzODEzYWMzZDIy