Skip to content

Instantly share code, notes, and snippets.

Avatar

Arne Sund arnesund

View GitHub Profile
@arnesund
arnesund / fetch_netatmo.py
Last active Aug 25, 2020
Fetch Netatmo Weather Station measurements and store in InfluxDB
View fetch_netatmo.py
#!/usr/bin/env python
import os
import sys
import json
import time
import requests
# Get your client ID and secret by creating an App at https://dev.netatmo.com/
NETATMO_CLIENT_ID = ""
NETATMO_CLIENT_SECRET = ""
@arnesund
arnesund / hashtag_word_count.py
Created Jul 13, 2015
Word count of Twitter hashtags using Apache Spark
View hashtag_word_count.py
# Count the number of occurrences for each hashtag,
# by first extracting the hashtag and lowercasing it,
# then do a standard word count with map and reduceByKey
countsRDD = (filteredTweetsRDD
.flatMap(lambda tweet: [hashtag['text'].lower() for hashtag in tweet['entities']['hashtags']])
.map(lambda tag: (tag, 1))
.reduceByKey(lambda a, b: a + b)
)
# Get the most used hashtags (order countsRDD descending by count)
@arnesund
arnesund / load_and_filter_tweets.py
Created Jul 13, 2015
Load tweets into Spark and filter
View load_and_filter_tweets.py
# Extract tweets from MongoDB
allTweets = []
for doc in db.tweets.find():
allTweets.append(doc['tweet'])
# Load tweets into Spark for analysis
allTweetsRDD = sc.parallelize(allTweets, 8)
# Set up filter to only get tweets from the last week
DAYS_LIMIT=7
@arnesund
arnesund / get_tweets.py
Last active Jan 30, 2017
Fetch tweets from Twitter using Python module Tweepy
View get_tweets.py
# Get details about own user
me = api.me()
friends = api.friends_ids(me.id)
# Initialize data structure
tweets = {}
# Fetch lists recent tweets for each of the user IDs in the list 'friends'
for user in friends:
# Only query Twitter for data not already cached
@arnesund
arnesund / cloudinit.conf
Created Feb 5, 2015
Simple Cloud-init Config Example for Apache
View cloudinit.conf
#cloud-config
packages:
- apache2
runcmd:
- [ a2ensite, "000-default" ]
You can’t perform that action at this time.