Skip to content

Instantly share code, notes, and snippets.

Mark Needham mneedham

Block or report user

Report or block mneedham

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@mneedham
mneedham / 00_install.sh
Last active May 29, 2019
Getting Neo4j Tweets
View 00_install.sh
pip install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint
pip install confluent-kafka[avro]
@mneedham
mneedham / import.cql
Last active Jun 1, 2019
Neo4j Twitter Graph
View import.cql
CREATE CONSTRAINT ON(u:User)
ASSERT u.id IS unique;
:param keysToKeep => ["name", "username", "bio", "following", "followers"];
CALL apoc.load.json("https://gist.github.com/mneedham/3c6a59fb5e7d87e20a2f5f1ae4fa2920/raw/9d7c57997c09b3a105556adb6c6f1819792a4db4/query.json")
YIELD value
MERGE (u:User {id: value.user.id })
SET u += value.user
FOREACH (following IN value.following |
MERGE (f1:User {id: following})
MERGE (u)-[:FOLLOWS]->(f1))
@mneedham
mneedham / query.json
Created May 14, 2019
Neo4j Twitter Graph
View query.json
create constraint on (u:User)
ASSERT u.id IS unique
:param keysToKeep => ["name", "username", "bio", "following", "followers"];
CALL apoc.load.json("http://localhost:8000/query.json")
YIELD value
MERGE (u:User {id: value.user.id })
SET u += apoc.map.fromLists($keysToKeep, apoc.map.values(value.user, $keysToKeep))
FOREACH (following IN value.following |
MERGE(f1:User {id: following})
MERGE (u)-[:FOLLOWS]->(f1)
View Followers.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@mneedham
mneedham / gh_releases.py
Created Apr 7, 2019
Get the downloads of a GitHub Releases Project
View gh_releases.py
import requests
import json
import os
query = """
query($owner:String!, $name: String!) {
repository(owner: $owner, name: $name) {
nameWithOwner
releases(first: 50, orderBy: {field:CREATED_AT, direction:DESC}) {
totalCount
View mps.csv
person motionId vote
Graham P Jones 1 FOR
Graham P Jones 2 FOR
Graham P Jones 3 FOR
Graham P Jones 4 DID_NOT_VOTE
Graham P Jones 5 DID_NOT_VOTE
Graham P Jones 6 AGAINST
Graham P Jones 7 AGAINST
Graham P Jones 8 AGAINST
View dblp-ref-0.json
{
"abstract":"The purpose of this study is to develop a learning tool for high school students studying the scientific aspects of information and communication net- works. More specifically, we focus on the basic principles of network proto- cols as the aim to develop our learning tool. Our tool gives students hands-on experience to help understand the basic principles of network protocols.",
"authors":[
"Makoto Satoh",
"Ryo Muramatsu",
"Mizue Kayama",
"Kazunori Itoh",
"Masami Hashimoto",
"Makoto Otani",
"Michio Shimizu",
View 00_articles_header.csv
index:ID(Article) title:string abstract:string year:int
View import.sh
export DATA_DIR=/home/markhneedham/projects/dblp/data
./bin/neo4j-admin import \
--database=foo.db \
--nodes:Author=${DATA_DIR}/authors_header.csv,${DATA_DIR}/authors.csv \
--nodes:Article=${DATA_DIR}/articles_header.csv,${DATA_DIR}/articles.csv \
--nodes:Venue=${DATA_DIR}/venues_header.csv,${DATA_DIR}/venues.csv \
--relationships:REFERENCES=${DATA_DIR}/article_REFERENCES_article_header.csv,${DATA_DIR}/article_REFERENCES_article.csv \
--relationships:AUTHOR=${DATA_DIR}/article_AUTHOR_author_header.csv,${DATA_DIR}/article_AUTHOR_author.csv \
--relationships:VENUE=${DATA_DIR}/article_VENUE_venue_header.csv,${DATA_DIR}/article_VENUE_venue.csv
View json_to_csv.py
import glob
import json
import csv
articles = {}
authors = set()
venues = set()
def write_header(file, fields):
You can’t perform that action at this time.