aseemk

This is a Neo4j 1.9 (pre-2.0) query:

START user=node:nodes(type='user')
MATCH user <-[:follows]- follower -[?:follows]-> other
WITH user, follower, 1.0 / COUNT(other) AS weighted
WITH user, COUNT(follower) AS numFollowers, SUM(weighted) as totalWeighted
RETURN user, numFollowers,
  ROUND(totalWeighted * 100) / 100.0 AS totalWeighted,
 ROUND(totalWeighted * 100 / numFollowers) / 100.0 AS avgFollowerWeight

ikwattro

Github Events Analysis with Neo4j

On July 22, Github announced the 3rd Annual Github Data Challenge presenting multiple sources of data available.

This sounded to me a good opportunity to use their available data and import it in Neo4j in order to have a lot of fun at analyzing the data that fits naturally in a graph.

As I work mainly offline or behind military proxies that do not permit me to use the ReST API, I decided to go for the Github Archive available here, you can then download json files representing Github Events on a daily/hour basis.

jexp

package jexp.adoc;

import org.asciidoctor.Asciidoctor;
import org.asciidoctor.ast.ContentPart;
import org.asciidoctor.ast.DocumentHeader;
import org.asciidoctor.ast.StructuredDocument;

import java.io.File;
import java.util.*;

jexp

Unusable on spinning disk, these results are from an SSD.

1M x create node+rel in 38s, 26k r/s
1M x create 2 nodes, 2 relationships, 2 properties in 47s, 21k r/s -> 80k records / s
1M x create 2 nodes with labels, 2 rels, 0 properties in 140s, 7k r/s
100k x create 100 rel + node -> 10M in 20s, 5k r/s
1M lookups by id in 22s, 43k r/s
1M lookup by id compiled runtime in 21s, 47k r/s

jexp

// running spark on a large single machine
// 6 workers, with 12G RAM each -> 72G total and 8G for the driver -> 80G RAM in total
// the machine has 6 physical CPUs
// the jar contains just AnormCypher.org + Dependencies 
neo@neo:/tmp/spark$ bin/spark-shell --jars ../neo4j/target/scala-2.10/Neo4j-Spark-Demo-assembly-1.0.jar --driver-memory 8G --executor-memory 12G --master local[6]
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/

jexp

javascript: (function() { 
var createTable = function() { $("neo-table > table.table.data:not(datatable)").addClass("datatable").DataTable(); };
if ($("#style-datatable").length == 0) {
	$("body").append($('<link id="style-datatable" rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.13/css/jquery.dataTables.min.css">'));
	$.getScript('//cdn.datatables.net/1.10.13/js/jquery.dataTables.min.js',createTable);
} else {
	createTable();
}
})();

jexp

export NEO4J_HOME=${NEO4J_HOME-~/Downloads/neo4j-community-3.0.1}

if [ ! -f data-csv.zip ]; then
  curl -OL https://cloudfront-files-1.publicintegrity.org/offshoreleaks/data-csv.zip
fi
	
export DATA=${PWD}/import

rm -rf $DATA

jexp

// On https://analytics.twitter.com/user/<name>/home
// run this in developer console
// essentially:

$(".home-group-row").map(function(x) { var group=$(this); return group.find(".home-group-header").text()+","+ ['tweets','tweetviews','profile-views','mentions','followers'].map(function(tag) { return group.find(".metric-"+tag).text(); }).join(","); }).toArray().join("\n")

// prettied up:

var metrics = ['tweets','profile-views','mentions','followers','tweetviews'];
"month\t" + metrics.join("\t") + "\n" + 

jexp

"C:\Program Files\Neo4j Community\jre\bin\java" -cp "C:\Program Files\Neo4j Community\bin\neo4j-desktop-3.0.2.jar" \
org.neo4j.tooling.ImportTool %*

jexp

How is X done

To display the nodes with the most properties defined, run the following Cypher Statement: