gitmathub/agilityscales-neo4j-gist.adoc

## agilityscales-neo4j-gist.adoc

      
    Raw
  

              agilityscales-neo4j-gist.adoc
            
          
    Node Database for Agility Scales


Introduction


With this study I looked at the graph database Neo4j and wanted to find out if it can be suitable for an app called Agility Scales App. More details about the app, which is in a early prototype phase, can be found on http://agilityscales.com and http://agilityscales.com/topic/group-name/


This study is based mainly on the functions shown on youtube: https://www.youtube.com/watch?v=HXKH5NNF4m4


Document Overview


Each section is a use case of the app and I design and build the database as we go.


In the second last section I draw a conclusion of the study.


Last section is the "Nerd Playground" with some hints how to install the graph database Neo4j and load the data.


First Thoughts: Why a database with nodes?


Suitable data model: Many topics with many relations


Flexibility. Relations and properties can be much easier added, removed or changed


Design and presentation is closer to human thinking, so easier to develop when it becomes complicated.


A non standard approach attracts good developers.


Elaborate and fast when it comes to queries of rated topics.


First Draft


The model was rapidly created. At the beginning it might look chaotic but it was easier to design than working with tables and foreign keys, what would be a usual way if you work with a SQL database. For me the draft on the paper was telling a story.


The App


One goal of the app is helping people to solve problems in a project. Similar to a Kanban board a card with a topic is moved through several stages.


The starting point is a user’s search. In this study the user looks for the term "team direction" and gets a list of topics and activities.


// by default: all entities are looked by name, so make name unique and build index
CREATE CONSTRAINT ON (term:Term) ASSERT term.name IS UNIQUE;
//FIXME: exchange with: user-id, email etc.
CREATE CONSTRAINT ON (user:User) ASSERT user.name IS UNIQUE;
CREATE CONSTRAINT ON (topic:Topic) ASSERT topic.name IS UNIQUE;
CREATE CONSTRAINT ON (activity:Activity) ASSERT activity.name IS UNIQUE;

CREATE
  (user:User {name: 'Mat'}),
  (term:Term {name: 'team direction'}),
  (:Topic:Alignment {name: 'Core Values', rate: 85, briefexpl:'A brief explanation of the topic and what to do...'}),
  (:Topic:Grouping {name: 'Formal Leadership', rate: 65}),
  (:Topic:Alignment {name: 'Group Symbol', rate: 75}),
  (:Topic:Alignment {name: 'Group Name', rate: 76}),
  (:Activity:Tool {name: 'Kudo Cards', rate: 84}),
  (:Activity:Tool {name: 'Slogan', rate: 74}),
  (:Activity:Tool {name: 'Culture Books', rate: 71}),
  (:Activity:Exercise {name: 'Inkigai', rate: 53});

// connect user with term
MATCH (user:User {name: 'Mat'})
   WITH user MATCH (term:Term {name: 'team direction'})
   CREATE (user)-[:ASKS]->(term);
// connect term with topic and activity
MATCH (term:Term)
  WITH term MATCH (t:Topic)
  CREATE (term)-[:RELATE_TO]->(t);
MATCH (term:Term)
  WITH term MATCH (a:Activity)
  CREATE (term)-[:RELATE_TO]->(a);


This is how the data looks like. The user (red) ask for team directions (green), which has relate_to connections to several topics (blue) and activities (pink).


Use Case: Order Topics and Activities


We query the database and in this case we want the output as table, so that we see the topics ordered by rate.


// use case: find all topics for a term
MATCH (term:Term {name: 'team direction'})-[:RELATE_TO]->(topic:Topic)
  RETURN topic.name, labels(topic), topic.rate
  ORDER BY topic.rate DESC LIMIT 10


Use case: User registers


When a user registers she/he gets a smiles account. The first 5 smiles are earned for registering.


The user has a dedicated smiles account (as a node). It will accumulate all smiles for the user. The property account will be updated every time the user earns smiles.


MATCH (user:User {name: 'Mat'})
  MERGE (user)-[:HAS]->(smiles {name:'Smiles', account:5})
  RETURN user, smiles


The Neo4j database comes with a interactive tool (Neo4j Browser) which is perfect for designing a database. As you can see on the screenshot, the node Smiles is highlighted blue and at the bottom its properties are shown. The name of the node is "Smiles" and the account is set to 5.


Use case: User starts studying a topic and gets smiles


The user gets another 5 smiles for starting a topic. 4 things have to be done:


Create a relation between user and topic called "studies"


Create a relation called account which holds the sum of all smiles the user has earned using this topic.


Create another relation called earned which indicates the context the smiles were earned. In this case, the smiles were earned when the user started with the topic.


Update the user’s overall smiles account. '


This proves, that designing the model with graphs is quite straight forward. Ideally the nodes and relations are named in a way, that the queries can be read like sentences in natural language.


Of course these steps result in a "net of nodes":


// create/update topic relation
MATCH (u:User {name:'Mat'})
WITH u
MATCH (t:Topic {name:'Core Values'})
MERGE (u)-[r:STUDIES]->(t)
SET r.stage='start';

// add smiles account to each topic # update the overall smiles
MATCH (:User {name:'Mat'})-[:HAS]->(smiles)
WITH smiles
MATCH (t:Topic {name:'Core Values'})
MERGE (smiles)-[a:ACCOUNT]->(t)
ON CREATE SET a.account=0;

// create smiles relation to topic, indicating how many smiles are earned by starting
MATCH (:User {name:'Mat'})-[:HAS]->(smiles)-[a:ACCOUNT]->(t:Topic {name:'Core Values'})
MERGE (smiles)-[earned:EARNED {reason:'start'}]->(t)
ON CREATE SET
  earned.count = 5,
  a.account = a.account + earned.count,
  smiles.account = smiles.account + earned.count;


Use case: User chooses a learning resource


The user chooses a learning resource. The following steps are in pseudo language:


(user)-[studies stage:learn]→(topic) … update the stage to *learn*


(user)-[learns]→(resource)  … new relation


(smiles)-[earned count:5]→(resource)  …  create smiles relation and update smiles accounts


// create the learning resources for this topic
MATCH (t:Topic {name:'Core Values'})
WITH t MERGE (t)-[:HAS]->(:Resource:Book {name: 'Words to Live by: 5', likes:5, link:'https://amazon.com'})
WITH t MERGE (t)-[:HAS]->(:Resource:Blog {name:'How to live with...', likes:-3, link:'http...'});

// 1. update the stage to *learn*
MATCH (u:User {name:'Mat'})-[r:STUDIES]->(t:Topic {name:'Core Values'})
SET r.stage='learn';

// 2. create/update resource relation
MATCH (u:User {name:'Mat'})
WITH u
MATCH (resource:Resource {name:'Words to Live by: 5'})
MERGE (u)-[r:LEARNS]->(resource);

// 3. create smiles relation and update smiles accounts
MATCH
  (u:User {name:'Mat'})-[:HAS]->(smiles)-[account:ACCOUNT]->(t:Topic {name:'Core Values'}),
  (u)-[:LEARNS]->(resource:Resource {name:'Words to Live by: 5'})
MERGE (smiles)-[earned:EARNED {reason:'learn'}]->(resource)
ON CREATE SET
  earned.count = 5,
  account.account = account.account + earned.count,
  smiles.account = smiles.account + earned.count;


On the screenshot of the Neo4j Browser you see the learning resource in yellow. The user 'Mat' (red node) 'learns' the resource. The relation between the smiles account (grey) and the resource (yellow) is selected (blue). On the bottom of the browser the properties of the relation can be seen: id:234 reason:learn count:5


Use case: Rating of a learning resource


Neo4j has a long list of customers (https://neo4j.com/customers/) and some used it for recommendations, like Walmart (https://neo4j.com/case-studies/walmart/).


A query could look like this (pseudo language):


count (:User)-[like]→(:Resource)


If you want to count only the likes of a resource in the context of a topic (pseudo language):


count (:User)-[like]→(:Resource)←[has]-(:Topic)


Of course the calculation can be more elaborated if needed. For example it could be the ratings of users are weighted higher, if they have finished the topic successfully. Here again shown in a pseudo language:


(:Resource)←[like]-(:User)-[studies stage:finished result:success]→(:Topic)


For our prototype model we keep it simple. We let the user rate +1 or a -1 and accumulate all likes as a property of the resource.


MATCH (u:User {name:'Mat'})-[:LEARNS]->(resource:Resource {name: 'Words to Live by: 5'})
MERGE (u)-[like:LIKE {rate:1}]->(resource)
ON CREATE SET
  resource.likes = resource.likes + like.rate
RETURN u, resource, like


The resource had 5 likes before and now 6.


Conclusion


The first impression is very good. The Neo4j seems to be a good candidate for implementing the database for this app.


Fun and good learning curve


I learned the Neo4j in one week and it was really fun. The concept, tools and documentation are great and I haven’t experienced the frustration I had when I learned for example JavaScript react or docker because I didn’t hit a blocker where only a friend’s advice helped me to solve it.


More elaborated setup


Of course there are many questions regarding performance and load, as well as server cluster setups that have to be evaluated. It’s likely that other databases and caches have to be added to the setup.


API and Microservices


Some programming languages for querying Neo4J are available: Python, Java, JavaScript and more. See: https://neo4j.com/developer/language-guides/


Of course the database could be connected directly by the (web) server. But usually you want to enrich the data with some business logic. Example: adding smiles depending on certain conditions or making recommendations. In order to reduce complexity and dependencies you encapsulate the business logic and put it into microservices. So developers building the microservices will enjoy working with the node database and they will be responsible for building good APIs and make the data easy accessible for everyone.


Nerd Playground


Playing around with Neo4j is easy aslong you are familiar with git. Follow the Local installation


A 'real' installation is also possible. You want this if you want to use neo4j shell tools. If you are familiar with docker, follow the Docker installation.


Local installation


Download community edition and install on your computer: https://neo4j.com/download/community-edition/


Clone git repository: https://github.com/gitmathub/agilityscales.git


Copy database files into your installation folder (something like: cp db/neostore* ~/neo4j/)


Start program Neo4j follow displayed instructions and have fun :-)


Docker Installation


TODO: Make screenshots for each step


Clone git repository: https://github.com/gitmathub/agilityscales.git


You retrieve a folder called: db. You are going to share this folder later in Docker.


Install Docker on your computer https://www.docker.com/community-edition


Start Docker


Go to Docker preferences and share your db folder in file sharing


Open Kitematic (see Docker menus)


In Kitematic: Login to dockerhub.com


Select Create button and search for docker image: neo4j


Select official neo4j image and it get downloaded and starts


Go to Settings tab and then to Ports tab.


Change localhost ports to 7474 and 7687 (same like docker port)


Go to Volumes tab and change local folder to your shared db folder (see Docker preferences file sharing)


Press the Restart button in Kitematik


Open your browser with http://localhost:7474 and have fun :-)


If you don’t want to use your homebrewn docker, you’ll find some help in the git repository in the docker folder.