Fitness Function for Detecting Cyclic Package Dependencies with Jupyter, jQAssistant and Neo4j

apt.txt

openjdk-8-jdk

fitness_function_cyclic_dependencies.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fitness Function for Detecting Cyclic Package Dependencies\n",
    "\n",
    "## Context\n",
    "Developers may have a hard time to learn from past failures or bugs in the application due to team autonomy, constant pressure to deliver and a heavily distributed code base. How can they achieve that past errors are avoided in future development activities?\n",
    "\n",
    "## Idea\n",
    "Neal Ford, Rebecca Parsons and Patrick Kua are proposing fitness functions  in their book \"Building Evolutionary Architectures: Support Constant Change\" that check for certain qualities in software applications.\n",
    "\n",
    "## Use Case\n",
    "\n",
    "In this example, we want to find possible cyclic package dependencies in a small Java code base.\n",
    "\n",
    "## Implementation\n",
    "In this notebook, I take this idea to show you a possible implementation of fitness functions using [Jupyter](https://jupyter.org/) notebooks, the structural code analysis scanner [jQAssistant](https://jqassistant.org/) and the graph database [Neo4j](https://neo4j.com/).\n",
    "\n",
    "\n",
    "\n",
    "Note: There is an interactive version of this notebook. It contains all the scanned software data with it and sets up a running Neo4j database instance in the background. This means that you can play around with the queries below by yourself! All this is made possible by the fantastic [BinderHub](https://binderhub.readthedocs.io/en/latest/) project!\n",
    "\n",
    "## Analysis\n",
    "We are using the Cypher notebook extension to directly execute Cypher query (the query language for Neo4j) in  this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext cypher\n",
    "%env NEO4J_URL=http://neo4j:neo4j@localhost:7474/db/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we are executing the actual Cypher query. This query returns all Java packages in the codebase that contain cycles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4 rows affected.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "    <tr>\n",
       "        <th>Package</th>\n",
       "        <th>Cycle</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>{&#x27;fqn&#x27;: &#x27;org.springframework.samples.petclinic.model&#x27;, &#x27;fileName&#x27;: &#x27;/org/springframework/samples/petclinic/model&#x27;, &#x27;name&#x27;: &#x27;model&#x27;}</td>\n",
       "        <td>[&#x27;org.springframework.samples.petclinic.repository&#x27;, &#x27;org.springframework.samples.petclinic.model&#x27;]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>{&#x27;fileName&#x27;: &#x27;/org/springframework/samples/petclinic/repository&#x27;, &#x27;fqn&#x27;: &#x27;org.springframework.samples.petclinic.repository&#x27;, &#x27;name&#x27;: &#x27;repository&#x27;}</td>\n",
       "        <td>[&#x27;org.springframework.samples.petclinic.model&#x27;, &#x27;org.springframework.samples.petclinic.repository&#x27;]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>{&#x27;fileName&#x27;: &#x27;/org/springframework/samples/petclinic/service&#x27;, &#x27;fqn&#x27;: &#x27;org.springframework.samples.petclinic.service&#x27;, &#x27;name&#x27;: &#x27;service&#x27;}</td>\n",
       "        <td>[&#x27;org.springframework.samples.petclinic.web&#x27;, &#x27;org.springframework.samples.petclinic.service&#x27;]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>{&#x27;fqn&#x27;: &#x27;org.springframework.samples.petclinic.web&#x27;, &#x27;fileName&#x27;: &#x27;/org/springframework/samples/petclinic/web&#x27;, &#x27;name&#x27;: &#x27;web&#x27;}</td>\n",
       "        <td>[&#x27;org.springframework.samples.petclinic.service&#x27;, &#x27;org.springframework.samples.petclinic.web&#x27;]</td>\n",
       "    </tr>\n",
       "</table>"
      ],
      "text/plain": [
       "[[{'fqn': 'org.springframework.samples.petclinic.model',\n",
       "   'fileName': '/org/springframework/samples/petclinic/model',\n",
       "   'name': 'model'},\n",
       "  ['org.springframework.samples.petclinic.repository',\n",
       "   'org.springframework.samples.petclinic.model']],\n",
       " [{'fileName': '/org/springframework/samples/petclinic/repository',\n",
       "   'fqn': 'org.springframework.samples.petclinic.repository',\n",
       "   'name': 'repository'},\n",
       "  ['org.springframework.samples.petclinic.model',\n",
       "   'org.springframework.samples.petclinic.repository']],\n",
       " [{'fileName': '/org/springframework/samples/petclinic/service',\n",
       "   'fqn': 'org.springframework.samples.petclinic.service',\n",
       "   'name': 'service'},\n",
       "  ['org.springframework.samples.petclinic.web',\n",
       "   'org.springframework.samples.petclinic.service']],\n",
       " [{'fqn': 'org.springframework.samples.petclinic.web',\n",
       "   'fileName': '/org/springframework/samples/petclinic/web',\n",
       "   'name': 'web'},\n",
       "  ['org.springframework.samples.petclinic.service',\n",
       "   'org.springframework.samples.petclinic.web']]]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%cypher\n",
    "MATCH\n",
    "    (p1:Package)-[:DEPENDS_ON]->(p2:Package),\n",
    "    path=shortestPath((p2)-[:DEPENDS_ON*]->(p1))\n",
    "WHERE\n",
    "    p1<>p2\n",
    "RETURN\n",
    "    p1 AS Package, EXTRACT(p IN nodes(path) | p.fqn) AS Cycle\n",
    "ORDER BY\n",
    "    Package.fqn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "After executing the cell above (with `Strg` + `Enter`), we get a table with cylic dependencies:\n",
    "\n",
    "- In the left column, we see the graph's nodes that are in a cycle\n",
    "- In the right column, we see the paths of the package cycle.\n",
    "\n",
    "With this result, we can think about possible dependency breaking mechanisms to get rid of these unwanted, cyclic dependencies."
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

postBuild

#!/bin/bash

wget https://github.com/feststelltaste/software-analytics/raw/master/notebooks/data/spring-petclinic/petclinic_joa_neo4j.dump

wget http://dist.neo4j.org/neo4j-community-3.5.8-unix.tar.gz
tar -xvf neo4j-community-3.5.8-unix.tar.gz
echo "dbms.connectors.default_listen_address=0.0.0.0" >> neo4j-community-3.5.8/conf/neo4j.conf
neo4j-community-3.5.8/bin/neo4j-admin set-initial-password neo4j
mkdir -p neo4j-community-3.5.8/data/databases/graph.db
neo4j-community-3.5.8/bin/neo4j-admin load --from=petclinic_joa_neo4j.dump --force

requirements.txt

jupyter-server-proxy
ipython-cypher

start

#!/bin/bash
set -eux

neo4j-community-3.5.8/bin/neo4j start

exec "$@"