Last active
October 31, 2019 08:58
-
-
Save feststelltaste/95c11bee82f9ceda4cdef587d1cf7eec to your computer and use it in GitHub Desktop.
Checking the modularization based on changes (3D Version)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# Introduction\n", | |
| "In my [previous blog post](https://www.feststelltaste.de/checking-the-modularization-of-software-systems-by-analyzing-co-changing-source-code-files/), we looked at the similarity within and across modules by only looking at the change data of each source code file.\n", | |
| "\n", | |
| "In this analysis, we use same data analysis approach, but visualize the result in a 3D scatter plot." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# Data Wrangling\n", | |
| "We just repeat the stuff explained in the mentioned blog post. The only difference is, that we are going from a 2D representation of the distance matrix to a 3D representation." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 103, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "<div>\n", | |
| "<style scoped>\n", | |
| " .dataframe tbody tr th:only-of-type {\n", | |
| " vertical-align: middle;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe tbody tr th {\n", | |
| " vertical-align: top;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe thead th {\n", | |
| " text-align: right;\n", | |
| " }\n", | |
| "</style>\n", | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>x</th>\n", | |
| " <th>y</th>\n", | |
| " <th>z</th>\n", | |
| " <th>module</th>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>file</th>\n", | |
| " <th></th>\n", | |
| " <th></th>\n", | |
| " <th></th>\n", | |
| " <th></th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>backend/src/main/java/at/dropover/comment/boundary/AddCommentRequestModel.java</th>\n", | |
| " <td>0.204507</td>\n", | |
| " <td>-0.500405</td>\n", | |
| " <td>0.328621</td>\n", | |
| " <td>comment</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>backend/src/main/java/at/dropover/comment/boundary/ChangeCommentRequestModel.java</th>\n", | |
| " <td>0.420250</td>\n", | |
| " <td>-0.287899</td>\n", | |
| " <td>0.239667</td>\n", | |
| " <td>comment</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>backend/src/main/java/at/dropover/comment/boundary/CommentData.java</th>\n", | |
| " <td>0.384984</td>\n", | |
| " <td>-0.418287</td>\n", | |
| " <td>0.185055</td>\n", | |
| " <td>comment</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>backend/src/main/java/at/dropover/comment/boundary/GetCommentRequestModel.java</th>\n", | |
| " <td>0.305661</td>\n", | |
| " <td>-0.373012</td>\n", | |
| " <td>0.299084</td>\n", | |
| " <td>comment</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>backend/src/main/java/at/dropover/comment/boundary/GetCommentResponseModel.java</th>\n", | |
| " <td>0.176830</td>\n", | |
| " <td>-0.580814</td>\n", | |
| " <td>0.123574</td>\n", | |
| " <td>comment</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>\n", | |
| "</div>" | |
| ], | |
| "text/plain": [ | |
| " x y \\\n", | |
| "file \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.204507 -0.500405 \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.420250 -0.287899 \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.384984 -0.418287 \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.305661 -0.373012 \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.176830 -0.580814 \n", | |
| "\n", | |
| " z module \n", | |
| "file \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.328621 comment \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.239667 comment \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.185055 comment \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.299084 comment \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.123574 comment " | |
| ] | |
| }, | |
| "execution_count": 103, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "import pandas as pd\n", | |
| "from sklearn.metrics.pairwise import cosine_distances\n", | |
| "from sklearn.manifold import MDS\n", | |
| "import numpy as np\n", | |
| "from matplotlib import cm\n", | |
| "from matplotlib.colors import rgb2hex\n", | |
| "import ipyvolume as ipv\n", | |
| "\n", | |
| "# read, filter and prepare data\n", | |
| "git_log = pd.read_csv(\"https://git.io/Jez2h\")\n", | |
| "prod_code = git_log.copy()\n", | |
| "prod_code = prod_code[prod_code.file.str.endswith(\".java\")]\n", | |
| "prod_code = prod_code[prod_code.file.str.startswith(\"backend/src/main\")]\n", | |
| "prod_code = prod_code[~prod_code.file.str.endswith(\"package-info.java\")]\n", | |
| "prod_code['hit'] = 1\n", | |
| "\n", | |
| "# pivot table to get a change vector per file\n", | |
| "commit_matrix = prod_code.reset_index().pivot_table(\n", | |
| " index='file',\n", | |
| " columns='sha',\n", | |
| " values='hit',\n", | |
| " fill_value=0)\n", | |
| "commit_matrix.iloc[0:5,50:55]\n", | |
| "\n", | |
| "# calculate distance between files based on changes\n", | |
| "dissimilarity_matrix = cosine_distances(commit_matrix)\n", | |
| "\n", | |
| "# break down matrix to 3D representation\n", | |
| "model = MDS(dissimilarity='precomputed', random_state=0, n_components=3)\n", | |
| "dissimilarity_3d = model.fit_transform(dissimilarity_matrix)\n", | |
| "\n", | |
| "# extract module names\n", | |
| "dissimilarity_3d_df = pd.DataFrame(\n", | |
| " dissimilarity_3d,\n", | |
| " index=commit_matrix.index,\n", | |
| " columns=[\"x\", \"y\", \"z\"])\n", | |
| "dissimilarity_3d_df['module'] = dissimilarity_3d_df.index.str.split(\"/\").str[6]\n", | |
| "dissimilarity_3d_df.head()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# Visualization\n", | |
| "So this part is new: We brew a color for each module." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 104, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "<div>\n", | |
| "<style scoped>\n", | |
| " .dataframe tbody tr th:only-of-type {\n", | |
| " vertical-align: middle;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe tbody tr th {\n", | |
| " vertical-align: top;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe thead th {\n", | |
| " text-align: right;\n", | |
| " }\n", | |
| "</style>\n", | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>x</th>\n", | |
| " <th>y</th>\n", | |
| " <th>z</th>\n", | |
| " <th>module</th>\n", | |
| " <th>color</th>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>file</th>\n", | |
| " <th></th>\n", | |
| " <th></th>\n", | |
| " <th></th>\n", | |
| " <th></th>\n", | |
| " <th></th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>backend/src/main/java/at/dropover/comment/boundary/AddCommentRequestModel.java</th>\n", | |
| " <td>0.204507</td>\n", | |
| " <td>-0.500405</td>\n", | |
| " <td>0.328621</td>\n", | |
| " <td>comment</td>\n", | |
| " <td>[0.6196078431372549, 0.00392156862745098, 0.25...</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>backend/src/main/java/at/dropover/comment/boundary/ChangeCommentRequestModel.java</th>\n", | |
| " <td>0.420250</td>\n", | |
| " <td>-0.287899</td>\n", | |
| " <td>0.239667</td>\n", | |
| " <td>comment</td>\n", | |
| " <td>[0.6196078431372549, 0.00392156862745098, 0.25...</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>backend/src/main/java/at/dropover/comment/boundary/CommentData.java</th>\n", | |
| " <td>0.384984</td>\n", | |
| " <td>-0.418287</td>\n", | |
| " <td>0.185055</td>\n", | |
| " <td>comment</td>\n", | |
| " <td>[0.6196078431372549, 0.00392156862745098, 0.25...</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>backend/src/main/java/at/dropover/comment/boundary/GetCommentRequestModel.java</th>\n", | |
| " <td>0.305661</td>\n", | |
| " <td>-0.373012</td>\n", | |
| " <td>0.299084</td>\n", | |
| " <td>comment</td>\n", | |
| " <td>[0.6196078431372549, 0.00392156862745098, 0.25...</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>backend/src/main/java/at/dropover/comment/boundary/GetCommentResponseModel.java</th>\n", | |
| " <td>0.176830</td>\n", | |
| " <td>-0.580814</td>\n", | |
| " <td>0.123574</td>\n", | |
| " <td>comment</td>\n", | |
| " <td>[0.6196078431372549, 0.00392156862745098, 0.25...</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>\n", | |
| "</div>" | |
| ], | |
| "text/plain": [ | |
| " x y \\\n", | |
| "file \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.204507 -0.500405 \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.420250 -0.287899 \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.384984 -0.418287 \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.305661 -0.373012 \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.176830 -0.580814 \n", | |
| "\n", | |
| " z module \\\n", | |
| "file \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.328621 comment \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.239667 comment \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.185055 comment \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.299084 comment \n", | |
| "backend/src/main/java/at/dropover/comment/bound... 0.123574 comment \n", | |
| "\n", | |
| " color \n", | |
| "file \n", | |
| "backend/src/main/java/at/dropover/comment/bound... [0.6196078431372549, 0.00392156862745098, 0.25... \n", | |
| "backend/src/main/java/at/dropover/comment/bound... [0.6196078431372549, 0.00392156862745098, 0.25... \n", | |
| "backend/src/main/java/at/dropover/comment/bound... [0.6196078431372549, 0.00392156862745098, 0.25... \n", | |
| "backend/src/main/java/at/dropover/comment/bound... [0.6196078431372549, 0.00392156862745098, 0.25... \n", | |
| "backend/src/main/java/at/dropover/comment/bound... [0.6196078431372549, 0.00392156862745098, 0.25... " | |
| ] | |
| }, | |
| "execution_count": 104, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "modules = dissimilarity_3d_df[['module']].drop_duplicates()\n", | |
| "rgb_colors = [x for x in cm.Spectral(np.linspace(0,1,len(modules)))]\n", | |
| "modules['color'] = rgb_colors\n", | |
| "modules = modules.set_index(\"module\", drop=True)\n", | |
| "dissimilarity_3d_df['color'] = dissimilarity_3d_df['module'].map(modules['color'].to_dict())\n", | |
| "dissimilarity_3d_df.head()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "And then, we visualize this data with [ipyvolume](https://ipyvolume.readthedocs.io/en/latest/index.html#)." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 106, | |
| "metadata": {}, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "application/vnd.jupyter.widget-view+json": { | |
| "model_id": "1bd327b383cc46a8a12aecb8c6afdd6b", | |
| "version_major": 2, | |
| "version_minor": 0 | |
| }, | |
| "text/plain": [ | |
| "VBox(children=(Figure(camera=PerspectiveCamera(fov=46.0, position=(0.0, 0.0, 2.0), quaternion=(0.0, 0.0, 0.0, …" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "x = dissimilarity_3d_df['x']\n", | |
| "y = dissimilarity_3d_df['y']\n", | |
| "z = dissimilarity_3d_df['z']\n", | |
| "color = dissimilarity_3d_df['color'].values.tolist()\n", | |
| "ipv.quickscatter(x, y, z, color=color, size=7, marker=\"sphere\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# Summary\n", | |
| "That's it! A nice 3D representation of our little software program.\n", | |
| "\n", | |
| "\n", | |
| "\n", | |
| "We see some spheres with the same color near to each other. These modules the were change together in the first place. But there are also some mixed up areas. The reasons for this are explained [here](https://www.feststelltaste.de/checking-the-modularization-of-software-systems-by-analyzing-co-changing-source-code-files/#Modules-%22comment%22,-%22framework%22-and-%22site%22)." | |
| ] | |
| } | |
| ], | |
| "metadata": { | |
| "kernelspec": { | |
| "display_name": "Python 3", | |
| "language": "python", | |
| "name": "python3" | |
| }, | |
| "language_info": { | |
| "codemirror_mode": { | |
| "name": "ipython", | |
| "version": 3 | |
| }, | |
| "file_extension": ".py", | |
| "mimetype": "text/x-python", | |
| "name": "python", | |
| "nbconvert_exporter": "python", | |
| "pygments_lexer": "ipython3", | |
| "version": "3.6.4" | |
| }, | |
| "varInspector": { | |
| "cols": { | |
| "lenName": 16, | |
| "lenType": 16, | |
| "lenVar": 40 | |
| }, | |
| "kernels_config": { | |
| "python": { | |
| "delete_cmd_postfix": "", | |
| "delete_cmd_prefix": "del ", | |
| "library": "var_list.py", | |
| "varRefreshCmd": "print(var_dic_list())" | |
| }, | |
| "r": { | |
| "delete_cmd_postfix": ") ", | |
| "delete_cmd_prefix": "rm(", | |
| "library": "var_list.r", | |
| "varRefreshCmd": "cat(var_dic_list()) " | |
| } | |
| }, | |
| "types_to_exclude": [ | |
| "module", | |
| "function", | |
| "builtin_function_or_method", | |
| "instance", | |
| "_Feature" | |
| ], | |
| "window_display": false | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 2 | |
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # This are the requirements for the notebook above | |
| # Especially useful when running the notebooks with mybinder.org | |
| pandas==0.23.4 | |
| matplotlib | |
| sklearn | |
| ipyvolume |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment