Last active
September 16, 2015 16:10
-
-
Save vitillo/fb3089f78ed9b93b442c to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"nbformat_minor": 0, "cells": [{"execution_count": 1, "cell_type": "code", "source": "import binascii\nimport pandas as pd\n\nfrom operator import attrgetter, itemgetter\nfrom moztelemetry import get_pings, get_pings_properties, get_one_ping_per_client, get_clients_history\nfrom collections import defaultdict\nfrom __future__ import division\n\n%pylab inline", "outputs": [{"output_type": "stream", "name": "stdout", "text": "Populating the interactive namespace from numpy and matplotlib\n"}], "metadata": {"scrolled": true, "collapsed": false, "trusted": true}}, {"execution_count": 2, "cell_type": "code", "source": "sc.defaultParallelism", "outputs": [{"execution_count": 2, "output_type": "execute_result", "data": {"text/plain": "80"}, "metadata": {}}], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Get all main pings for a set of recent build-ids:", "cell_type": "markdown", "metadata": {}}, {"execution_count": 3, "cell_type": "code", "source": "build_ids = (\"20150820000000\", \"20150827999999\")\n\npings = get_pings(sc,\n app=\"Firefox\",\n channel=\"aurora\",\n build_id=build_ids,\n doc_type=\"main\",\n schema=\"v4\")\n\ncrashes = get_pings(sc,\n app=\"Firefox\",\n channel=\"aurora\",\n build_id=build_ids,\n doc_type=\"crash\",\n schema=\"v4\")", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Take a subset of clients:", "cell_type": "markdown", "metadata": {}}, {"execution_count": 4, "cell_type": "code", "source": "def sample(ping):\n client_id = ping.get(\"clientId\", None)\n return client_id and binascii.crc32(ping[\"clientId\"]) % 100 < 10\n\nsampled_pings = pings.filter(sample)\nsampled_crashes = crashes.filter(sample)", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 5, "cell_type": "code", "source": "crashes_by_client = sampled_crashes.map(lambda c: (c[\"clientId\"], c[\"meta\"])).groupByKey().collectAsMap()", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Get a subset of fields:", "cell_type": "markdown", "metadata": {}}, {"execution_count": 6, "cell_type": "code", "source": "subset = get_pings_properties(sampled_pings, [\"clientId\",\n \"meta/documentId\",\n \"meta/submissionDate\",\n \"meta/creationTimestamp\",\n \"environment/system/os/name\",\n \"payload/info/reason\",\n \"payload/info/sessionId\",\n \"payload/info/subsessionId\",\n \"payload/info/previousSessionId\",\n \"payload/info/previousSubsessionId\",\n \"payload/info/subsessionCounter\",\n \"payload/info/profileSubsessionCounter\",\n \"payload/simpleMeasurements/firstPaint\",\n \"payload/simpleMeasurements/savedPings\",\n \"payload/simpleMeasurements/uptime\",\n \"payload/histograms/STARTUP_CRASH_DETECTED\",\n \"payload/histograms/E10S_AUTOSTART\",\n \"environment/settings/e10sEnabled\"]).\\\n filter(lambda p: p[\"payload/info/profileSubsessionCounter\"] is not None)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Group fragments by client and dedupe by documentId:", "cell_type": "markdown", "metadata": {}}, {"execution_count": 7, "cell_type": "code", "source": "def dedupe_and_sort(group):\n key, history = group\n \n seen = set()\n result = []\n \n for fragment in history:\n id = fragment[\"meta/documentId\"]\n if id in seen:\n continue\n \n seen.add(id)\n result.append(fragment)\n \n result.sort(key=itemgetter(\"payload/info/profileSubsessionCounter\"))\n return result\n\ngrouped = subset.groupBy(lambda x: x[\"clientId\"]).map(dedupe_and_sort).collect()", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "**< Digression>** What's the percentage of clients that have at least one pair of fragments with different documentIds but the same profileSubsessionCounter?", "cell_type": "markdown", "metadata": {}}, {"execution_count": 8, "cell_type": "code", "source": "def duplicate_pssc(grouped):\n dupes = 0\n dupe_clients = set()\n\n for history in grouped:\n counts = defaultdict(int)\n\n for fragment in history:\n key = fragment[\"payload/info/profileSubsessionCounter\"]\n counts[key] += 1\n\n for _, v in counts.iteritems():\n if v > 1:\n dupes += 1\n dupe_clients.add(history[0][\"clientId\"])\n break\n\n print 100.0*dupes/len(grouped)\n return dupe_clients\n \ndupe_clients = duplicate_pssc(grouped)", "outputs": [{"output_type": "stream", "name": "stdout", "text": "1.72101917412\n"}], "metadata": {"scrolled": true, "collapsed": false, "trusted": true}}, {"source": "**< /Digression\\>** Let's remove those clients to be safe.", "cell_type": "markdown", "metadata": {}}, {"execution_count": 9, "cell_type": "code", "source": "dd_grouped = filter(lambda h: h[0][\"clientId\"] not in dupe_clients, grouped)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Let's take only clients with e10s enabled:", "cell_type": "markdown", "metadata": {}}, {"execution_count": 10, "cell_type": "code", "source": "def has_e10s(grouped):\n return grouped[0][\"environment/settings/e10sEnabled\"]", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 11, "cell_type": "code", "source": "dd_grouped = filter(has_e10s, grouped)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Number of clients:", "cell_type": "markdown", "metadata": {}}, {"execution_count": 12, "cell_type": "code", "source": "len(dd_grouped)", "outputs": [{"execution_count": 12, "output_type": "execute_result", "data": {"text/plain": "14971"}, "metadata": {}}], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Given the set of chain breaks, how many of them are due to missing starting/ending fragments?", "cell_type": "markdown", "metadata": {}}, {"execution_count": 13, "cell_type": "code", "source": "class AdjacentBreaks:\n def __init__(self):\n self.missing_total = 0\n self.missing_start = 0\n self.missing_end = 0\n self.missing_both = 0\n self.crashed_prev = 0\n self.reason = defaultdict(int)\n \n \n def process(self, prev, curr): \n if prev[\"payload/info/sessionId\"] == curr[\"payload/info/previousSessionId\"]:\n # Ignore fake missing fragments? See IncrementError class\n if prev[\"payload/info/reason\"] in (\"aborted-session\", \"shutdown\") and \\\n curr[\"payload/info/subsessionCounter\"] == 1:\n return\n \n self.missing_total += 1\n self.reason[\"{} -> {}\".format(prev[\"payload/info/reason\"], curr[\"payload/info/reason\"])] += 1\n \n # Are there missing starting fragments?\n missing_start = curr[\"payload/info/subsessionCounter\"] != 1\n \n # Are there missing ending fragments?\n missing_end = prev[\"payload/info/reason\"] not in (\"aborted-session\", \"shutdown\")\n \n if missing_start and missing_end:\n self.missing_both += 1\n elif missing_start:\n self.missing_start += 1\n elif missing_end:\n self.missing_end += 1\n \n self.crashed_prev += curr[\"payload/histograms/STARTUP_CRASH_DETECTED\"] or has_crash_ping(prev, curr)\n \n \n def stats(self, total):\n print \"ADJACENT SESSIONS STATS\"\n print \"{:5.2f}% of edges have fragments missing\".format(100*self.missing_total/total)\n print \"{:5.2f}% of edges are missing one or more starting fragments\".format(100*self.missing_start/total)\n print \"{:5.2f}% of edges are missing one or more ending fragments\".format(100*self.missing_end/total)\n print \"{:5.2f}% of edges are missing both starting and ending fragments\".format(100*self.missing_both/total)\n print \"{:5.2f}% of edges have a crash in-between\".format(100*self.crashed_prev/self.missing_total)\n \n print \"\"\n print \"Reason distribution:\"\n print dict(self.reason)\n print \"\"\n \n\nclass WithinBreaks:\n def __init__(self):\n self.missing_total = 0\n self.crashed_prev = 0\n self.reason = defaultdict(int)\n \n \n def process(self, prev, curr):\n if prev[\"payload/info/sessionId\"] == curr[\"payload/info/sessionId\"]:\n self.missing_total += 1\n self.reason[\"{} -> {}\".format(prev[\"payload/info/reason\"], curr[\"payload/info/reason\"])] += 1\n self.crashed_prev += curr[\"payload/histograms/STARTUP_CRASH_DETECTED\"] or has_crash_ping(prev, curr)\n \n \n def stats(self, total):\n print \"WITHIN SESSIONS STATS\"\n print \"{:5.2f}% of edges have fragments missing\".format(100*self.missing_total/total)\n print \"{:5.2f}% of edges have a crash in-between\".format(100*self.crashed_prev/self.missing_total)\n \n print \"\"\n print \"Reason distribution:\"\n print dict(self.reason)\n print \"\"\n\n \nclass NonAdjacentBreaks:\n def __init__(self):\n self.missing_total = 0\n self.reason = defaultdict(int)\n self.difference = defaultdict(int)\n self.crashed_prev = 0\n \n \n def process(self, prev, curr):\n if prev[\"payload/info/sessionId\"] != curr[\"payload/info/sessionId\"] and \\\n prev[\"payload/info/sessionId\"] != curr[\"payload/info/previousSessionId\"]:\n self.missing_total += 1\n self.reason[\"{} -> {}\".format(prev[\"payload/info/reason\"], curr[\"payload/info/reason\"])] += 1\n self.difference[curr[\"payload/info/profileSubsessionCounter\"] - prev[\"payload/info/profileSubsessionCounter\"]] += 1\n self.crashed_prev += curr[\"payload/histograms/STARTUP_CRASH_DETECTED\"] or has_crash_ping(prev, curr)\n \n \n def stats(self, total):\n print \"NON-ADJACENT SESSIONS STATS\"\n print \"{:5.2f}% of edges have fragments missing\".format(100*self.missing_total/total)\n print \"{:5.2f}% of edges have a crash in-between\".format(100*self.crashed_prev/self.missing_total)\n print \"\"\n \n print \"Reason distribution:\"\n print dict(self.reason)\n print \"\"\n \n print \"Difference distribution:\"\n dist = pd.Series(self.difference)\n dist.sort_index()\n print dist\n \n print \"\"\n \n \nclass IncrementError:\n def __init__(self):\n self.errors_total = 0\n self.reason = defaultdict(int)\n \n def process(self, prev, curr):\n if prev[\"payload/info/sessionId\"] == curr[\"payload/info/previousSessionId\"] and \\\n prev[\"payload/info/reason\"] in (\"aborted-session\", \"shutdown\") and \\\n curr[\"payload/info/subsessionCounter\"] == 1:\n self.errors_total += 1\n self.reason[\"{} -> {}\".format(prev[\"payload/info/reason\"], curr[\"payload/info/reason\"])] += 1\n \n def stats(self, total):\n print \"PROFILESUBSESSIONCOUNTER INCREMENT ERRORS\"\n print \"{:5.2f}% of edges have a mismatching profileSubsessionCounter\".format(100*self.errors_total/total)\n print \"\"\n \n print \"Reason distribution:\"\n print dict(self.reason) \n print \"\"\n\n\ndef has_crash_ping(prev, curr):\n client_id = prev[\"clientId\"]\n client_crashes = crashes_by_client.get(client_id, None)\n \n if client_crashes:\n for crash in list(client_crashes):\n if crash[\"creationTimestamp\"] >= prev[\"meta/creationTimestamp\"] and \\\n crash[\"creationTimestamp\"] <= curr[\"meta/creationTimestamp\"]:\n return True\n \n return False\n \n\ndef missing(grouped):\n broken_clients = set()\n correct_clients = set()\n num_broken_chains = 0\n num_crashed = 0\n total_edges = 0\n \n adjacent_breaks = AdjacentBreaks()\n within_breaks = WithinBreaks()\n non_adjacent_breaks = NonAdjacentBreaks()\n increment_errors = IncrementError()\n \n for history in grouped:\n correct_clients.add(history[0][\"clientId\"])\n\n for i in range(1, len(history)):\n prev_fragment = history[i - 1]\n prev_pss_counter = prev_fragment[\"payload/info/profileSubsessionCounter\"]\n \n curr_fragment = history[i]\n current_pss_counter = curr_fragment[\"payload/info/profileSubsessionCounter\"]\n\n num_crashed += curr_fragment[\"payload/histograms/STARTUP_CRASH_DETECTED\"] or has_crash_ping(prev_fragment, curr_fragment)\n total_edges += 1\n\n # Is a fragment missing?\n if prev_pss_counter + 1 != current_pss_counter:\n broken_clients.add(curr_fragment[\"clientId\"])\n num_broken_chains += 1\n \n adjacent_breaks.process(prev_fragment, curr_fragment)\n within_breaks.process(prev_fragment, curr_fragment)\n non_adjacent_breaks.process(prev_fragment, curr_fragment)\n increment_errors.process(prev_fragment, curr_fragment)\n \n correct_clients = correct_clients.difference(broken_clients)\n \n print \"GENERAL STATS\"\n print \"{:5.2f}% clients have a broken session chain\".format(100*len(broken_clients)/len(grouped)) \n print \"{:5.2f}% of clients with a missing fragment experienced at least one crash\".format(100*len(broken_clients.intersection(crashes_by_client.keys()))/len(broken_clients))\n print \"{:5.2f}% of clients without a missing fragment experienced at least one crash\".format(100*len(correct_clients.intersection(crashes_by_client.keys()))/len(correct_clients))\n print \"{:5.2f}% of edges have a crash in-between\\n\".format(100*num_crashed/total_edges)\n \n increment_errors.stats(num_broken_chains)\n adjacent_breaks.stats(num_broken_chains)\n within_breaks.stats(num_broken_chains)\n non_adjacent_breaks.stats(num_broken_chains)\n \nmissing(dd_grouped)", "outputs": [{"output_type": "stream", "name": "stdout", "text": "GENERAL STATS\n 4.90% clients have a broken session chain\n17.05% of clients with a missing fragment experienced at least one crash\n 5.44% of clients without a missing fragment experienced at least one crash\n 1.63% of edges have a crash in-between\n\nPROFILESUBSESSIONCOUNTER INCREMENT ERRORS\n 4.35% of edges have a mismatching profileSubsessionCounter\n\nReason distribution:\n{'aborted-session -> shutdown': 46, 'aborted-session -> environment-change': 11, 'aborted-session -> aborted-session': 14, 'aborted-session -> daily': 16, 'shutdown -> shutdown': 1}\n\nADJACENT SESSIONS STATS\n10.18% of edges have fragments missing\n 5.68% of edges are missing one or more starting fragments\n 4.50% of edges are missing one or more ending fragments\n 0.00% of edges are missing both starting and ending fragments\n 3.88% of edges have a crash in-between\n\nReason distribution:\n{'shutdown -> shutdown': 21, 'aborted-session -> environment-change': 1, 'environment-change -> shutdown': 16, 'environment-change -> environment-change': 10, 'daily -> shutdown': 30, 'aborted-session -> aborted-session': 24, 'environment-change -> aborted-session': 7, 'daily -> daily': 19, 'daily -> aborted-session': 5, 'shutdown -> environment-change': 1, 'environment-change -> daily': 3, 'daily -> environment-change': 1, 'shutdown -> aborted-session': 68}\n\nWITHIN SESSIONS STATS\n 9.29% of edges have fragments missing\n 2.66% of edges have a crash in-between\n\nReason distribution:\n{'aborted-session -> daily': 27, 'aborted-session -> environment-change': 39, 'daily -> shutdown': 1, 'environment-change -> aborted-session': 41, 'daily -> daily': 1, 'daily -> aborted-session': 42, 'aborted-session -> shutdown': 14, 'shutdown -> aborted-session': 23}\n\nNON-ADJACENT SESSIONS STATS\n76.17% of edges have fragments missing\n 9.34% of edges have a crash in-between\n\nReason distribution:\n{'shutdown -> shutdown': 1034, 'environment-change -> environment-change': 58, 'daily -> shutdown': 31, 'shutdown -> daily': 51, 'aborted-session -> daily': 6, 'daily -> daily': 8, 'daily -> aborted-session': 5, 'shutdown -> environment-change': 62, 'aborted-session -> environment-change': 3, 'aborted-session -> aborted-session': 26, 'shutdown -> aborted-session': 86, 'environment-change -> shutdown': 62, 'environment-change -> aborted-session': 9, 'environment-change -> daily': 7, 'aborted-session -> shutdown': 85, 'daily -> environment-change': 8}\n\nDifference distribution:\n0 1075\n2 234\n3 49\n4 23\n5 14\n6 7\n7 4\n8 6\n9 4\n10 7\n11 1\n12 4\n14 3\n15 2\n16 5\n...\n183 1\n192 1\n193 1\n201 1\n204 1\n205 1\n226 1\n241 1\n263 1\n272 1\n278 1\n289 1\n330 1\n406 1\n900 1\nLength: 88, dtype: int64\n\n"}], "metadata": {"scrolled": false, "collapsed": false, "trusted": true}}], "nbformat": 4, "metadata": {"kernelspec": {"display_name": "Python 2", "name": "python2", "language": "python"}, "language_info": {"mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "2.7.9", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython2", "codemirror_mode": {"version": 2, "name": "ipython"}}}} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment