Find a sane thread hangs stack limit

thread_hang_stack_limit.ipynb

{"nbformat_minor": 0, "cells": [{"execution_count": 56, "cell_type": "code", "source": "import datetime as dt\nimport ujson as json\nimport pandas as pd\nimport numpy as np\n\nfrom moztelemetry import get_pings, get_pings_properties, get_one_ping_per_client, get_clients_history", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"source": "Define some utility functions to filter the pings.", "cell_type": "markdown", "metadata": {}}, {"execution_count": 57, "cell_type": "code", "source": "def sample_by_clientId(pings):\n    return pings.filter(lambda p: p.get(\"meta/sampleId\", \"\") < 10)\n\ndef get_interesting_reasons(pings):\n    return pings.filter(lambda p: p.get(\"payload/info/reason\", \"\") in [ \"shutdown\" ])", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"source": "Get the main (shutdown) pings from the past 4 weeks.", "cell_type": "markdown", "metadata": {}}, {"execution_count": 58, "cell_type": "code", "source": "build_ids = (\"20150722000000\", \"20150929999999\")\nlast_weeks = (dt.datetime.now() - dt.timedelta(weeks=4)).strftime(\"%Y%m%d\")\nmain_pings = get_pings(sc, app=\"Firefox\", channel=\"nightly\", build_id=build_ids, submission_date=last_weeks, doc_type=\"main\", schema=\"v4\", fraction=1.0)", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 59, "cell_type": "code", "source": "main_pings.count()", "outputs": [{"execution_count": 59, "output_type": "execute_result", "data": {"text/plain": "169335"}, "metadata": {}}], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Just keep the fields we're interested. Then, further restrict the computation to 10% of the clients.", "cell_type": "markdown", "metadata": {}}, {"execution_count": 60, "cell_type": "code", "source": "subset = get_pings_properties(main_pings, [\"meta/sampleId\",\n                                           \"payload/info/reason\",\n                                           \"payload/threadHangStats\"])\n\nsampled_pings = sample_by_clientId(subset)\nfilteredSubset = get_interesting_reasons(sampled_pings)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 61, "cell_type": "code", "source": "filteredSubset.count()", "outputs": [{"execution_count": 61, "output_type": "execute_result", "data": {"text/plain": "14451"}, "metadata": {}}], "metadata": {"collapsed": false, "trusted": true}}, {"source": "This function finds the depth of each thread hang stack reported in a ping. Each stack is processed by removing repeated \"(chrome script)\" and \"(content script)\" entries.", "cell_type": "markdown", "metadata": {}}, {"execution_count": 62, "cell_type": "code", "source": "def get_stack_depths(ping):\n    threadHangs = ping[\"payload/threadHangStats\"]\n\n    stack_depths = []\n    \n    # Safety?\n    if not threadHangs:\n        return stack_depths\n    \n    # Iterate the threads which got stuck...\n    for thread in threadHangs:\n        if \"hangs\" not in thread:\n            continue\n\n        # ...then through each hang.\n        for hang in thread[\"hangs\"]:\n            if \"stack\" not in hang or len(hang[\"stack\"]) < 1:\n                continue\n                \n            # Get the hang stack, if available.\n            stack = hang[\"stack\"]\n    \n            # Collapse (* script) entries in the stack and get the depth of the new stack.\n            depth = 1\n            for prev, curr in zip(stack, stack[1:]):\n                if (curr == \"(chrome script)\" or curr == \"(content script)\") and prev == curr:\n                    continue\n\n                depth = depth + 1\n            \n            # Append the depth for this stack to the list of stack depths for this ping.\n            stack_depths.append(depth)\n\n    return stack_depths", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Collect the stack depths and summarize the counts.", "cell_type": "markdown", "metadata": {}}, {"execution_count": 63, "cell_type": "code", "source": "stack_lengths = filteredSubset.flatMap(get_stack_depths).collect()\ntest_stacks = pd.Series(stack_lengths)\ntest_stacks.value_counts()", "outputs": [{"execution_count": 63, "output_type": "execute_result", "data": {"text/plain": "3      78511\n2      64200\n4      64002\n5      49545\n6      38353\n7      31287\n8      22330\n1      12988\n9      12539\n10      8197\n11      6966\n12      4655\n13      3290\n15      2257\n14      2136\n16      1759\n17      1066\n19       662\n18       657\n24       402\n20       367\n22       346\n21       288\n23       254\n25       214\n26       159\n27       122\n28       117\n30       110\n29        91\n31        77\n33        46\n32        41\n34        37\n35        16\n36        15\n41        10\n39         9\n45         8\n37         8\n38         7\n40         6\n42         5\n43         4\n44         3\n49         1\n55         1\n81         1\n46         1\n123        1\ndtype: int64"}, "metadata": {}}], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Find a stack depth which would successfully represent most of the stacks (95th percentile).", "cell_type": "markdown", "metadata": {}}, {"execution_count": 64, "cell_type": "code", "source": "np.percentile(stack_lengths, 95)", "outputs": [{"execution_count": 64, "output_type": "execute_result", "data": {"text/plain": "11.0"}, "metadata": {}}], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": null, "cell_type": "code", "source": "", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}], "nbformat": 4, "metadata": {"kernelspec": {"display_name": "Python 2", "name": "python2", "language": "python"}, "language_info": {"mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "2.7.9", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython2", "codemirror_mode": {"version": 2, "name": "ipython"}}}}