Analysis of MediaSearch interleaved A/B Test https://phabricator.wikimedia.org/T261759

T261759.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MediaSearch Interleaved A/B Test Analysis\n",
    "\n",
    "The Structured Data team ran an interleaved A/B test of search on Commons, and we're wanting to understand if there was a difference in preference between the two options.\n",
    "\n",
    "The phabricator task for this analysis is [T261759](https://phabricator.wikimedia.org/T261759)\n",
    "\n",
    "> The test started on 2020-09-10, and ended on 2020-09-17. The timestamps are roughly accurate to within a minute and are gathered from the [Server Admin Log](https://wikitech.wikimedia.org/wiki/Server_Admin_Log)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "── \u001b[1mAttaching packages\u001b[22m ─────────────────────────────────────── tidyverse 1.3.0 ──\n",
      "\n",
      "\u001b[32m✔\u001b[39m \u001b[34mggplot2\u001b[39m 3.3.2     \u001b[32m✔\u001b[39m \u001b[34mpurrr  \u001b[39m 0.3.4\n",
      "\u001b[32m✔\u001b[39m \u001b[34mtibble \u001b[39m 3.0.3     \u001b[32m✔\u001b[39m \u001b[34mdplyr  \u001b[39m 1.0.2\n",
      "\u001b[32m✔\u001b[39m \u001b[34mtidyr  \u001b[39m 1.1.2     \u001b[32m✔\u001b[39m \u001b[34mstringr\u001b[39m 1.4.0\n",
      "\u001b[32m✔\u001b[39m \u001b[34mreadr  \u001b[39m 1.3.1     \u001b[32m✔\u001b[39m \u001b[34mforcats\u001b[39m 0.5.0\n",
      "\n",
      "── \u001b[1mConflicts\u001b[22m ────────────────────────────────────────── tidyverse_conflicts() ──\n",
      "\u001b[31m✖\u001b[39m \u001b[34mdplyr\u001b[39m::\u001b[32mfilter()\u001b[39m masks \u001b[34mstats\u001b[39m::filter()\n",
      "\u001b[31m✖\u001b[39m \u001b[34mdplyr\u001b[39m::\u001b[32mlag()\u001b[39m    masks \u001b[34mstats\u001b[39m::lag()\n",
      "\n",
      "\n",
      "Attaching package: ‘lubridate’\n",
      "\n",
      "\n",
      "The following objects are masked from ‘package:base’:\n",
      "\n",
      "    date, intersect, setdiff, union\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# install.packages(c(\"tidyverse\", \"remotes\", \"furrr\"))\n",
    "library(tidyverse)\n",
    "library(lubridate)\n",
    "library(future)\n",
    "plan(multiprocess(workers = getOption(\"mc.cores\", 2)))\n",
    "library(furrr) # future-enabled purrr\n",
    "\n",
    "library(wmfdata) # remotes::install_github(\"wikimedia/wmfdata-r\")\n",
    "library(wmfastr) # remotes::install_github(\"wikimedia/wmfastr\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Name of the test configuration:\n",
    "subtest_name = 'mediasearch_commons_int'\n",
    "\n",
    "# Start and end timestamps (approximate is fine, as we also filter on subtest):\n",
    "start_ts = ymd_hms('2020-09-10T11:00:00Z')\n",
    "end_ts = ymd_hms('2020-09-17T12:00:00Z')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_query = paste0(\"USE event;\n",
    "SELECT\n",
    "  coalesce(client_dt, meta.dt) AS event_ts,\n",
    "  event.uniqueId AS event_id,\n",
    "  event.mwSessionId AS mw_session_id,\n",
    "  event.pageViewId AS page_id,\n",
    "  event.articleId AS article_id,\n",
    "  event.searchSessionId AS session_id,\n",
    "  event.subTest AS group,\n",
    "  wiki,\n",
    "  event.source AS search_type, -- fulltext or autocomplete\n",
    "  event.searchtoken AS search_token, -- uniquely links visitPage events to specific SRP events\n",
    "  MD5(LOWER(TRIM(event.query))) AS query_hash,\n",
    "  event.action AS event,\n",
    "  IF(event.position < 0, NULL, event.position) AS event_position,\n",
    "  CASE\n",
    "    WHEN event.action = 'searchResultPage' AND event.hitsReturned > -1 THEN event.hitsReturned\n",
    "    WHEN event.action = 'searchResultPage' AND event.hitsReturned IS NULL THEN 0\n",
    "    ELSE NULL END AS n_results,\n",
    "  event.extraParams AS event_extraparams\n",
    "FROM searchsatisfaction\n",
    "WHERE year = 2020\n",
    "  AND month = 9\n",
    "  AND day BETWEEN 10 AND 17\n",
    "  AND event.subtest = '\", subtest_name, \"'\n",
    "  AND coalesce(client_dt, meta.dt) IS NOT NULL\n",
    "  AND coalesce(client_dt, meta.dt) > '\", format_ISO8601(start_ts), \"'\n",
    "  AND coalesce(client_dt, meta.dt) < '\", format_ISO8601(end_ts), \"'\n",
    "  AND wiki = 'commonswiki'\n",
    "  AND event.action IN ('searchResultPage', 'visitPage')\n",
    "  AND CASE\n",
    "    WHEN event.action = 'searchResultPage' THEN event.msToDisplayResults IS NOT NULL\n",
    "    WHEN event.action = 'visitPage' THEN event.pageViewId IS NOT NULL\n",
    "    ELSE TRUE END\n",
    "  AND useragent.is_bot = false;\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Don't forget to authenticate with Kerberos using kinit\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Events Sessions \n",
      "  661053    59250 \n"
     ]
    }
   ],
   "source": [
    "if (!fs::file_exists(\"events.rds\")) {\n",
    "    # De-duplicated dataset does not exist yet\n",
    "    if (fs::file_exists(\"events_backup.rds\")) {\n",
    "        events <- readr::read_rds(\"events_backup.rds\") # load cached\n",
    "    } else {\n",
    "        events = query_hive(data_query)\n",
    "        readr::write_rds(events, \"events_backup.rds\", compress = \"gz\") # cache\n",
    "    }\n",
    "    print(c(Events = nrow(events), Sessions = length(unique(events$session_id))))\n",
    "} else {\n",
    "    # De-duplicated dataset exists, do nothing\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Events Sessions \n",
      "  660962    59250 \n"
     ]
    }
   ],
   "source": [
    "if (fs::file_exists(\"events.rds\")) {\n",
    "    events <- readr::read_rds(\"events.rds\") # load cached\n",
    "} else {\n",
    "    # De-duplicate events:\n",
    "    events <- events %>%\n",
    "      dplyr::mutate(\n",
    "        timestamp = lubridate::ymd_hms(event_ts),\n",
    "        date = as.Date(timestamp)\n",
    "      ) %>%\n",
    "      dplyr::arrange(session_id, event_id, timestamp) %>%\n",
    "      dplyr::distinct(session_id, event_id, .keep_all = TRUE)\n",
    "    readr::write_rds(events, \"events.rds\", compress = \"gz\") # cache\n",
    "    print(c(Events = nrow(events), Sessions = length(unique(events$session_id))))\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`process_session()` takes an input of events belonging to a session, matches the visited pages with the team drafts in the search result page events, and outputs the memberships."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "process_session <- function(events_subset) {\n",
    "  serp_events <- events_subset %>%\n",
    "    filter(event == \"searchResultPage\", event_extraparams != \"NULL\", n_results > 0) %>%\n",
    "    # Extract team drafts from the extraParams JSON:\n",
    "    mutate(team_drafts = map(event_extraparams, ~ jsonlite::fromJSON(.x)$teamDraft)) %>%\n",
    "    select(page_id, search_token, n_results, team_drafts) %>%\n",
    "    # Remove SERP events missing team draft info:\n",
    "    filter(!map_lgl(team_drafts, is.null))\n",
    "  visit_page_events <- events_subset %>%\n",
    "    filter(event == \"visitPage\") %>%\n",
    "    select(search_token, article_id)\n",
    "  visited_teams <- visit_page_events %>%\n",
    "    left_join(serp_events, by = \"search_token\") %>%\n",
    "    mutate(team = map2_chr(article_id, team_drafts, function(id, teams) {\n",
    "      dplyr::case_when(\n",
    "        id %in% teams$a ~ \"A\",\n",
    "        id %in% teams$b ~ \"B\",\n",
    "        TRUE ~ as.character(NA)\n",
    "      )\n",
    "    })) %>%\n",
    "    select(team)\n",
    "  return(visited_teams)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "events <- split(events, events$session_id)\n",
    "visited <- future_map_dfr(events, process_session, .id = \"session_id\", .progress = FALSE)\n",
    "write_rds(visited, \"visited.rds\", compress = \"gz\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "    A     B \n",
       "19496 18949 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "table(visited$team) # counts of visited pages from the two teams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE**: Just because \"A\" has more total visits does not indicate there is a preference for it. The preference statistic is calculated based on *sessions*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAC/VBMVEUAAAABAQECAgIDAwME\nBAQFBQUGBgYHBwcICAgJCQkKCgoLCwsMDAwNDQ0ODg4PDw8QEBARERESEhITExMUFBQVFRUW\nFhYXFxcYGBgZGRkaGhobGxscHBwdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQlJSUmJiYnJyco\nKCgpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEyMjIzMzM0NDQ1NTU2NjY3Nzc4ODg5OTk6\nOjo7Ozs8PDw9PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NERERFRUVGRkZHR0dISEhJSUlKSkpLS0tM\nTExNTU1OTk5PT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dYWFhZWVlaWlpbW1tdXV1eXl5f\nX19gYGBhYWFiYmJjY2NkZGRlZWVmZmZnZ2doaGhpaWlqampra2tsbGxtbW1ubm5vb29wcHBx\ncXFycnJzc3N0dHR1dXV2dnZ3d3d4eHh5eXl6enp7e3t8fHx9fX1+fn5/f3+AgICBgYGCgoKD\ng4OEhISFhYWGhoaHh4eIiIiJiYmKioqLi4uMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSV\nlZWWlpaXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+goKChoaGioqKjo6OkpKSlpaWmpqan\np6eoqKipqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS1tbW2tra3t7e4uLi5\nubm6urq7u7u8vLy9vb2+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrL\ny8vMzMzNzc3Ozs7Pz8/Q0NDR0dHS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd\n3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojp6enq6urr6+vs7Ozt7e3u7u7v\n7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7///9Vpqhr\nAAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO3dCZxcVZ3o8ZNOuptMAp2QhACJJEKA\npwwhIc8QRlwQHORpDA9JMAm2SxYUMAzgEnAGZiJu4emMjOt7DoGgjuNANIj6xAV4Cg4aMk4Q\n84ZIeApGBOKShSXL/by6Vd3Vt6rP3eqe5Z5zf9/PR/pW962qW7fuz3Sf/JOIAEBhwvYBAD4g\nJEABQgIUICRAAUICFCAkQAFCAhQgJEABQgIUICRAAUICFCAkQAFCAhQgJEABQgIUICRAAUIC\nFCAkQAFCAhQgJEABQgIUICRAAUICFCAkQAFCAhQgJEABQgIUICRAAUICFCAkQAFCAhQgJEAB\nQgIUICRAAUICFCAkQAFCAhQgJEABQurQSCHE/bYPImLf35/WN0KIC5ufsHOAww6jKggpgzm1\na/KcxmZ/bXNKuCG/Tu+49tpr7zB6bIPeJOoKhKTk2IcdRlUQUgbZQ1pZ++xKo8c24Kfh5XvI\nMdMubX4md0gqjn34YVQFIWUgC+mKVatWPTZsT2shfbb2xC/aFf2MlZCGH0ZVEFIGspDkrIV0\nQ+2Jz2r5jJWQhh9GVRBSBknf2j3/2dce3XvI1P+67LPPBAvEoAnhLg+ufOlh3ZP/8h/3Nu66\n529O6D1yycO7wq9vGXyEpy6bPurVQbDxqrOOnzDq0OPfvLGxb7jP/duWHjFm3m1BsPuvj+ud\n+s7ftR1U9NGvbD7xV5tfrz/87y6b3jv14h2Su7Tfbjv2oPVxdrxrWvNxoscdBFsu/fO+nqPe\n+NWDMYdRFYSUQUJIz85rXjx3tV6M+y4dvHHMT8M7PD2zfuOQf4mG9C8vqv3nVUFwSvOe570Q\n7hxuffLQ+iduePrk+sfj/xg9pNZHjwtp/VH1T056aPhd2m8nhvT5CZHHiR73gfeOGLjXa54h\nJKQIQ5p+Zd1JbSF9vB7HhFHDQ3pn84Y4fFvtDq8fuNEbDWlSeOOVjZAOG1P/+nXho4cb3Y39\newZTvTp6SK2PHhfSmIHPztg77C7ttxND6ok+TvS460/cNTb87yv2ERKSzREtoiG9sfbh9oPB\n/p9/6tXfC57cvqR2c8n27dt/FTwQ7vmyb23+SNjYgiC4J7w9++s/bPwyMBiSGHneey46Lwje\nesPWPUHw5BXhdRx+k1Tf6Q1feX/9/+8n/sPNx4VXceSI2h595/arax9fXnviPc1d6g9/3re/\nHR6h+OSwu7Tfjh57i/bHiR735trxjVj7bPBvx9Q+9XnpYVQFIWWQENI5tf9DjizeDf3Avjz8\nP/pw/er62saIJ+tfGRd+d3ZRNKSR97Y80b7Rtc89HDRCmrk/CF4ebnyr9jNU+HHv0I7tjx6s\nFbLFhtNqUR4Ij37e8LsMe4iYxYb2x4ked3iXi8KNW2sbcwPZYVQFIWWQEFL4a0jvGcv+x/ef\nre85dDG+pLZ1cbjx2/Autwen1v67LLx9bzSkiwae4cBXFp4wtqvx8N8NGiF9ovbx7bWPU2tX\n8fbwEzuGjqj90WNC+ly48Y+1jZ79w+4y7CESQoo+TvS4X1rbPPn1NWHwI18gJCRKWGz4Zd9A\nXYde9VwQvRjH17Y+Wt/6s9rWp4Kptf9+KLz5RDSk9Y1H3f2qSKdfCxoh3Vn7eFnt42tqH3eE\nn/j10BG1P3pMSN8JN74Z3vfpYXcZ9hAJIUUfJ3rc41v+D+Y3hIREScvfD5/fO3AdhRfh0MU4\nrrb1sfpW+O3ap4MpgyE9Hg3p7sajhj9aiJkXrVx5SO3jhmBg+bv2cZVojNv8ui2k9kePCel/\nhxvfaATQfpdhD5EQUvRxosc9riWk7YSERMkjQnt+9E+rZ9du9T6b9K1d+Bgrwtv3REMa+B3T\nE2qbl9U+7uvJGFLGb+0+E27cKAp/axd9nOhxh4/xiejOhIQECSHta3x6/5G1m48GQbgm95b6\nZ9p/lr948PZSSUiHDvRzp8gYUsbFhpcdqP34Ff50lmWxYejYEx8netwraptz9ze2d/w4ICQk\nSgjpyvNu/X8Hg4PfGNX4tudvax+O+v6j259prC7P/fa/D64219cYXva1exq/e9MW0otrm+f+\n4eDdR2cNqf3RY0ISb7jzzvnhx38YfpdhDzF07ImPEz3uTeHnFj0SBL+/8y2jr5UeRlUQUgYJ\nIYVXes+E+o9Jp9c++y3RsKrl9zvHR39DdowkpHeHnxs5RnR3Zwxp2KPLQxr8jdTj9kju0n47\ncuztj3No9HGiQ3x/Vf/06PrXr5UeRlUQUgYpITUcGc7P7J/dvBj3XTL4lWN+Et7hqcakz5jb\nw//+36Dlgnz6+PrXRq6bkDWk9keXh/T5ifVdJvxMdpf225Fjb3+cL06JPE40pAPvHxwREuIG\n6WFUBSFlkBDSr/5p+cuOGd19xCs//Pv615+57LiewYtx08qXjO0+4uwbB36bf89fz+g5cul/\nPhhedDuD1gty5+XTuictuC/IHFL7o8tDun/HO4/pmbLiN9K7DLsdPfa2x/ndZdOaj9M6Vr71\nilPHjxwz441rfyE/jKogJNPeX7vWjrd9EDmU7c/UlxQhmfLN/u/W/p//9x8Pf7b/oO2DyYGQ\nMiEkUzYI0TVxYv1HipkuzXQSUiaEZMqG5k/lZ+9I39umDRGElBEhmfL0p9504uGjxp38ju/a\nPpI0IoKQMiIktGsNCZlwpgAFCAlQgJAABQgJUICQAAUICVCAkAAFCAlQgJAABQgJUICQAAUI\nCVCAkAAFCAlQgJAABQgJUICQAAUICVCAkAAFCAlQgJAABQgJUICQAAUICVCAkAAFCAlQgJAA\nBQgJUICQAAUICVCAkAAFCAlQgJAABQgJUICQfPf8ihXftn0MFUBIvtsjxEdtH0MFEJLvCMkI\nQvIdIRlBSDntOvxx24eQj3shnfBD20fQAULK6Unxc9uHkI97IfV9zfYRdICQciIk7QipCghJ\nO0KqAkLSjpCqYOeIR2wfQj7uhTTpm7aPoAOElNcvbR9ATu6FtP2A7SPoACH5zr2QnERIviMk\nIwjJd4RkBCHlxGSDdkw2VAHL39qx/F0FhKQdIVUBIWlHSFVASNoRUhUw2aAdkw2VwGSDbkw2\noITcC8lJhOQ7QjKCkHxzR5t/FeKt7Z8bYPtQfUJIOZV+ssH9kJhsqILSL3+7HxLL31VASNoR\nUhUQknaEVAWEpB0hVUHpJxvcD4nJhkoo+2SD+yEx2YAScD8kJxGSbwjJCkLyDSFZQUg5Mdmg\nHZMNVcDyt3Ysf1cBIWlHSFVASNoRUhUQknaEVAVMNmjHZEMlMNmgG5MNKAH3Q3ISIfmGkKwg\nJN8QkhWElBOTDdox2VAFLH9rx/J3FRCSdpUL6eDWjetv2bj1oLKDcQEhaVexkPaumSLqpq7Z\nq/CAyo6QtKtWSLtPE12zFy1fsWhWl5i3R+UhlRuTDdpVa7LharH0icbW44vFNaoOxwFMNuhW\nrcmGY+c0X+6BU2eoORgo4H5ITuo4pJ7Lh7ZX9ao4FChBSFZ0HNKkBUPb8yerOBQoQUhWdBzS\n4q6bBzdvGrFEzcG4gMkG7ao12bCtT8xevW7DhnWrZ4lx21QeUrmx/K1dtZa/gy1zxYC5WxQe\nUNkRknYVCykINq1dtnDhsrWblB2MCwhJu8qFVEmEpB0hVQGTDdpVa7IhqOjQKpMN2lVrsqGq\nQ6ul535ITmJo1TeEZAVDq74hJCsYWs2JyQbtqjXZkDi0+uvT5jSdPMHFnx1jsfytXbWWvxOH\nVp/7wuea3iue7/Q5yoiQtKtWSJmHVn9ESEYRkhX6h1YJySxCskL/0KpnITHZoF3lJhuyDa16\nFhKTDdpVbLIhK99CKjv3Q3ISIfmGkKwgJN8QkhWdh3TgSyvffVdj84ZzEvbzLCQmG7Sr1mTD\n/teHC3bn/zHc7k96FM9CYvlbu2otf39GTP7Ip+eKOb8PCKlUCMmKjkM6fdTW2rd3fyPm/pGQ\nSoWQrOg4pENfVf9wo3j5bkIqE0KyouOQehc2Pq4VZ+6tUkhMNmhXrcmGGacPbFwrXre4QiEx\n2aBdtSYbLuj5w8DWX4mRVQqp7NwPyUkdh/RF8ZnBzeWCkMqDkKzoOKQ/feK2wc0DH3tfwo6E\nZBYhWcGIUE5MNmhXrcmGzDwLieVv7aq1/J0ZIZlFSFYQUk6EpB0hSRGSWYRkBSHlxGSDdtWa\nbMjMs5CYbNCuWpMNmfkWUtm5H5KTCMk3hGQFIfmGkKwgpJyYbNCOyQYpz0Ji+Vs7lr+lCMks\nQrKCkHIiJO0ISYqQzCIkKwgpJyYbtGOyQcqzkJhs0I7JBinfQio790NyEiH5hpCsICTfEJIV\nhJQTkw3aMdkg5VlILH9rx/K3FCGZRUhWEFJOhKQdIUkRklmEZAUh5cRkg3ZMNkh5FhKTDdox\n2SDlW0hl535ITiIk3xCSFYTkG0KygpByYrJBOyYbpDwLieVv7Vj+liIkswjJCkLKiZC0IyQp\nQjKLkKwgpJyYbNCOyQYpz0JiskE7JhukfAup7NwPyUmE5BtCsoKQfENIVhBSTkw2aMdkg5Rn\nIbH8rR3L31KEZBYhWUFIORGSdoQkRUhmEZIVhJQTkw3aMdkg5VlITDZox2SDlG8hlZ37ITmJ\nkHxDSFYQkm8IyQpCyonJBu2YbJDyLCSWv7Vj+VuKkMwiJCsIKSdC0o6QpAjJLEKygpByYrJB\nOyYbpDwLickG7ZhskPItpLJzPyQnEZJvCMkKQvINIVlBSDkx2aAdkw1SnoXE8rd2LH9LEZJZ\nhGQFIeVESNoRkhQhmUVIVhBSTkw2aMdkg5RnITHZoB2TDVK+hVR27ofkJELyDSFZQUi+ISQr\nCCknJhu0Y7JByrOQWP7WjuVvKUIyi5CsIKScCEk7QpIiJLMIyQpCyonJBu2YbJDyLCQmG7Rj\nskHKt5DKzv2QnERIviEkKwjJN4RkBSHlxGSDdkw2SHkWEsvf2rH8LUVIZhGSFYSUEyFpR0hS\nhGQWIVlBSDkx2aAdkw1SnoXEZIN2TDZI+RZS2bkfkpMIyTeEZAUh+YaQrCCknJhs0I7JBinP\nQmL5WzuWv6UIySxCsoKQciIk7QhJipDMIiQrCCknJhu0q9xkw8GtG9ffsnHrweS9PAuJyQbt\nKjbZsHfNFFE3dc3epP18C6ns3A/JSR2HtPs00TV70fIVi2Z1iXl7EnYkJLMIyYqOQ7paLH2i\nsfX4YnFNwo6EZBYhWdFxSMfOaX4ne+DUGQk7ehYSkw3aVWuyoefyoe1VvQk7ehYSy9/aVWv5\ne9KCoe35kxN2JCSzCMmKjkNa3HXz4OZNI5Yk7EhIZhGSFR2HtK1PzF69bsOGdatniXHbEnYk\nJLMIyYrOfx9py1wxYO6WpP08C4nJBu0qN9mwae2yhQuXrd2UvJdnITHZoF3FJhuy8i2ksssR\nUg62X1XpEZJvCMkKQvINIVlRMKR7z50w9pQb9iXt4llIPk02lDSkak02TH537T9fGllftluQ\n9CcpPAvJp+XvkoZUreVv0R8ET4/t+sCjO28/StyasCMhmUVIVhQK6bOiPnD3Y/Ha9q/+5pdN\nXyUkowjJikIhvWvgopo1qe2L20TUcwWOr3QISbvqhfQ20fizsed3t3/1V97+iuTTZENJQ6rW\nZEMY0vXit/XtMycm7OjZz0g+TTaUNKRqTTaIrt7ebvHd+vb02Qk7+hZS2bkfkpM6DunEug+F\nm5vExQk7EpJZhGSFismGB9Y+mPBVQjKLkKxgRCgnJhu0q9ZkQ2aehcTyt3bVWv7OjJDMIiQr\nlIR05bSELxKSWYRkhZKQ+pMehZDMIiQrCCknJhu0q9Zkw4UR0ysUEpMN2lVssqFFwo6+hVR2\n7ofkpI5DGnPC0Fk+i5B06zwBQjKi45BOP2zoj8VW6WckSwip5DoO6RIx9NerVikkS5MNVQqp\nWpMNt825e2i7Qv8+kqXl7yqFVN3l70SEpAIhlRwh5URI2hGSFCGpQEglR0g5WZpsqFJI1Zps\nyMyzkCxNNlQppGpNNmTmW0h2VCkkJxGSGwip5AjJDYRUcoSUE5MN2lVrsiEzz0Ji+Vs7lr+l\nCEkFQio5QsqJkLQjJClCUoGQSo6QcmKyQTsmG6Q8C4nJBu2YbJDyLSQ7qhSSkwjJDYRUcoTk\nBkIqOULKickG7ZhskPIsJJa/tWP5W4qQVCCkkiOknAhJO0KSIiQVCKnkCCknJhu0Y7JByrOQ\nmGzQjskGKd9CsqNKITmJkNxASCVHSG4gpJIjpJyYbNCOyQYpz0Ji+Vs7lr+lCEkFQio5QsqJ\nkLQjJClCUoGQSo6QcmKyQTsmG6Q8C4nJBu2YbJDyLSQ7qhSSkwjJDYRUcoTkBkIqOULKickG\n7ZhskPIsJJa/tWP5W4qQVCCkkiOknAhJO+dD2qnlGQhJBUIquWhIo/vv0/AMnoXEZIN2zk82\nHC/EzE/9UfUzeBaSysmG4td3Bu6F5Pxkw8HvLeoRY97xgNpn8C0khYpf3xm4F5KT2hYbnvzI\ncUKc+rldCp+BkGIVv74zICQjhq3aHfzOm7rFoRdvUfYMhBSr+PWdASEZMXz5+7EPHCGEGPHm\nPyh6Bs9CUjnZUPz6zsC9kHyYbNj/9f/WJV70d49/89ViqaJn8Cwklcvfxa/vDNwLyfnl7+BX\n104RI167YX9t8+D88YqegZBiFb++MyAkI6IhvWGkOPyKwd8luV7VzAMhxSp+fWdASEZEcxEv\nu+nZ5o1NNyp6BkKKVfz6zoCQjIiG9FMtz+BZSConG4pf3xm4F5Lzkw16eBYSkw3aOT/Z8JVX\n/7r+8dev+leFz+BbSAoVv74zcC8kJ0VDOnvOwMYp5yh8BkKKVfz6zoCQjIiGNHnlwMY7jlb4\nDIQUq/j1nQEhGRENqfvqgY3VPQqfwbOQmGzQzvnJhiMXDmwsPELhM3gWEsvf2jm//H1B7y/q\nHx/u/e8Kn4GQYhW/vjMgJCOiId3XdfgnH9n7yCcP7/qRwmcgpFjFr+8MCMmIlt9H+sxIERr5\nGZXPQEixil/fGRCSEa2/IfuzlbOmz7r4P5Q+g2chMdmgHZMNUp6FxGSDds5PNujhW0gKFb++\nM3AvJCcRkkXFr+8MCMmIlpDunj+5Z2SdwmcgpFjFr+8MCMmIaEh3dIm+k06pU/gMnoXEZIN2\nzk82zBn5xYPqn8GzkFj+1s755e/eC3Q8AyHFKn59Z0BIRkRDOvxdOp6BkGIVv74zICQjoiEt\nmhO7WwGEFKv49Z0BIRkRDemxydftV/8MnoXEZIN2zk829L9GTFvQX6fwGTwLickG7ZyfbBBD\nFD6DbyEpVPz6zkBNSNnZPqmWRJPZPEThMxBSLCMXNiEZwYiQRUYubEIyoi2kx+5T9Y9QNHkW\nUkUnG7Irflqcn2wI7p8pxF1B8OWT7lb4DJ6FVNHl7+yKnxbnl78fHjN2QRjSrjGXKHwGQopl\n5MImJCOiIS3p+Y+nwpCCNzC0GouQUhQ/Lc6HNPnCoBHSVRMUPgMhxTJyYROSEdGQRr1vIKT3\n8RdExqroZEN2xU+L85MNk94+ENJfTlP4DJ6FVNHJhuyKnxbnJxvOm/xcPaTvjehX+Ay+haSQ\nkQvbvZCcFA3ph13n3iM2PnBFd/fPFD4DIcUycmETkhGtf0HkqPqgXffNKp+BkGIZubAJyYjW\nyYaHLp0z/ZSVDyl9Bs9CYrIhRfHT4v5kgxaehcTyd4rip8X55W89CCmWkQubkIwgpJwIKUXx\n0+J8SMcNUfgMhBTLyIVNSEZEQ+qrGyXEYX0Kn8GzkJhsSFH8tDg/2dDwwr+dPv8Fhc/gWUhM\nNqQoflqcn2wYtPOo6xQ+g28hKWTkwnYvJCdJFxsuerHCZyCkWEYubEIyQhrSsmzT3we3blx/\ny8atKX9fOCHFMnJhE5IRspB2TM7yK9LeNVMaf3XX1DV7k/bzLCQmG1IUPy3OTzZcW3fNWw4T\nf5d+x92nia7Zi5avWDSrS8zbk7CjZyGx/J2i+Glxfvl78G+HHH1VhmWTq8XSJxpbjy8W1yTs\nSEixjFzYhGREyz80Vnfnfbuy3PHYOc3aDpw6I2FHQopl5MImJCM6HhHquXxoe1Vvwo6EFMvI\nhU1IRnQc0qQFQ9vzJyfs6FlITDakKH5aPJlsyGhxV/OP/900YknCjp6FxGRDiuKnxfnJhmmt\nku+4rU/MXr1uw4Z1q2eJcdsSdvQtJIWMXNjuheSkaEgTxgkhxtT+N25CKOWeW+YOrvLN3ZK0\nHyHFMnJhE5IR0ZB2nXHqnbuCXXfOPiPTul2wae2yhQuXrd2UvBchxTJyYROSEdGQLj+28Rur\ne469XL5zRzwLicmGFMVPi/OTDVOuGti4aqrCZ/AsJJa/UxQ/Lc4vf/dcObBxZdLvCw2p5NAq\nIaUoflqcD+mE6bvrH3dP+y8Z7lnRoVVCSlH8tDgf0sfFKRueCZ7ZcIr4RPodGVotzsiFTUhG\nREM6sLz260v4l62uYGg1FpMNKYqfFg8mG77fP3PazP4fZLlj4tDq8+s+1/Rev0JisiFF8dPi\n/GRDLolDq796ybFNR4vnOn0O3xm5sN0LyUkd/6vmVR1aVcnIhU1IRnT8r5pXdmhVISMXNiEZ\n0fG/al7VoVUmG1IUPy3OTzbk+1fNKzq0yvJ3iuKnxfnl77z/qnklh1YJKUXx0+J8SPyr5hkQ\nUorip8X5kPhXzTMgpBTFT4vzIfGvmmfAZEOK4qfF+cmGjv9V8yuTfgXzLCQmG1IUPy3uTzZ0\n+q+a9yfNR/gWkkJGLmz3QnKSkn/VnJA6Y+TCJiQjogncvznHHS+MmE5IHTFyYROSEdEERrwp\nzx1bJOzoWUhMNqQoflqcn2yYeFGOO445YejcnVWhkFj+TlH8tDi//H3Bifuz3/H0w4b+roYq\n/YxESCmKnxbnQ/rPCZcm/ZnxVpeIoUlVQuqMkQubkIyIJtB/pph49lv6Q+l3vG3O0J+1uK1C\nf9SckFIUPy3Oh5Rt8SAvz0JisiFF8dPi/GTD5iEKn8GzkJhsSFH8tLg/2aCFbyEpZOTCdi8k\nJzVD+vKPNT0DIcUycmETkhHNkER/7T83nKP+GQgplpELm5CMaA0pcR27Q56FxGRDiuKnxe3J\nBkLKhuXvFMVPi9vL34SUDSGlKH5aCEmKkGIZubAJyQhCyomQUhQ/LY6H1N3X19ct+hoUPoNn\nITHZkKL4aXF7siHzny/Ky7OQmGxIUfy0uD3Z8GwLhc/gW0gKGbmw3QvJSYwIWWTkwiYkIwjJ\nIiMXNiEZQUg5MdmQovhpcXuyQRvPQmL5O0Xx0+L28rc2hBTLyIVNSEYQUk6ElKL4aSEkKUKK\nZeTCJiQjCCknJhtSFD8tbk82aONZSEw2pCh+WtyebNDGt5AUMnJhuxeSkwjJIiMXNiEZQUgW\nGbmwCckIQsqJyYYUxU8Lkw1SnoXE8neK4qeF5W8pQopl5MImJCMIKSdCSlH8tBCSFCHFMnJh\nE5IRhJQTkw0pip8WJhukPAuJyYYUxU8Lkw1SvoWkkJEL272QnERIFhm5sAnJCEKyyMiFTUhG\nEFJOTDakKH5amGyQ8iwklr9TFD8tLH9LEVIsIxc2IRlBSDkRUorip4WQpAgplpELm5CMIKSc\nmGxIUfy0MNkg5VlITDakKH5amGyQ8i0khYxc2O6F5CRCssjIhU1IRhCSRUYubEIygpByYrIh\nRfHTwmSDlGchpS5/m7xos3AvJJa/pQjJLkIygpByIqQUxU8xIUkRkl2EZAQh5ZQ62WDyos3C\nvZCYbJDyLKTUyQaTF20W7oXEZIOUbyGlMXnRZuFeSE4iJNVMXrRZEJIRhKSayYs2C0IygpBy\nSp1sMHnRZuFeSEw2SHkWEsvfKYqfYpa/pQjJLkIygpByIqQUxU8xIUkRkl2EZAQh5cRkQ4ri\np5jJBinPQmKyIUXxM8xkg5RvIaUxedFm4V5ITiIk1UxetFkQkhGEpJrJizYLQjKCkHJisiFF\n8VPMZIOUZyGx/J2i+Clm+VuKkOwiJCMIKSdCSlH8FBOSFCHZRUhGEFJOTDakKH6KmWyQ8iwk\nJhtSFD/DTDZI+RZSGpMXbRbuheQkQlLN5EWbBSEZQUiqmbxosyAkIwgpJyYbUhQ/xUw2SHkW\nEsvfKYqfYpa/pQjJLkIygpByIqQUxU8xIUkRkl2EZAQh5cRkQ4rip5jJBinPQmKyIUXxM8xk\ng5RvIaUxedFm4V5ITiIk1UxetFkQkhGEpJrJizYLQjKCkHJisiFF8VPMZIOUZyGx/J2i+Clm\n+VuKkOwiJCMIKSdCUifmDBKSFCHZRUhGEFJOTDaoE3MGmWyQ8iwkJhvUiTmDTDZI+RZSGttX\nZzv3QnISIalm++psR0hGEJJqtq/OdoRkBCHlxGSDOjFnkMkGKc9CYvlbnZgzyPK3FCHZRUhG\nEFJOhKROzBkkJClCsouQjCCknJhsUCfmDDLZIOVZSEw2qBNzBplskPItpDS2r8527oXkpCIh\nHdy6cf0tG7ceTN6LkOwiJCM6D2nvmimibuqavUn7EZJdhGRExyHtPk10zV60fMWiWV1i3p6E\nHT0LickGdWLOYLUmG64WS59obD2+WFyTsKNnIbH8rU7MGazW8vexc5prKwdOnZGwIyHZRUhG\ndBxSz+VD26t6E3YkJLsIyYiOQ5q0YGh7/uSEHQnJLkIyouOQFnfdPLh504glCTt6FhKTDerE\nnMFqTTZs6xOzV6/bsGHd6lli3LaEHT0LickGdWLOYMUmG7bMFQPmbknaz7eQ0ti+Otu5F5KT\nikw2bFq7bOHCZWs3Je9FSHYRkhF6Zu2eWrKw6TXiOS3PUVa2r852hGSEnpD+sGpF0wK/fkVi\nskGdmDNYrcmGzDz71o7lb3VizmC1lr8b7j13wthTbtiXtAsh2UVIRnQc0uR31/7zpZH1ZbsF\nSX+SgpDsIiQjOg5J9AfB02O7PvDoztuPErcm7EhIdhGSEYVC+qyoD9z9WLw2YUfPQmKyQZ2Y\nM1ityYYwpHcN/L/zrFDLfvUAAA2iSURBVEkJO3oWEpMN6sScwWpNNoQhvU00/mzs+d0JO/oW\nUhrbV2c790JyUqGQrhe/rW+fOTFhR0Kyi5CM6Dykrt7ebvHd+vb02Qk7EpJdhGRExyGdWPeh\ncHOTuDhhR89CYrJBnZgzWNnJhgfWPpjwVc9CYvlbnZgzWK3l78wIyS5CMoKQciIkdWLOICFJ\nEZJdhGQEIeXEZIM6MWewWpMNmfkR0tC7/z/tXXgdcS+kak02ZOZbSK5xLyQnEVI2tq+5zhGS\nEYSUje1rrnOEZAQhZdN887966DqLl14H3AupspMNyTwL6VbxaYuXXgfcC4nlbylCsouQjCCk\nbJpvPiGpE3OuCUmKkOwiJCMIKZvmm//lEZ+3eOl1wL2QmGyQ8iwkJhvUiTnXTDZI+RaSa9wL\nyUmElI3ta65zhGQEIWVj+5rrHCEZQUjZNN98JhvUiTnXTDZIeRYSy9/qxJxrlr+lCMkuQjKC\nkLJpvvmEpE7MuSYkKUKyq8QhxRjzgeGfs/3+pyKkbJrvKJMN2vVdN/xztt//VISUzdBbymSD\nbl/YOPxztt//VISUjfnLSRX3QpKx/f6nIqRsbF9InSMkIwgpG9sXUucIyQhCyqb5jjLZoN2U\njw3/nO33PxUhZdN8R1n+1o7lbylCsouQjCCkbJrvKCFpR0hShGQXIRlBSNk031EmG7RjskHK\ns5CYbNCOyQYp30JyjXshydh+/1MRUja2L6TOEZIRhJSN7Qupc4RkBCFl03xHmWzQjskGKc9C\nYvlbO5a/pQjJLkIygpCyab6jhKQdIUkRkl2EZAQhZdN8R5ls0I7JBinPQmKyQTsmG6R8C8k1\n7oUkY/v9T0VI2di+kDpHSEYQUja2L6TOEZIRhJRN8x1lskE7JhukPAuJ5W/tWP6WIiS7CMkI\nQsqm+Y4SknaEJEVIdhGSEYSUTfMdZbJBOyYbpDwLickG7ZhskPItJNe4F5KM7fc/FSFlY/tC\n6hwhGUFI2di+kDpHSEYQUjbNd5TJBu2YbJDyLCSWv7Vj+VuKkOwiJCMIKZvmO0pI2hGSFCHZ\nRUhGEFI2zXeUyQbtmGyQ8iwkJhu0Y7JByreQXONeSDK23/9UhJSN7Qupc4RkBCFlY/tC6hwh\nGUFI2TTfUSYbtGOyQcqzkFj+1o7lbylCsouQjCCkbJrvKCFpR0hShGQXIRlBSNk031EmG7Rj\nskHKs5CYbNCOyQYp30JyjXshydh+/1MRUja2L6TOEZIRhJSN7Qupc4RkBCFl03xHmWzQjskG\nKc9CYvlbO5a/pQjJLkIygpCyab6jhKQdIUkRkl2EZAQhZdN8R5ls0I7JBinPQmKyQTsmG6R8\nC8k17oUkY/v9T1XpkGxfHUYQkhGE5DtCMoKQcmKyQTsmG6Q8C4nlb+1Y/pYiJLsIyQhCyomQ\ntCMkKUKyi5CMIKScmGzQjskGKc9CYrJBOyYbpHwLyTXuhSRj+1JJRUi+IyQjCMl3foSUnaVr\niZByYrJBO9lkQ3aWriVCyonlb+1ky9/ZWbqWCCknQtKOkKQIyS5CMoKQciIk7QhJyrOQmGzQ\nTjbZkJ2la4mQ8mKyQTfZZEN2lq4lQvKdeyEVY+laIiTfEZIRhOQ7QjKCkHJiskE7JhukPAuJ\n5W/tWP6WIiS7CMkIQsqJkLQjJClCsouQjCCknJhs0K7YZEMOKq8lQsqLyQbdik025KDyWioS\n0sGtG9ffsnHrweS9fAvJNe6FZIzKa6nzkPaumSLqpq7Zm7QfIdlFSLFUXksdh7T7NNE1e9Hy\nFYtmdYl5exJ2JCS7CCmWymup45CuFkufaGw9vlhck7BjQkhaXrO2097AZIN2xSYbcsh+UaXr\nOKRj5xwY3Dxw6oyEHT0LieVv7Yotf+eQ/aJK13FIPZcPba/qbfvio5PGNx0qXoh7DC2vWdtp\nbyAk7aoV0qQFQ9vzJ7d98cAP7mr6zq2dPkUpPSl+bvsQ8tkjxEdtH0M+fV+zfQQd6DikxV03\nD27eNGKJmoNxASFpV62QtvWJ2avXbdiwbvUsMW6bykMqt50jHrF9CPm4F9Kkb9o+gg50/vtI\nW+aKAXO3KDyg0vul7QPIyb2Qth9I36d0ikw2bFq7bOHCZWs3KTsYaOBeSE7SP2sHuwjJCELy\nHSEZQUg57Tr8cduHkI97IZ3wQ9tH0AFCyonlb+2qtfxdVYSkHSFVASFpR0hVQEjaEVIVMNmg\nXcUmG6qKyQbdKjfZABe4F5KTCMl3hGQEIfmOkIwgpJyYbNCOyYYqYPlbO5a/q4CQtCOkKiAk\n7QipCghJO0Iqg3UCrpgnfwuZbCiDO0b/VK+va358qdF/3/l9Lz1/o7oDUWl+zF8+xWRDGXxj\njO0j0GHMN2wfgQZvfavtI1CIkJxASGVHSE4gpLIjJCcQUtkRkhMIqewIyQmEVHaE5ARCKjtC\ncgIhlR0hOYGQyo6QnEBIZeddSN8Zb/sIdBj/HdtHoMGKFbaPQCHvQjqw3fYR6ODk+FmanTtt\nH4FC3oUE2EBIgAKEBChASIAChAQoQEiAAoQEKEBIgAKEBChASIAChAQoQEiAAoQEKEBIgAKE\nBChASIACLoe0bcnk3hnX7Bn6xP6/fd0xo8fPuu6Z+q3bLv2LMeLC2J1LKuVFtXz9xMY/6jDZ\nwmHmlOdVOfNWtXA4pC3jRsxfdaqYt7f5mWfFka+84HWTxNGPhbfmiMNOaIY0fOdySntRLV8/\nsas/9G5Lx5pdrlflylvVyuGQ5oqbguDAYrGm+ZmD9Xfl+aViefjxB48cvKMZ0vCdyyntRbV8\n/cReC0fYiVyvypW3qpW7IW0Ss8IPj3dNPdj2lbvFqwe2miHF71wuaS+q9euuhJTrVbnyVrVx\nN6S1YnX94yyxte0rl4lVA1vNkOJ3Lpe0F9X69RO7r3/7JZ97xugRdiLXq3LlrWrjbkjLxLr6\nx0ViY+Szq1a+eYaY+buBW82Q5DuXT9qLav16Y7Fh7JeMH2ZOuV6VK29VG3dDWig21D+uELdE\nPjumdmW97reDt5ohyXcun7QX1fr1D921Y+9Dl3aNvNf0YeaU61W58la1cS+kA5eEftk84cvF\n+uiXD+7452lHbhq4MSyktp1LI+uLkn39GnGuuQPNpaNXVfa3KoZ7Ie2rfz/zfxK+BXhIzBzY\ncuZbu6wvSvb1R8UEQ0eZV0evquxvVQz3Qho0+EPp7OE/lB4lBv4Oz2GLDZKdSyXtRcm+vlOM\nNXR0ncr1qlx5q9q4G9ImMTv88ETXlPZl0j+NFH9qbEWWv+N2Lpe0FyX7+gZxirkD7EiuV+XK\nW9XG3ZCCueLm2rfhSxu/cXfTJ54Mgvv/Pdx8+jzxyoFdor8hG9m5xNJeVPTrD/ws/MxPjhY3\n2DrarPK8KmfeqlYOh7Slr2vB5XPEafVRkuPET4Lgw+LYsy44Y7Q46hfhp27r7z9LTO/vv3LY\nziWW9qKiX18rjjv7/NkjxBtfsHrIGeR5Vc68Va0cDinYtnhSz7FX765v19+dh6+cM3Fk39zr\nGj8hXdMY6RTThu1cZikvKvr1B5effPioia9d78C3QDlelTtvVQuXQwJKg5AABQgJUICQAAUI\nCVCAkAAFCAlQgJAABQgJUICQAAUICVCAkAAFCAlQgJAABQgJUICQAAUICVCAkAAFCAlQgJAA\nBQgJUICQAAUICVCAkAAFCAlQgJAABQgJUICQAAUICVCAkAAFCAlQgJAABQgJUICQSmWz6Nfz\nwAc+dOIh4kY9j42AkEomU0iPNP+J6exuFPOu+/Cm/AeEjAipVLSFdI54Kv/RIDtCKhVtIZ08\nMv/BIAdCKpVaSD+fP/7PXvH9xs0vn3HoIX/+4edatz/c+Mfa18fdrXZj24WTRtwfBPedP7n7\nqKW/CIIr6/foM/1qqoSQSmWzOKPvzA+8Y/TIDeGt94gj3nnVS8SrXmjZfugGMW/9+vWPxt1t\ns3jNhBMvOn9z8PmuSW9736KeMT8ONq2f2rV+/VesvKSKIKRS2SzE+2ofHuyeuCcI7hUv/l0Q\n7DtXXN+6Pfxbu5a71W5cur926+Huc/bWPvxs7Mzaf0/iWzu9CKlUNotxu8KP/eKfg+Ct4qZw\n++ERL27dloUUudtmEeYUBJeKe54KLRCPEZJ2hFQqm8WZ9Y//K/wVZmZYQM3R4vct27KQInfb\nLM6u35gjBt1PSNoRUqlsFm+uf7xDXBwE00RjmWFOLaLotiykyN02i7fUb0wXG+9q+AMhaUdI\npaLmV6T++o1TxANDexCSZoRUKi0/7PSLdeH21vDnouj2o+KCpLsNhrRSXDG0ByFpRkil0rL8\ndo847ukg2Pd68cHW7T+KuUl3Gwxpy6ju74Ufd9XiIiTdCKlUmr8hdHt46wox+ZL3vFS84vm2\n7dPEm69bsyXubs3piC+MGnHO+98zf8xJASFpR0ilUh9RGDf6jO81bt76F2N7T/rgs+3bj7xh\n/Ijhkw2DdxsaM9p80Yt6xp908Q8CQtKOkHyg7U9fICtC8gEhWUdIrtq3o2kfIVlHSK76SXNw\nQfyEkKwjJFf96a6mP9k+FhASoAIhAQoQEqAAIQEKEBKgACEBChASoAAhAQoQEqAAIQEKEBKg\nACEBChASoAAhAQoQEqAAIQEKEBKgACEBChASoAAhAQoQEqAAIQEKEBKgACEBChASoAAhAQr8\nf0jb4v36gatyAAAAAElFTkSuQmCC",
      "text/plain": [
       "Plot with title “Histogram of boot_pref”"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 420,
       "width": 420
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "boot_pref <- interleaved_bootstraps(visited$session_id, visited$team)\n",
    "preference <- interleaved_preference(visited$session_id, visited$team)\n",
    "hist(boot_pref, col = \"gray70\", border = NA)\n",
    "abline(v = quantile(boot_pref, c(0.025, 0.975)), lty = \"dashed\")\n",
    "abline(v = preference, lwd = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Positive statistic would indicate a preference for team \"A\" results (control, legacy search), but since the statistic and the 95% confidence interval are negative, that indicates that there was a strong preference for team \"B\" results (from the new search algorithm)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}