ElasticON 2020 - Using machine learning to detect DGA with >99.9% accuracy

README.md

Using machine learning to detect DGA with >99.9% accuracy

Supporting material for ElasticON 2020 talk on detecting DGA activity.

Note, full model, configuration and details on how to deploy the model will shortly be available in https://github.com/elastic/detection-rules

dga_ElasticON_2020.ipynb shows how the model was trained.

Supporting data files are available https://storage.googleapis.com/elastic-ml-public/elasticon_2020_dga/dga_ElasticON_2020.tar

dga_ElasticON_2020.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "Train a production DGA model based on a'realistic' DGA dataset similar to feeds from [DGArchive](https://dgarchive.caad.fkie.fraunhofer.de/) and [netlab360](https://data.netlab.360.com/dga/).\n",
    "\n",
    "## Elasticsearch Cluster\n",
    "\n",
    "Elastic cloud was used to train and test the model. The Elasticsearch cluster was provisioned on GCP and consisted of 2 instances:\n",
    "\n",
    "```\n",
    "Instance #0, v7.9.2, 8 GB RAM, GCP.DATA.HIGHIO.1, (data, master, coordinating, ingest)\n",
    "Instance #1, v7.9.2, 64 GB RAM, GCP.ML.1, (ml)\n",
    "```\n",
    "\n",
    "Smaller ML instances can be used, but training may take longer and the number of threads for training should be reduced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import eland as ed\n",
    "import elasticsearch\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import datetime\n",
    "import json\n",
    "import os\n",
    "import re\n",
    "from elasticsearch import Elasticsearch\n",
    "from elasticsearch import helpers\n",
    "import time\n",
    "import gzip\n",
    "import io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Training Data\n",
    "\n",
    "The training dataset was generated from a combination of a netlab360 feed + generator python scripts. The resulting dataset contains 75 different variants of malware. The proportions of the different dga families are similar to netlab and dgarchive feeds.\n",
    "\n",
    "This dataset is then used to make real DNS lookups and the network traffic is captured by packetbeat. The results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train_dedup = pd.read_pickle('train_437564_2020-07-06.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "437564"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df_train_dedup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tinba               93759\n",
       "banjori             72443\n",
       "emotet              52496\n",
       "gameover            36344\n",
       "necurs              25487\n",
       "rovnix              24541\n",
       "ramnit              19422\n",
       "qakbot              18693\n",
       "murofet             16791\n",
       "simda               10972\n",
       "pykspa2s            10719\n",
       "ranbyus              7983\n",
       "virut                6049\n",
       "urlzone              6014\n",
       "dyre                 4269\n",
       "cryptolocker         3236\n",
       "locky                3161\n",
       "symmi                2678\n",
       "monerodownloader     2500\n",
       "qadars               2302\n",
       "xxhex                2176\n",
       "shifu                1429\n",
       "vawtrak              1339\n",
       "qsnatch              1310\n",
       "ramdo                1000\n",
       "ebury                 983\n",
       "chinad                953\n",
       "pushdo                818\n",
       "dircrypt              736\n",
       "fobber                597\n",
       "                    ...  \n",
       "bamital               176\n",
       "oderoor               169\n",
       "matsnu                154\n",
       "pykspa                134\n",
       "hesperbot             128\n",
       "zloader               128\n",
       "sphinx                128\n",
       "feodo                 125\n",
       "pykspa_v2_fake         93\n",
       "dmsniff                58\n",
       "enviserv               57\n",
       "downloader             49\n",
       "gspy                   44\n",
       "sisron                 40\n",
       "wd                     39\n",
       "gozi                   37\n",
       "nymaim2                34\n",
       "infy                   33\n",
       "darkshell              32\n",
       "tsifiri                30\n",
       "redyms                 16\n",
       "kingminer              15\n",
       "omexo                  14\n",
       "qhost                  13\n",
       "tofsee                 12\n",
       "torpig                  5\n",
       "diamondfox              3\n",
       "mirai                   2\n",
       "blackhole               1\n",
       "madmax                  1\n",
       "Name: dga_family, Length: 75, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('max_rows', 60)\n",
    "pd.set_option('min_rows', 60)\n",
    "\n",
    "df_train_dedup.dga_family.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df_train_dedup.domain.to_csv('/tmp/train_437564', index=False, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now generate real packetbeat data for these domains:\n",
    "\n",
    "```\n",
    "#!/bin/bash\n",
    "\n",
    "ROOT=\"/data/dga/machine-learning-data/solutions/security/dga\"\n",
    "\n",
    "DATE=\"`date --iso-8601`T07+00:00\"\n",
    "\n",
    "TRAIN_PCAP=train_437564-$DATE.pcap\n",
    "TRAIN_JSON=train_437564-$DATE.json\n",
    "tcpdump -i any udp port 53 -w $TRAIN_PCAP &\n",
    "TCPDUMP_PID=$!\n",
    "sleep 5\n",
    "\n",
    "cat /tmp/train_437564 | $ROOT/dns_benchmark/1000000/c_code/zdns/zdns/zdns A\n",
    "\n",
    "sleep 5\n",
    "echo \"kill $TCPDUMP_PID\"\n",
    "kill $TCPDUMP_PID\n",
    "\n",
    "$ROOT/dns_benchmark/packetbeat-7.7.0-linux-x86_64/packetbeat -e -I $TRAIN_PCAP -t -c $ROOT/dns_benchmark/packetbeat.console.yml > $TRAIN_JSON\n",
    "sleep 2\n",
    "\n",
    "gzip $TRAIN_PCAP\n",
    "gzip $TRAIN_JSON\n",
    "gsutil cp $TRAIN_PCAP.gz gs://ml-dga/train/\n",
    "gsutil cp $TRAIN_JSON.gz gs://ml-dga/train/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add data to Elasticsearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# connect to cloud instance\n",
    "es = elasticsearch.Elasticsearch(\n",
    "    cloud_id=\"dga-train-test:xxx\",\n",
    "    http_auth=(\"elastic\", \"xxx\"),\n",
    "    retry_on_timeout=True, \n",
    "    timeout=120\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'instance-0000000000',\n",
       " 'cluster_name': '3aacc9e46db848bc9af9c7dbca54f5cd',\n",
       " 'cluster_uuid': 'CKvYxc2eTBeIIEWICO60vw',\n",
       " 'version': {'number': '7.9.2',\n",
       "  'build_flavor': 'default',\n",
       "  'build_type': 'docker',\n",
       "  'build_hash': 'd34da0ea4a966c4e49417f2da2f244e3e97b4e6e',\n",
       "  'build_date': '2020-09-23T00:45:33.626720Z',\n",
       "  'build_snapshot': False,\n",
       "  'lucene_version': '8.6.2',\n",
       "  'minimum_wire_compatibility_version': '6.8.0',\n",
       "  'minimum_index_compatibility_version': '6.0.0-beta1'},\n",
       " 'tagline': 'You Know, for Search'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'acknowledged': True}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add packetbeat template (this can also be done by calling ./packetbeat setup)\n",
    "template_definition = {\"index_patterns\":[\"packetbeat-7.9.2-*\"],\"settings\":{\"index\":{\"lifecycle\":{\"name\":\"packetbeat\",\"rollover_alias\":\"packetbeat-7.9.2\"},\"mapping\":{\"total_fields\":{\"limit\":\"10000\"}},\"refresh_interval\":\"5s\",\"number_of_shards\":\"1\",\"max_docvalue_fields_search\":\"200\",\"query\":{\"default_field\":[\"message\",\"tags\",\"agent.ephemeral_id\",\"agent.id\",\"agent.name\",\"agent.type\",\"agent.version\",\"as.organization.name\",\"client.address\",\"client.as.organization.name\",\"client.domain\",\"client.geo.city_name\",\"client.geo.continent_name\",\"client.geo.country_iso_code\",\"client.geo.country_name\",\"client.geo.name\",\"client.geo.region_iso_code\",\"client.geo.region_name\",\"client.mac\",\"client.registered_domain\",\"client.top_level_domain\",\"client.user.domain\",\"client.user.email\",\"client.user.full_name\",\"client.user.group.domain\",\"client.user.group.id\",\"client.user.group.name\",\"client.user.hash\",\"client.user.id\",\"client.user.name\",\"cloud.account.id\",\"cloud.availability_zone\",\"cloud.instance.id\",\"cloud.instance.name\",\"cloud.machine.type\",\"cloud.provider\",\"cloud.region\",\"container.id\",\"container.image.name\",\"container.image.tag\",\"container.name\",\"container.runtime\",\"destination.address\",\"destination.as.organization.name\",\"destination.domain\",\"destination.geo.city_name\",\"destination.geo.continent_name\",\"destination.geo.country_iso_code\",\"destination.geo.country_name\",\"destination.geo.name\",\"destination.geo.region_iso_code\",\"destination.geo.region_name\",\"destination.mac\",\"destination.registered_domain\",\"destination.top_level_domain\",\"destination.user.domain\",\"destination.user.email\",\"destination.user.full_name\",\"destination.user.group.domain\",\"destination.user.group.id\",\"destination.user.group.name\",\"destination.user.hash\",\"destination.user.id\",\"destination.user.name\",\"dns.answers.class\",\"dns.answers.data\",\"dns.answers.name\",\"dns.answers.type\",\"dns.header_flags\",\"dns.id\",\"dns.op_code\",\"dns.question.class\",\"dns.question.name\",\"dns.question.registered_domain\",\"dns.question.subdomain\",\"dns.question.top_level_domain\",\"dns.question.type\",\"dns.response_code\",\"dns.type\",\"ecs.version\",\"error.code\",\"error.id\",\"error.message\",\"error.stack_trace\",\"error.type\",\"event.action\",\"event.category\",\"event.code\",\"event.dataset\",\"event.hash\",\"event.id\",\"event.kind\",\"event.module\",\"event.original\",\"event.outcome\",\"event.provider\",\"event.timezone\",\"event.type\",\"file.device\",\"file.directory\",\"file.extension\",\"file.gid\",\"file.group\",\"file.hash.md5\",\"file.hash.sha1\",\"file.hash.sha256\",\"file.hash.sha512\",\"file.inode\",\"file.mode\",\"file.name\",\"file.owner\",\"file.path\",\"file.target_path\",\"file.type\",\"file.uid\",\"geo.city_name\",\"geo.continent_name\",\"geo.country_iso_code\",\"geo.country_name\",\"geo.name\",\"geo.region_iso_code\",\"geo.region_name\",\"group.domain\",\"group.id\",\"group.name\",\"hash.md5\",\"hash.sha1\",\"hash.sha256\",\"hash.sha512\",\"host.architecture\",\"host.geo.city_name\",\"host.geo.continent_name\",\"host.geo.country_iso_code\",\"host.geo.country_name\",\"host.geo.name\",\"host.geo.region_iso_code\",\"host.geo.region_name\",\"host.hostname\",\"host.id\",\"host.mac\",\"host.name\",\"host.os.family\",\"host.os.full\",\"host.os.kernel\",\"host.os.name\",\"host.os.platform\",\"host.os.version\",\"host.type\",\"host.user.domain\",\"host.user.email\",\"host.user.full_name\",\"host.user.group.domain\",\"host.user.group.id\",\"host.user.group.name\",\"host.user.hash\",\"host.user.id\",\"host.user.name\",\"http.request.body.content\",\"http.request.method\",\"http.request.referrer\",\"http.response.body.content\",\"http.version\",\"log.level\",\"log.logger\",\"log.origin.file.name\",\"log.origin.function\",\"log.original\",\"log.syslog.facility.name\",\"log.syslog.severity.name\",\"network.application\",\"network.community_id\",\"network.direction\",\"network.iana_number\",\"network.name\",\"network.protocol\",\"network.transport\",\"network.type\",\"observer.geo.city_name\",\"observer.geo.continent_name\",\"observer.geo.country_iso_code\",\"observer.geo.country_name\",\"observer.geo.name\",\"observer.geo.region_iso_code\",\"observer.geo.region_name\",\"observer.hostname\",\"observer.mac\",\"observer.name\",\"observer.os.family\",\"observer.os.full\",\"observer.os.kernel\",\"observer.os.name\",\"observer.os.platform\",\"observer.os.version\",\"observer.product\",\"observer.serial_number\",\"observer.type\",\"observer.vendor\",\"observer.version\",\"organization.id\",\"organization.name\",\"os.family\",\"os.full\",\"os.kernel\",\"os.name\",\"os.platform\",\"os.version\",\"package.architecture\",\"package.checksum\",\"package.description\",\"package.install_scope\",\"package.license\",\"package.name\",\"package.path\",\"package.version\",\"process.args\",\"text\",\"process.executable\",\"process.hash.md5\",\"process.hash.sha1\",\"process.hash.sha256\",\"process.hash.sha512\",\"process.name\",\"text\",\"text\",\"text\",\"text\",\"text\",\"process.thread.name\",\"process.title\",\"process.working_directory\",\"server.address\",\"server.as.organization.name\",\"server.domain\",\"server.geo.city_name\",\"server.geo.continent_name\",\"server.geo.country_iso_code\",\"server.geo.country_name\",\"server.geo.name\",\"server.geo.region_iso_code\",\"server.geo.region_name\",\"server.mac\",\"server.registered_domain\",\"server.top_level_domain\",\"server.user.domain\",\"server.user.email\",\"server.user.full_name\",\"server.user.group.domain\",\"server.user.group.id\",\"server.user.group.name\",\"server.user.hash\",\"server.user.id\",\"server.user.name\",\"service.ephemeral_id\",\"service.id\",\"service.name\",\"service.node.name\",\"service.state\",\"service.type\",\"service.version\",\"source.address\",\"source.as.organization.name\",\"source.domain\",\"source.geo.city_name\",\"source.geo.continent_name\",\"source.geo.country_iso_code\",\"source.geo.country_name\",\"source.geo.name\",\"source.geo.region_iso_code\",\"source.geo.region_name\",\"source.mac\",\"source.registered_domain\",\"source.top_level_domain\",\"source.user.domain\",\"source.user.email\",\"source.user.full_name\",\"source.user.group.domain\",\"source.user.group.id\",\"source.user.group.name\",\"source.user.hash\",\"source.user.id\",\"source.user.name\",\"threat.framework\",\"threat.tactic.id\",\"threat.tactic.name\",\"threat.tactic.reference\",\"threat.technique.id\",\"threat.technique.name\",\"threat.technique.reference\",\"tracing.trace.id\",\"tracing.transaction.id\",\"url.domain\",\"url.extension\",\"url.fragment\",\"url.full\",\"url.original\",\"url.password\",\"url.path\",\"url.query\",\"url.registered_domain\",\"url.scheme\",\"url.top_level_domain\",\"url.username\",\"user.domain\",\"user.email\",\"user.full_name\",\"user.group.domain\",\"user.group.id\",\"user.group.name\",\"user.hash\",\"user.id\",\"user.name\",\"user_agent.device.name\",\"user_agent.name\",\"text\",\"user_agent.original\",\"user_agent.os.family\",\"user_agent.os.full\",\"user_agent.os.kernel\",\"user_agent.os.name\",\"user_agent.os.platform\",\"user_agent.os.version\",\"user_agent.version\",\"text\",\"agent.hostname\",\"timeseries.instance\",\"cloud.project.id\",\"cloud.image.id\",\"host.os.build\",\"host.os.codename\",\"kubernetes.pod.name\",\"kubernetes.pod.uid\",\"kubernetes.namespace\",\"kubernetes.node.name\",\"kubernetes.replicaset.name\",\"kubernetes.deployment.name\",\"kubernetes.statefulset.name\",\"kubernetes.container.name\",\"kubernetes.container.image\",\"jolokia.agent.version\",\"jolokia.agent.id\",\"jolokia.server.product\",\"jolokia.server.version\",\"jolokia.server.vendor\",\"jolokia.url\",\"type\",\"server.process.name\",\"server.process.args\",\"server.process.executable\",\"server.process.working_directory\",\"server.process.start\",\"client.process.name\",\"client.process.args\",\"client.process.executable\",\"client.process.working_directory\",\"client.process.start\",\"flow.id\",\"status\",\"method\",\"resource\",\"path\",\"query\",\"params\",\"request\",\"response\",\"amqp.reply-text\",\"amqp.exchange\",\"amqp.exchange-type\",\"amqp.consumer-tag\",\"amqp.routing-key\",\"amqp.queue\",\"amqp.content-type\",\"amqp.content-encoding\",\"amqp.delivery-mode\",\"amqp.correlation-id\",\"amqp.reply-to\",\"amqp.expiration\",\"amqp.message-id\",\"amqp.timestamp\",\"amqp.type\",\"amqp.user-id\",\"amqp.app-id\",\"cassandra.request.headers.flags\",\"cassandra.request.headers.stream\",\"cassandra.request.headers.op\",\"cassandra.request.query\",\"cassandra.response.headers.flags\",\"cassandra.response.headers.stream\",\"cassandra.response.headers.op\",\"cassandra.response.result.type\",\"cassandra.response.result.rows.meta.keyspace\",\"cassandra.response.result.rows.meta.table\",\"cassandra.response.result.rows.meta.flags\",\"cassandra.response.result.rows.meta.paging_state\",\"cassandra.response.result.keyspace\",\"cassandra.response.result.schema_change.change\",\"cassandra.response.result.schema_change.keyspace\",\"cassandra.response.result.schema_change.table\",\"cassandra.response.result.schema_change.object\",\"cassandra.response.result.schema_change.target\",\"cassandra.response.result.schema_change.name\",\"cassandra.response.result.schema_change.args\",\"cassandra.response.result.prepared.prepared_id\",\"cassandra.response.result.prepared.req_meta.keyspace\",\"cassandra.response.result.prepared.req_meta.table\",\"cassandra.response.result.prepared.req_meta.flags\",\"cassandra.response.result.prepared.req_meta.paging_state\",\"cassandra.response.result.prepared.resp_meta.keyspace\",\"cassandra.response.result.prepared.resp_meta.table\",\"cassandra.response.result.prepared.resp_meta.flags\",\"cassandra.response.result.prepared.resp_meta.paging_state\",\"cassandra.response.authentication.class\",\"cassandra.response.warnings\",\"cassandra.response.event.type\",\"cassandra.response.event.change\",\"cassandra.response.event.host\",\"cassandra.response.event.schema_change.change\",\"cassandra.response.event.schema_change.keyspace\",\"cassandra.response.event.schema_change.table\",\"cassandra.response.event.schema_change.object\",\"cassandra.response.event.schema_change.target\",\"cassandra.response.event.schema_change.name\",\"cassandra.response.event.schema_change.args\",\"cassandra.response.error.msg\",\"cassandra.response.error.type\",\"cassandra.response.error.details.read_consistency\",\"cassandra.response.error.details.write_type\",\"cassandra.response.error.details.keyspace\",\"cassandra.response.error.details.table\",\"cassandra.response.error.details.stmt_id\",\"cassandra.response.error.details.num_failures\",\"cassandra.response.error.details.function\",\"cassandra.response.error.details.arg_types\",\"dhcpv4.transaction_id\",\"dhcpv4.flags\",\"dhcpv4.client_mac\",\"dhcpv4.server_name\",\"dhcpv4.op_code\",\"dhcpv4.hardware_type\",\"dhcpv4.option.message_type\",\"dhcpv4.option.parameter_request_list\",\"dhcpv4.option.class_identifier\",\"dhcpv4.option.domain_name\",\"dhcpv4.option.hostname\",\"dhcpv4.option.message\",\"dhcpv4.option.boot_file_name\",\"dns.question.etld_plus_one\",\"dns.authorities.name\",\"dns.authorities.type\",\"dns.authorities.class\",\"dns.additionals.name\",\"dns.additionals.type\",\"dns.additionals.class\",\"dns.additionals.data\",\"dns.opt.version\",\"dns.opt.ext_rcode\",\"http.response.status_phrase\",\"icmp.version\",\"icmp.request.message\",\"icmp.response.message\",\"memcache.protocol_type\",\"memcache.request.line\",\"memcache.request.command\",\"memcache.response.command\",\"memcache.request.type\",\"memcache.response.type\",\"memcache.response.error_msg\",\"memcache.request.opcode\",\"memcache.response.opcode\",\"memcache.response.status\",\"memcache.request.raw_args\",\"memcache.request.automove\",\"memcache.response.version\",\"mongodb.error\",\"mongodb.fullCollectionName\",\"mongodb.startingFrom\",\"mongodb.query\",\"mongodb.returnFieldsSelector\",\"mongodb.selector\",\"mongodb.update\",\"mongodb.cursorId\",\"mysql.insert_id\",\"mysql.num_fields\",\"mysql.num_rows\",\"mysql.query\",\"mysql.error_message\",\"nfs.tag\",\"nfs.opcode\",\"nfs.status\",\"rpc.xid\",\"rpc.status\",\"rpc.auth_flavor\",\"rpc.cred.gids\",\"rpc.cred.machinename\",\"pgsql.error_message\",\"pgsql.error_severity\",\"pgsql.num_fields\",\"pgsql.num_rows\",\"redis.return_value\",\"redis.error\",\"thrift.params\",\"thrift.service\",\"thrift.return_value\",\"thrift.exceptions\",\"tls.client.x509.version\",\"tls.client.x509.version_number\",\"tls.client.x509.serial_number\",\"tls.client.x509.issuer.distinguished_name\",\"tls.client.x509.issuer.common_name\",\"tls.client.x509.issuer.organizational_unit\",\"tls.client.x509.issuer.organization\",\"tls.client.x509.issuer.locality\",\"tls.client.x509.issuer.province\",\"tls.client.x509.issuer.state_or_province\",\"tls.client.x509.issuer.country\",\"tls.client.x509.signature_algorithm\",\"tls.client.x509.subject.distinguished_name\",\"tls.client.x509.subject.common_name\",\"tls.client.x509.subject.organizational_unit\",\"tls.client.x509.subject.organization\",\"tls.client.x509.subject.locality\",\"tls.client.x509.subject.province\",\"tls.client.x509.subject.state_or_province\",\"tls.client.x509.subject.country\",\"tls.client.x509.public_key_algorithm\",\"tls.client.x509.alternative_names\",\"tls.server.x509.version\",\"tls.server.x509.version_number\",\"tls.server.x509.serial_number\",\"tls.server.x509.issuer.distinguished_name\",\"tls.server.x509.issuer.common_name\",\"tls.server.x509.issuer.organizational_unit\",\"tls.server.x509.issuer.organization\",\"tls.server.x509.issuer.locality\",\"tls.server.x509.issuer.province\",\"tls.server.x509.issuer.state_or_province\",\"tls.server.x509.issuer.country\",\"tls.server.x509.signature_algorithm\",\"tls.server.x509.subject.distinguished_name\",\"tls.server.x509.subject.common_name\",\"tls.server.x509.subject.organizational_unit\",\"tls.server.x509.subject.organization\",\"tls.server.x509.subject.locality\",\"tls.server.x509.subject.province\",\"tls.server.x509.subject.state_or_province\",\"tls.server.x509.subject.country\",\"tls.server.x509.public_key_algorithm\",\"tls.server.x509.alternative_names\",\"tls.detailed.version\",\"tls.detailed.resumption_method\",\"tls.detailed.client_hello.version\",\"tls.detailed.client_hello.session_id\",\"tls.detailed.client_hello.supported_compression_methods\",\"tls.detailed.client_hello.extensions.server_name_indication\",\"tls.detailed.client_hello.extensions.application_layer_protocol_negotiation\",\"tls.detailed.client_hello.extensions.session_ticket\",\"tls.detailed.client_hello.extensions.supported_versions\",\"tls.detailed.client_hello.extensions.supported_groups\",\"tls.detailed.client_hello.extensions.signature_algorithms\",\"tls.detailed.client_hello.extensions.ec_points_formats\",\"tls.detailed.client_hello.extensions._unparsed_\",\"tls.detailed.server_hello.version\",\"tls.detailed.server_hello.selected_compression_method\",\"tls.detailed.server_hello.session_id\",\"tls.detailed.server_hello.extensions.application_layer_protocol_negotiation\",\"tls.detailed.server_hello.extensions.session_ticket\",\"tls.detailed.server_hello.extensions.supported_versions\",\"tls.detailed.server_hello.extensions.ec_points_formats\",\"tls.detailed.server_hello.extensions._unparsed_\",\"tls.detailed.client_certificate.version_number\",\"tls.detailed.client_certificate.serial_number\",\"tls.detailed.client_certificate.public_key_algorithm\",\"tls.detailed.client_certificate.signature_algorithm\",\"tls.detailed.client_certificate.alternative_names\",\"tls.detailed.client_certificate.subject.country\",\"tls.detailed.client_certificate.subject.organization\",\"tls.detailed.client_certificate.subject.organizational_unit\",\"tls.detailed.client_certificate.subject.province\",\"tls.detailed.client_certificate.subject.common_name\",\"tls.detailed.client_certificate.subject.locality\",\"tls.detailed.client_certificate.subject.distinguished_name\",\"tls.detailed.client_certificate.issuer.country\",\"tls.detailed.client_certificate.issuer.organization\",\"tls.detailed.client_certificate.issuer.organizational_unit\",\"tls.detailed.client_certificate.issuer.province\",\"tls.detailed.client_certificate.issuer.common_name\",\"tls.detailed.client_certificate.issuer.locality\",\"tls.detailed.client_certificate.issuer.distinguished_name\",\"tls.detailed.server_certificate.version_number\",\"tls.detailed.server_certificate.serial_number\",\"tls.detailed.server_certificate.public_key_algorithm\",\"tls.detailed.server_certificate.signature_algorithm\",\"tls.detailed.server_certificate.alternative_names\",\"tls.detailed.server_certificate.subject.country\",\"tls.detailed.server_certificate.subject.organization\",\"tls.detailed.server_certificate.subject.organizational_unit\",\"tls.detailed.server_certificate.subject.province\",\"tls.detailed.server_certificate.subject.state_or_province\",\"tls.detailed.server_certificate.subject.common_name\",\"tls.detailed.server_certificate.subject.locality\",\"tls.detailed.server_certificate.subject.distinguished_name\",\"tls.detailed.server_certificate.issuer.country\",\"tls.detailed.server_certificate.issuer.organization\",\"tls.detailed.server_certificate.issuer.organizational_unit\",\"tls.detailed.server_certificate.issuer.province\",\"tls.detailed.server_certificate.issuer.state_or_province\",\"tls.detailed.server_certificate.issuer.common_name\",\"tls.detailed.server_certificate.issuer.locality\",\"tls.detailed.server_certificate.issuer.distinguished_name\",\"tls.detailed.alert_types\",\"fields.*\"]}}},\"mappings\":{\"_meta\":{\"beat\":\"packetbeat\",\"version\":\"7.9.1\"},\"dynamic_templates\":[{\"labels\":{\"path_match\":\"labels.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"container.labels\":{\"path_match\":\"container.labels.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"dns.answers\":{\"path_match\":\"dns.answers.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"log.syslog\":{\"path_match\":\"log.syslog.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"network.inner\":{\"path_match\":\"network.inner.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"observer.egress\":{\"path_match\":\"observer.egress.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"observer.ingress\":{\"path_match\":\"observer.ingress.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"fields\":{\"path_match\":\"fields.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"docker.container.labels\":{\"path_match\":\"docker.container.labels.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"kubernetes.labels.*\":{\"path_match\":\"kubernetes.labels.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"*\"}},{\"kubernetes.annotations.*\":{\"path_match\":\"kubernetes.annotations.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"*\"}},{\"amqp.headers\":{\"path_match\":\"amqp.headers.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"cassandra.response.supported\":{\"path_match\":\"cassandra.response.supported.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"http.request.headers\":{\"path_match\":\"http.request.headers.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"http.response.headers\":{\"path_match\":\"http.response.headers.*\",\"mapping\":{\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}},{\"strings_as_keyword\":{\"mapping\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"match_mapping_type\":\"string\"}}],\"properties\":{\"container\":{\"properties\":{\"image\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"tag\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"runtime\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"labels\":{\"type\":\"object\"}}},\"kubernetes\":{\"properties\":{\"container\":{\"properties\":{\"image\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"node\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"pod\":{\"properties\":{\"uid\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"statefulset\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"namespace\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"annotations\":{\"properties\":{\"*\":{\"type\":\"object\"}}},\"replicaset\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"labels\":{\"properties\":{\"*\":{\"type\":\"object\"}}},\"deployment\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"agent\":{\"properties\":{\"hostname\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ephemeral_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"notes\":{\"path\":\"error.message\",\"type\":\"alias\"},\"source\":{\"properties\":{\"nat\":{\"properties\":{\"port\":{\"type\":\"long\"},\"ip\":{\"type\":\"ip\"}}},\"address\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"top_level_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ip\":{\"type\":\"ip\"},\"mac\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"packets\":{\"type\":\"long\"},\"geo\":{\"properties\":{\"region_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"continent_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"city_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"location\":{\"type\":\"geo_point\"},\"region_name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"as\":{\"properties\":{\"number\":{\"type\":\"long\"},\"organization\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}}}}}},\"registered_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"port\":{\"type\":\"long\"},\"bytes\":{\"type\":\"long\"},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"user\":{\"properties\":{\"full_name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}},\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"email\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"group\":{\"properties\":{\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}}}},\"icmp\":{\"properties\":{\"request\":{\"properties\":{\"code\":{\"type\":\"long\"},\"message\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"type\":\"long\"}}},\"response\":{\"properties\":{\"code\":{\"type\":\"long\"},\"type\":{\"type\":\"long\"},\"message\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"redis\":{\"properties\":{\"return_value\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"error\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"cloud\":{\"properties\":{\"image\":{\"properties\":{\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"availability_zone\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"instance\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"provider\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"machine\":{\"properties\":{\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"project\":{\"properties\":{\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"region\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"account\":{\"properties\":{\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"path\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"observer\":{\"properties\":{\"product\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"os\":{\"properties\":{\"kernel\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"family\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"platform\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"full\":{\"ignore_above\":1024,\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}},\"type\":\"keyword\"}}},\"ip\":{\"type\":\"ip\"},\"serial_number\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"mac\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"egress\":{\"type\":\"object\",\"properties\":{\"vlan\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"zone\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"interface\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"alias\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"geo\":{\"properties\":{\"region_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"continent_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"city_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"region_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"location\":{\"type\":\"geo_point\"}}},\"ingress\":{\"type\":\"object\",\"properties\":{\"vlan\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"zone\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"interface\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"alias\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"hostname\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"vendor\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"timeseries\":{\"properties\":{\"instance\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"ecs\":{\"properties\":{\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"code_signature\":{\"properties\":{\"valid\":{\"type\":\"boolean\"},\"trusted\":{\"type\":\"boolean\"},\"subject_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"exists\":{\"type\":\"boolean\"},\"status\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"host\":{\"properties\":{\"os\":{\"properties\":{\"build\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"kernel\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"codename\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"family\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"platform\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"full\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}}}},\"ip\":{\"type\":\"ip\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"mac\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"uptime\":{\"type\":\"long\"},\"geo\":{\"properties\":{\"region_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"continent_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"city_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"location\":{\"type\":\"geo_point\"},\"region_name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"hostname\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"containerized\":{\"type\":\"boolean\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"user\":{\"properties\":{\"full_name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"email\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"group\":{\"properties\":{\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"architecture\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"mysql\":{\"properties\":{\"error_message\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"insert_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"query\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"num_fields\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"num_rows\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"error_code\":{\"type\":\"long\"},\"affected_rows\":{\"type\":\"long\"}}},\"memcache\":{\"properties\":{\"protocol_type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"request\":{\"properties\":{\"count_values\":{\"type\":\"long\"},\"opaque\":{\"type\":\"long\"},\"sleep_us\":{\"type\":\"long\"},\"noreply\":{\"type\":\"boolean\"},\"initial\":{\"type\":\"long\"},\"line\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"delta\":{\"type\":\"long\"},\"flags\":{\"type\":\"long\"},\"cas_unique\":{\"type\":\"long\"},\"automove\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"opcode\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"command\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"raw_args\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"exptime\":{\"type\":\"long\"},\"bytes\":{\"type\":\"long\"},\"dest_class\":{\"type\":\"long\"},\"source_class\":{\"type\":\"long\"},\"vbucket\":{\"type\":\"long\"},\"quiet\":{\"type\":\"boolean\"},\"opcode_value\":{\"type\":\"long\"},\"verbosity\":{\"type\":\"long\"}}},\"response\":{\"properties\":{\"count_values\":{\"type\":\"long\"},\"opaque\":{\"type\":\"long\"},\"error_msg\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"status_code\":{\"type\":\"long\"},\"flags\":{\"type\":\"long\"},\"cas_unique\":{\"type\":\"long\"},\"opcode\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"command\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"bytes\":{\"type\":\"long\"},\"opcode_value\":{\"type\":\"long\"},\"value\":{\"type\":\"long\"},\"status\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"flow\":{\"properties\":{\"vlan\":{\"type\":\"long\"},\"final\":{\"type\":\"boolean\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"group\":{\"properties\":{\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"tracing\":{\"properties\":{\"trace\":{\"properties\":{\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"transaction\":{\"properties\":{\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"dhcpv4\":{\"properties\":{\"transaction_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"client_mac\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"server_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"relay_ip\":{\"type\":\"ip\"},\"flags\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"assigned_ip\":{\"type\":\"ip\"},\"op_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"seconds\":{\"type\":\"long\"},\"hardware_type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"server_ip\":{\"type\":\"ip\"},\"client_ip\":{\"type\":\"ip\"},\"hops\":{\"type\":\"long\"},\"option\":{\"properties\":{\"class_identifier\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"max_dhcp_message_size\":{\"type\":\"long\"},\"boot_file_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"message_type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"utc_time_offset_sec\":{\"type\":\"long\"},\"ntp_servers\":{\"type\":\"ip\"},\"message\":{\"norms\":False,\"type\":\"text\"},\"broadcast_address\":{\"type\":\"ip\"},\"dns_servers\":{\"type\":\"ip\"},\"router\":{\"type\":\"ip\"},\"domain_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hostname\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"requested_ip_address\":{\"type\":\"ip\"},\"parameter_request_list\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"renewal_time_sec\":{\"type\":\"long\"},\"vendor_identifying_options\":{\"type\":\"object\"},\"subnet_mask\":{\"type\":\"ip\"},\"time_servers\":{\"type\":\"ip\"},\"server_identifier\":{\"type\":\"ip\"},\"ip_address_lease_time_sec\":{\"type\":\"long\"},\"rebinding_time_sec\":{\"type\":\"long\"}}}}},\"package\":{\"properties\":{\"installed\":{\"type\":\"date\"},\"build_version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"description\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"reference\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"license\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"path\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"install_scope\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"size\":{\"type\":\"long\"},\"checksum\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"architecture\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"method\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"resource\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"query\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"dns\":{\"properties\":{\"resolved_ip\":{\"type\":\"ip\"},\"response_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"question\":{\"properties\":{\"registered_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"top_level_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"etld_plus_one\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"subdomain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"class\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"answers\":{\"type\":\"object\",\"properties\":{\"data\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"class\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ttl\":{\"type\":\"long\"}}},\"flags\":{\"properties\":{\"truncated_response\":{\"type\":\"boolean\"},\"authoritative\":{\"type\":\"boolean\"},\"checking_disabled\":{\"type\":\"boolean\"},\"recursion_available\":{\"type\":\"boolean\"},\"recursion_desired\":{\"type\":\"boolean\"},\"authentic_data\":{\"type\":\"boolean\"}}},\"additionals_count\":{\"type\":\"long\"},\"header_flags\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"authorities\":{\"type\":\"object\",\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"class\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"op_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"additionals\":{\"type\":\"object\",\"properties\":{\"data\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ttl\":{\"type\":\"long\"},\"class\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"opt\":{\"properties\":{\"ext_rcode\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"udp_size\":{\"type\":\"long\"},\"do\":{\"type\":\"boolean\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"answers_count\":{\"type\":\"long\"},\"authorities_count\":{\"type\":\"long\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"vulnerability\":{\"properties\":{\"severity\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"reference\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"score\":{\"properties\":{\"environmental\":{\"type\":\"float\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"temporal\":{\"type\":\"float\"},\"base\":{\"type\":\"float\"}}},\"report_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"scanner\":{\"properties\":{\"vendor\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"description\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"classification\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"category\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"enumeration\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"params\":{\"norms\":False,\"type\":\"text\"},\"pgsql\":{\"properties\":{\"error_message\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"error_severity\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"num_fields\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"num_rows\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"error_code\":{\"type\":\"long\"}}},\"labels\":{\"type\":\"object\"},\"tags\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"as\":{\"properties\":{\"number\":{\"type\":\"long\"},\"organization\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}}}}}},\"bytes_out\":{\"path\":\"destination.bytes\",\"type\":\"alias\"},\"cassandra\":{\"properties\":{\"request\":{\"properties\":{\"headers\":{\"properties\":{\"op\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"stream\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"length\":{\"type\":\"long\"},\"flags\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"type\":\"long\"}}},\"query\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"response\":{\"properties\":{\"result\":{\"properties\":{\"keyspace\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"schema_change\":{\"properties\":{\"args\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"keyspace\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"change\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"table\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"target\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"object\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"prepared\":{\"properties\":{\"prepared_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"req_meta\":{\"properties\":{\"pkey_columns\":{\"type\":\"long\"},\"keyspace\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"paging_state\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"flags\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"table\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"col_count\":{\"type\":\"long\"}}},\"resp_meta\":{\"properties\":{\"pkey_columns\":{\"type\":\"long\"},\"keyspace\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"paging_state\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"flags\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"col_count\":{\"type\":\"long\"},\"table\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"rows\":{\"properties\":{\"meta\":{\"properties\":{\"pkey_columns\":{\"type\":\"long\"},\"keyspace\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"paging_state\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"flags\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"table\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"col_count\":{\"type\":\"long\"}}},\"num_rows\":{\"type\":\"long\"}}}}},\"headers\":{\"properties\":{\"op\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"stream\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"length\":{\"type\":\"long\"},\"flags\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"type\":\"long\"}}},\"warnings\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"event\":{\"properties\":{\"schema_change\":{\"properties\":{\"args\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"keyspace\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"change\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"table\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"object\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"target\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"port\":{\"type\":\"long\"},\"change\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"host\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"error\":{\"properties\":{\"msg\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"code\":{\"type\":\"long\"},\"details\":{\"properties\":{\"alive\":{\"type\":\"long\"},\"stmt_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"received\":{\"type\":\"long\"},\"write_type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"num_failures\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"required\":{\"type\":\"long\"},\"read_consistency\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"keyspace\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"function\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"arg_types\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"data_present\":{\"type\":\"boolean\"},\"table\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"blockfor\":{\"type\":\"long\"}}},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"supported\":{\"type\":\"object\"},\"authentication\":{\"properties\":{\"class\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"no_request\":{\"type\":\"boolean\"}}},\"http\":{\"properties\":{\"request\":{\"properties\":{\"referrer\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"headers\":{\"type\":\"object\"},\"method\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"bytes\":{\"type\":\"long\"},\"body\":{\"properties\":{\"bytes\":{\"type\":\"long\"},\"content\":{\"ignore_above\":1024,\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}},\"type\":\"keyword\"}}}}},\"response\":{\"properties\":{\"headers\":{\"type\":\"object\"},\"status_code\":{\"type\":\"long\"},\"status_phrase\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"bytes\":{\"type\":\"long\"},\"body\":{\"properties\":{\"bytes\":{\"type\":\"long\"},\"content\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}}}}}},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"nfs\":{\"properties\":{\"tag\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"opcode\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"type\":\"long\"},\"status\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"minor_version\":{\"type\":\"long\"}}},\"fields\":{\"type\":\"object\"},\"hash\":{\"properties\":{\"sha1\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha256\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha512\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"md5\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"status\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"request\":{\"norms\":False,\"type\":\"text\"},\"server\":{\"properties\":{\"nat\":{\"properties\":{\"port\":{\"type\":\"long\"},\"ip\":{\"type\":\"ip\"}}},\"process\":{\"properties\":{\"args\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"start\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"working_directory\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"executable\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"address\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"top_level_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ip\":{\"type\":\"ip\"},\"packets\":{\"type\":\"long\"},\"mac\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"geo\":{\"properties\":{\"continent_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"region_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"city_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"location\":{\"type\":\"geo_point\"},\"region_name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"as\":{\"properties\":{\"number\":{\"type\":\"long\"},\"organization\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}}}}}},\"registered_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"port\":{\"type\":\"long\"},\"bytes\":{\"type\":\"long\"},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"user\":{\"properties\":{\"full_name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"email\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"group\":{\"properties\":{\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}}}},\"log\":{\"properties\":{\"original\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"level\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"logger\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"origin\":{\"properties\":{\"file\":{\"properties\":{\"line\":{\"type\":\"long\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"function\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"syslog\":{\"type\":\"object\",\"properties\":{\"severity\":{\"properties\":{\"code\":{\"type\":\"long\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"priority\":{\"type\":\"long\"},\"facility\":{\"properties\":{\"code\":{\"type\":\"long\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}}}},\"bytes_in\":{\"path\":\"source.bytes\",\"type\":\"alias\"},\"destination\":{\"properties\":{\"nat\":{\"properties\":{\"port\":{\"type\":\"long\"},\"ip\":{\"type\":\"ip\"}}},\"address\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"top_level_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ip\":{\"type\":\"ip\"},\"packets\":{\"type\":\"long\"},\"mac\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"geo\":{\"properties\":{\"continent_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"region_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"city_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"location\":{\"type\":\"geo_point\"},\"region_name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"as\":{\"properties\":{\"number\":{\"type\":\"long\"},\"organization\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}},\"type\":\"keyword\"}}}}},\"registered_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"port\":{\"type\":\"long\"},\"bytes\":{\"type\":\"long\"},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"user\":{\"properties\":{\"full_name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"email\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"group\":{\"properties\":{\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}}}},\"rule\":{\"properties\":{\"reference\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"license\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"author\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ruleset\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"description\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"category\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"uuid\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"error\":{\"properties\":{\"code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"stack_trace\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"message\":{\"norms\":False,\"type\":\"text\"}}},\"interface\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"alias\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"docker\":{\"properties\":{\"container\":{\"properties\":{\"labels\":{\"type\":\"object\"}}}}},\"network\":{\"properties\":{\"transport\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"inner\":{\"type\":\"object\",\"properties\":{\"vlan\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"packets\":{\"type\":\"long\"},\"community_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"forwarded_ip\":{\"type\":\"ip\"},\"protocol\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"vlan\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"application\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"bytes\":{\"type\":\"long\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"iana_number\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"direction\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"geo\":{\"properties\":{\"region_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"continent_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"city_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"location\":{\"type\":\"geo_point\"},\"region_name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"file\":{\"properties\":{\"owner\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"extension\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"gid\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"drive_letter\":{\"ignore_above\":1,\"type\":\"keyword\"},\"created\":{\"type\":\"date\"},\"accessed\":{\"type\":\"date\"},\"mtime\":{\"type\":\"date\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"directory\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"target_path\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"mode\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"inode\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"path\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"uid\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"code_signature\":{\"properties\":{\"valid\":{\"type\":\"boolean\"},\"trusted\":{\"type\":\"boolean\"},\"subject_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"exists\":{\"type\":\"boolean\"},\"status\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"size\":{\"type\":\"long\"},\"mime_type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"pe\":{\"properties\":{\"file_version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"product\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"description\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"original_file_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"company\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ctime\":{\"type\":\"date\"},\"attributes\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"device\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"properties\":{\"sha1\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha256\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha512\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"md5\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"group\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"vlan\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"related\":{\"properties\":{\"ip\":{\"type\":\"ip\"},\"user\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"thrift\":{\"properties\":{\"return_value\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"service\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"params\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"exceptions\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"client\":{\"properties\":{\"nat\":{\"properties\":{\"port\":{\"type\":\"long\"},\"ip\":{\"type\":\"ip\"}}},\"process\":{\"properties\":{\"args\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"start\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"working_directory\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"executable\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"address\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"top_level_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ip\":{\"type\":\"ip\"},\"mac\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"packets\":{\"type\":\"long\"},\"geo\":{\"properties\":{\"region_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"continent_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"city_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_iso_code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"location\":{\"type\":\"geo_point\"},\"region_name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"as\":{\"properties\":{\"number\":{\"type\":\"long\"},\"organization\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}},\"type\":\"keyword\"}}}}},\"registered_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"port\":{\"type\":\"long\"},\"bytes\":{\"type\":\"long\"},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"user\":{\"properties\":{\"full_name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"email\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"group\":{\"properties\":{\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}}}},\"event\":{\"properties\":{\"severity\":{\"type\":\"long\"},\"original\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"code\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"risk_score\":{\"type\":\"float\"},\"created\":{\"type\":\"date\"},\"kind\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"timezone\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"module\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"start\":{\"type\":\"date\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"url\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"reference\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"duration\":{\"type\":\"long\"},\"sequence\":{\"type\":\"long\"},\"ingested\":{\"type\":\"date\"},\"provider\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"risk_score_norm\":{\"type\":\"float\"},\"action\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"end\":{\"type\":\"date\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"category\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"dataset\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"outcome\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"mongodb\":{\"properties\":{\"fullCollectionName\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"query\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"numberReturned\":{\"type\":\"long\"},\"numberToSkip\":{\"type\":\"long\"},\"update\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"selector\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"error\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"startingFrom\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"returnFieldsSelector\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"numberToReturn\":{\"type\":\"long\"},\"cursorId\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"user_agent\":{\"properties\":{\"original\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"os\":{\"properties\":{\"kernel\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"family\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"platform\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"full\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}}}},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"device\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"jolokia\":{\"properties\":{\"server\":{\"properties\":{\"product\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"vendor\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"agent\":{\"properties\":{\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"secured\":{\"type\":\"boolean\"},\"url\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"registry\":{\"properties\":{\"hive\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"path\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"data\":{\"properties\":{\"strings\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"bytes\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"value\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"key\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"process\":{\"properties\":{\"parent\":{\"properties\":{\"pgid\":{\"type\":\"long\"},\"start\":{\"type\":\"date\"},\"pid\":{\"type\":\"long\"},\"working_directory\":{\"ignore_above\":1024,\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}},\"type\":\"keyword\"},\"thread\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"type\":\"long\"}}},\"title\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"entity_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"executable\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"ppid\":{\"type\":\"long\"},\"uptime\":{\"type\":\"long\"},\"args\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"code_signature\":{\"properties\":{\"valid\":{\"type\":\"boolean\"},\"trusted\":{\"type\":\"boolean\"},\"subject_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"exists\":{\"type\":\"boolean\"},\"status\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"exit_code\":{\"type\":\"long\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"args_count\":{\"type\":\"long\"},\"command_line\":{\"ignore_above\":1024,\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}},\"type\":\"keyword\"},\"hash\":{\"properties\":{\"sha1\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha256\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha512\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"md5\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"pgid\":{\"type\":\"long\"},\"start\":{\"type\":\"date\"},\"working_directory\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"pid\":{\"type\":\"long\"},\"thread\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"type\":\"long\"}}},\"title\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"entity_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"executable\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"ppid\":{\"type\":\"long\"},\"uptime\":{\"type\":\"long\"},\"args\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"code_signature\":{\"properties\":{\"valid\":{\"type\":\"boolean\"},\"trusted\":{\"type\":\"boolean\"},\"subject_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"exists\":{\"type\":\"boolean\"},\"status\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"pe\":{\"properties\":{\"file_version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"product\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"description\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"company\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"original_file_name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"exit_code\":{\"type\":\"long\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"args_count\":{\"type\":\"long\"},\"command_line\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"hash\":{\"properties\":{\"sha1\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha256\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha512\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"md5\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"rpc\":{\"properties\":{\"cred\":{\"properties\":{\"gids\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"uid\":{\"type\":\"long\"},\"gid\":{\"type\":\"long\"},\"stamp\":{\"type\":\"long\"},\"machinename\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"xid\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"auth_flavor\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"status\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"os\":{\"properties\":{\"kernel\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}},\"type\":\"keyword\"},\"family\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"platform\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"full\":{\"ignore_above\":1024,\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}},\"type\":\"keyword\"}}},\"dll\":{\"properties\":{\"path\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"code_signature\":{\"properties\":{\"valid\":{\"type\":\"boolean\"},\"trusted\":{\"type\":\"boolean\"},\"subject_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"exists\":{\"type\":\"boolean\"},\"status\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"pe\":{\"properties\":{\"file_version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"product\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"description\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"company\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"original_file_name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"properties\":{\"sha1\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha256\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha512\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"md5\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"amqp\":{\"properties\":{\"content-encoding\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"correlation-id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"no-wait\":{\"type\":\"boolean\"},\"method-id\":{\"type\":\"long\"},\"reply-code\":{\"type\":\"long\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"consumer-tag\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"mandatory\":{\"type\":\"boolean\"},\"consumer-count\":{\"type\":\"long\"},\"durable\":{\"type\":\"boolean\"},\"class-id\":{\"type\":\"long\"},\"delivery-tag\":{\"type\":\"long\"},\"exclusive\":{\"type\":\"boolean\"},\"message-id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"no-ack\":{\"type\":\"boolean\"},\"content-type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"no-local\":{\"type\":\"boolean\"},\"reply-to\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"timestamp\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"headers\":{\"type\":\"object\"},\"message-count\":{\"type\":\"long\"},\"app-id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"user-id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"multiple\":{\"type\":\"boolean\"},\"if-unused\":{\"type\":\"boolean\"},\"priority\":{\"type\":\"long\"},\"passive\":{\"type\":\"boolean\"},\"redelivered\":{\"type\":\"boolean\"},\"delivery-mode\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"reply-text\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"immediate\":{\"type\":\"boolean\"},\"auto-delete\":{\"type\":\"boolean\"},\"arguments\":{\"type\":\"object\"},\"expiration\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"exchange\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"routing-key\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"exchange-type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"queue\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"if-empty\":{\"type\":\"boolean\"}}},\"message\":{\"norms\":False,\"type\":\"text\"},\"url\":{\"properties\":{\"extension\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"original\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"scheme\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"top_level_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"query\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"path\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"password\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"fragment\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"registered_domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"port\":{\"type\":\"long\"},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"username\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"full\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}}}},\"@timestamp\":{\"type\":\"date\"},\"pe\":{\"properties\":{\"file_version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"product\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"description\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"original_file_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"company\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"response\":{\"norms\":False,\"type\":\"text\"},\"service\":{\"properties\":{\"node\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"state\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ephemeral_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"type\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"organization\":{\"properties\":{\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"tls\":{\"properties\":{\"established\":{\"type\":\"boolean\"},\"cipher\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"handshake_completed\":{\"path\":\"tls.established\",\"type\":\"alias\"},\"server\":{\"properties\":{\"not_after\":{\"type\":\"date\"},\"x509\":{\"properties\":{\"not_after\":{\"type\":\"date\"},\"subject\":{\"properties\":{\"country\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"state_or_province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organization\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"distinguished_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"locality\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"common_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organizational_unit\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"not_before\":{\"type\":\"date\"},\"public_key_algorithm\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"signature_algorithm\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"serial_number\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"public_key_size\":{\"type\":\"long\"},\"version_number\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"alternative_names\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"issuer\":{\"properties\":{\"state_or_province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organization\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"distinguished_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"locality\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"common_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organizational_unit\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"subject\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ja3s\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"not_before\":{\"type\":\"date\"},\"certificate\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"certificate_chain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"issuer\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"properties\":{\"sha1\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha256\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"md5\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"curve\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"alert_types\":{\"path\":\"tls.detailed.alert_types\",\"type\":\"alias\"},\"next_protocol\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version_protocol\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"fingerprints\":{\"properties\":{\"ja3\":{\"path\":\"tls.client.ja3\",\"type\":\"alias\"}}},\"server_certificate\":{\"properties\":{\"not_after\":{\"path\":\"tls.detailed.server_certificate.not_after\",\"type\":\"alias\"},\"not_before\":{\"path\":\"tls.detailed.server_certificate.not_before\",\"type\":\"alias\"},\"subject\":{\"properties\":{\"country\":{\"path\":\"tls.detailed.server_certificate.subject.country\",\"type\":\"alias\"},\"province\":{\"path\":\"tls.detailed.server_certificate.subject.province\",\"type\":\"alias\"},\"organization\":{\"path\":\"tls.detailed.server_certificate.subject.organization\",\"type\":\"alias\"},\"locality\":{\"path\":\"tls.detailed.server_certificate.subject.locality\",\"type\":\"alias\"},\"common_name\":{\"path\":\"tls.detailed.server_certificate.subject.common_name\",\"type\":\"alias\"},\"organizational_unit\":{\"path\":\"tls.detailed.server_certificate.subject.organizational_unit\",\"type\":\"alias\"}}},\"public_key_algorithm\":{\"path\":\"tls.detailed.server_certificate.public_key_algorithm\",\"type\":\"alias\"},\"signature_algorithm\":{\"path\":\"tls.detailed.server_certificate.signature_algorithm\",\"type\":\"alias\"},\"serial_number\":{\"path\":\"tls.detailed.server_certificate.serial_number\",\"type\":\"alias\"},\"public_key_size\":{\"path\":\"tls.detailed.server_certificate.public_key_size\",\"type\":\"alias\"},\"alternative_names\":{\"path\":\"tls.detailed.server_certificate.alternative_names\",\"type\":\"alias\"},\"version\":{\"path\":\"tls.detailed.server_certificate.version\",\"type\":\"alias\"},\"issuer\":{\"properties\":{\"country\":{\"path\":\"tls.detailed.server_certificate.issuer.country\",\"type\":\"alias\"},\"province\":{\"path\":\"tls.detailed.server_certificate.issuer.province\",\"type\":\"alias\"},\"organization\":{\"path\":\"tls.detailed.server_certificate.issuer.organization\",\"type\":\"alias\"},\"locality\":{\"path\":\"tls.detailed.server_certificate.issuer.locality\",\"type\":\"alias\"},\"common_name\":{\"path\":\"tls.detailed.server_certificate.issuer.common_name\",\"type\":\"alias\"},\"organizational_unit\":{\"path\":\"tls.detailed.server_certificate.issuer.organizational_unit\",\"type\":\"alias\"}}}}},\"resumption_method\":{\"path\":\"tls.detailed.resumption_method\",\"type\":\"alias\"},\"client_certificate\":{\"properties\":{\"not_after\":{\"path\":\"tls.detailed.client_certificate.not_after\",\"type\":\"alias\"},\"subject\":{\"properties\":{\"country\":{\"path\":\"tls.detailed.client_certificate.subject.country\",\"type\":\"alias\"},\"province\":{\"path\":\"tls.detailed.client_certificate.subject.province\",\"type\":\"alias\"},\"organization\":{\"path\":\"tls.detailed.client_certificate.subject.organization\",\"type\":\"alias\"},\"locality\":{\"path\":\"tls.detailed.client_certificate.subject.locality\",\"type\":\"alias\"},\"common_name\":{\"path\":\"tls.detailed.client_certificate.subject.common_name\",\"type\":\"alias\"},\"organizational_unit\":{\"path\":\"tls.detailed.client_certificate.subject.organizational_unit\",\"type\":\"alias\"}}},\"not_before\":{\"path\":\"tls.detailed.client_certificate.not_before\",\"type\":\"alias\"},\"public_key_algorithm\":{\"path\":\"tls.detailed.client_certificate.public_key_algorithm\",\"type\":\"alias\"},\"signature_algorithm\":{\"path\":\"tls.detailed.client_certificate.signature_algorithm\",\"type\":\"alias\"},\"serial_number\":{\"path\":\"tls.detailed.client_certificate.serial_number\",\"type\":\"alias\"},\"public_key_size\":{\"path\":\"tls.detailed.client_certificate.public_key_size\",\"type\":\"alias\"},\"alternative_names\":{\"path\":\"tls.detailed.client_certificate.alternative_names\",\"type\":\"alias\"},\"version\":{\"path\":\"tls.detailed.client_certificate.version\",\"type\":\"alias\"},\"issuer\":{\"properties\":{\"country\":{\"path\":\"tls.detailed.client_certificate.issuer.country\",\"type\":\"alias\"},\"province\":{\"path\":\"tls.detailed.client_certificate.issuer.province\",\"type\":\"alias\"},\"organization\":{\"path\":\"tls.detailed.client_certificate.issuer.organization\",\"type\":\"alias\"},\"locality\":{\"path\":\"tls.detailed.client_certificate.issuer.locality\",\"type\":\"alias\"},\"common_name\":{\"path\":\"tls.detailed.client_certificate.issuer.common_name\",\"type\":\"alias\"},\"organizational_unit\":{\"path\":\"tls.detailed.client_certificate.issuer.organizational_unit\",\"type\":\"alias\"}}}}},\"detailed\":{\"properties\":{\"server_certificate\":{\"properties\":{\"not_after\":{\"type\":\"date\"},\"subject\":{\"properties\":{\"state_or_province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organization\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"locality\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"distinguished_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"common_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organizational_unit\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"not_before\":{\"type\":\"date\"},\"public_key_algorithm\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"signature_algorithm\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"public_key_size\":{\"type\":\"long\"},\"version_number\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"serial_number\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"alternative_names\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"type\":\"long\"},\"issuer\":{\"properties\":{\"country\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"state_or_province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organization\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"locality\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"distinguished_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"common_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organizational_unit\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"resumption_method\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"client_certificate\":{\"properties\":{\"not_after\":{\"type\":\"date\"},\"not_before\":{\"type\":\"date\"},\"subject\":{\"properties\":{\"country\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organization\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"locality\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"distinguished_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"common_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organizational_unit\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"public_key_algorithm\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"signature_algorithm\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"public_key_size\":{\"type\":\"long\"},\"version_number\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"serial_number\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"alternative_names\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"type\":\"long\"},\"issuer\":{\"properties\":{\"country\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organization\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"locality\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"distinguished_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"common_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organizational_unit\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"alert_types\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"client_certificate_requested\":{\"type\":\"boolean\"},\"client_hello\":{\"properties\":{\"extensions\":{\"properties\":{\"application_layer_protocol_negotiation\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"_unparsed_\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"session_ticket\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"server_name_indication\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ec_points_formats\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"supported_versions\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"supported_groups\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"signature_algorithms\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"supported_compression_methods\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"session_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"server_hello\":{\"properties\":{\"extensions\":{\"properties\":{\"application_layer_protocol_negotiation\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"_unparsed_\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"session_ticket\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ec_points_formats\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"supported_versions\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"session_id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"selected_compression_method\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"client_certificate_requested\":{\"path\":\"tls.detailed.client_certificate_requested\",\"type\":\"alias\"},\"client_hello\":{\"properties\":{\"extensions\":{\"properties\":{\"application_layer_protocol_negotiation\":{\"path\":\"tls.detailed.client_hello.extensions.application_layer_protocol_negotiation\",\"type\":\"alias\"},\"_unparsed_\":{\"path\":\"tls.detailed.client_hello.extensions._unparsed_\",\"type\":\"alias\"},\"session_ticket\":{\"path\":\"tls.detailed.client_hello.extensions.session_ticket\",\"type\":\"alias\"},\"server_name_indication\":{\"path\":\"tls.detailed.client_hello.extensions.server_name_indication\",\"type\":\"alias\"},\"ec_points_formats\":{\"path\":\"tls.detailed.client_hello.extensions.ec_points_formats\",\"type\":\"alias\"},\"supported_versions\":{\"path\":\"tls.detailed.client_hello.extensions.supported_versions\",\"type\":\"alias\"},\"supported_groups\":{\"path\":\"tls.detailed.client_hello.extensions.supported_groups\",\"type\":\"alias\"},\"signature_algorithms\":{\"path\":\"tls.detailed.client_hello.extensions.signature_algorithms\",\"type\":\"alias\"}}},\"supported_ciphers\":{\"path\":\"tls.client.supported_ciphers\",\"type\":\"alias\"},\"supported_compression_methods\":{\"path\":\"tls.detailed.client_hello.supported_compression_methods\",\"type\":\"alias\"},\"session_id\":{\"path\":\"tls.detailed.client_hello.session_id\",\"type\":\"alias\"},\"version\":{\"path\":\"tls.detailed.client_hello.version\",\"type\":\"alias\"}}},\"client\":{\"properties\":{\"not_after\":{\"type\":\"date\"},\"x509\":{\"properties\":{\"not_after\":{\"type\":\"date\"},\"not_before\":{\"type\":\"date\"},\"subject\":{\"properties\":{\"state_or_province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organization\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"locality\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"distinguished_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"common_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organizational_unit\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"public_key_algorithm\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"signature_algorithm\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"public_key_size\":{\"type\":\"long\"},\"version_number\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"serial_number\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"alternative_names\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"version\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"issuer\":{\"properties\":{\"state_or_province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"country\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"province\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organization\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"distinguished_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"locality\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"common_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"organizational_unit\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"server_name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"not_before\":{\"type\":\"date\"},\"subject\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"supported_ciphers\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"certificate\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"ja3\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"certificate_chain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"properties\":{\"sha1\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"sha256\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"md5\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"issuer\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"resumed\":{\"type\":\"boolean\"},\"server_hello\":{\"properties\":{\"extensions\":{\"properties\":{\"application_layer_protocol_negotiation\":{\"path\":\"tls.detailed.server_hello.extensions.application_layer_protocol_negotiation\",\"type\":\"alias\"},\"_unparsed_\":{\"path\":\"tls.detailed.server_hello.extensions._unparsed_\",\"type\":\"alias\"},\"session_ticket\":{\"path\":\"tls.detailed.server_hello.extensions.session_ticket\",\"type\":\"alias\"},\"ec_points_formats\":{\"path\":\"tls.detailed.server_hello.extensions.ec_points_formats\",\"type\":\"alias\"},\"supported_versions\":{\"path\":\"tls.detailed.server_hello.extensions.supported_versions\",\"type\":\"alias\"}}},\"selected_cipher\":{\"path\":\"tls.cipher\",\"type\":\"alias\"},\"session_id\":{\"path\":\"tls.detailed.server_hello.session_id\",\"type\":\"alias\"},\"selected_compression_method\":{\"path\":\"tls.detailed.server_hello.selected_compression_method\",\"type\":\"alias\"},\"version\":{\"path\":\"tls.detailed.server_hello.version\",\"type\":\"alias\"}}}}},\"threat\":{\"properties\":{\"framework\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"technique\":{\"properties\":{\"reference\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}},\"tactic\":{\"properties\":{\"reference\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}},\"user\":{\"properties\":{\"full_name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\",\"fields\":{\"text\":{\"norms\":False,\"type\":\"text\"}}},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"email\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"hash\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"group\":{\"properties\":{\"domain\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"name\":{\"ignore_above\":1024,\"type\":\"keyword\"},\"id\":{\"ignore_above\":1024,\"type\":\"keyword\"}}}}}},\"date_detection\":False},\"aliases\":{}}\n",
    "\n",
    "es.indices.put_template(\"packetbeat-7.9.2\", template_definition)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_437564-2020-07-06T07+00:00.json.gz\n",
      "Adding 10000 items\n",
      "Adding 20000 items\n",
      "Adding 30000 items\n",
      "Adding 40000 items\n",
      "Adding 50000 items\n",
      "Adding 60000 items\n",
      "Adding 70000 items\n",
      "Adding 80000 items\n",
      "Adding 90000 items\n",
      "Adding 100000 items\n",
      "Adding 110000 items\n",
      "Adding 120000 items\n",
      "Adding 130000 items\n",
      "Adding 140000 items\n",
      "Adding 150000 items\n",
      "Adding 160000 items\n",
      "Adding 170000 items\n",
      "Adding 180000 items\n",
      "Adding 190000 items\n",
      "Adding 200000 items\n",
      "Adding 210000 items\n",
      "Adding 220000 items\n",
      "Adding 230000 items\n",
      "Adding 240000 items\n",
      "Adding 250000 items\n",
      "Adding 260000 items\n",
      "Adding 270000 items\n",
      "Adding 280000 items\n",
      "Adding 290000 items\n",
      "Adding 300000 items\n",
      "Adding 310000 items\n",
      "Adding 320000 items\n",
      "Adding 330000 items\n",
      "Adding 340000 items\n",
      "Adding 350000 items\n",
      "Adding 360000 items\n",
      "Adding 370000 items\n",
      "Adding 380000 items\n",
      "Adding 390000 items\n",
      "Adding 400000 items\n",
      "Adding 410000 items\n",
      "Adding 420000 items\n",
      "Adding 430000 items\n",
      "Adding 437554 items\n"
     ]
    }
   ],
   "source": [
    "count = 0\n",
    "match = 0\n",
    "\n",
    "# only add domains in original train set - otherwise there are some domains (e.g. loggingapi.google.com) which\n",
    "# are in the background noise when packetbeat runs\n",
    "domains = set(df_train_dedup.domain.values)\n",
    "\n",
    "json_filenames = [\"train_437564-2020-07-06T07+00:00.json.gz\"]\n",
    "\n",
    "index = \"packetbeat-7.9.2-train\"\n",
    "\n",
    "actions = []\n",
    "\n",
    "es.indices.delete(index, ignore=[400,404])\n",
    "\n",
    "for filename in json_filenames:\n",
    "    print(filename)\n",
    "    with gzip.open(filename) as f:\n",
    "        for line in f:\n",
    "            count = count + 1\n",
    "            j = json.loads(line)\n",
    "            domain = j['dns']['question']['name']\n",
    "            if (domain in domains):\n",
    "                # remove from set to avoid duplicates\n",
    "                domains.remove(domain)\n",
    "                match = match + 1\n",
    "                action = {\n",
    "                    \"_index\": index\n",
    "                }\n",
    "                j['malicious'] = 1\n",
    "                action['_source'] = j\n",
    "    \n",
    "                actions.append(action)\n",
    "        \n",
    "                if match % 10000 == 0:\n",
    "                    print(\"Adding \" + str(match) + \" items\")\n",
    "                    try:\n",
    "                        helpers.bulk(es, actions)\n",
    "                        actions = []\n",
    "                    except elasticsearch.TransportError as err:\n",
    "                        print(err)\n",
    "                    except:\n",
    "                        print(actions)\n",
    "                        raise\n",
    "\n",
    "print(\"Adding \" + str(match) + \" items\")\n",
    "try:\n",
    "    helpers.bulk(es, actions)\n",
    "    actions = []\n",
    "except:\n",
    "    print(actions)\n",
    "    raise\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add benign data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "benign-2020-06-23T11+00:00.json.gz\n",
      "Adding 10000 items\n",
      "Adding 20000 items\n",
      "Adding 30000 items\n",
      "Adding 40000 items\n",
      "Adding 50000 items\n",
      "Adding 60000 items\n",
      "Adding 70000 items\n",
      "Adding 80000 items\n",
      "Adding 90000 items\n",
      "Adding 100000 items\n",
      "Adding 110000 items\n",
      "Adding 120000 items\n",
      "Adding 130000 items\n",
      "Adding 140000 items\n",
      "Adding 150000 items\n",
      "Adding 160000 items\n",
      "Adding 170000 items\n",
      "Adding 180000 items\n",
      "Adding 190000 items\n",
      "Adding 200000 items\n",
      "Adding 210000 items\n",
      "Adding 220000 items\n",
      "Adding 230000 items\n",
      "Adding 240000 items\n",
      "Adding 250000 items\n",
      "Adding 260000 items\n",
      "Adding 270000 items\n",
      "Adding 280000 items\n",
      "Adding 290000 items\n",
      "Adding 300000 items\n",
      "Adding 310000 items\n",
      "Adding 320000 items\n",
      "Adding 330000 items\n",
      "Adding 340000 items\n",
      "Adding 350000 items\n",
      "Adding 360000 items\n",
      "Adding 370000 items\n",
      "Adding 380000 items\n",
      "Adding 390000 items\n",
      "Adding 400000 items\n",
      "Adding 410000 items\n",
      "Adding 420000 items\n",
      "Adding 430000 items\n",
      "Adding 437555 items\n"
     ]
    }
   ],
   "source": [
    "# benign-2020-06-23T11+00:00.json.gz contains 1,000,000+ domains\n",
    "# only add same number of domains as malicious set\n",
    "ed_df = ed.DataFrame(es, index)\n",
    "\n",
    "benign_size = len(ed_df)\n",
    "\n",
    "count = 0\n",
    "\n",
    "actions = []\n",
    "\n",
    "for filename in ['benign-2020-06-23T11+00:00.json.gz']:\n",
    "    print(filename)\n",
    "    with gzip.open(filename) as f:\n",
    "        for line in f:\n",
    "            count = count + 1\n",
    "            j = json.loads(line)\n",
    "            action = {\n",
    "                \"_index\": index\n",
    "            }\n",
    "            j['malicious'] = 0\n",
    "            action['_source'] = j\n",
    "\n",
    "            actions.append(action)\n",
    "            \n",
    "            if count > benign_size:\n",
    "                break\n",
    "\n",
    "            if (count % 10000 == 0):\n",
    "                print(\"Adding \" + str(count) + \" items\")\n",
    "                try:\n",
    "                    helpers.bulk(es, actions)\n",
    "                    actions = []\n",
    "                except elasticsearch.TransportError as err:\n",
    "                    print(err)\n",
    "                except:\n",
    "                    print(actions)\n",
    "                    raise\n",
    "            \n",
    "print(\"Adding \" + str(count) + \" items\")\n",
    "try:\n",
    "    helpers.bulk(es, actions)\n",
    "    actions = []\n",
    "except:\n",
    "    print(actions)\n",
    "    raise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'eland.dataframe.DataFrame'>\n",
      "Index: 875109 entries, 69zmAnUBq5qB3a8CdGym to reXrAnUBq5qB3a8CkNbX\n",
      "Columns: 1046 entries, @metadata.beat to vulnerability.severity\n",
      "dtypes: bool(47), datetime64[ns](28), float64(5), int64(141), object(825)\n",
      "memory usage: 96.0 bytes\n"
     ]
    }
   ],
   "source": [
    "ed_df = ed.DataFrame(es, index)\n",
    "\n",
    "ed_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reindex data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'task': 'KDm4yuWVTkqhomnQIRIY5g:95946'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analysis_index = f\"{index}_expanded\"\n",
    "\n",
    "body = {\n",
    "  \"source\": {\n",
    "    \"index\": index\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": analysis_index,\n",
    "    \"pipeline\": \"dga_ngrams_create\"\n",
    "  }\n",
    "}\n",
    "\n",
    "es.reindex(body=body, wait_for_completion=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train model\n",
    "\n",
    "(wait for reindex to complete first)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "packetbeat-7.9.2-train_expanded_8_analysis\n",
      "TEXT EXCLUDED FROM NOTEBOOK TO REDUCE NOTEBOOK SIZE\n"
     ]
    }
   ],
   "source": [
    "for num_threads in [8]:\n",
    "    INDEX_NAME=analysis_index\n",
    "    ANALYSIS_NAME=f\"{INDEX_NAME}_{num_threads}_analysis\"\n",
    "\n",
    "    print(ANALYSIS_NAME)\n",
    "\n",
    "    ret = es.indices.delete(index=ANALYSIS_NAME, ignore=[400, 404])\n",
    "    ret = es.ml.stop_data_frame_analytics(ANALYSIS_NAME, ignore=[400, 404])\n",
    "    ret = es.ml.delete_data_frame_analytics(ANALYSIS_NAME, ignore=[400, 404])\n",
    "    #ret = es.transport.perform_request(\"GET\", f\"/_ml/inference/{ANALYSIS_NAME}-*\")\n",
    "    #if ret['count']>0:\n",
    "    #    model_id = ret['trained_model_configs'][0]['model_id']\n",
    "    #    ret = es.transport.perform_request(\"DELETE\", \"/_ml/inference/\" + model_id)\n",
    "\n",
    "    analysis_config = {\n",
    "      \"description\": \"\",\n",
    "      \"source\": {\n",
    "        \"index\": INDEX_NAME\n",
    "      },\n",
    "      \"dest\": {\n",
    "        \"index\": ANALYSIS_NAME\n",
    "      },\n",
    "      \"analyzed_fields\": {\n",
    "        \"includes\": [\"f.*\",\"dns.response_code\", \"malicious\"]\n",
    "      },\n",
    "      \"analysis\": {\n",
    "        \"classification\": {\n",
    "          \"dependent_variable\": \"malicious\",\n",
    "          \"randomize_seed\": 1\n",
    "        }\n",
    "      },\n",
    "      \"model_memory_limit\": \"3gb\",\n",
    "      \"max_num_threads\": num_threads\n",
    "    }\n",
    "\n",
    "    es.ml.put_data_frame_analytics(ANALYSIS_NAME, analysis_config)\n",
    "\n",
    "    es.ml.start_data_frame_analytics(ANALYSIS_NAME)\n",
    "\n",
    "    while True:\n",
    "        results = es.ml.get_data_frame_analytics_stats(ANALYSIS_NAME)\n",
    "        state = results['data_frame_analytics'][0]['state']\n",
    "        progress = results['data_frame_analytics'][0]['progress']\n",
    "        status = state + \" \"\n",
    "        for p in progress:\n",
    "            status = status + p['phase'] + \":\" + str(p['progress_percent']) + \" \"\n",
    "        print(str(pd.Timestamp.now()), status)\n",
    "        if state == 'stopped':\n",
    "            break\n",
    "        time.sleep(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'classification': {'accuracy': {'classes': [{'class_name': '0', 'accuracy': 0.9996788971430988}, {'class_name': '1', 'accuracy': 0.9996788971430988}], 'overall_accuracy': 0.9996788971430988}, 'multiclass_confusion_matrix': {'confusion_matrix': [{'actual_class': '0', 'actual_class_doc_count': 437555, 'predicted_classes': [{'predicted_class': '0', 'count': 437416}, {'predicted_class': '1', 'count': 139}], 'other_predicted_class_doc_count': 0}, {'actual_class': '1', 'actual_class_doc_count': 437554, 'predicted_classes': [{'predicted_class': '0', 'count': 142}, {'predicted_class': '1', 'count': 437412}], 'other_predicted_class_doc_count': 0}], 'other_actual_class_count': 0}}}\n",
      "Dataset = all\n",
      "Accuracy = 0.9996788971430988\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    437558\n",
       "1    437551\n",
       "dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>437416</td>\n",
       "      <td>139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>142</td>\n",
       "      <td>437412</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        0       1\n",
       "0  437416     139\n",
       "1     142  437412"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>99.967547</td>\n",
       "      <td>0.031768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.032453</td>\n",
       "      <td>99.968232</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           0          1\n",
       "0  99.967547   0.031768\n",
       "1   0.032453  99.968232"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'classification': {'accuracy': {'classes': [{'class_name': '0', 'accuracy': 0.9996788971430988}, {'class_name': '1', 'accuracy': 0.9996788971430988}], 'overall_accuracy': 0.9996788971430988}, 'multiclass_confusion_matrix': {'confusion_matrix': [{'actual_class': '0', 'actual_class_doc_count': 437555, 'predicted_classes': [{'predicted_class': '0', 'count': 437416}, {'predicted_class': '1', 'count': 139}], 'other_predicted_class_doc_count': 0}, {'actual_class': '1', 'actual_class_doc_count': 437554, 'predicted_classes': [{'predicted_class': '0', 'count': 142}, {'predicted_class': '1', 'count': 437412}], 'other_predicted_class_doc_count': 0}], 'other_actual_class_count': 0}}}\n",
      "Dataset = training\n",
      "Accuracy = 0.9996788971430988\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    437558\n",
       "1    437551\n",
       "dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>437416</td>\n",
       "      <td>139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>142</td>\n",
       "      <td>437412</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        0       1\n",
       "0  437416     139\n",
       "1     142  437412"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>99.967547</td>\n",
       "      <td>0.031768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.032453</td>\n",
       "      <td>99.968232</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           0          1\n",
       "0  99.967547   0.031768\n",
       "1   0.032453  99.968232"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate_all = {\"index\":ANALYSIS_NAME,\"evaluation\":{\"classification\":{\"actual_field\":\"malicious\",\"predicted_field\":\"ml.malicious_prediction\",\"metrics\":{\"accuracy\":{},\"multiclass_confusion_matrix\":{}}}}}\n",
    "evaluate_training = {\"index\":ANALYSIS_NAME,\"query\":{\"bool\":{\"filter\":[{\"term\":{\"ml.is_training\":True}}]}},\"evaluation\":{\"classification\":{\"actual_field\":\"malicious\",\"predicted_field\":\"ml.malicious_prediction\",\"metrics\":{\"accuracy\":{},\"multiclass_confusion_matrix\":{}}}}}\n",
    "#evaluate_test = {\"index\":ANALYSIS_NAME,\"query\":{\"bool\":{\"filter\":[{\"term\":{\"ml.is_training\":False}}]}},\"evaluation\":{\"classification\":{\"actual_field\":\"test_malicious\",\"predicted_field\":\"ml.malicious_prediction\",\"metrics\":{\"accuracy\":{},\"multiclass_confusion_matrix\":{}}}}}\n",
    "\n",
    "results = dict()\n",
    "results['all'] = es.ml.evaluate_data_frame(evaluate_all)\n",
    "results['training'] = es.ml.evaluate_data_frame(evaluate_training)\n",
    "#results['test'] = es.ml.evaluate_data_frame(evaluate_test)\n",
    "from IPython.display import display\n",
    "for key, value in results.items():\n",
    "    accuracy = value['classification']['accuracy']['overall_accuracy']\n",
    "    row_0 = [\n",
    "        value['classification']['multiclass_confusion_matrix']['confusion_matrix'][0]['predicted_classes'][0]['count'],\n",
    "        value['classification']['multiclass_confusion_matrix']['confusion_matrix'][0]['predicted_classes'][1]['count']\n",
    "    ]\n",
    "    row_1 = [\n",
    "        value['classification']['multiclass_confusion_matrix']['confusion_matrix'][1]['predicted_classes'][0]['count'],\n",
    "        value['classification']['multiclass_confusion_matrix']['confusion_matrix'][1]['predicted_classes'][1]['count']\n",
    "    ]\n",
    "    matrix = [row_0, row_1]\n",
    "    df = pd.DataFrame(matrix)\n",
    "    print(value)\n",
    "    cm = df\n",
    "    normalized_cm = cm/cm.sum()*100\n",
    "    \n",
    "    print(\"Dataset =\",key)\n",
    "    print(\"Accuracy =\", accuracy)\n",
    "    display(cm.sum())\n",
    "    display(cm)\n",
    "    display(normalized_cm)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

Is this file truncated?

@michaelschem - apologies, file was truncated due to browser upload. I've reduced the size and re-uploaded. Let me know if you have any issues (note github often has problems rendering notebooks, so it's normally best to download them and view in jupyter).