Skip to content

Instantly share code, notes, and snippets.

@markharwood
Last active January 5, 2017 13:45
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save markharwood/c478ea0192857b9cdb24ad41d4d14fcd to your computer and use it in GitHub Desktop.
Graph overview of network comms
// Delete any existing index
DELETE ipcomms
// Define a mapping with original fields and derived fields
PUT ipcomms
{
"mappings": {
"log": {
"properties": {
"src": {
"type": "ip",
"copy_to":"ips"
},
"dest": {
"type": "ip",
"copy_to":"ips"
},
"comms_pair": {
"type": "keyword"
},
"ips": {
"type": "keyword"
}
}
}
}
}
// Create a an ingest pipeline to take documents and derive a "comms pair" field for indexing
PUT _ingest/pipeline/graphing_pipeline_id
{
"description" : "Combines src and destination IPs into a sorted comms_pair",
"processors":[
{
"script": {
"lang": "painless",
"inline": "ctx.comms_pair = [ctx.src, ctx.dest]",
"params": {
"param_c": 10
}
}
},
{
"sort": {
"field": "comms_pair",
"order": "asc"
}
},
{
"join": {
"field": "comms_pair",
"separator": "-"
}
}
]
}
// Add some example docs, using the ingest pipeline outlined above to process the docs
POST ipcomms/log?pipeline=graphing_pipeline_id
{
"src": "216.58.201.36",
"dest": "151.101.1.69"
}
POST ipcomms/log?pipeline=graphing_pipeline_id
{
"src": "216.58.201.35",
"dest": "151.101.1.69"
}
POST ipcomms/log?pipeline=graphing_pipeline_id
{
"src": "216.58.201.36",
"dest": "151.101.1.68"
}
// Run a graph query to summarise a sample of comms using settings
// to ensure the sample isn't dominated by any particularly "chatty"
// pair of IPs.
GET ipcomms/_xpack/_graph/_explore
{
"query": {
"query_string": {
"query": "*"
}
},
"controls": {
"use_significance": false,
"sample_size": 20000,
"timeout": 5000,
"sample_diversity": {
"field": "comms_pair",
"max_docs_per_value": 1
}
},
"connections": {
"vertices": [
{
"field": "ips",
"size": 500,
"min_doc_count": 1
}
]
},
"vertices": [
{
"field": "ips",
"size": 500,
"min_doc_count": 1
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment