test-beats-processors.md

How to test Beats processors

To test Beats processors, beats playground can be helpful.

Input:

2019-09-29 STATUS_OK Server started normally. 92ms
2019-09-30 STATUS_OK Server started normally. 120ms
2019-10-01 STATUS_NG Server stopped abnormally. Timed out. 3010ms

Output:

[
  {
    "@timestamp": "2021-09-07T04:30:50.543Z",
    "message": "Server started normally.",
    "status": "STATUS_OK",
    "took": "92ms"
  },
  {
    "@timestamp": "2021-09-07T04:30:50.543Z",
    "message": "Server started normally.",
    "status": "STATUS_OK",
    "took": "120ms"
  },
  {
    "@timestamp": "2021-09-07T04:30:50.543Z",
    "message": "Server stopped abnormally. Timed out.",
    "status": "STATUS_NG",
    "took": "3010ms"
  }
]

Processors Example:

- dissect:
    tokenizer: '%{@timestamp} %{status} %{message}'
    target_prefix: ''
    overwrite_keys: true
- script:
    lang: javascript
    id: my_filter
    source: >
      function process(event) {
          var str = event.Get('message');
          var idx = str.lastIndexOf(' ');
          event.Put('message', str.substr(0, idx));
          event.Put('took', str.substr(idx + 1));
      }

test-ingest-pipeline.md

POST _ingest/pipeline/_simulate
{
  "docs": [
    {"_source": {}}
  ],
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "message",
          "value": "hello world"
        }
      }
    ]
  }
}

PUT _ingest/pipeline/my_pipeline
{
  "processors": [
    {
      "set": {
        "field": "message",
        "value": "hello world"
      }
    }
  ]
}

POST _ingest/pipeline/my_pipeline/_simulate
{
  "docs": [
    {"_source": {}}
  ]
}

test-logstash-filters.md

How to test Logstash filters?

The proposal to add Logstash pipeline simulate API endpoint staled from security considerations. elastic/logstash#7832

Usually, we can test filters by changing input and output for testing purpose.

input {
  generator {
    message => '{"environment":"urban","envType":"outdoor","positionNumber":1,"frameIndex":15,"fcntUp":22324,"numberOfGateways":3,"numberOfTimestamps":0,"serverTimestamp":"23:44:36","lnsAppEui":"4883C7DF30040000","lnsDevEui":"4883C7DF3004223A","lnsNetId":"000007","lnsServerTimestamp":"27/10/2017 23:44:36","gtwFrequency":868.1,"gtwDataRate":"SF10_BW_125","estimatedLongitudeWGS84":null,"estimatedLatitudeWGS84":null,"deviceLongitudeWGS84":2.289511063734628,"deviceLatitudeWGS84":48.88373587201055,"spreadingFactor":10,"accuracy":null,"estimatedAccuracy":null,"averageISD":null,"hdopDevice":null,"hdopEstimated":null,"gateways":[{"antennaLongitude":2.2957080835783388,"rssi":-107.7376019773414,"rssiStandardDeviation":-95.0,"elapsedTimeSince1PPS":0.0,"antennaID":"1","antennaLatitude":48.89112433031253,"gatewayTimestamp":"27/10/2017 23:44:31","snr":-12.5,"frequencyOffset":0,"lnsServerTimestamp":"27/10/2017 23:44:36","rssiSignal":-95.0,"gatewayID":"M15279"},{"antennaLongitude":2.2932473657963164,"rssi":-107.1244260279434,"rssiStandardDeviation":-103.0,"elapsedTimeSince1PPS":0.0,"antennaID":"1","antennaLatitude":48.88193968215203,"gatewayTimestamp":"27/10/2017 23:44:32","snr":-2.0,"frequencyOffset":0,"lnsServerTimestamp":"27/10/2017 23:44:36","rssiSignal":-103.0,"gatewayID":"M15073"},{"antennaLongitude":2.3014574627818787,"rssi":-104.59612087980607,"rssiStandardDeviation":-94.0,"elapsedTimeSince1PPS":0.0,"antennaID":"1","antennaLatitude":48.87684647097623,"gatewayTimestamp":"27/10/2017 23:44:32","snr":-10.2,"frequencyOffset":0,"lnsServerTimestamp":"27/10/2017 23:44:36","rssiSignal":-94.0,"gatewayID":"M15913"}],"loraDevAddr":"0F127D87"}'
    count => 1
  }
}

filter {
  json {
    source => "message"
  }
  ruby {
    code => '
      gateways_size = event.get("[gateways]").size
      gateways_size.times do |index|
        event.set("[gateways][#{index}][position][lat]", event.get("[gateways][#{index}][antennaLatitude]"))
        event.set("[gateways][#{index}][position][lon]", event.get("[gateways][#{index}][antennaLongitude]"))
      end
    '
  }
}

output {
  stdout { codec => rubydebug }
}

If you'd like to write some test suite, logstash filter test may be helpful.