Fingerprint in Logstash

README

We use Logstash to transform specific business logs that don't contain any unique value which could be used to calculate fingerprint in order to avoid duplicates. That's why we calculate fingerprints with the entire document/log. 

That approach works correctly for the same document which has been sent once again to Logstash we received the same fingerprints which are expecting behavior. 

Please see the below steps to replicate that issue.
I attached logstash.conf which is a simple pipeline calculating fingerprint and saving the output in a file. 

1. Run Logstash instance:

docker run -p 41311:41311 -v ${PWD}/pipeline/:/usr/share/logstash/pipeline  logstash:6.8.4

2. Send the example document to Logstash

curl -XPOST -d@example.json http://localhost:41311

3. The output is attached in a file response-1.json. If you send the document once again you will recieve the same output - see the file response-2.json

  "@timestamp": "2019-11-15T09:52:44.742Z",
  "fingerprint": "69f6f3db31db2566ea176fe3f5a91aad21c6b407",


 "@timestamp": "2019-11-15T09:53:17.859Z",
 "fingerprint": "69f6f3db31db2566ea176fe3f5a91aad21c6b407",
 
 The fingerprints are the same. It works like a charm!
 
 
 4. Restart Logstash instance using the command from step 1 and send the log once again using the command from step 2. 
 
 The output is attached to the file response-1-restated.json
 
 We recieved different fingerprint:  
  "fingerprint": "f79d50c9c109a9b94c90c7b587e4da348df41bf9"

because the order of the document has been changed. 

example.json

{
    "type": "document_type",
    "logId": "123",
    "messageJson": {
        "Request": [
            {
                "name": "John",
                "age": 31,
                "cars": {
                    "car1": "Ford",
                    "car2": "BMW",
                    "car3": "Fiat"
                }
            },
            {
                "name": "Julian",
                "age": 12,
                "bikes": {
                    "bike1": "Canyon",
                    "bike2": "Trek",
                    "bike3": "Orbea"
                }
            }
        ],
        "Response": [
            {
                "name": "Tom",
                "age": 45,
                "animals": {
                    "animal1": "Dog",
                    "animal2": "Cat",
                    "animal3": "Horse"
                }
            }
        ]
    }
}

logstash.conf

input { 
   http {
       port => 41311
       codec => "json"
       max_content_length => 1073741824
   } 
}

filter {
    mutate {
       id => "filter_1"
       rename => {
              "[messageJson]" => "[message]"
        }
    }
     fingerprint {
        id => "filter_2"
        method => "SHA1"
        key => "[B;o^Vy%-GK]%=t1Cp\Q>(:C("
        target => "fingerprint"
     } 
}

output {
   file {
       path => "/usr/share/logstash/pipeline/log.json"
       codec => "json"

   }
}

response-1-restarted-logstash.json

{
    "host": "172.17.0.1",
    "@timestamp": "2019-11-15T09:55:46.425Z",
    "@version": "1",
    "type": "document_type",
    "message": {
        "Request": [
            {
                "cars": {
                    "car3": "Fiat",
                    "car2": "BMW",
                    "car1": "Ford"
                },
                "name": "John",
                "age": 31
            },
            {
                "name": "Julian",
                "age": 12,
                "bikes": {
                    "bike2": "Trek",
                    "bike1": "Canyon",
                    "bike3": "Orbea"
                }
            }
        ],
        "Response": [
            {
                "animals": {
                    "animal2": "Cat",
                    "animal1": "Dog",
                    "animal3": "Horse"
                },
                "name": "Tom",
                "age": 45
            }
        ]
    },
    "logId": "123",
    "headers": {
        "request_method": "POST",
        "http_accept": "*/*",
        "content_length": "840",
        "http_host": "localhost:41311",
        "content_type": "application/x-www-form-urlencoded",
        "request_path": "/",
        "http_user_agent": "curl/7.64.1",
        "http_version": "HTTP/1.1"
    },
    "fingerprint": "f79d50c9c109a9b94c90c7b587e4da348df41bf9"
}

response-1.json

{
    "headers": {
        "http_accept": "*/*",
        "request_method": "POST",
        "http_user_agent": "curl/7.64.1",
        "http_host": "localhost:41311",
        "content_type": "application/x-www-form-urlencoded",
        "http_version": "HTTP/1.1",
        "content_length": "840",
        "request_path": "/"
    },
    "message": {
        "Request": [
            {
                "name": "John",
                "cars": {
                    "car1": "Ford",
                    "car2": "BMW",
                    "car3": "Fiat"
                },
                "age": 31
            },
            {
                "name": "Julian",
                "bikes": {
                    "bike2": "Trek",
                    "bike1": "Canyon",
                    "bike3": "Orbea"
                },
                "age": 12
            }
        ],
        "Response": [
            {
                "name": "Tom",
                "animals": {
                    "animal2": "Cat",
                    "animal1": "Dog",
                    "animal3": "Horse"
                },
                "age": 45
            }
        ]
    },
    "@timestamp": "2019-11-15T09:52:44.742Z",
    "fingerprint": "69f6f3db31db2566ea176fe3f5a91aad21c6b407",
    "@version": "1",
    "logId": "123",
    "host": "172.17.0.1",
    "type": "document_type"
}

response-2.json

{
    "headers": {
        "http_accept": "*/*",
        "request_method": "POST",
        "http_user_agent": "curl/7.64.1",
        "http_host": "localhost:41311",
        "content_type": "application/x-www-form-urlencoded",
        "http_version": "HTTP/1.1",
        "content_length": "840",
        "request_path": "/"
    },
    "message": {
        "Request": [
            {
                "name": "John",
                "cars": {
                    "car1": "Ford",
                    "car2": "BMW",
                    "car3": "Fiat"
                },
                "age": 31
            },
            {
                "name": "Julian",
                "bikes": {
                    "bike2": "Trek",
                    "bike1": "Canyon",
                    "bike3": "Orbea"
                },
                "age": 12
            }
        ],
        "Response": [
            {
                "name": "Tom",
                "animals": {
                    "animal2": "Cat",
                    "animal1": "Dog",
                    "animal3": "Horse"
                },
                "age": 45
            }
        ]
    },
    "@timestamp": "2019-11-15T09:53:17.859Z",
    "fingerprint": "69f6f3db31db2566ea176fe3f5a91aad21c6b407",
    "@version": "1",
    "logId": "123",
    "host": "172.17.0.1",
    "type": "document_type"
}