Skip to content

Instantly share code, notes, and snippets.

@yaauie
Created September 6, 2022 15:30
Show Gist options
  • Save yaauie/37c8168ff88408ed92d02bc808354031 to your computer and use it in GitHub Desktop.
Save yaauie/37c8168ff88408ed92d02bc808354031 to your computer and use it in GitHub Desktop.
2022 high-level docs for logstash-to-logstash using the HTTP input/output pair

We have had some success using LS-to-LS over HTTP(S), which supports an HTTP(s) Load Balancer or Proxy in the middle, and can be secured with TLS/SSL. It can be made to be quite performant, but doing so requires some specific tuning.

Upstream (HTTP Output)

The upstream pipelie would contain a single HTTP output plugin aimed either directly at a downstream Logstash or at a Load Balancer, importantly configured with:

  • format => json_batch (for performance; without this one event will be sent at a time) and
  • retry_non_idempotent => true (for resilience; without this, some failures cannot be safely retried).

Depending on whether we ar sending directly to another Logstash or through an SSL-terminating Load Balancer or proxy, the output may need to be configured

  • with HTTP Basic credentials (user/password),
  • with an X509 Certificate Authority (cacert), and/or
  • with its own X509 Certificate keypair to assert its identity (client_cert/client_key).
output {
    http {
        id => 'ls1-output'
        url => 'https://load.balancer.address:8888'
        http_method => post

        #user => "logstash"
        #password => "SeCReT"

        retry_non_idempotent => true
        format => json_batch
        
        cacert => "./cert-bundle/ca/ca.crt"
        client_key => "./cert-bundle/upstream/upstream.key.pk8"
        client_cert => "./cert-bundle/upstream/upstream.crt"
    }
}

Downstream (HTTP Input)

The downstream pipeline would be configured with an HTTP input, importantly configured with:

  • additional_codecs => { "application/json" => "json" } (to read the batched JSON payloads), and
  • ecs_compatibility => disabled (to avoid adding ECS-related fields in LS 8+ where ECS is on-by-default).

If this downstream Logstash is publicly accessible it should be guarded to prevent unauthorized events from being injected by an adversary, which would require either SSL+Basic Auth or mutual-TLS.

If the customer is re-establishing TLS between the LB and the downstream Logstash, then the input would need to be configured

  • with an X509 Certificate keypair to identify itself (ssl_certificate/ssl_key),
  • possibly HTTP Basic auth (user/password), and
  • possibly with an X509 Certificate Authority with which to trust the upstream or LB's certificate (if ssl_verify_mode => peer or ssl_verify_mode => force_peer).
input {
    http {
        id => 'ls2-input'
        host => '0.0.0.0'
        port => 8888
        #user => "logstash"
        #password => "SeCReT"
        additional_codecs => { "application/json" => "json_lines" }
        ecs_compatibility => disabled
        
        ssl => true
        ssl_key => "./cert-bundle/downstream.key.pk8"
        ssl_certificate => "./cert-bundle/logstash2.crt"
        ssl_certificate_authorities => "./cert-bundle/ca/ca.crt"
        ssl_verify_mode => peer # will need CN to match host
    }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment