Skip to content

Instantly share code, notes, and snippets.

@jgoodall
Last active September 19, 2023 18:06
Show Gist options
  • Save jgoodall/6323951 to your computer and use it in GitHub Desktop.
Save jgoodall/6323951 to your computer and use it in GitHub Desktop.
This is a sample of how to send some information to logstash via the TCP input from node.js or python.

This is a sample of how to send some information to logstash via the TCP input in nodejs or python. It assumes the logstash host is on 10.10.10.100 and the TCP listening input is 9563.

The logstash.conf should look something like the sample file.

The log message should be a stringified JSON object with the log message in the @message field.

To use, run the node script node sendMessageToLogstash.js, or the python script python sendMessageToLogstash.js

input {
tcp {
'charset' => 'UTF-8'
'format' => 'json_event'
'port' => '9563'
'type' => 'stucco-tcp'
}
}
output {
elasticsearch_http {
'host' => 'localhost'
'port' => '9200'
}
file {
'path' => '/usr/local/logstash/server/log/output.log'
}
}
/* globals require:true, console:true, process:true */
// This script will send a message to a [logstash](http://logstash.net/)
// server using the [TCP input](http://logstash.net/docs/1.1.13/inputs/tcp)
// and then quit. If there is no listener, it will just quit.
'use strict';
var net = require('net');
var logHost = '10.10.10.100'
, logPort = 9563
, sender = require('os').hostname();
var conn = net.createConnection({host: logHost, port: logPort}, function() {
var message = {
'@tags': ['nodejs', 'test']
, '@message': 'tcp test ' + Math.floor(Math.random() * 10000)
, '@fields': {'sender': sender}
}
conn.write(JSON.stringify(message));
process.exit(0);
})
.on('error', function(err) {
console.error(err);
process.exit(1);
});
import socket
import json
import sys
HOST = '10.10.10.100'
PORT = 9563
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
sys.stderr.write("[ERROR] %s\n" % msg[1])
sys.exit(1)
try:
sock.connect((HOST, PORT))
except socket.error, msg:
sys.stderr.write("[ERROR] %s\n" % msg[1])
sys.exit(2)
msg = {'@message': 'python test message', '@tags': ['python', 'test']}
sock.send(json.dumps(msg))
sock.close()
sys.exit(0)
@hbinduni
Copy link

why node program sends data to logstash after quitting?

you need to end your msg with new line. so it should be something like client.write(msg + '\n')

@tamaroth
Copy link

tamaroth commented Jun 4, 2021

I stumbled upon this while looking for a simple way to send logs directly to Logstash.

I've used your basic idea and adjusted it to my needs:

  • i have a log file with structured logs, where each line is a JSON object and I wanted to feed this to Logstash
  • I do not care about exceptions, it's a quick hack and if it throws I'd prefer to see the entire stack
  • My personal ELK stack was installed from docker-elk, with docker-elk/logstash/pipeline/logstash.conf being the following:
input {
        beats {
                port => 5044
        }

        tcp {
                port => 5000
                codec => json
        }
}

## Add your filters / logstash plugins configuration here

output {
        elasticsearch {
                hosts => "elasticsearch:9200"
                user => "elastic"
                password => "changeme"
                ecs_compatibility => disabled
        }
}

The script I used is this:

import socket

host = "localhost"
port = 5000


with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    sock.connect((host, port))

    with open("/path/to/log/file.log", mode="rb") as f:
        for line in f.readlines():
            sock.send(line)
    print("all data sent to Logstash")

If you don't mind, I'm dropping it here in hopes it helps someone else when they scour the internet for similar problem. Thanks for the base code!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment