Splunk provides a library that can be used to create Java loggers needed for forwarding logs to a Splunk instance via multiple means such as HTTP, TCP and TCP-SSL. For more details on this, check the Splunk Logging for Java page. The rest of this article assumes you already have an up and running Splunk instance (on a UNIX box)
Note that details given in this write-up are specific to Splunk 8.0. Configuration parameters may have been different in previous versions and may also change in subsequent versions.
Details on forwarding logs directly via TCP and HTTP are quite clear, however, trying to log securely over TCP requires some additional processes. The stages involved in this process are listed below. Note that there may be other ways of achieving this same result, and the proceedure highlighted in the article is just one out of many other possible ways.
- Generate Self Signed Server Certificate
- Create a TCP Data Input Using the Splunk Web UI
- Import Splunk Server Certificate Into The Trust Store Used by Your Java (Log Forwarder) Application
- Ensure That Your Application Loads the Relevant Trust Store at Start Up
- Configure the Java Logging Library (e.g log4j2) to Use The Trust Store Containing the Server Certificate
Details about how to generate a self signed certificate can be found on Splunk's website. After generating this certificates, you should now have 3 files
- myServerCertificate.pem
- myServerPrivateKey.key
- myCACertificate.pem
These 3 files can be concatenated into 1 (myNewServerCertificate.pem), following the instructions here
Details on how to set up a data receiver/input in Splunk can be found here. In case the documentation feels overwhelming, you can just log in straight on the Web UI to get it done. It is quite intuitive. Much easier than documentation makes it look like.
- Log in to Splunk
- Click 'Settings'
- Click 'Data Inputs'
- Select TCP 'Add New'
- Click 'New Local TCP'
- Input the number of the port you want Splunk to listen for these inputs on
- Source type = raw not log4j
- The rest of the selection will be based on your preference
For secure communication between your application (forwarding the logs) and your Splunk receiver, the trust store of your application should include your myNewServerCertificate.pem. For a Java application, you can go about this in multiple ways which are highlighted below.
-
Export the myNewServerCertificate.pem generated in step 1 into the default trust store of the JDK your application is being run with.
-
Create your own brand new trust store and import only the myNewServerCertificate.pem into it
To perform these imports/exports of certificates between trust stores, you may need the java keytool (a Java based key and certificate management command line tool). It is usually located in %JAVA_HOME/bin/keytool. Also, the
JDK's default trust store can be found in %JAVA_HOME%/jre/lib/security/cacerts
Sample code below shows how to create a brand new trust store having only myNewServerCertificate.pem in it:
keytool -import -alias yourSplunkCertAlias -file myNewServerCertificate.pem -keystore truststore.jks -deststorepass yourTrustStorePassword
truststore.jks will be a brand new trust store having only the myNewServerCertificate.pem
To export myServerCertificate.pem into the trust store of the current Java installation:
keytool -import -trustcacerts -file myServerCertificate.pem -alias yourSplunkCertAlias -keystore $JAVA_HOME/jre/lib/security/cacerts
Option 1 above means at run-time, other SSL related operations may fail, as the trust store in use would be missing a lot of certificate at run time. What is recommended to go along with option 1 is to update the newly created trust store with certificates from the JDK's default trust store.
-
Helpful example on copying certificates to and from a trust store in Java can be found here
-
Information on other helpful
keytoolcommands can be found here
If you are making use of a newly created trust store and not the Java default, ensure that at start up, you program points to this new trust store. This is important especially when using the java.util.logging.StreamHandler class. A helper code snippet to help you do this can be found below.
public static void loadTrustStore(String pathToTrustStore) {
try {
URL trustStoreURL = SecurityHelper.class.getResource(pathToTrustStore);
if (trustStoreURL != null) {
System.setProperty("javax.net.ssl.trustStore", trustStoreURL.getPath());
System.setProperty("javax.net.ssl.trustStorePassword", "yourTrustStorePassword");
RuntimeState.isSslTrustStoreLoaded = true;
} else {
Logger.getGlobal().log(Level.SEVERE, "Could not load SSL trust store. " +
"This will affect secure logging");
RuntimeState.isSslTrustStoreLoaded = false;
}
} catch (Exception ex) {
Logger.getGlobal().log(Level.SEVERE, "Could not load SSL trust store. " +
"This will affect secure logging");
RuntimeState.isSslTrustStoreLoaded = false;
}
}Depending on the java logging library in use, this can be configured differently. Also, you need the Splunk Logging for Java library imported in your project.
<Configuration status="OFF" name="yourLogger" packages="">
<Appenders>
<Socket name="ssl" host="localhost" port="9338">
<PatternLayout charset="UTF-8">
<pattern>%date{ISO8601} [%thread] [%level] %msg %ex{short} %n</pattern>
</PatternLayout>
<SSL>
<TrustStore location="security/truststore.jks" password="yourTrustStorePassword"/>
</SSL>
</Socket>
</Appenders>
<Loggers>
<Root level="INFO">
</Root>
<Logger name="splunk.secure.logger" level="DEBUG">
<AppenderRef ref="ssl"/>
</Logger>
</Loggers>
</Configuration>public class SecureRemoteLogHelper {
private Logger globalLogger = Logger.getGlobal();
private Logger secureLogger;
private Socket socket;
public BasicLoggerSecureRemoteLogHelper() {
}
public Logger getSecureRemoteLogger(String remoteHostAddress, int remoteHostPort) {
if (secureLogger == null
|| socket == null
|| socket.isClosed()
|| !socket.isConnected()) {
return setUpSecureRemoteLogger(remoteHostAddress, remoteHostPort);
}
return secureLogger;
}
private Logger setUpSecureRemoteLogger(String remoteHostAddress, int remoteHostPost) {
if (socket != null) closeSocket();
secureLogger = null;
StreamHandler streamHandler;
if (!isSslTrustStoreLoaded) {
SecurityHelper
.loadTrustStore(TRUSTSTORE_FILE_PATH);
}
if (isSslTrustStoreLoaded) {
secureLogger = Logger
.getLogger(secureRemoteLoggerDefaultName);
secureLogger
.setUseParentHandlers(false);
streamHandler = getStreamHandler(remoteHostAddress, remoteHostPost, null);
if (streamHandler != null) {
secureLogger.addHandler(streamHandler);
} else {
secureLogger = null;
}
}
return secureLogger;
}
private StreamHandler getStreamHandler(String remoteHostAddress, int remoteHostPost) {
StreamHandler streamHandler = null;
if (getOutputStream(remoteHostAddress, remoteHostPost) != null) {
try {
streamHandler = new StreamHandler(getOutputStream(remoteHostAddress, remoteHostPost),
new SimpleFormatter()) {
@Override
public synchronized void publish(final LogRecord record) {
super.publish(record);
try {
flush();
getOutputStream(remoteHostAddress, remoteHostPost).flush();
} catch (IOException e) {
globalLogger.log(Level.SEVERE, "Error occurred while " +
"trying to flush ssl output stream. " + e.getMessage());
}
}
};
streamHandler.setLevel(Level.WARNING);
} catch (Exception ex) {
globalLogger.log(Level.SEVERE, "Error occurred while " +
"trying to create stream handler. " + ex.getMessage());
return null;
}
}
return streamHandler;
}
private Socket getSocket(String remoteHostAddress, int remoteHostPort) {
SSLSocketFactory sslSocketFactory;
if (socket == null || !socket.isConnected()) {
try {
sslSocketFactory = (SSLSocketFactory) SSLSocketFactory.getDefault();
socket = sslSocketFactory.createSocket(remoteHostAddress, remoteHostPort);
socket.setKeepAlive(true); //2 hours 11 minutes by default
} catch (Exception e) {
globalLogger.log(Level.SEVERE, "Error occurred while trying" +
" to create ssl socket. " + e.getMessage());
return null;
}
}
return socket;
}
private OutputStream getOutputStream(String remoteHostAddress, int remoteHostPost) {
OutputStream outputStream;
try {
if (getSocket(remoteHostAddress, remoteHostPost) != null) outputStream
= getSocket(remoteHostAddress, remoteHostPost).getOutputStream();
else return null;
} catch (Exception e) {
globalLogger.log(Level.SEVERE, "Error occurred while trying" +
" to create ssl output stream. " + e.getMessage());
return null;
}
return outputStream;
}
private void closeSocket() {
try {
socket.close();
} catch (IOException e) {
globalLogger.severe("Error occurred while trying to close ssl socket. "+ e.getMessage());
}
}
}This does not currently work well with sending logs to splunk via tcp-ssl. Find more details about this issue on Stack Overflow
Maven Import Snippet For Splunk Logging For Java
<dependency>
<groupId>com.splunk.logging</groupId>
<artifactId>splunk-library-javalogging</artifactId>
<version>1.6.2</version>
</dependency>Gradle Import Snippet For Splunk Logging For Java
compile group: 'com.splunk.logging', name: 'splunk-library-javalogging', version: '1.6.2'SecureRemoteLogHelper helper = new SecureRemoteLogHelper();
java.util.logging.Logger secureJuliRemoteLogger = helper.getSecureRemoteLogger("localhost", 9338);
secureJuliRemoteLogger.severe("A Critical Error Just Occurred");org.apache.logging.log4j.core.Logger secureLog4J2RemoteLogger = (org.apache.logging.log4j.core.Logger) LogManager.getLogger("splunk.secure.logger");
secureLog4J2RemoteLogger.info("App Status: OK");A shorter direct way of testing is to load the trust store, create raw socket connection and write directly to the port.
public class Main {
public static void main(String[] args) throws IOException {
loadTrustStore("truststore.jks");
String host = "localhost";
Integer port = 9338;
SSLSocketFactory sslsocketfactory = (SSLSocketFactory) SSLSocketFactory.getDefault();
SSLSocket sslsocket = (SSLSocket) sslsocketfactory.createSocket(host, port);
InputStream in = sslsocket.getInputStream();
OutputStream out = sslsocket.getOutputStream();
System.out.println("isConnected: "+sslsocket.isConnected());
System.out.println("Local Port: "+sslsocket.getLocalPort());
System.out.println("Remote Port: "+sslsocket.getPort());
out.write(1);
while (in.available() > 0) {
System.out.print("Value gotten in: "+in.read());
}
System.out.println("Secured interaction successful");
}On splunk web
- Click 'Apps'
- Click 'Search and Reporting'
- Click Data Sumamry
- Select the souce or IP address the data was meant to come from
- You sheould see this data already collected by the splunk indexer
For challenges you may encounter while trying to set-up log farwarding from Java to splunk, the tips below will help in troubleshooting
Default configurations used by your Splunk instance can be found in /opt/spunk/etc/system/default/*. To override/add to any of the configurations in that location, create a file with the same name in location /opt/spunk/etc/system/local/* and add the configurations you wish to add or amend.
To get logging over ssl working, we will need to add some more configurations to 2 main files. The third (props.conf) is optional.
- input.conf
- server.conf
- props.conf
The snippets below show a rough example of what your ..local/input.conf files should look like if you want to create an indexer that receives data over plain TCP-SSL on port 9336 and another that uses TCP-SSL on port 9338.
/opt/spunk/etc/system/local/input.conf
[tcp://:9336]
acceptFrom=*
connection_host=dns
sourcetype=log4j
[tcp-ssl://:9338]
disabled = 0
[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/myNewServerCertificate.pem
sslPassword = passwordOfYourmyNewServerCertificate.pem
requireClientCert = false
/opt/spunk/etc/system/local/props.conf (Optional)
[log4j]
CHARSET = UTF-16LE
SHOULD_LINEMERGE = false
/opt/spunk/etc/system/local/server.conf
[sslConfig]
enableSplunkdSSL = true
serverCert = /opt/splunk/etc/auth/mycerts/myNewServerCertificate.pem
sslPassword = passwordOfYourmyNewServerCertificate.pem
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCACertificate.pem
[general]
pass4SymmKey = {auto generated}
On start up of Splunk, you can check logs to be sure everything is fine. Important logs can be found in /opt/splunk/var/log/splunk/splunkd.log
You should see logs similar to the below, if everything is Ok
01-23-2020 22:25:32.696 +0000 INFO TcpInputConfig - IPv4 port 9336 is reserved for raw input
01-23-2020 22:25:32.696 +0000 INFO TcpInputConfig - IPv4 port 9336 will negotiate s2s protocol level 6
01-23-2020 22:25:32.696 +0000 INFO TcpInputConfig - Creating FwdDataSSLConfig SSL context. Will open port=IPv4 port 9338 with compression=1
01-23-2020 22:25:32.696 +0000 INFO TcpInputConfig - IPv4 port 9338 is reserved for raw input (SSL)
01-23-2020 22:25:32.696 +0000 INFO TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 22:25:32.696 +0000 INFO TcpInputProc - Creating raw Acceptor for IPv4 port 9336 with Non-SSL
01-23-2020 22:25:32.697 +0000 INFO TcpInputProc - Creating raw Acceptor for IPv4 port 9338 with SSL
Misconfigured firewall settings related to the port you have set for Splunk to ingest data from could lead to data loss, and consequently, nothing being indexed. To fix this, remember to enable the input port. See example below.
sudo ufw enable 9938
To confirm that the port has been unblocked:
sudo ufw status
Use the code snippet below to confirm that the trust store you are using is OK
keytool -list -v -keystore truststore.jks
If you your are forwarding data to Splunk from an external network. Check to be sure you have set up port forwarding on your local router to make sure data sent to Splunk is not being dropped by your router. How to do this differs from router-to-router, and you may need to check with your provider to know how best to go about this.
Once you start forwarding data to the port, you can also do a verbose TCP dump to be sure data gets to the port. This is especially useful when you are unable to ascertain that the indexer is getting any data at all.
sudo tcpdump port 9338 -vv