Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
Logstash Multiline Filter for Java Stacktrace (tested on field)
# stacktrace java as one message
multiline {
#type => "all" # no type means for all inputs
pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
what => "previous"
}
juil. 25, 2012 10:49:46 AM hudson.triggers.SafeTimerTask run
Grave: Timer task com.base2services.jenkins.SqsQueueHandler@32eea79d failed
com.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Empty key
at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:55)
at com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:83)
at com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:46)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:238)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:776)
at com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:564)
at com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:732)
at com.base2services.jenkins.SqsProfile.createQueue(SqsProfile.java:72)
at com.base2services.jenkins.SqsProfile.getQueueUrl(SqsProfile.java:62)
at com.base2services.jenkins.SqsQueueHandler.doRun(SqsQueueHandler.java:37)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Empty key
at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
... 14 more
Caused by: java.lang.IllegalArgumentException: Empty key
at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)
at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:87)
... 15 more
juil. 25, 2012 10:49:54 AM hudson.slaves.SlaveComputer tryReconnect
Infos: Attempting to reconnect CentosVagrant
{"hits":[
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"u5S8Bg2RRo-YutkXuSCoKQ","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.457000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"juil. 25, 2012 10:49:46 AM hudson.triggers.SafeTimerTask run\r"},"sort":[1343309704457]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"zyeCfYboStC1oofKFMW8-w","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":["multiline"]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.474000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"Grave: Timer task com.base2services.jenkins.SqsQueueHandler@32eea79d failed\r\ncom.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Empty key\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:55)\r\n\tat com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:83)\r\n\tat com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:46)\r\n\tat com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:238)\r\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:776)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:564)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:732)\r\n\tat com.base2services.jenkins.SqsProfile.createQueue(SqsProfile.java:72)\r\n\tat com.base2services.jenkins.SqsProfile.getQueueUrl(SqsProfile.java:62)\r\n\tat com.base2services.jenkins.SqsQueueHandler.doRun(SqsQueueHandler.java:37)\r\n\tat hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)\r\n\tat java.util.TimerThread.mainLoop(Timer.java:555)\r\n\tat java.util.TimerThread.run(Timer.java:505)\r\nCaused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Empty key\r\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)\r\n\t... 14 more\r\nCaused by: java.lang.IllegalArgumentException: Empty key\r\n\tat javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:87)\r\n\t... 15 more\r"},"sort":[1343309704474]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"efd9s2rgTFeNiatd__pofg","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.599000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"\r"},"sort":[1343309704599]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"xcxgNxRUTb2syet--n3CsQ","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.610000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"juil. 25, 2012 10:49:54 AM hudson.slaves.SlaveComputer tryReconnect\r"},"sort":[1343309704610]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"WwoynUlNSl6iAMzFw-4gZg","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.631000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"Infos: Attempting to reconnect CentosVagrant\r"},"sort":[1343309704631]}
]}

vimil commented Mar 6, 2013

Wont this cause an issue if two stacktraces are printed one right after the other? won't the first line of the second stacktrace become part of the first message?

Could you please post the complete configuration? Because as for me, just pasting your snippet into one filter just does not work

ksonu commented Sep 23, 2014

Thanks for posting this , But Can you please explain how to use this multiline codec. Is this will be added to the shipper.conf or indexer.conf . and If in indexer.conf then at which part of the configuration, filter or input or some other part. A detail explanation of how to use this codec will help every one.

steveash commented Jun 7, 2015

I would update the pattern line to:
pattern => "(^[a-zA-Z.]+(?:Error|Exception): .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"

That will get both exceptions and error and reduce the chance of accidentally throwing any line with "Exception:" in to the previous message. The typical stack trace print outs start with the fully qualified exception class name.

This also separates the first line of the log from the exception itself. Notice sample result json #1 is the first line of the exception log and the second json sample response if the multiline exception. This isnt ideal. Really we need the exception AND the initial log line in one single entry.

Got it I think. Tiny tweak...
(^[a-zA-Z.]+(?:Error|Exception).+)|(^\s+at .+)|(^\s+... \d+ more)|(^\t+)|(^\s*Caused by:.+)

It does not work if there is a break of one or two lines in java stack trace.

2016-05-03 07:47:02,504 [C3P0PooledConnectionPoolManager[identityToken->2s8lue9e179a5c21ofo3nn|27d57a2c]-HelperThread-#5] WARN c.m.v.r.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@4736c113 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:
java.sql.SQLException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:673)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:715)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:385)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:30)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:564)

Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.net.ns.NSProtocolStream.negotiateConnection(NSProtocolStream.java:272)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:263)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1360)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:486)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:715)

I think colon : is responsible for breakup of log in kibana . Could you please help me and provide an updated pattern for my use-case

nagaraj200788 commented Feb 17, 2017

Thanks good codec pattern but for stack trace there is a space in between and also different word like ORA- and blank line

java.lang.Exception: And binfilename=TIS_BACARDI_BasisSwapRA_LISP-PRINCIPAL_2017-01-04_Scratt_1483574512000_14_Input.bin.IN09718909L.ISSUE
at sgcib.solstis.persist.service.impl.BinFileHandler.handle(BinFileHandler.java:126) ~[classes/:na]
at sgcib.solstis.persist.service.impl.PersistCallable.call(PersistCallable.java:32) [classes/:na]
at sgcib.solstis.persist.service.impl.PersistCallable.call(PersistCallable.java:13) [classes/:na]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [na:1.6.0_26]
at java.util.concurrent.FutureTask.run(FutureTask.java:138) [na:1.6.0_26]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_26]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_26]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_26]
Caused by: sgcib.solstis.core.exception.SolstisDAOException: Exception occurred while inserting Analysis and BinCache.
at sgcib.solstis.persist.service.SplittedBinsService.insertAnalysisAndBinCache(SplittedBinsService.java:151) ~[classes/:na]
at sgcib.solstis.persist.service.SplittedBinsService.initBinCache(SplittedBinsService.java:106) ~[classes/:na]
at sgcib.solstis.persist.service.impl.BinFileHandler.handle(BinFileHandler.java:106) ~[classes/:na]
... 7 common frames omitted
Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-01400: cannot insert NULL into ("SOLSTIS_LOCAL_BAN"."ANALYSISPROCESS"."COMMANDNAME")
ORA-06512: at line 1

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:205) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1043) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1336) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3613) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3694) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OracleCallableStatement.executeUpdate(OracleCallableStatement.java:4780) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1354) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at com.mchange.v2.c3p0.impl.NewProxyCallableStatement.executeUpdate(NewProxyCallableStatement.java:2160) ~[c3p0-0.9.1.2.jar:0.9.1.2]
at sgcib.solstis.core.dao.referential.AnalysisProcessDao.insertAnalysisProcess(AnalysisProcessDao.java:101) ~[classes/:na]
at sgcib.solstis.persist.service.SplittedBinsService.insertAnalysisAndBinCache(SplittedBinsService.java:138) ~[classes/:na]
... 9 common frames omitted

How to read all in single event.Now i see three events

Tried

(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)|(^ORA-06512:.+)

and

pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\sCaused by:.+)|(^\sORA-06512:.+)"

not worked

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment