Skip to content

Instantly share code, notes, and snippets.

@smougenot

smougenot/A_Logstash.conf

Created July 26, 2012 13:59
Show Gist options
  • Star 59 You must be signed in to star a gist
  • Fork 11 You must be signed in to fork a gist
  • Save smougenot/3182192 to your computer and use it in GitHub Desktop.
Save smougenot/3182192 to your computer and use it in GitHub Desktop.
Download ZIP
Logstash Multiline Filter for Java Stacktrace (tested on field)
Raw
A_Logstash.conf
# stacktrace java as one message
multiline {
#type => "all" # no type means for all inputs
pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
what => "previous"
}
Raw
sample_original_stacktrace.log
juil. 25, 2012 10:49:46 AM hudson.triggers.SafeTimerTask run
Grave: Timer task com.base2services.jenkins.SqsQueueHandler@32eea79d failed
com.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Empty key
at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:55)
at com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:83)
at com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:46)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:238)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:776)
at com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:564)
at com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:732)
at com.base2services.jenkins.SqsProfile.createQueue(SqsProfile.java:72)
at com.base2services.jenkins.SqsProfile.getQueueUrl(SqsProfile.java:62)
at com.base2services.jenkins.SqsQueueHandler.doRun(SqsQueueHandler.java:37)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Empty key
at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
... 14 more
Caused by: java.lang.IllegalArgumentException: Empty key
at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)
at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:87)
... 15 more
juil. 25, 2012 10:49:54 AM hudson.slaves.SlaveComputer tryReconnect
Infos: Attempting to reconnect CentosVagrant
Raw
sample_result_message.json
{"hits":[
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"u5S8Bg2RRo-YutkXuSCoKQ","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.457000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"juil. 25, 2012 10:49:46 AM hudson.triggers.SafeTimerTask run\r"},"sort":[1343309704457]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"zyeCfYboStC1oofKFMW8-w","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":["multiline"]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.474000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"Grave: Timer task com.base2services.jenkins.SqsQueueHandler@32eea79d failed\r\ncom.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Empty key\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:55)\r\n\tat com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:83)\r\n\tat com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:46)\r\n\tat com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:238)\r\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:776)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:564)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:732)\r\n\tat com.base2services.jenkins.SqsProfile.createQueue(SqsProfile.java:72)\r\n\tat com.base2services.jenkins.SqsProfile.getQueueUrl(SqsProfile.java:62)\r\n\tat com.base2services.jenkins.SqsQueueHandler.doRun(SqsQueueHandler.java:37)\r\n\tat hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)\r\n\tat java.util.TimerThread.mainLoop(Timer.java:555)\r\n\tat java.util.TimerThread.run(Timer.java:505)\r\nCaused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Empty key\r\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)\r\n\t... 14 more\r\nCaused by: java.lang.IllegalArgumentException: Empty key\r\n\tat javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:87)\r\n\t... 15 more\r"},"sort":[1343309704474]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"efd9s2rgTFeNiatd__pofg","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.599000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"\r"},"sort":[1343309704599]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"xcxgNxRUTb2syet--n3CsQ","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.610000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"juil. 25, 2012 10:49:54 AM hudson.slaves.SlaveComputer tryReconnect\r"},"sort":[1343309704610]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"WwoynUlNSl6iAMzFw-4gZg","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.631000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"Infos: Attempting to reconnect CentosVagrant\r"},"sort":[1343309704631]}
]}
@ksonu
Copy link

ksonu commented Sep 23, 2014

Thanks for posting this , But Can you please explain how to use this multiline codec. Is this will be added to the shipper.conf or indexer.conf . and If in indexer.conf then at which part of the configuration, filter or input or some other part. A detail explanation of how to use this codec will help every one.

@steveash
Copy link

steveash commented Jun 7, 2015

I would update the pattern line to:
pattern => "(^[a-zA-Z.]+(?:Error|Exception): .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"

That will get both exceptions and error and reduce the chance of accidentally throwing any line with "Exception:" in to the previous message. The typical stack trace print outs start with the fully qualified exception class name.

@bobbyhubbard
Copy link

This also separates the first line of the log from the exception itself. Notice sample result json #1 is the first line of the exception log and the second json sample response if the multiline exception. This isnt ideal. Really we need the exception AND the initial log line in one single entry.

@bobbyhubbard
Copy link

Got it I think. Tiny tweak...
(^[a-zA-Z.]+(?:Error|Exception).+)|(^\s+at .+)|(^\s+... \d+ more)|(^\t+)|(^\s*Caused by:.+)

@Aditya-prit
Copy link

It does not work if there is a break of one or two lines in java stack trace.

2016-05-03 07:47:02,504 [C3P0PooledConnectionPoolManager[identityToken->2s8lue9e179a5c21ofo3nn|27d57a2c]-HelperThread-#5] WARN c.m.v.r.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@4736c113 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:
java.sql.SQLException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:673)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:715)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:385)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:30)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:564)

Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.net.ns.NSProtocolStream.negotiateConnection(NSProtocolStream.java:272)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:263)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1360)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:486)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:715)

@Aditya-prit
Copy link

I think colon : is responsible for breakup of log in kibana . Could you please help me and provide an updated pattern for my use-case

@nagaraj200788
Copy link

nagaraj200788 commented Feb 17, 2017 •
edited

Thanks good codec pattern but for stack trace there is a space in between and also different word like ORA- and blank line

java.lang.Exception: And binfilename=TIS_BACARDI_BasisSwapRA_LISP-PRINCIPAL_2017-01-04_Scratt_1483574512000_14_Input.bin.IN09718909L.ISSUE
at sgcib.solstis.persist.service.impl.BinFileHandler.handle(BinFileHandler.java:126) ~[classes/:na]
at sgcib.solstis.persist.service.impl.PersistCallable.call(PersistCallable.java:32) [classes/:na]
at sgcib.solstis.persist.service.impl.PersistCallable.call(PersistCallable.java:13) [classes/:na]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [na:1.6.0_26]
at java.util.concurrent.FutureTask.run(FutureTask.java:138) [na:1.6.0_26]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_26]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_26]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_26]
Caused by: sgcib.solstis.core.exception.SolstisDAOException: Exception occurred while inserting Analysis and BinCache.
at sgcib.solstis.persist.service.SplittedBinsService.insertAnalysisAndBinCache(SplittedBinsService.java:151) ~[classes/:na]
at sgcib.solstis.persist.service.SplittedBinsService.initBinCache(SplittedBinsService.java:106) ~[classes/:na]
at sgcib.solstis.persist.service.impl.BinFileHandler.handle(BinFileHandler.java:106) ~[classes/:na]
... 7 common frames omitted
Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-01400: cannot insert NULL into ("SOLSTIS_LOCAL_BAN"."ANALYSISPROCESS"."COMMANDNAME")
ORA-06512: at line 1

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:205) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1043) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1336) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3613) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3694) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OracleCallableStatement.executeUpdate(OracleCallableStatement.java:4780) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1354) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at com.mchange.v2.c3p0.impl.NewProxyCallableStatement.executeUpdate(NewProxyCallableStatement.java:2160) ~[c3p0-0.9.1.2.jar:0.9.1.2]
at sgcib.solstis.core.dao.referential.AnalysisProcessDao.insertAnalysisProcess(AnalysisProcessDao.java:101) ~[classes/:na]
at sgcib.solstis.persist.service.SplittedBinsService.insertAnalysisAndBinCache(SplittedBinsService.java:138) ~[classes/:na]
... 9 common frames omitted

How to read all in single event.Now i see three events

Tried

(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)|(^ORA-06512:.+)

and

pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\sCaused by:.+)|(^\sORA-06512:.+)"

not worked

@monotek
Copy link

monotek commented Aug 6, 2017

This could help: https://sematext.com/blog/2015/05/26/handling-stack-traces-with-logstash/

@brijsoni23
Copy link

Following pattern is working:

codec => multiline {
pattern => "^[^\[]"
what => "previous"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment