Skip to content

Instantly share code, notes, and snippets.

@smougenot

smougenot/A_Logstash.conf

Created July 26, 2012 13:59
Show Gist options
  • Save smougenot/3182192 to your computer and use it in GitHub Desktop.
Save smougenot/3182192 to your computer and use it in GitHub Desktop.
Download ZIP
Logstash Multiline Filter for Java Stacktrace (tested on field)
Raw
A_Logstash.conf
# stacktrace java as one message
multiline {
#type => "all" # no type means for all inputs
pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
what => "previous"
}
Raw
sample_original_stacktrace.log
juil. 25, 2012 10:49:46 AM hudson.triggers.SafeTimerTask run
Grave: Timer task com.base2services.jenkins.SqsQueueHandler@32eea79d failed
com.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Empty key
at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:55)
at com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:83)
at com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:46)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:238)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:776)
at com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:564)
at com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:732)
at com.base2services.jenkins.SqsProfile.createQueue(SqsProfile.java:72)
at com.base2services.jenkins.SqsProfile.getQueueUrl(SqsProfile.java:62)
at com.base2services.jenkins.SqsQueueHandler.doRun(SqsQueueHandler.java:37)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Empty key
at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
... 14 more
Caused by: java.lang.IllegalArgumentException: Empty key
at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)
at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:87)
... 15 more
juil. 25, 2012 10:49:54 AM hudson.slaves.SlaveComputer tryReconnect
Infos: Attempting to reconnect CentosVagrant
Raw
sample_result_message.json
{"hits":[
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"u5S8Bg2RRo-YutkXuSCoKQ","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.457000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"juil. 25, 2012 10:49:46 AM hudson.triggers.SafeTimerTask run\r"},"sort":[1343309704457]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"zyeCfYboStC1oofKFMW8-w","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":["multiline"]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.474000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"Grave: Timer task com.base2services.jenkins.SqsQueueHandler@32eea79d failed\r\ncom.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Empty key\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:55)\r\n\tat com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:83)\r\n\tat com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:46)\r\n\tat com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:238)\r\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:776)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:564)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:732)\r\n\tat com.base2services.jenkins.SqsProfile.createQueue(SqsProfile.java:72)\r\n\tat com.base2services.jenkins.SqsProfile.getQueueUrl(SqsProfile.java:62)\r\n\tat com.base2services.jenkins.SqsQueueHandler.doRun(SqsQueueHandler.java:37)\r\n\tat hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)\r\n\tat java.util.TimerThread.mainLoop(Timer.java:555)\r\n\tat java.util.TimerThread.run(Timer.java:505)\r\nCaused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Empty key\r\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)\r\n\t... 14 more\r\nCaused by: java.lang.IllegalArgumentException: Empty key\r\n\tat javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:87)\r\n\t... 15 more\r"},"sort":[1343309704474]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"efd9s2rgTFeNiatd__pofg","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.599000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"\r"},"sort":[1343309704599]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"xcxgNxRUTb2syet--n3CsQ","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.610000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"juil. 25, 2012 10:49:54 AM hudson.slaves.SlaveComputer tryReconnect\r"},"sort":[1343309704610]},
{"_index":"logstash-2012.07.26","_type":"dummy","_id":"WwoynUlNSl6iAMzFw-4gZg","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log"
,"@type":"dummy"
,"@tags":[]
,"@fields":{}
,"@timestamp":"2012-07-26T13:35:04.631000Z"
,"@source_host":"centos-6-vagrant.vagrantup.com"
,"@source_path":"/vagrant_projet/logstash/stacktrace.log"
,"@message":"Infos: Attempting to reconnect CentosVagrant\r"},"sort":[1343309704631]}
]}
@bobbyhubbard
Copy link

Got it I think. Tiny tweak...
(^[a-zA-Z.]+(?:Error|Exception).+)|(^\s+at .+)|(^\s+... \d+ more)|(^\t+)|(^\s*Caused by:.+)

@Aditya-prit
Copy link

It does not work if there is a break of one or two lines in java stack trace.

2016-05-03 07:47:02,504 [C3P0PooledConnectionPoolManager[identityToken->2s8lue9e179a5c21ofo3nn|27d57a2c]-HelperThread-#5] WARN c.m.v.r.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@4736c113 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:
java.sql.SQLException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:673)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:715)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:385)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:30)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:564)

Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.net.ns.NSProtocolStream.negotiateConnection(NSProtocolStream.java:272)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:263)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1360)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:486)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:715)

@Aditya-prit
Copy link

I think colon : is responsible for breakup of log in kibana . Could you please help me and provide an updated pattern for my use-case

@nagaraj200788
Copy link

nagaraj200788 commented Feb 17, 2017 •
edited
Loading

Thanks good codec pattern but for stack trace there is a space in between and also different word like ORA- and blank line

java.lang.Exception: And binfilename=TIS_BACARDI_BasisSwapRA_LISP-PRINCIPAL_2017-01-04_Scratt_1483574512000_14_Input.bin.IN09718909L.ISSUE
at sgcib.solstis.persist.service.impl.BinFileHandler.handle(BinFileHandler.java:126) ~[classes/:na]
at sgcib.solstis.persist.service.impl.PersistCallable.call(PersistCallable.java:32) [classes/:na]
at sgcib.solstis.persist.service.impl.PersistCallable.call(PersistCallable.java:13) [classes/:na]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [na:1.6.0_26]
at java.util.concurrent.FutureTask.run(FutureTask.java:138) [na:1.6.0_26]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_26]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_26]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_26]
Caused by: sgcib.solstis.core.exception.SolstisDAOException: Exception occurred while inserting Analysis and BinCache.
at sgcib.solstis.persist.service.SplittedBinsService.insertAnalysisAndBinCache(SplittedBinsService.java:151) ~[classes/:na]
at sgcib.solstis.persist.service.SplittedBinsService.initBinCache(SplittedBinsService.java:106) ~[classes/:na]
at sgcib.solstis.persist.service.impl.BinFileHandler.handle(BinFileHandler.java:106) ~[classes/:na]
... 7 common frames omitted
Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-01400: cannot insert NULL into ("SOLSTIS_LOCAL_BAN"."ANALYSISPROCESS"."COMMANDNAME")
ORA-06512: at line 1

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:951) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:513) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:205) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1043) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1336) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3613) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3694) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OracleCallableStatement.executeUpdate(OracleCallableStatement.java:4780) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1354) ~[ojdbc6-11.2.0.4.jar:11.2.0.3.0]
at com.mchange.v2.c3p0.impl.NewProxyCallableStatement.executeUpdate(NewProxyCallableStatement.java:2160) ~[c3p0-0.9.1.2.jar:0.9.1.2]
at sgcib.solstis.core.dao.referential.AnalysisProcessDao.insertAnalysisProcess(AnalysisProcessDao.java:101) ~[classes/:na]
at sgcib.solstis.persist.service.SplittedBinsService.insertAnalysisAndBinCache(SplittedBinsService.java:138) ~[classes/:na]
... 9 common frames omitted

How to read all in single event.Now i see three events

Tried

(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)|(^ORA-06512:.+)

and

pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\sCaused by:.+)|(^\sORA-06512:.+)"

not worked

@monotek
Copy link

monotek commented Aug 6, 2017

This could help: https://sematext.com/blog/2015/05/26/handling-stack-traces-with-logstash/

@brijsoni23
Copy link

Following pattern is working:

codec => multiline {
pattern => "^[^\[]"
what => "previous"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment