gustafa

Sample logs

Frontend logs	Backend logs
Sending request:Rest.li endpoint: d2://nuageHTTP Method: POSTHTTP Headers {“x-req-id”: 123456,“user”: “lideveloper”}HTTP Body { “databaseType”: “VOLDEMORT” “name”: “XYZ”}HTTP Response:...	[req-id-123456]: Handling new request.[req-id-123456]: User lideveloper has appropriate privileges.[req-id-123456]: Creating Voldemort store XYZException ... Status: 500

gustafa

Example email sent to Nuage development team when a user experiences an exception

####Unexpected Exception observed in Nuage Recipients: nuage-debugging

Request ID: 123456
User: lideveloper

Frontend Info

gustafa

Log4j.xml - Emit the request ID on every log line.

<layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="%d{yyyy/MM/dd HH:mm:ss.SSS} [req-id-%X{requestId}] ..." />
</layout>

gustafa

Webserver.java - Adding request ID to the thread of the request.

import org.apache.log4j.MDC;

public void onRequest(request) {
  String requestId = request.get(“x-req-id”);
  MDC.put(“requestId”, requestId);
}

gustafa

Kibana - Getting all related loglines stored in Elasticsearch from the Nuage frontend and backend.

Search: request_id: 123456

Results:

| Date | Service | Host | Request ID | Message | | ----- | ----- | ----- | ----- | -----| ------: | | ... | nuage-frontend | host1 | 123456 | User lidev creating Espresso database | | ... |nuage-backend |host2 |123456 |Handling new request from nuage-frontend. | | ... | nuage-backend | host2 | 123456 | User lidev is authenticated to create DB | | ... | nuage-backend | host2 |123456 | Uploading DB definition to Espresso host3| |... | nuage-backend | host2 |123456 |DB Creation failed: Connection to host3 timed out.|

gustafa

Logstash - Parse the log line and put the request ID into its own field.

# Add the request ID to a separate field if one is present.
  grok {
    match => ["message", "\[req[\-]id[\-]%{POSINT:request_id}\]$"]
  }