Config files used to send Windows Eventlog data with NXLOG to Logstash

Example output with no filters

{
  "_index": "logstash-2013.12.23",
  "_type": "logs",
  "_id": "eMsyB2q4TA6sH0bW3kCBZA",
  "_score": null,
  "_source": {
    "EventTime": "2013-12-22 22:45:27",
    "Hostname": "aapc",
    "Keywords": -9187343239835812000,
    "EventType": "INFO",
    "SeverityValue": 2,
    "Severity": "INFO",
    "EventID": 7036,
    "SourceName": "Service Control Manager",
    "ProviderGuid": "{555908D1-A6D7-4695-8E1E-26931D2012F4}",
    "Version": 0,
    "Task": 0,
    "OpcodeValue": 0,
    "RecordNumber": 57035,
    "ProcessID": 632,
    "ThreadID": 5772,
    "Channel": "System",
    "Message": "The nxlog service entered the stopped state.",
    "param1": "nxlog",
    "param2": "stopped",
    "EventReceivedTime": "2013-12-22 22:45:29",
    "SourceModuleName": "in",
    "SourceModuleType": "im_msvistalog",
    "@timestamp": "2013-12-23T03:45:40.544Z",
    "@version": "1",
    "type": "WindowsEventLog",
    "host": "10.10.1.2",
    "tags": [
      "_grokparsefailure"
    ],
    "priority": 13,
    "severity": 5,
    "facility": 1,
    "facility_label": "user-level",
    "severity_label": "Notice"
  },
  "sort": [
    1387770340544
  ]
}

nxlog.conf

define ROOT C:\Program Files (x86)\nxlog
 
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension syslog>
    Module      xm_syslog
</Extension>

<Extension json>
    Module      xm_json
</Extension>

<Input in>
    Module      im_msvistalog
    Exec $raw_event = to_json();
   </Input>

<Processor buffer>
    Module  pm_buffer
    MaxSize 1024
    Type Mem
    WarnLimit 512
</Processor>

<Output out>
    Module      om_tcp
    Host        10.1.150.59
    Port        1514
</Output>

<Route r>
    Path        in => buffer => out
</Route>

nxloginput.conf

input {
  syslog {
    type => "WindowsEventLog"
    port => 1514
    codec => json { 
	charset => [ "CP1252" ]
}
}
}


filter {
        if [type] == "WindowsEventLog" {
		mutate { remove_field => [ "Keywords","SourceModuleType","facility","facility_label","host","priority","severity","severity_label","SourceModuleName","RecordNumber","ProviderGuid","Task","ThreadID","ProcessID","Severity","SeverityValue","OpcodeValue","Opcode","ThreadID" ] }
        }
}


output {
  elasticsearch {
  cluster => "Mint16ES"
     }
}

I am not a Windows SysAdmin, but I can see the value in this. Sending in JSON, and the only filter is pruning some named fields—well done!

Thanks! NXLOG is super lightweight and a single config file. (read: easy to deploy with puppet)

At this point I'm looking for a way to include ONLY certain fields in logstash; the windows logs are VERY verbose and include a ton of redundant information.

I'm using nxlog to sent evenlog/iis to logstash for quiet a long time now and it works great. Agree with you on a way to include only certain fields. Haven't tried it yet, but perhaps the prune filter can get the job done: http://logstash.net/docs/1.3.2/filters/prune

ERROR couldn't connect to tcp socket on 172.20.27.21:3515; No connection could be made because the target machine actively refused it. Can any tell me what did i do wrong ?

Module om_tcp Host 10.1.150.59 Port 1514 is 10.1.150.59 is logstash host or window host ? thanks

I don't have access to a windows machine, so I have not tried it with im_msvistalog, but I do similar with im_file. If you want to drop some fields inside nxlog and not send them to logstash at all try:

Module im_msvistalog Exec delete($Keywords); delete($SourceModuleType); to_json();

I know the below config works for me on linux (partial of my whole config)

Module im_file File "/var/log/messages" SavePos TRUE Exec parse_syslog_bsd(); Exec delete($EventReceivedTime); Exec to_json();

See my whole config for nxlog and logstash here: https://gist.github.com/dantheautomator/9255555

It would be better to do the field remove in the NXlog side, to avoid to transfer everything all over the network only to drop it on the logstash side. It is what I am looking for right now.