learning syslog-ng patterndb

example.xml

<patterndb version='4' pub_date='2010-10-17'>
    <ruleset name='ssh' id='123456678'>
        <pattern>ssh</pattern>
            <rules>
                <rule provider='me' id='182437592347598' class='system'>
                    <patterns>
                        <pattern>Accepted @ESTRING:SSH.AUTH_METHOD: @for @ESTRING:SSH_USERNAME: @from @ESTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
                    </patterns>
                    <examples>
                        <example>
                            <test_message program="ssh">Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
                            <test_values>
                                <test_value name="SSH.AUTH_METHOD">password</test_value>
                                <test_value name="SSH_USERNAME">sampleuser</test_value>
                                <test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
                                <test_value name="SSH_PORT_NUMBER">42156</test_value>
                            </test_values>
                       </example>
                    </examples>
                </rule>
            </rules>
    </ruleset>
</patterndb>

notes.md

Notes:

example.xml is a fixed[1] version of the documentation version

syslog-ng.conf is a copy of the documentation example

All testing and example carried out on CentOS:

   [nick@localhost ~]$ cat /etc/redhat-release ; uname -a; rpm -qa | grep syslog-ng
   CentOS Linux release 7.2.1511 (Core) 
   Linux CentOSvm.local 3.10.0-327.3.1.el7.x86_64 #1 SMP Wed Dec 9 14:09:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
   syslog-ng-3.5.6-1.el7.x86_64
   [nick@localhost ~]$

[1] fixed by Fabien Wernli

Update 23rd Dec 2015

Improvements with the help of Alex Deprez:

• New Test machine - CentOS7 with Syslog-NG 3.5.6
• Test logfile needed to be prepended with ${PROGRAM}, in this case that is ssh
• Output template needed variable escaping
• Additional output showing that tests can use the -M message switch instead of -f input file

output.log

sampleuser; 10.50.0.247; 
user; 10.51.0.27; 

pdbtool_match.output.txt

[nick@localhost ~]$ pdbtool match -p example.xml -f testfile.log --template "\${SSH_USERNAME}; \${SSH_CLIENT_ADDRESS}; \n" 
sampleuser; 10.50.0.247; 
user; 10.51.0.27;
[nick@localhost ~]$ pdbtool match -p example.xml -f testfile.log --template "\${SSH_USERNAME}; \${SSH_CLIENT_ADDRESS}; \n" -D -v
Module loaded and initialized successfully; module='syslogformat'
Module loaded and initialized successfully; module='basicfuncs'
Pattern matching part:
Accepted @ESTRING:SSH.AUTH_METHOD=password@for @ESTRING:SSH_USERNAME=sampleuser@from @ESTRING:SSH_CLIENT_ADDRESS=10.50.0.247@port @NUMBER:SSH_PORT_NUMBER=42156@ ssh2
Matching part:
Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2
Values:
MESSAGE=Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2
PROGRAM=ssh
LEGACY_MSGHDR=ssh: 
.classifier.class=system
.classifier.rule_id=182437592347598
SSH_USERNAME=sampleuser
SSH_CLIENT_ADDRESS=10.50.0.247
SSH.AUTH_METHOD=password
SSH_PORT_NUMBER=42156
TAGS=

Pattern matching part:
Accepted @ESTRING:SSH.AUTH_METHOD=password@for @ESTRING:SSH_USERNAME=user@from @ESTRING:SSH_CLIENT_ADDRESS=10.51.0.27@port @NUMBER:SSH_PORT_NUMBER=4256@ ssh2
Matching part:
Accepted password for user from 10.51.0.27 port 4256 ssh2
Values:
MESSAGE=Accepted password for user from 10.51.0.27 port 4256 ssh2
PROGRAM=ssh
LEGACY_MSGHDR=ssh: 
.classifier.class=system
.classifier.rule_id=182437592347598
SSH_USERNAME=user
SSH_CLIENT_ADDRESS=10.51.0.27
SSH.AUTH_METHOD=password
SSH_PORT_NUMBER=4256
TAGS=

Closing log transport fd; fd='3'
[nick@localhost ~]$ 

pdbtool_match_nofile.output.txt

[nick@localhost ~]$ pdbtool match -P "ssh" -M "Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2" -p example.xml --template "\${SSH_USERNAME}; \${SSH_CLIENT_ADDRESS}; \n" -c
sampleuser; 10.50.0.247;
[nick@localhost ~]$ pdbtool match -P "ssh" -M "Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2" -p example.xml --template "\${SSH_USERNAME}; \${SSH_CLIENT_ADDRESS}; \n" -c -D -v
Module loaded and initialized successfully; module='syslogformat'
Module loaded and initialized successfully; module='basicfuncs'
Pattern matching part:
Accepted @ESTRING:SSH.AUTH_METHOD=password@for @ESTRING:SSH_USERNAME=sampleuser@from @ESTRING:SSH_CLIENT_ADDRESS=10.50.0.247@port @NUMBER:SSH_PORT_NUMBER=42156@ ssh2
Matching part:
Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2
Values:
MESSAGE=Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2
PROGRAM=ssh
.classifier.class=system
.classifier.rule_id=182437592347598
SSH_USERNAME=sampleuser
SSH_CLIENT_ADDRESS=10.50.0.247
SSH.AUTH_METHOD=password
SSH_PORT_NUMBER=42156
TAGS=

[nick@localhost ~]$ 

pdbtool_test.output.txt

[nick@localhost ~]$ pdbtool test --validate example.xml 
example.xml validates
Testing message program='ssh' message='Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2'
[nick@localhost ~]$ 

syslog-ng.conf

@version:3.5

# Default options copied from distribution
options {

        flush_lines (0);
        time_reopen (10);
        log_fifo_size (1000);
        chain_hostnames (off);
        use_dns (no);
        use_fqdn (no);
        create_dirs (yes);
        keep_hostname (yes);
        owner("nick");
        group("nick");
        perm(0640);
};

source test_logfile {
        file("/home/nick/testfile.log");
};

parser test_pattern {
            db_parser(
                file("/home/nick/example.xml")
            );
        };

destination test_output {
        file("/home/nick/output.log"
                template("${SSH_USERNAME}; ${SSH_CLIENT_ADDRESS}; \n")
                template_escape(no)
        );
};

log { source (test_logfile); parser(test_pattern); destination (test_output); };

testfile.log

ssh: Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2
ssh: Accepted password for user from 10.51.0.27 port 4256 ssh2