Adjusting Filebeat when ES logs change

adjusting_filebeat_es_logs.md

Background

Filebeat consumes Elasticsearch logs via its elasticsearch module. Specifically, for each type of Elasticsearch log (server, gc, deprecation, etc.) there is a corresponding fileset under the Filebeat elasticsearch module. This fileset is responsible for parsing the Elasticsearch log files into structured event that can then be shipped to Elasticsearch or other outputs.

So whenever the structure of Elasticsearch logs changes, the changes must be tested with the Filebeat elasticsearch module to ensure two things:

• that the module can handle the new log structure, and
• that the module can continue to handle the previous log structure (in case a user is running an older version of Elasticsearch against a newer version of Filebeat)

If necessary, the ingest pipeline used by the fileset to do the parsing should be updated.

Steps

These steps show how to adjust Filebeat to handle structure changes to Elasticsearch logs. It uses the Elasticsearch server log as an example.

1. Make Elasticsearch generate some sample log lines that use the new structure.

2. Add a new file under https://github.com/elastic/beats/tree/master/filebeat/module/elasticsearch/server/test with the new lines. Note that we are adding under the elasticsearch/server folder because that's the fileset that is responsible for parsing Elasticsearch server logs.

3. Build Filebeat's test binary.

   cd $GOPATH/src/github.com/elastic/beats/filebeat
   make filebeat.test
   

4. Initialize and activate the Python virtualenv used by Filebeat for testing.

   make python-env
   . build/python-env/bin/activate
   

5. Clear out data from old test runs.

   rm -rf build/system-tests/*
   

6. Make sure Elasticsearch is running on localhost:9200.

7. Run the Filebeat system test for the elasticsearch/server fileset:

   GENERATE=1 INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=elasticsearch TESTING_FILEBEAT_FILESETS=server nosetests -v --nocapture tests/system/test_modules.py
   

   a. If this succeeeds, a new *.expected.json file will be added as a sibling of the new sample file you added in step 1. Check this file to make sure it looks good. If not, you probably need to adjust one or more of the ingest pipelines in https://github.com/elastic/beats/tree/master/filebeat/module/elasticsearch/server/ingest and repeat steps 5-6 again.

   b. If this fails, you will see errors in the console. Further, you can inspect the files under build/system-tests for additional details of the failure. You probably need to adjust one or more of the ingest pipelines in https://github.com/elastic/beats/tree/master/filebeat/module/elasticsearch/server/ingest and repeat steps 5-6 again.

8. Also make sure that no *.expected.json files other than the one corresponding to your new sample file are changed. If this happens, it indicates that any changes you made to the ingest pipeline in step 6a or 6b are not backwards compatible!