ELK Stack Example Apache Logs Import

elk-apache-example.md

Getting Started with Elastic Stack for Apache Logs

This document is an updated version of the official guide. The official guide is designed for ELK version 6.0, where as when this tutorial is written, the newest version of the ELK stack is 7.5.1.

This example provides sample files to ingest, analyze & visualize Apache Access Logs using the Elastic Stack, i.e. Elasticsearch, Filebeat and Kibana. The sample logs in this example are in the default apache combined log format.

In order to achieve this we use the Filebeat Apache module per Elastic Stack best practices.

Historically this example used Logstash. This configuration is provided for reference only.

Versions

Example has been tested in following versions:

• Elasticsearch 7.5
• Filebeat 7.5
• Kibana 7.5

Example Contents

• apache_logs - Sample apache log files

Legacy Files

• apache_logstash.conf: Logstash configuration. REFERENCE ONLY.
• apache_kibana.json: Custom Kibana dashboard associated with Logstash configuration. REFERENCE ONLY.
• apache_template.json: Template for Logstash ingestion. REFERENCE ONLY.

Installation & Setup

• Follow the Installation & Setup Guide to install and test the Elastic Stack (you can skip this step if you have a working installation of the Elastic Stack,)

• For a Debian ELK installation guide, you can refer to this Debian ELK Installation tutorial that I have written.

• Run Elasticsearch & Kibana

    <path_to_elasticsearch_root_dir>/bin/elasticsearch
    <path_to_kibana_root_dir>/bin/kibana

• Install the required plugins

  • ingest-user-agent and ingest no longer need to be installed manually as they are now come as a part of the ELK stack.

• Check that Elasticsearch and Kibana are up and running.

  • Open localhost:9200 in web browser -- should return status code 200
  • Open localhost:5601 in web browser -- should display Kibana UI.

  Note: By default, Elasticsearch runs on port 9200, and Kibana run on ports 5601. If you changed the default ports, change the above calls to use appropriate ports.

• Download and install Filebeat as described here. Do not start Filebeat

Ingesting Logs Using Filebeat

Option 1: Running Filebeat Command

From the Filebeat installation directory setup the apache2 module and ingest the sample provided. Modify the following command to include the location to the above sample data file. Further details on the apache2 module configuration can be found here.

cd <path_to_filebeat_root_dir>
./filebeat setup -e --modules=apache -M "apache2.access.var.paths=[<PATH_TO_APACHE_LOGS_FILE>]"

Verify that data is successfully indexed into Elasticsearch. Running http://localhost:9200/filebeat-*/_count should return a response a "count":10000.

Note: The module assumes that you are running Elasticsearch on the same host as Filebeat and have not changed the defaults. Modify the settings my appending the parameter to the -E switch:

-E output.elasticsearch.hosts=<HOST>:<PORT>

Option 2: Using the Filebeat Service

If the above method doesn't work for you, you can also use the Filebeat service to ingest the logs.

Download the apache_logs file. In this example we are saving it as /root/access.log.

curl -sL wget https://raw.githubusercontent.com/elastic/examples/master/Common%20Data%20Formats/apache_logs/apache_logs -o /root/access.log

Open /etc/filebeat/modules.d/apache.yml and edit the module/access/var.paths key. Add the path to the downloaded log file into the value like below.

- module: apache
  access:
    enabled: true
    var.paths: ["/root/access.log*"]

  error:
    enabled: true

Start Filebeat service and wait for Filebeat to ingest the logs into Elastic Search.

systemctl start filebeat

Keep on querying the Elastic Search database status using the following command.

curl http://localhost:9200/filebeat-*/_count

If the logs are fully ingested, you should be seeing something like:

{"count":10000,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}

Finally, stop the filebeat command and run filebeat setup to setup Kibana dashboards and other presets. This will install things like the Filebeat Apache dashboard.

filebeat -e setup --modules apache

Visualize Data in Kibana

Access Kibana by going to http://localhost:5601 in a web browser. You may be asked to select a default index pattern, if this is a clean Kibana/ES install. If so, select the "filebeat-*" index pattern and click the star button in the upper right.

For the timestamp field name, choose @timestamp.

Click on Dashboard tab and open [Filebeat Apache] Access and error logs ECS dashboard.

From the time range selector in the top right, select the time period 2015-05-17 00:00:00.000 to 2015-05-21 12:00:00.000 and click Go. You may also choose something like Last 15 Years to make the query easier if you have a clean ELK installation.

Voila! You should see the following dashboard. Happy Data Exploration!

We Would Love Your Feedback

If you found this example helpful and would like more such Getting Started examples for other standard formats, we would love to hear from you. If you would like to contribute Getting Started examples to this repo, we'd love that too!