This tutorial allows to setup an ELK Stack using Amazon ES (Elasticsearch Service) for Elasticsearch & Kibana, and an EC2 instance running Amazon Linux 2 AMI for Logstash.
For the following Steps, we'll work with the
EU (Ireland)(a.k.aeu-west-1) region. Replaceeu-west-1by your region when needed.
We're also assuming you already own an Amazon Web Services Account and you are already logged in.
Go to https://eu-west-1.console.aws.amazon.com/es
Then click "Create a new domain"
Select a domain name and select the last available version of the Elasticsearch engine.
Then click "Next".
To know excatly what you need in
Node configurationandStorage configuration, it really depends on multiple factors and the topic is out of the scope of this tutorial. Howewer you should find useful information here :
- Calculating Storage Requirements : https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/sizing-domains.html#aes-bp-storage
- Get started with Amazon Elasticsearch Service: T-shirt-size your domain : https://aws.amazon.com/blogs/database/get-started-with-amazon-elasticsearch-service-t-shirt-size-your-domain/
- Get Started with Amazon Elasticsearch Service: How Many Data Instances Do I Need ? : https://aws.amazon.com/blogs/database/get-started-with-amazon-elasticsearch-service-how-many-data-instances-do-i-need/
In this tutorial we'll keep things simple selecting "t2.small.elasticsearch" as instance type and keeping everything else as default.
Then click "Next".
Select "Public access" as Network configuration and select the template "Allow open access to the domain" as Access policy.
A prompt should popup, accept the risk and click "Ok".
Then click "Next".
Finally, click "Confirm".
While your domain is being configured by Amazon, you can proceed to the next instructions.
Go to https://eu-west-1.console.aws.amazon.com/ec2/v2
Then click "Launch instance"
Select "Amazon Linux 2 AMI (HVM), SSD Volume Type", it should be the first one.
Select the "t2.micro" instance type, click "Review and Launch" then "Launch".
Follow the instructions about selecting an existing key pair or create an new pair. Be sure to have the selected private key file on your computer, or you won't be able to log into your instance.
Then click on "Launch Instances"
For now, you should see your new EC2 instance as a running instance in your EC2 Dashboard, get your instance IP (IPv4 Public IP) and open a terminal to connect to it through SSH.
You will need to provide the path of your private SSH key file (the one you just downloaded if you created a new key pair when setuping the EC2 instance).
The SSH connection need to be proceeded as follows :
ssh -i SSH_PRIVATE_KEY_FILE_PATH ec2-user@EC2_INSTANCE_IP
| Variable | Description |
|---|---|
SSH_PRIVATE_KEY_FILE_PATH |
Path of your private SSH key file (the one you just downloaded if you created a new key pair when setuping the EC2 instance) |
EC2_INSTANCE_IP |
Your EC2 instance IP |
It should result in something like :
ssh -i ~/.ssh/private.pem ec2-user@35.180.58.201
Then, as it is the first time you are connecting to this IP, SSH will warned you that the authenticity of host can't be etablished and ask you if you are sure to continue connecting. Type yes and you shoud be finally logged into your EC2 instance.
At this stage, we're assuming you are connected to your EC2 instance, following the instructions above.
Before starting install any packages, you will need to have root privileges, so all you need to do is :
sudo su
We'll be working in /root folder for now so move there
cd /root/
Logstash need Java to work properly, so we need to install it before intalling Logstash.
As the time of writing, latest Java 8 version is 8u181, consider check http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html and adjust the version in the followings commands (including link of the RPM) if needed.
Download the RPM :
wget --no-check-certificate -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jre-8u181-linux-x64.rpm
Note that we need to specify options to
wgetto accept the License Agreement through the request
Then install Java using YUM :
yum install -y jre-8u181-linux-x64.rpm
Finally, check Java version to see if everything is fine :
java -version
You can remove the RPM file now :
rm -rf jre-8u181-linux-x64.rpm
As the time of writing , latest Logstash version is 6.3.2, consider check https://www.elastic.co/downloads/logstash and adjust the version in the followings commands if needed.
As you may have issues installing Logstash through the RPM, we will use downloadable binaries instead, so download the TAR.GZ file and unarchive it using tar :
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz
tar -zxf logstash-6.3.2.tar.gz
You can now remove the tar.gz file and move in the created folder :
rm -rf logstash-6.3.2.tar.gz
cd logstash-6.3.2
We'll now configure Logstash, make it work with a short example and finally use it with Amazon ES.
First important thing is as Elasticsearch, Logstash use a JVM (Java Virtual Machine), and by default, it tells the JVM to use a heap with a minimum and maximum size of 1 GB.
Furthermore, as mentioned here : https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html, both minimum heap size (Xms) and maximum heap size (Xmx) should be equal to each other, and Xmx should be set to no more than 50% of your physical RAM, to ensure that there is enough physical RAM left for kernel file system caches.
However, our EC2 instance is of type "t2.micro" for now, and has only 1GB of RAM. We must then adjust the JVM options, otherwise you'll be ne not able yo start your Logstash server.
To do this, you need to edit the jvm.options file in the config folder of Logstash, and set both minimum and maximum heap size to 512m instead of 1g.
We're assuming you're already in the logstash folder under
/root(ex:"/root/logstash-6.3.2")
nano config/jvm.options
Replace
-Xms1g
-Xmx1g
by
-Xms512m
-Xmx512m
Logstash needs a configuration file to need from where it reads data (input) and where to write it (output).
Here it a short example which simply read on stdin and write the Logstash created event from this input on stdout :
input {
stdin { }
}
output {
stdout { }
}
Copy the above content in a file config/example.conf and try it out with Logstash :
./bin/logstash -f config/example.conf
It can makes time to Logstash to launch, so be patient. When all is ready you should see this as last Logstah log :
[2018-08-23T09:42:41,454][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
Now Logstash is waiting for some input, try typing some word, "hello" for example. It should result in :
{
"@timestamp" => 2018-08-23T09:43:46.299Z,
"@version" => "1",
"host" => "ip-xxx-xx-xx-xx.eu-west-1.compute.internal",
"message" => "hello"
}Here we go, Logstash is working fine with a simple example, we can know make a new configuration which will work with Amazon ES.
To stop Logstash, press Ctrl + C.
Get the Endpoint to Elasticsearch in your Elasticsearch Domain Dashboard : https://eu-west-1.console.aws.amazon.com/es/.
You'll find it in the "Overview" Tab of the Dashboard.
./bin/logstash-plugin install logstash-output-amazon_es
Create a new configuration file config/amazones.conf with the following :
input {
stdin { }
}
output {
amazon_es {
hosts => ["ENDPOINT"]
region => "REGION"
aws_access_key_id => "AWS_ACCESS_KEY_ID"
aws_secret_access_key => "AWS_SECRET_ACCESS_KEY"
}
}Replace the right things using the following table :
| Variable | Description |
|---|---|
ENDPOINT |
The Endpoint you got above in your ES Dashboard, without https:// |
REGION |
The region where is hosted your Amazon ES (eu-west-1 for example) |
AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY |
You'll find them around Security Credentials in your Amazon Account |
It should look like this :
input {
stdin { }
}
output {
amazon_es {
hosts => ["some-name.eu-west-1.es.amazonaws.com"]
region => "eu-west-1"
aws_access_key_id => "XXXXXXXXXXXXXXXXXXXX"
aws_secret_access_key => "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
}Then launch Logstash with the new configuration file :
./bin/logstash -f config/amazones.conf
Type something like "hello" and press enter. You won't have output in Logstash this time cause we're not saying anymore to Logstash to output on stdout, but on Amazon ES.
Now go and your Amazon ES Kibana, you'll find the link in your Amazon ES Dashboard. It's also your Elasticsearch endpoint following by /_plugin/kibana/.
On the left, go in "Management" and click "Index Patterns".
Note that if you didn't send any data to the Elasticsearch endpoint through Logstash, you won't be able to define the index pattern.
Define the following index pattern :
logstash-*
Then click "Next step"
Select @timestamp as "Time Filter field name"
Then click "Create index pattern"
When the index pattern is created, you can finally check in the Kibana "Discover" panel that your message sent through Logstash is displayed.
./bin/logstash-plugin install logstash-input-syslog
Copy the Amazon ES configuration file to a new one named herokulogs-to-amazones.conf
cp config/amazones.conf config/herokulogs-to-amazones.conf
And replace
input {
stdin { }
}
by
input {
syslog { }
}
Then Launch Logstash with the new configuration file :
./bin/logstash -f config/herokulogs-to-amazones.conf
We finally need to tell Heroku where to forward your Heroku logs, in our case it will be an external syslog server (through Logstash).
If you don't provide a specific port to the syslog input in the Logstash configuration file, it will be 514 by default.
Your EC2 instance will be hosting a syslog server on syslog://YOUR_EC2_PUBLIC_DNS_OR_IP:514
Replace YOUR_EC2_PUBLIC_DNS_OR_IP by your EC2 instance's Public DNS or Public IP which can be easily found in your EC2 Dashboard.
You can now using the Heroku CLI to add a new Log Drain :
heroku drains:add -a YOUR_APP syslog://YOUR_EC2_PUBLIC_DNS_OR_IP:514
Replacing the right things using the following table :
| Variable | Description |
|---|---|
YOUR_APP |
Name of your Heroku app |
YOUR_EC2_PUBLIC_DNS_OR_IP |
Your EC2 instance's Public DNS or Public IP which can be easily found in your EC2 Dashboard |
We setuped a new Log drain for Heroku, but it won't be able to send the logs right now, cause our EC2 instance is only accessible by SSH using the port 22. We need to add a new port rule on the associated Security Group to reach the final behavior we want.
To do this, go on your EC2 Dashboard, select your instance and in the "Description" tab, look for "Security groups". As we didn't setup a security group when we created the EC2 instance, it should be named like "launch-wizard-x".
Click on it, you'll be redirected on the "Security Groups" page. Select the good one and in the available "Actions", click "Edit inbound rules".
A popup should show up, add a new rule with the following fields :
Type : Custom TCP Rule
Port Range : 514
Source : Anywhere
Then save, you can know see on Kibana that the Heroku logs are received.