Elastic Beat Development 101

elastic-beat-development-101.md

Elastic Beats Development 101

This is a short guide to get up and building Elastic Beats on a new Linux host.

Start a VM

This uses Google Compute Engine (GCE) to start an Ubuntu 20.04 virtual machine. You can use other versions of Linux or different virtualization platforms (or no virtualization), but those are not guaranteed to work with the commands here.

 gcloud auth login
 
 gcloud config set project elastic-beats
 
 gcloud compute instances create $USER-beats-dev \
  --image-project=ubuntu-os-cloud \
  --image-family ubuntu-2004-lts \
  --zone us-central1-a \
  --machine-type=n1-standard-1 \
  --boot-disk-size=100GB
  
 gcloud compute ssh --zone=us-central1-a $USER-beats-dev

NOTE: To clean up this resource afterwards use gcloud compute instances delete --zone us-central1-a $USER-beats-dev.

Install Deps

sudo apt-get update
sudo apt-get install -y vim docker.io make gcc python3-pip python3-venv git libpcap-dev libsystemd-dev librpm-dev

sudo curl -L "https://github.com/andrewkroh/gvm/releases/download/v0.4.1/gvm-linux-amd64" -o /usr/local/bin/gvm
sudo chmod +x /usr/local/bin/gvm

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

sudo usermod -a -G docker $USER

Clone Beats

mkdir -p ~/go/src/github.com/elastic
git clone https://github.com/elastic/beats.git ~/go/src/github.com/elastic/beats

Update .bash_aliases

This configures your bash shell to setup your Go development envionrment on login.

echo 'eval "$(gvm $(cat ~/go/src/github.com/elastic/beats/.go-version))"' >> ~/.bash_aliases
echo 'export PATH=$(go env GOPATH)/bin:${PATH}' >> ~/.bash_aliases
source ~/.bash_aliases

Install Mage

cd ~/go/src/github.com/elastic/beats
make mage

Listing Build Targets and Getting Usage Details

The main build tool is Mage. You can use mage -l to list the available build targets. And mage -h <target> to get detailed usage information about a target.

cd filebeat # or any Beat
mage -l
mage -h exportDashboard

Update Generated Content

mage update

Build

You can use mage -v <targets> to enable verbose output from Mage.

mage build

Unit Test

mage unitTest

Integration Test

mage integTest

Executing Multiple Targets

If you are going to execute multiple targets it's more efficient to run them in a single command.

mage update build integTest

Packaging

mage package

You can control the target platforms with the PLATFORMS environment variable. Use SNAPSHOT=true to build a snapshot. And use VERSION_QUALIFIER=rc1 to add a qualifier to version. Some examples:

PLATFORMS='linux/amd64' SNAPSHOT=true mage package

PLATFORMS='linux/armv7' mage package

PLATFORMS='windows darwin' mage package

Packages are written to build/distributions.

Hello, this is an excellent tutorial.

When running this per the instructions, I received only filebeat-oss-8.2.0-SNAPSHOT-amd64.X file outputs and the process ended with the following output.

Status: Image is up to date for docker.elastic.co/beats-dev/golang-crossbuild:1.17.6-main-debian8
>> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64]
>> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64]
/tmp/ccy6qz4n.o: In function `main':
god.c:(.text+0x2a6): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
god.c:(.text+0x23e): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking




W: Failed to fetch http://archive.debian.org/debian/dists/jessie/InRelease

W: Failed to fetch http://archive.debian.org/debian/dists/jessie/Release.gpg  Unable to connect to archive.debian.org:http: [IP: 193.62.202.28 80]

W: Some index files failed to download. They have been ignored, or old ones used instead.
>> package: Building filebeat-oss type=docker for platform=linux/amd64
>> package: Building filebeat-oss type=tar.gz for platform=linux/amd64
>> package: Building filebeat-oss type=deb for platform=linux/amd64

and

useradd: UID 0 is not unique
>> package: Building filebeat-oss type=rpm for platform=linux/amd64
useradd: UID 0 is not unique
The command '/bin/sh -c for iter in {1..10}; do         apt-get update -y &&         DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes ca-certificates curl gawk libcap2-bin xz-utils &&         apt-get clean all &&         exit_code=0 && break || exit_code=$? && echo "apt-get error: retry $iter in 10s" && sleep 10;     done;     (exit $exit_code)' returned a non-zero code: 100
>> Building docker images again (after 10 s)
The command '/bin/sh -c for iter in {1..10}; do         apt-get update -y &&         DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes ca-certificates curl gawk libcap2-bin xz-utils &&         apt-get clean all &&         exit_code=0 && break || exit_code=$? && echo "apt-get error: retry $iter in 10s" && sleep 10;     done;     (exit $exit_code)' returned a non-zero code: 100
>> Building docker images again (after 10 s)
The command '/bin/sh -c for iter in {1..10}; do         apt-get update -y &&         DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes ca-certificates curl gawk libcap2-bin xz-utils &&         apt-get clean all &&         exit_code=0 && break || exit_code=$? && echo "apt-get error: retry $iter in 10s" && sleep 10;     done;     (exit $exit_code)' returned a non-zero code: 100
>> Building docker images again (after 10 s)
The command '/bin/sh -c for iter in {1..10}; do         apt-get update -y &&         DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes ca-certificates curl gawk libcap2-bin xz-utils &&         apt-get clean all &&         exit_code=0 && break || exit_code=$? && echo "apt-get error: retry $iter in 10s" && sleep 10;     done;     (exit $exit_code)' returned a non-zero code: 100
package ran for 17m10.589291866s
Error: failed building filebeat-oss type=docker for platform=linux/amd64: failed to build docker: running "docker build -t docker.elastic.co/beats/filebeat-oss:8.2.0-SNAPSHOT build/package/filebeat-oss/filebeat-oss-linux-amd64.docker/docker-build" failed with exit code 100

Questions:

1. How can we generate non-oss packages for use with regular beats?
2. Is there an environment variable that would prevent the generation of oss packages to speed up the process?
3. How do we get past the build errors above?
   Thank you - Jared

1. How can we generate non-oss packages for use with regular beats?

Build from beats/x-pack/filebeat.

2. Is there an environment variable that would prevent the generation of oss packages to speed up the process?

No, but if you build from the x-pack/* dirs you will produce Elastic licensed packages.

3. How do we get past the build errors above?

Those look like connectivity issues with the apt repo ("Unable to connect to archive.debian.org").