Document how to run ingest pipelines tests in Beats

README.md

I couldn't find how to run ingest pipeline tests in Beats, so I'm writing this Gist to capture what I know so far.

Assumption

• Using ubuntu 20.04
• Installed Golang 1.18+
• Installed mage https://github.com/magefile/mage/

Steps

Checking the Linux version:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal

# install mage
mkdir -p ~/bin
cd ~/bin
wget https://github.com/magefile/mage/releases/download/v1.14.0/mage_1.14.0_Linux-64bit.tar.gz
tar zxvf mage_1.14.0_Linux-64bit.tar.gz
export PATH=$PATH:~/bin

# install go
sudo apt install golang
go version


# docker
# https://docs.docker.com/engine/install/ubuntu/
sudo apt-get update
sudo apt-get install \
	ca-certificates \
	curl \
	gnupg \
	lsb-release

sudo mkdir -p /etc/apt/keyrings
	curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \
	  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
	  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
	  
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-compose


# https://docs.docker.com/engine/install/linux-postinstall/
sudo groupadd docker || echo
sudo usermod -aG docker $USER
# Log out and log back in so that your group membership is re-evaluated.

	
# clone 
mkdir -p ~/code/projects/elastic
cd ~/code/projects/elastic

git clone https://github.com/elastic/beats.git
cd beats/x-pack/filebeat/

# python
sudo apt install python3-pip

# 
mage pythonVirtualEnv

# run tests for activitylogs
TESTING_FILEBEAT_MODULES=azure TESTING_FILEBEAT_FILESETS=activitylogs MODULES_PATH=module PYTEST_ADDOPTS="-k test_xpack_modules.py"  mage -v pythonIntegTest

I am using Filebeat as an example.

First, walk into the Filebeat root directory:

cd x-pack/filebeat/

In general you can use the following command:

TESTING_FILEBEAT_MODULES=<module> \
TESTING_FILEBEAT_FILESETS=<fileset> \
MODULES_PATH=module \
mage -v pythonIntegTest

For example, here's the command line to test the activitylogs fileset in the azure module:

TESTING_FILEBEAT_MODULES=azure \
TESTING_FILEBEAT_FILESETS=activitylogs \
MODULES_PATH=module \
mage -v pythonIntegTest

Additional environment variables:

# To rebuild the expected logs.
GENERATE=1

# To skip everything but the modules test.
PYTEST_ADDOPTS="-k test_modules"
PYTEST_ADDOPTS="-k test_modules -vv"