Splunk Tutorial: Install & Configure Universal Forwarders

splunk-Universal-Forwarders.txt

Download File URL - https://download.splunk.com/products/splunk/releases/9.0.1/linux/splunk-9.0.1-82c987350fde-Linux-x86_64.tgz

Splunk Linux Tar file - wget -O splunk-9.0.1-82c987350fde-Linux-x86_64.tgz "https://download.splunk.com/products/splunk/releases/9.0.1/linux/splunk-9.0.1-82c987350fde-Linux-x86_64.tgz"

Splunk Linux rpm file - wget -O splunk-9.0.1-82c987350fde-linux-2.6-x86_64.rpm "https://download.splunk.com/products/splunk/releases/9.0.1/linux/splunk-9.0.1-82c987350fde-linux-2.6-x86_64.rpm"

Splunk Linux Debian file - wget -O splunk-9.0.1-82c987350fde-linux-2.6-amd64.deb "https://download.splunk.com/products/splunk/releases/9.0.1/linux/splunk-9.0.1-82c987350fde-linux-2.6-amd64.deb"

Splunk Linux Windows file - wget -O splunk-9.0.1-82c987350fde-x64-release.msi "https://download.splunk.com/products/splunk/releases/9.0.1/windows/splunk-9.0.1-82c987350fde-x64-release.msi"

Splunk Universal Forwarder

MSI file
$ wget -O splunkforwarder-9.0.1-82c987350fde-x64-release.msi "https://download.splunk.com/products/universalforwarder/releases/9.0.1/windows/splunkforwarder-9.0.1-82c987350fde-x64-release.msi"

Linux tar file
$ wget -O splunkforwarder-9.0.1-82c987350fde-Linux-x86_64.tgz "https://download.splunk.com/products/universalforwarder/releases/9.0.1/linux/splunkforwarder-9.0.1-82c987350fde-Linux-x86_64.tgz"



PORTS SPECIFICATIOON OF SPLUNK SERER
-----------------------------------------

  Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open


Setting up a Forwarders

Pre-Requisite to add forwarders
----------------------------------	
1. Settings => Monitoring console => Setting => Forwarder Monitoring Setup => Forwarder Monitoring (ENABLE with 15 mins)
2. Settings => Forwarding and Recieving => Receive data => Add New ==> Listen on this port (For example, 9997 will receive data on TCP port 9997)
3. Restart a Splunk Instance
Settings => Server Controls => Restart Splunk

# Make sure in Fireall (Port should be enabled or Firewall

# Make sure in Fireall (Port should be enabled or Firewall
==============================================

$ cd /opt/
$ wget wget -O splunkforwarder-7.2.5-088f49762779-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.2.5&product=universalforwarder&filename=splunkforwarder-7.2.5-088f49762779-Linux-x86_64.tgz&wget=true'

$ tar -zxvf splunkforwarder-7.2.5-088f49762779-Linux-x86_64.tgz
$ cd /opt/splunk-dir/bin/ 
3. cd  /opt/splunkforwarder/bin	
	./splunk start
	./splunk stop
	./splunk restart
	./splunk help
4. sudo ./splunk start --accept-license 
5. sudo ./splunk enable boot-start
 
==============HOSTED==============
sudo ./splunk add forward-server X.X.X.X:9997
OR sudo ./splunk add forward-server X.X.X.X:9997 -auth admin:goodpass
sudo ./splunk list forward-server
sudo ./splunk list monitor 
sudo ./splunk add monitor /var/log 
sudo ./splunk list monitor 
sudo ./splunk list forward-server
sudo ./splunk restart
sudo ./splunk list forward-server
C:\Program Files\SplunkUniversalForwarder\etc\system\local
Filename - outputs.conf

==============CLOUD==============
6. sudo ./splunk edit user admin -password goodpassword -role admin -auth admin:changeme
7. sudo ./splunk install app /opt/splunkforwarder/splunkclouduf.spl -auth admin:goodpassword
8. sudo ./splunk restart
9. sudo ./splunk add monitor -auth admin:goodpassword /opt/log/www1

========== In Splunk Web Servers=============================	
1. Go to "Settings" and click on "Monitoring console"
2. On the Second Top Menu --
		Click on the "Forwarders Deployment" and Visualize
		Click on the "Forwarders Instance" and Visualize
3. Run a Search

=====================Forwarders Troubleshooting========================
========== In Forwarders Servers=============================
bin> sudo ./splunk list forward-server
bin> netstat -a | grep 9997

========== In Splunk Web Servers=============================
> netstat -a

# Check firewall setting as well
# Add execption forwarders ports
# Make sure in Fireall (Port should be enabled or Firewall
		
	
=================================================
sudo ./splunk start --accept-license
=================================================
# On branch master
# Changes not staged for commit:
#   (use "git add/rm <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       deleted:    ../ftr
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       ../etc/apps/learned/local/
#       ../etc/apps/learned/metadata/local.meta
#       ../etc/auth/ca.pem
#       ../etc/auth/ca.srl
#       ../etc/auth/cacert.pem
#       ../etc/auth/server.pem
#       ../etc/auth/splunk.secret
#       ../etc/instance.cfg
#       ../etc/myinstall/splunkd.xml
#       ../etc/passwd
#       ../etc/splunk-launch.conf
#       ../etc/system/local/inputs.conf
#       ../etc/system/local/server.conf
#       ../etc/system/metadata/local.meta
#       ../var/

=================================================
sudo ./splunk add forward-server 13.234.32.244:9997 -auth admin:admin123
=================================================
[root@ip-172-31-19-160 bin]# sudo ./splunk add forward-server 13.234.32.244:9997 -auth admin:admin123
Added forwarding to: 13.234.32.244:9997.
[root@ip-172-31-19-160 bin]# git status
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   ../etc/system/metadata/local.meta
#       modified:   ../var/log/splunk/audit.log
#       modified:   ../var/log/splunk/health.log
#       modified:   ../var/log/splunk/metrics.log
#       modified:   ../var/log/splunk/splunkd.log
#       modified:   ../var/log/splunk/splunkd_access.log
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       ../etc/login-info.cfg
#       ../etc/system/local/outputs.conf
no changes added to commit (use "git add" and/or "git commit -a")
----------------------------------------
[root@ip-172-31-19-160 bin]# more ../etc/system/local/outputs.conf
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 13.234.32.244:9997

[tcpout-server://13.234.32.244:9997]