cleesmith/Security Onion installation

## Security Onion installation
Security Onion
Aug 2014:

... my purpose for installing this was to:
    - learn more about security stuff
    - steal the packet captures (pcap) provided so I can replay them using tcpreplay for snort testing,
       as it's not so sexy to just test using ICMP ping data or local rules that match anything

see:
http://blog.securityonion.net/
https://www.youtube.com/channel/UCEt-VQX8pxvDPuB7fip5b0g/videos

the following is from:
https://github.com/Security-Onion-Solutions/security-onion/wiki/IntroductionWalkthrough
... the above refers to installing SO 12.04 on a VirtualBox VM, but new installation
    guides for Xubuntu 14.04 64-bit no longer refer to VirtualBox -- see:
(1) https://github.com/Security-Onion-Solutions/security-onion/wiki/Installation
(2) Download our Security Onion ISO image and Quickly Evaluate
    https://github.com/Security-Onion-Solutions/security-onion/wiki/QuickISOImage
(3) Post Installation page:
    https://github.com/Security-Onion-Solutions/security-onion/wiki/PostInstallation


______________________________________________________________________________________
Security Onion is configured to run on version 12.04 of any Ubuntu-based Linux server or
desktop distribution, such as Ubuntu, Lubuntu, Xubuntu, and Kubuntu. Your base operating
system choice really depends on personal preference, your hardware and how you intend to
interact with Security Onion. If you're experienced with the flavors of Ubuntu you
probably have already made this decision. We're going to walkthrough setting up the Security
Onion Live Xubuntu 12.04 distribution in a virtual machine (VM) and installing Security Onion
using the Quick Setup option.
Having Security Onion installed in a VM gives you an isolated environment which can act as a
"client" for interacting with a remote Security Onion server. In an Ubuntu Server deployment,
where access to the server is limited to SSH and command line, the client VM will let us setup
remote servers and sensors graphically. It is also recommended for analysts to run Security
Onion in a virtual machine for client access to ensure you have all the tools needed to manage
and monitor a deployment in an isolated environment.
You'll need a computer with at least 4GB of RAM (ideally 8GB) for
best results. We'll use VirtualBox, a free desktop virtualization tool, but the process is very
similar for VMware or others.
You can download a copy of VirtualBox for Windows, Mac OS X or Linux at
 http://www.virtualbox.org.
We'll also need to download the Security Onion 12.04 Live distribution from
https://sourceforge.net/projects/security-onion/files/12.04/.

Once downloaded, install VirtualBox, then launch it and click the "New" button.
First we'll provide a name for our virtual machine ("Security Onion Standalone Client"
for example) and specify
the type ("Linux") and version ("Ubuntu" or "Ubuntu 64 bit"), then click "Continue."
We'll next define how much memory we want to make available to our virtual machine.
For a Security Onion client, you should dedicate at least 2GB RAM, but ideally 4GB if possible.
2GB RAM will provide decent performance for the Sguil client and retrieving packet
captures from the server but also enough to run Security Onion in standalone mode
for monitoring the local client and testing packet captures with tools like tcpreplay,
but you could potentially experience some delays underheavy use or retrieving
large packet captures.

Next we'll create a virtual hard drive for our client.
Specify "Create a virtual hard drive now" then click "Create" to choose the
hard drive file type
"VDI (VirtualBox Disk Image)" and "Continue."
For storage, we have the options of "Dynamically allocated" or "Fixed size."
For a client virtual machine, "Dynamically allocated" is the best choice as it
will grow the hard disk up to whatever we define as the maximum size on an as
needed basis until full, at
which point Security Onion's disk cleanup routines will work to keep disk space available.
If you happen to be running a dedicated sensor in a virtual machine, I would
suggest using "Fixed size,"
which will allocate all of the disk space you define up front and save you some disk
performance early on.
Once you've settled on the storage allocation, click "Continue" and provide a name from
your hard disk image file and specify the location where you want the disk file to be
created if other than the default location.
For disk size, you'll want enough disk capacity for retrieving/testing packet captures and
downloading system updates.
At a minimum for a client, I would designate at least 20GB, with 40GB recommended.
Click "Create" and your Security Onion client virtual machine will be created.

______________________________________________________________________________________
At this point, you can click "Settings" for your new virtual machine so we can get it configured.
You might want to increase your display virtual memory to 128MB of RAM, but most other settings
should be fine.
We do, however, need to do a couple of things. First, mount the Security Onion Live 12.04 .iso file
we downloaded so our virtual machine can boot from it to install Linux.

ISO install:
Click the "Storage" icon, then under "Controller: IDE" select the "Empty" CD icon.
To the right, you'll see "CD/DVD Drive" with "IDE Secondary" specified with another CD icon.
Click the icon, then select "Choose a virtual CD/DVD disk file" and browse to where you
downloaded the Security Onion Live 12.04 .iso file, select it then choose "Open."

______________________________________________________________________________________
Next click "Network"
- click "Adapter 1" and select "Bridged Adapter" ... I added this, but it's probably the default anyways
- click "Adapter 2."
You'll need to click the checkbox to enable it then attach it to "Internal Network."
Under the "Advanced" options, set "Promiscuous Mode" to "Allow All."
Click "Ok" and we are ready to install the operating system.
______________________________________________________________________________________

Hit the "Start" button with your new virtual machine selected and after a few seconds
the boot menu will load.
Choose "live - boot the Live System" or wait and the Security Onion desktop will load.
Double-click the "Install SecurityOnion 12.04" icon on the desktop to initiate the first of a handful
of operating system setup screens.
After language selection we'll see some information about our system in
"Preparing to install SecurityOnion."
Go ahead and check the box to "Download updates while installing" if you have an active
network connection, then
click "Continue."
On the "Installation type" screen we want to "Erase disk and install SecurityOnion"
which will partition and format the
virtual disk we created. Click "Continue" again then "Install Now."
After a moment, you'll be prompted to specify a time zone, which isn't important for
us as Security Onion will default
the system to UTC time, so you can just click "Continue" to move on to
"Keyboard layout" selection where the defaults are usually sufficient.
"Continue" again to create a user account for accessing the system and set the virtual
machine's hostname.
The username and password defined here will be the only account that can access the system
once the installation is complete until you add additional accounts.
Click "Continue" and after a few minutes the installation will complete and you'll
be prompted to "Restart Now."

There are two things we need to do once our virtual machine boots up on it's own for the
first time to prepare for Security Onion setup:
install all Ubuntu/SO updates and install VirtualBox Guest Additions
(or VMware Tools if you're using VMware).
Go ahead and login with the username and password you created during the install and
double-click the "Terminal Emulator" icon on the desktop.
Now that we have a command prompt, type:

sudo apt-get update -y
sudo apt-get dist-upgrade ... takes a long time :(
sudo reboot

You'll be asked to enter your password since we're executing a "sudo" command.
By default, "root" access is disabled in Ubuntu.
The "sudo" command allows authorized users, like system administrators, to execute commands
that would otherwise require root access without having to know the root password.
It is similar to how recent versions of Microsoft Windows may prompt you to confirm administrative
actions, when installing new applications for example.
The command we execute tells Linux: update the package and repository listings with root
access without prompting me to tell you yes & again with root access upgrade this Linux
distribution with the latest patches and again with root access, reboot.
Patching will take a few minutes on a fresh installation and when it's done the system will restart.
When the system comes back up, log in again and open another terminal window.

Out-of-date, so don't do the following: ______________________________________________________
At the top of your virtual machine window you'll notice menu items for VirtualBox.
Click on your virtual machine window, then on the menu click "Devices"
then "Install Guest Additions..."
Doing so will mount the VirtualBox guest additions CD on your virtual machine and
 it will open the folder showing
you the files now available.
Click on your terminal window and type "cd /media/VBOX" then hit the
<tab>
key to autofill the folder name and
<enter>
to change to that directory.

To install the Guest Additions type:
sudo ./VBoxLinuxAdditions.run
You'll again be prompted for your password since we're running sudo for the first time
after a reboot.
The installation will launch and after a couple minutes you'll return to the command prompt
when it's complete.
In the upper right hand corner of your Xubuntu desktop, click your username then "Shut down"
to shut down the system.
______________________________________________________________________________________

Before we install Security Onion, this is an excellent time to take a snapshot of your virtual machine.
While the system is shutdown, you'll notice two icons on the top right in VirtualBox Manager when
you select your virtual machine:
Details and Snapshots.
Click "Snapshots" then click the camera icon and give your snapshot a name and description.
I recommend something descriptive here, perhaps naming it "New Build SO Client" with a
description including details that the system was patched and updated with VirtualBox Guest
Additions installed and provide the date.
Once we have a snapshot, we'll be able to make changes to the system and revert those changes
back to the state we are preserving.
For a Security Onion client this is useful, as we can setup Security Onion as a standalone
server for testing, then later revert to the snapshot and reinstall Security Onion to only
use the client tools.
Boot up the system again once you've completed the snapshot and we'll install Security Onion.

______________________________________________________________________________________
At this point, without running the Security Onion setup script, you have a fully functioning Security
Onion client workstation environment with which to access a Security Onion server.  Next we're going
to install Security Onion using the Quick Setup to familiarize ourselves with the setup and get started
learning the tools. Once we're done experimenting we can revert our VM to the snapshot we just
took at be back to a clean Security Onion client only state.
* note: snort and other stuff is not installed at this point!
______________________________________________________________________________________

When you're logged in again, double-click the "Setup" icon on the desktop to begin Security Onion
setup.
You'll first be prompted for your password then asked to confirm that you want to continue with
the installation.
Choose "Yes, Continue!" and you'll be asked if you would like to configure
/etc/network/interfaces now.
You'll first be asked "Which network interface should be the management interface?"
Choose "eth0" and you'll then be asked whether to use static or DHCP IP addressing, with
static being highly recommended.
When installing a production server or sensor you should make sure to use static IP addressing,
but for our client VM we can use DHCP if static addressing isn't available or the VM is going to
change networks. Just be aware that IP address changes can cause issues with some of the
Security Onion agents, so if you have a test VM using DHCP, you may need to re-run the
Security Onion setup.
If you can assign a static IP, choose static and you'll be asked to provide the IP address,
gateway, netmask, broadcast address and DNS servers. You'll then be asked to configure
the monitor interface.
Choose "Yes, configure monitor interfaces" then choose "eth1" and click OK.
Once you've made your network selections, click "Yes, make changes and reboot!"

______________________________________________________________________________________
Log back in and double-click the "Setup" icon again.
Security Onion setup will detect that we've already configured the network interfaces, so choose
"Yes, skip network configuration!" when prompted.
You'll next be asked whether you want to install Security Onion using "Advanced Setup" or
"Quick Setup."
For this purpose, we'll use "Quick Setup" which will automatically configure most of your system
to monitor one network interface.
You'll first be asked "Which network interface should snort listen on?" Choose "eth1."
You'll then be prompted for usernames and passwords for Sguil, Squert, ELSA and Snorby and
whether or not you want to enable ELSA and that's it.
"Advanced Setup" lets you specify whether the Security Onion instance will be running as a Server,
Sensor or in Standalone mode, which IDS engine you'd prefer (Snort or Suricata), how many
CPU cores you want to assign to Snort/Suricata and Bro IDS, which Snort/Suricata rule set you
want to use, your username/passwords and whether you want to install ELSA.

When you're done,
click "Yes, proceed with the changes!"
and Security Onion Setup will do the following:
Set the OS timezone to UTC;
Delete any existing NSM data/configuration;
Create a Sguil server and user;
Create a Snorby user;
Configure Snort and Bro to monitor interface eth0;
Run a single IDS process per interface;
Run a single Bro process per interface; and
Configure ELSA as both a Log Node and Web Node.
It will take a minute for the setup to complete, but trust me it's a lot less time than it would
take if you didn't have Security Onion guiding the way.
When setup is finished you'll see several informative prompts providing you valuable information
about your new Security Onion host, which we'll cover here in more detail as a future reference.
______________________________________________________________________________________

"Setup log can be found here: /var/log/nsm/sosetup.log"
- sosetup.log documents changes to the system and details about what occurred during the setup
process.
If you have any suspicions that some part of the setup failed, this should be your first stop.

"You may view IDS alerts using Sguil, Squert, Snorby or ELSA (if enabled)"
- Sguil is a client application while Squert, Snorby and ELSA are web interfaces.

"Bro logs can be found in ELSA (if enabled) and the following location:
/nsm/bro/"
- Bro logs are rotated on an hourly basis and provide a wealth of information about the network
being monitored, which can be invaluable during an investigation.

"Rules downloaded by PulledPork are stored in /etc/nsm/rules/downloaded.rules"
- PulledPork is a tool to update and manage Snort IDS signatures.
All rules that PulledPork downloads from Snort or Emerging Threats are stored in the
downloaded.rules file.
"Local rules can be added to: /etc/nsm/rules/local.rules"
- Local rules allow you to deploy custom signatures that will not be overwritten when PulledPork's
scheduled updates occur.
"You can have PulledPork modify the downloaded rules by modifying the files in:
/etc/nsm/pulledpork/"
- There are a number of configuration file options for PulledPork:
pulledpork.conf
enablesid.conf
disablesid.conf
dropsid.conf
modifysid.conf
"Rules will be updated every day at 7:01 AM UTC.
You can manually update them by running: /usr/bin/rule-update"
- The rule-update script allows you to manually run PulledPork to update signatures,
which is most useful when tuning signatures by modifying the /etc/nsm/pulledpork .conf files.

"Sensors can be tuned by modifying the files in:
/etc/nsm/HOSTNAME-INTERFACE/"
- There are multiple configuration files in this path:
sensor.conf - contains a number of variables that are used throughout the Security Onion network
monitoring services, such as which interfaces are being monitored, paths to config files, and more.
Typically you'll only need to modify the following files if you are monitoring IP address ranges
other than private RFC1918 address space (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12).
snort.conf - configuration file used to define variables and configuration settings specific to Snort.
suricata.yaml - configuration file used to define variables and configuration settings specific to Suricata.
sancp.conf - configuration file used to define variables and configuration settings specific to SANCP.
prads.conf - configuration file used to define variables and configuration settings specific to PRADS.

"If you have any questions or problems, please visit our website where you can find the following links:
FAQ, Wiki, Mailing Lists, IRC channel and more!" - If you need help or assistance, it's not far away.
The Security Onion Mailing List is the most efficient means of getting community support or help if you
experience problems or have questions.
Security Onion - http://blog.securityonion.net/
Security Onion Google Code Project - http://code.google.com/p/security-onion/
FAQ - http://code.google.com/p/security-onion/wiki/FAQ
Wiki - http://code.google.com/p/security-onion/w/list
Mailing Lists - https://groups.google.com/forum/?fromgroups#!forum/security-onion
IRC channel - http://webchat.freenode.net/?channels=securityonion
______________________________________________________________________________________

At this point we have a fully functioning Security Onion standalone system up and running ... at last :)
To confirm, open up a terminal and type:
ps aux | grep -i snort
... or:
sudo nsm_sensor_ps-status
and you should see results showing the various agent and service components of Security Onion
with an "OK" or "FAIL" status.
You can also type:
sudo nsm_server_ps-status
to check the status of Sguil server. If any of the agents or server fail, a reference to the log
file will be included that will be useful for troubleshooting.

A useful script to provide performance and health status of your Security Onion hosts is "sostat."
You should run it periodically on any deployment with the command:
sudo sostat | less
to review and monitor all aspects of Security Onion. It includes nsm_server|sensor_ps-status results,
network interface status, disk usage, network sockets, IDS rule update status, CPU usage,
log archive size, IDS engine packet drops, pf_ring stats, Sguil uncategorized events and summaries,
top 50 URLs for previous day, and Snorby events and summaries.
It provides powerful visibility into the health of Security Onion and should be adopted as part of your
monitoring routine.
______________________________________________________________________________________

If everything looks ok, we can quickly test Sguil and Snort/Suricata detections.
Double-click the Sguil icon on the desktop and enter your Sguil username and password
(created during the Security Onion Setup).
You'll be prompted to choose which network(s) to monitor:
the monitored network interface(s) and/or OSSEC events.
Choose "Select All" then "Start SGUIL" and the Sguil client will load.
You might already have some events showing up, but just to confirm type:
curl http://testmyids.com
in a terminal window and you should see an event appear in Sguil for "GPL ATTACK_RESPONSE
id check returned root."

Security Onion includes a number of useful links on the desktop in addition to the Security Onion
application menu which provides access to man pages for tools included in Security Onion.
The "README" icon on the desktop is a good starting point and will open https://localhost in
a web browser with local links to Squert, Snorby, ELSA, and Xplico and external links to
additional useful Security Onion information.
Sguil, Squert and ELSA all share the same username/password, while Snorby uses
e-mail addresses for usernames.
______________________________________________________________________________________

Here's a brief description of the primary tools available in Security Onion for security monitoring:
Sguil (http://sguil.sourceforge.net/) - THE analyst console for security monitoring. There isn't a more
powerful and capable solution available for event analysis, correlation and review.

Squert (http://www.squertproject.org/) - A web interface to query and view Sguil event data that
was designed to supplement Sguil by providing additional context around events.

Snorby (https://snorby.org/) - A slick and effective Ruby on Rails app for monitoring Snort/Suricata
IDS events.

ELSA (http://code.google.com/p/enterprise-log-search-and-archive/) - Enterprise Log Search and
Archive is a distributed log archive system similar to Splunk and allows for event searching and
 visualization of all the log data Security Onion consumes, including OSSEC, Snort/Suricata,
and most importantly, Bro IDS.

If you want to take some time to experiment with the tools, Security Onion includes some sample
packet capture files we can replay to generate event data. I highly recommend physically disabling
the network connection from your host operating system prior to replaying the packets.
Since we're using a VM, simply unplugging the physical network connection or disabling wireless
on the host system where your VM is running will work. We will be replaying samples of live traffic,
some of which are malicious, so use caution.
The samples provided are available in /opt/samples/ and once we're disconnected from the network,
we can replay them by opening up a terminal and typing:
... at long last, the reason I installed this onion:
**************************************************
sudo tcpreplay -i eth1 -M10 /opt/samples/*.pcap
**************************************************
If you have the Sguil client up and running, you'll see events start to roll in to the console.
You now have a decent sample of data to play with.

______________________________________________________________________________________
* Note: at this point, I can finally copy snort's unified2 file to use with
            idstools + my indexer + Elasticsearch,
            and totally avoid using Barnyard2
            ... as it's very slow to read/join multiple tables from MySQL and index into Elasticsearch

... how to find those unified2 files:
sudo find / -name "*.unified2.*"
/nsm/sensor_data/onion-eth1/snort-1/snort.unified2.1408489449 (the suffix varies)
ls -la /nsm/sensor_data/onion-eth1/snort-1/
... copy them to your home:
cd ~
sudo cp /nsm/sensor_data/onion-eth1/snort-1/snort.unified2.*
sudo chown your_username:your_username snort.unified2.*
now, scp the unified2 files to wherever you need them
______________________________________________________________________________________

note: the following is a great demo, if you can get the right-click to work in Sguil, mine doesn't :(

Let's dig into one example to get a feeling for the capabilities at our disposal.
You'll notice several events with a source IP of 188.72.243.72 and a destination IP of 192.168.3.65 that
indicate a potentially suspicious executable file has been downloaded causing several Snort IDS ET
(Emerging Threats) signatures to fire: "ET INFO Packed Executable Download,"
"ET POLICY PE EXE or DLL Windows file download" and "ET INFO EXE IsDebuggerPresent
(Used in Malware Anti-Debugging)."
If we look at "ET INFO Packed Executable Download" we can see under the CNT column the number
of correlated events and if you right-click on the value you can open a new tab showing all the
correlated events.
Additionally, you can right-click on the source or destination IP and port columns to query for additional
IDS, SANCP or PADS events or perform DShield lookups. At this point, about all we know is a file
was downloaded.
We can click on the event entry for "ET INFO Packed Executable Download" then in the bottom right
corner of the Sguil client we can click the "Show Packet Data" and "Show Rule" checkboxes.
"Show Rule" will display the Snort/Suricata rule that created the event, while "Show Packet Data"
will load theTCP headers and packet data. Poking through the packet data, we really only get
confirmation of what we already know:
Content-Type: application/x-msdownload tells us it was an exe, but we don't know
what it was at this point.

A better option with Sguil is to right-click on the Alert ID column for the event and choose "Transcript."
The transcript reaches out to the full packet captured events stored in /nsm and rebuilds a transcript
of the session from the archived packet captures. Now we have context.
Just from the top header information we almost immediately know the domain name
(hosted-by.leaseweb.com) and the fingerprinted OS of our host involved (Windows XP SP1+, 2000 SP3).
Following the header, we see the color coded conversation between the source (SRC) and
destination (DST) IPs involved.
First we see our SRC make a HTTP POST request to ishi-bati . com/kartos/youyou.php followed
by the server’s response.
Then our SRC issues a GET to ishi-bati . com/kartos/krt.exe.
The DST then replies with file content-type that is application/x-msdownload,
indicating a DOS executable.
(the string "This program cannot be run in DOS mode" is also a giveaway.)

In a matter of seconds, we know a client downloaded krt.exe from a suspect domain.
Want to know more about krt.exe?
Right-click on the "ET INFO Packed Executable Download" event Alert ID field again, but this
time choose "Network Miner."  Sguil will load the packet captures we just reviewed in Network
Miner, which is useful for getting details on the hosts  involved and, in this case, for it's ability to
extract files and certificates from packet captures.
When the packet capture loads, you'll notice a "Files" tab.
You might have to expand the window and column names a bit.
In the "Filename" column you'll see "krt.exe.x-msdownload."
Right-click the file and choose "Open folder" and you now have the file that triggered the alert.
From here you can scan a copy of the file with whatever endpoint security product you use to
see if it might have been detected or submit the file or check it's MD5 hash with VirusTotal to
identify what security vendors might be detecting it. You could also use a CIF
(Collective Intelligence Framework) server to check the MD5 against known threats based
on community intel.  Or you might look into Cuckoo Sandbox or Razorback
as an option for local automated analysis.

The choice is yours but the options are available. Lack of financial support for network security
monitoring is no longer excusable in information security. The tools are there, and in most cases are
easier to support and maintain than vendor-based solutions, and the capabilities rival or surpass
best-of-breed commercial solutions. If you care about security, take what I've written, practice it and
learn the process, then show someone else. There's a community working very hard to make this type
of monitoring possible for such a small expense as time and hardware on your part.   But there's also
a community out there that needs your help and needs you to take these tools that have been
harmoniously woven together with a most complex thread and learn how they work.
Then learn to use them and teach others.
I call it "crash, burn and learn" but it's what it's all about.
Try hard to learn and improve and advance and share what Doug Burks and the Security Onion
community is offering.
You'll fail, but you will also have a dedicated and faithful group of people committed to
Security Onion helping you succeed.
__________________________________fin_________________________________________________