Any VMs have public ip numbers and registered domain names. Therefore these machines are accessible to the internet and vulternable to attack. Using available Azure VM images has its advantages but a verification of security measures is required to ensure that the VMs are not immediately open to intrusion and do not become compromised over time.
Ideally a vm should be created from distro sources and provisioned and uploaded to Azure accordingly. The following outlines the process to build a CentOS 7.x minimal installation and load it to Azure : Prepare a CentOS-based virtual machine for Azure, 05/09/2016. The current Swarm Nodes were built on top of publically available CentOS vm images. These should likely be replaced once the integration of all the moving parts is complete and the setup and provisioning of this configuration is scripted and automated. The current vms are based on "CentOS-based 7.2" by OpenLogic and available on Azure when creating a compute node.
If an image is built from source distro then only the required services and applications need be installed making the image both smaller and easier to manage and control. Public images may have many additional features installed that require additional administrative consideration as well as footprint.
At the very least the following security measures should be taken for any public facing linux system, but especially if you are using one from the open vm registry available on Azure
- Change the root password - the root password could be well known as it is a public image
- Obfuscate the root password and allow only administrative users access to root functionality using sudo to prevent root login
- Install, configure and enable DenyHosts to prevent brute force SSH Dictionary attacks
- Install, configure and enable firwalld daemon
- Update all installed packages and run yum.cron so that security patches and fixes are installed regularily as these are released.
- Consider other recommended options below
In order to force certificate authenitcation to the vms, steps must be performed on both the client machines as well as the servers. For more information check here
Each User Needs to Generate an SSH key-pair on their Client Machine (if one doesn't already exist or existing ones can be destroyed inadvertantly!)
For password-less access to the system(recommended), do not provide a password when prompted and simply hit return.
Peters-MacBook-Pro:~ peter$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/peter/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again
Your identification has been saved in /home/peter/.ssh/id_rsa.
Your public key has been saved in /home/peter/.ssh/id_rsa.pub.
The key fingerprint is:
15:60:97:ac:0f:96:9a:df:0c:32:61:fd:65:fb:77:dc peter@Peters-MacBook-Pro
The key's randomart image is:
+--[ RSA 2048]----+
| ooo. |
| . .o. |
| . o. |
| o *. o |
| . =S+ o . |
| = . o . |
| + + . ..|
| . o . E|
| ..|
+-----------------+The public part of the private / public key pair that was just generated needs to be uploaded to the server. This can be added manually by the user or an administrator but there is a utility that makes this easy called ssh-copy-id. Since the key is not copied over yet, you will be prompted for the password for the vm. If all worked well, then you should not be prompted with a subsequent ssh login from that client. If you have issues, it may be related to permissions on the server. Check here to verify permissions.
Peters-MBP:~ peter$ ssh-copy-id root@streamworksnode2.westus.cloudapp.azure.com
/usr/local/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/peter/.ssh/id_rsa.pub"
/usr/local/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/local/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
(if you think this is a mistake, you may want to use -f option)
Peters-MBP:~ peter$ ssh root@streamworksnode2.westus.cloudapp.azure.com
Last login: Thu Nov 24 10:09:41 2016 from c-73-153-102-2.hsd1.co.comcast.net
[root@node02 ~]$The user that created the vms will have sudo privileges to the host vm.
[petergdoyle@Node0 ~]$ sudo su -
[sudo] password for petergdoyle:
[root@Node0 ~]$
REQUIRED - Install Minimal Additional OS Utilities and Programs - Enable the epel repository and install additional cli tools
[root@Node0 ~] $ yum -y install epel-release && yum -y install git vim htop curl wget tree unzip bash-completion net-tools telnet emacs jq apg denyhostsIT IS STRONGLY RECOMMENDED THAT YOU CONNECT TO THE VM WITH AT LEAST 1, PREFERRABLY 2 OTHER SHELLS AND SUDO IN AS ROOT TO ALL OF THEM. ANY OF THE FOLLOWING PROCEDURES COULD RESULT IN LOCKING YOU OUT OF THE SYSTEM PERMANENTLY AND THE VM WILL HAVE TO BE DESTROYED WITH THE CLOUD PORTAL AND RECREATED. TEST EACH STEP THOROUGHTLY BEFORE PROCEEDING. IF YOU DO EXPERIENCE UNEXPECTED RESULTS YOU HAVE ACCESS TO THE SYSTEM WITH THE OTHER SHELLS TO FIX THE PROBLEMS - CHANGES TO THE SSH CONFIG AND INTRUSION DETECTION MECHANISMS DO NOT FORCE OFF A CONNECTED USER.
If you do get locked out on one session but have another open, then follow these instructions to allow that use to get back in once Denyhosts has locked them out:
Stop DenyHosts
$ systemctl stop denyhosts.service- Remove the IP address from
/etc/hosts.deny - Edit
/var/lib/denyhosts/hostsand remove the lines containing the IP address. Save the file. - Edit
/var/lib/denyhosts/hosts-restrictedand remove the lines containing the IP address. Save the file. - Edit
/var/lib/denyhosts/hosts-rootand remove the lines containing the IP address. Save the file. - Edit
/var/lib/denyhosts/hosts-validand remove the lines containing the IP address. Save the file. - Edit
/var/lib/denyhosts/users-hostsand remove the lines containing the IP address. Save the file. - (optional) Consider adding the IP address to
/var/lib/denyhosts/allowed-hosts
Start DenyHosts
$ systemctl start denyhosts.serviceThe apg utility can generate strong passwords. A password length of 15 with mixed case characters, special characters and digits is strong. This password should not be saved anywhere preventing direct login with the root users. All root activity then is controlled by a single user or an admin group but prevent direct root login.
One liner - if you choose to NOT keep track of the root password Preferred sinde there is not copy and pasted involved and the password is never seen. This one-liner will generate 6 strong passwords, select one randomly and set it for the root user.
[root@Node0 ~]$ apg -a 1 -m 15 | sed -n $(shuf -i1-6 -n1)p| passwd root --stdinStep-by-step - if you choose to keep track of the root password - and remember it. Generate strong passwords, run the passwd command for root and copy and paste one in.
[root@Node0 ~]$ apg -a 1 -m 15
1'jE/-8iEeyo{MD
Fv?23.F/nx?W||6
{M'bpW-JzW?G,YD
lbHH8m(CbH`7->]
6FsdzI[FU4e1=Na
{"eIwlfBwnbN,)[
[root@Node0 ~]$ su -
Last login: Mon Oct 17 11:47:32 UTC 2016 on pts/0
[root@Node0 ~] $ passwd
Changing password for user root.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Rather than run commands directly as root or use sudo commands, it is helpful to use a combination of the two so that you can separate root history from promary user history commands. The primary user on the machine should have sudo nopass rights. The user should therefore be added or created with sudo access. The he can be added to the wheel group to bypass password when running sudo.
root@Node0 ~]$ usermod -aG wheel <username>
This is a convenience and possibly weaker security rather than having the user re-enter their password every time they enter a sudo command. Note there is a tab after ALL=(ALL)
root@Node0 ~]$ cp /etc/sudoers /etc/sudoers.bkp
root@Node0 ~]$ echo '%wheel ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
NOTE: the vm may have to be restarted for group permissions to be applied and allow nopass sudo access.
- install denyhosts (requires epel-release to be installed first)
- in order to unlock accounts that are inadvertantly locked out make sure you can allow them back by stopping denyhosts, clearing out the bad entry in /etc/hosts.deny, changing the
RESET_ON_SUCCESSin the/etc/denyhosts.conffile and then restarting the service. Gotcha - keep at least two terminals ssh'd into the system, so that in case you get locked out in one, you can continue on the other one (or two as a further safety measure). SSHD / Denyhosts cannot disconnect sessions in progress, even though they will prevent future connections if that connection has been marked for denial. - set
RESET_ON_SUCCESS=trueif required - suggested way is to comment outRESET_ON_SUCCESS = nothen uncommentRESET_ON_SUCCESS = yes. Gotcha - If you have more than one entry the first one is taken - turn on the service
- enable the service to start up automatically
- check the status of the service
[root@Node0 ~] $ grep RESET_ON_SUCCESS /etc/denyhosts.conf
[root@Node0 ~] $ sed -i "s/#RESET_ON_SUCCESS = yes/RESET_ON_SUCCESS = yes/g" /etc/denyhosts.conf
[root@Node0 ~] $ grep RESET_ON_SUCCESS /etc/denyhosts.conf
[root@Node0 ~] $ systemctl start denyhosts.service && systemctl enable denyhosts.service && systemctl status denyhosts.service
● denyhosts.service - SSH log watcher
Loaded: loaded (/usr/lib/systemd/system/denyhosts.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2016-10-14 10:54:30 UTC; 2 days ago
Main PID: 632 (denyhosts.py)
CGroup: /system.slice/denyhosts.service
└─632 /usr/bin/python2 /usr/bin/denyhosts.py --foreground --unlock --config=/etc/denyhosts.conf
Oct 14 10:54:30 Node0 systemd[1]: Started SSH log watcher.
Oct 14 10:54:30 Node0 systemd[1]: Starting SSH log watcher...
And within minutes, you can see denyhosts blocking attempts to connect to the vms by reading /var/log/secure and blacklisting ips that are trying to attempt access to the systems by adding them to the /etc/hosts.deny entries.
[root@Node0 ~]$ less /etc/hosts.deny
#
# hosts.deny This file contains access rules which are used to
# deny connections to network services that either use
# the tcp_wrappers library or that have been
# started through a tcp_wrappers-enabled xinetd.
#
# The rules in this file can also be set up in
# /etc/hosts.allow with a 'deny' option instead.
#
# See 'man 5 hosts_options' and 'man 5 hosts_access'
# for information on rule syntax.
# See 'man tcpd' for information on tcp_wrappers
#
# DenyHosts: Thu Oct 13 15:50:40 2016 | sshd: 50.235.31.47
sshd: 50.235.31.47
# DenyHosts: Thu Oct 13 15:50:40 2016 | sshd: 212.83.133.230
sshd: 212.83.133.230
# DenyHosts: Thu Oct 13 19:24:53 2016 | sshd: 82.85.187.101
sshd: 82.85.187.101
...
[root@Node0 ~] $ systemctl start firewalld.service && systemctl enable firewalld && systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2016-10-14 10:55:32 UTC; 2 days ago
Main PID: 633 (firewalld)
CGroup: /system.slice/firewalld.service
└─633 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
Oct 16 16:17:03 Node0 systemd[1]: Started firewalld - dynamic firewall daemon.Check the active FirewalID zone
[root@Node0 ~]$ firewall-cmd --get-default-zone
public
[root@Node0 ~]$ firewall-cmd --get-active-zones
public
interfaces: eth0
Notice that the interface is eth0, firewallID is configured around the concept of zones and zones bound around interfaces - for moe information on firewallID refer to the link here How do we know what rules are associated with the public zone though? We can print out the default zone's configuration by typing:
[root@Node0 ~] $ firewall-cmd --list-all
public (default, active)
interfaces: eth0
sources:
services: dhcpv6-client ssh
ports:
masquerade: no
forward-ports:
icmp-blocks:
rich rules:
[root@Node0 ~] $ yum -y update && yum -y install yum-cronSee yum-cron details to configure it properly https://community.centminmod.com/threads/automatic-nightly-yum-updates-with-yum-cron.1507/
Once the admin user has strengthened the default SSH Security measures outlined previously, and is able to still sign into the system (*see caution about reconfiguring ssh without having at least 1 additional ssh connection, likely 2, so that the admin user doesn not get locked out of the system) then Password-base Authentication can be disable. Follow steps outlined here
Internet Protocol version 6 (IPv6) provides a new Internet layer of the TCP/IP protocol suite that replaces Internet Protocol version 4 (IPv4) and provides many benefits. Currently there are no good tools out which are able to check a system over network for IPv6 security issues. Most Linux distro began enabling IPv6 protocol by default. Crackers can send bad traffic via IPv6 as most admins are not monitoring it. Unless network configuration requires it, disable IPv6 or configure Linux IPv6 firewall:
RedHat / Centos Disable IPv6 Networking
More info here
- Log in to the server as root using SSH.
- Open the /etc/ssh/sshd_config file in your preferred text editor (nano, vi, etc.).
- Change
PermitRootLogin yestoPermitRootLogin no