Ideas for an ssh bastion hosts

gistfile1.txt

I was investigating how to realize a modular, comprehensive and secure solution to log all activities run from an ssh bastion.

The idea is to record in text format all input and output from any command run on the server by a set (potentially everybody, but root can circumveen it) of users. Those operators do need to log on the server via SSH and not tunnel through it (this can be blocked via ssh configs) as they  already perform now. 

I developed this solution using uniquely open source software and I tested on Amazon Linux server. This solution works at a very low level (session bytes copy) and therefore should be compatible with every user activity (like ansible, screen or tmux).


Let’s analyze all the components before seeing how we stitch all together.

SSH daemon configuration : we ensure a specific command is run _before_ every user gets logged on the server, This is done transparently via the configuration stored in .ssh/authorized_keys

Text console logging:  there’s a nice forgotten utility called script, which records any type of character sent/received from a console, this data can be sent over a separate file

Output buffer: writing to a regular file is a brittle security solution because if the script process will need write access to the file and therefore can empty it using a text editor, truncate utility or bash redirection. A structure like a fifo or posix queue will be more secure but it cannot be easily integrated with a bash (any shell) without writing custom code.

Luckily there is a specially developed kernel module to do that:  emlog (https://github.com/nicupavel/emlog) implements a simple character device driver which acts like a named pipe that has a finite, circular buffer. 

Data collection: the session data needs to be collected from the above buffer and stored safely on a secure disk location or shipped to an ELK stack. There are multiple ways to achieve this but there is a caveat, this activity is not reading a regular text file so while you can use cat,less,tail to see it,  a program cannot use fsnotify to ‘follow’ a file. 

Logrotation does not work: the special file cannot be renamed, flushed or opened for write access.



The whole process work in this way:

The emlog module is loaded and a special buffer file is created:

insmod emlog.ko emlog_max_size=2048

mknod /tmp/logger c 248 1

We can create a file per user and its size is 2Mb, the file needs to writable by the user processes

A user  open an ssh connection, the server checks in his home directory the file .ssh/authotized_keys which dictates to run the following command before any shell:

ssh-rsa AAAA………..AN aws

command="script -f /tmp/logger"

On another shell you can simply run cat /tmp/logger and you’ll see the user activity including terminal colours

A program can collect these bytes (potentially from multiple files) and send over using for example execbeat (https://github.com/christiangalsterer/execbeat) or simply copy to a regular file. 

I sketched a simple golang program to read the buffer and write to file, if we want to proceed I can add parallel buffer consumption (one per user buffer).

 



package main



import (

	"fmt"

	"io"

	"os"

	"path/filepath"

	"strings"

	"time"

)



var (

	srcDir = "/tmp/logger"

	dstDir = "/tmp"

)



func main() {

	for {

		files, _ := filepath.Glob(srcDir + ".[0-9]")

		time.Sleep(2 * time.Second)

	}

}



func wrapper(files string[]) {

		for _, f := range files {

			fmt.Printf("working on file %s\n", f)

			go bufferChaser(f)

		}



}



func bufferChaser(filename string) {

	// open input file

	fi, err := os.Open(filename)

	if err != nil {

		fmt.Println(err)

	}

	// close fi on exit and check for its returned error

	defer func() {

		if err := fi.Close(); err != nil {

			fmt.Println(err)

		}

	}()



	// open output file

	outfile := strings.Split(filename, ".")

	fo, err := os.Create(dstDir + "/session_user_" + outfile[1])

	if err != nil {

		fmt.Println(err)

	}

	// close fo on exit and check for its returned error

	defer func() {

		if err := fo.Close(); err != nil {

			fmt.Println(err)

		}

	}()



	// make a buffer to keep chunks that are read

	buf := make([]byte, 1024)

	for {

		// read a chunk

		n, err := fi.Read(buf)

		if err != nil && err != io.EOF {

			fmt.Println(err)

		}

		if n == 0 {

			break

		}



		// write a chunk

		if _, err := fo.Write(buf[:n]); err != nil {

			fmt.Println(err)

		}

	}

}