Skip to content

Instantly share code, notes, and snippets.

Last active February 24, 2024 14:11
Show Gist options
  • Save lizthegrey/9c21673f33186a9cc775464afbdce820 to your computer and use it in GitHub Desktop.
Save lizthegrey/9c21673f33186a9cc775464afbdce820 to your computer and use it in GitHub Desktop.
Hardening SSH with 2fa
default['sshd']['sshd_config']['AuthenticationMethods'] = 'publickey,keyboard-interactive:pam'
default['sshd']['sshd_config']['ChallengeResponseAuthentication'] = 'yes'
default['sshd']['sshd_config']['PasswordAuthentication'] = 'no'

Hi! I'm Liz, a Developer Advocate at, and I spent my first weeks at the company doing security hardening of our infrastructure. I'd like to share what I'd learned with you, so that you can benefit from my reading of dozens of scattered pages of documentation and my ruling out of numerous dead ends.

Why you should take security and usability seriously

Developers and administrators have historically used SSH keys to provide authentication between hosts. By adding passphrase encryption, the private keys become resistant to theft when at rest. But what about when in use? Unfortunately, the usability challenges of re-entering the passphrase on every connection means that engineers began caching keys unencrypted in memory of their workstations, and worse yet, forwarding the agent to allow remote hosts to use the cached keys without further confirmation. The recent breach at Matrix underscores how dangerous it is to allow authenticated sessions to propagate across hosts and environments without a human in the loop.

Thus, we need solutions that prevent key theft from the systems we connect to, while maintaining ease of use. Two-factor authentication stops malicious automated propagation in its tracks by having a second factor protect use of our keys. There are two primary ways of preventing an attacker from misusing our credentials: either using a separate device that generates, using a shared secret, numerical codes that we can transfer over out of band and enter alongside our key, or having the separate device perform all the cryptography for us only when physically authorized by us.

Google, where I previously worked, employs short-lived SSH certificates issued by a central piece of infrastructure, stored on secure hardware tokens. But this is a serious change to developer workflow, and requires extensive infrastructure to set up. What will work for a majority of developers who are used to simply loading their SSH key into the agent at the start of their login session and SSHing everywhere?

Design considerations & threat models

I'm assuming that you have a publicly exposed bastion host for each environment that intermediates accesses to the rest of each environment's VPC, and use SSH keys to authenticate from laptops to the bastion and from the bastion to each VM/container in the VPC. If you don't yet have a bastion host and a VPC, start there!

It was important to me to make Honeycomb safe from compromise, even if malicious worm-like code were executed on a developer's laptop while SSH keys were unlocked, or if a developer accidentally forwarded an SSH agent to a hostile remote system. I also thought it important to build on existing work to disk encrypt all endpoints by ensuring the loss of physical control over a phone or hardware token could not itself grant production access. However, I consider it out of scope to prevent active local intervention and session hijacking (since someone who controls your active console or keyboard has you pretty well pwned).

I'm also assuming you have a mix of operating systems, hardware, and preferences about carrying dongles vs. wanting to use phones for second factor, etc.

How to get started!

First, start by enabling numerical time-based one time password (TOTP) for SSH authentication. Is it perfect? No, since a malicious host could impersonate the real bastion (if strict host checking isn't on), intercept your OTP, and then use it to authenticate to the real bastion. But it's better than being wormed or compromised because you forgot to take basic measures against even a passive adversary.

Server-side setup

You'll want a root shell open just in case, and the following snippets added to your Chef cookbooks (from this gist):

  • metadata.rb
  • attributes/default.rb (from attributes.rb)
  • files/sshd
  • recipes/default.rb (copy from recipe.rb)
  • templates/default/users.oath.erb

Okay, now we can set this running on our hosts… and go through the client setup for ourselves at least.

Client-side setup

Now, each user authenticating needs a shared key to be present, encrypted, in SSM (or equivalent for your choice of cloud provider). Have each user install an OTP app such as Google Authenticator, Authy, Duo, or Lastpass, then do the following on their laptop:

Install dependencies:

brew install oath-toolkit OR apt install oathtool openssl

Generate a random base16 string to use as your key:

➜ openssl rand -hex 10
##### ^^^ that's an example output used here - don't use it!

Convert it and put it into a phone-based authenticator app: Run oathtool -v [key] to convert it to the format (“Base32 secret”) that mobile authenticators use.

➜ oathtool -v 22ea2966afefd82660e1
Hex secret: 22ea2966afefd82660e1
Base32 secret: ELVCSZVP57MCMYHB
... more stuff down here we don't need
  • For 1Password, add a one time password and enter the “Base32 secret” output from oathtool -v [key]
  • For Duo, select “other” and use the Base32 secret.
  • for Authy click “Enter key manually” and use the Base32 secret

Verify that generated codes are correct: Run oathtool --totp [key] and check that it returns the same value as your authenticator application.

➜ oathtool --totp 22ea2966afefd82660e1

Store our key into the cloud secrets manager: Run aws ssm put-parameter --name /2fa/totp/$USER --value [key] --type SecureString --key-id alias/parameter_store_key to put your key into SSM Parameter Store. $USER should be the same as the username you use when you log in to a bastion. If you are updating the key instead of pushing it for the first time, add the --overwrite flag to the end of the command.

➜ aws ssm put-parameter --name /2fa/totp/lizf --value 22ea2966afefd82660e1 --type SecureString --key-id alias/parameter_store_key 
    "Version": 1

Log in for the first time: Now, when we ssh to the bastion host, we can ensure that the SSH agent can only be trampolined to other hosts within the VPC, but any attempt to programatically use from the outside the forwarded agent (or loaded in-memory keys) to access a bastion will fail because no TOTP from the separate mobile device was provided.

Let's check that we're asking for TOTPs:

➜ ssh -A bastion
Enter passphrase for key '[snip]': 
One-time password (OATH) for '[user]': 
Welcome to Ubuntu 18.04.1 LTS...

Now there's a value proposition for hardware auth…

People might get sick and tired of entering a numerical OTP every time they have to log into the bastion! It's almost like the old days of passphrase-encrypted SSH keys that motivated us to use agents! So let's leverage this inherent laziness to get people more, rather than less, secure!

Server-side setup

Change the beginning of files/sshd in your Chef module to begin as follows:

auth	required
auth	optional

# If it's a hardware or secure enclave SSH key, no need for a numerical OTP.
auth    sufficient file=/etc/2fa_token_keys

# Check a TOTP code as a second resort, using a time slip of +/- 150 seconds.
auth	sufficient usersfile=/etc/users.oath digits=6 window=5

# People without OTPs will need to add an OTP secret to AWS SSM and wait an hour.
auth	requisite

And if your openssh is older than 7.8 and/or your libpam-ssh-agent-auth is older than 0.10.4, add the following additional lines to recipes/default.rb (a note to the nervous: my source modifications to openssh-server and libpam-ssh-agent-auth are available from Launchpad):

apt_repository 'openssl-pam-bindings' do
  uri ''

packages = %w{ openssh-server libpam-ssh-agent-auth }
packages.each do |p|
  r = package p do
    action :upgrade

service 'sshd' do
  subscribes :reload, 'package[openssh-server]'

Now you'll need to use Chef to populate /etc/2fa_token_keys with keys that you know are generated and stored securely (e.g. using one of the below methods). I don't know how you maintain your lists of ssh key mappings to users, nor how you add ssh keys to your ~/.ssh/authorized_keys files, so I can't provide general advice.

Mac client setup

People with Touchbar Macs should use TouchID to authenticate logins, as they'll have their laptop and their fingers with them anyways. sekey lets us support this.

Install the binary:

brew cask install sekey

Add to ~/.ssh/config on your local machine:

IdentityAgent ~/.sekey/ssh-agent.ssh

Generate a key and export it:

sekey --generate-keypair "bastion key"
sekey --export-key $(sekey --list-keys|grep "bastion key"|grep --only-matching -E '[a-f0-9]{40}')

And then store the resulting key to /etc/2fa_token_keys and ~/.ssh/authorized_keys in Chef. setup for iOS and Android

Instead of generating OTPs and sending them over manually with our fingers, our mobile devices can securely store our SSH keys and only remotely authorize usage (and send the signed challenge to the remote server) if a human presses a button on the phone.

This is the theory behind, and is even more secure than a TOTP app so long as you supply appropriate parameters to force hardware coprocessor storage (NIST P-256 for iOS, and 3072-bit RSA for Android, on new enough devices). Make sure people use screen locks!

Follow the instructions here: and then supply the generated key to both ~/.ssh/authorized_keys and /etc/2fa_token_keys in your Chef automation, and you won't be prompted for a TOTP.

YubiKey hardware token & Linux/ChromeOS client setup

Initial per-YubiKey setup

Follow these instructions from a Linux host to set up a basic working hardened YubiKey SSH key:

Install Dependencies

sudo apt-add-repository ppa:yubico/stable && sudo apt-get update
sudo apt-get install gpg yubikey-manager-qt pinentry-curses scdaemon pcscd
echo "reader-port Yubico YubiKey" > .gnupg/scdaemon.conf

Hardening to prevent a rogue host from authenticating without your permission

ykman openpgp touch sig on
ykman openpgp touch aut on
ykman openpgp touch enc on

Hardening in case your security key is stolen

gpg --change-pin

Default user pin is 123456 and admin pin is 12345678, change both of them to something more secure; they can both be the same PIN.

Generate a random 24-byte hex-encoded reset key and save it somewhere, GPG encrypted with your normal daily use keys (ykman-gui can generate a 24-byte string for you in “PIV → Configure PINs → Change Management Key”)

Generating the keys:

gpg --card-edit
  • Input 4096 for all three modes (you'll need to enter the admin and user pins)
  • Don't back up the stubs when prompted.
  • Enter your full name and email address; make sure you leave a comment (e.g. desk computer) so you know which stub key is which in your GPG keyring.

Wait a minute, then enter the user PIN one more time, then wait about 5-10 minutes for the generation process to complete. It will print the UID of the master key before returning you to the card-edit prompt.


gpg --export-ssh-key UID_of_master_key

This will print out the ssh pubkey string you'll need to add to the remote ~/.ssh/authorized_keys and /etc/2fa_token_keys in Chef.

Usage for authentication


Once the per-key setup is done, the configured Yubikey can be used in a Linux machine configured like so: gpg --with-keygrip -K

Save the keygrip of the master key you just generated to .gnupg/sshcontrol

Ensure that you have gpg-agent configured correctly: Set curses pinentry. Why? So you don't randomly get X passphrase/passcode prompts all over the place (esp remotely):

Edit ~/.gnupg/gpg-agent.conf to contain:

pinentry-program /usr/bin/pinentry-curses

You'll update your ~/.bashrc to contain the following lines:

export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
export GPG_TTY=$(tty)
gpg-connect-agent updatestartuptty /bye >/dev/null

Run ssh-add -l to confirm you see your key in the list (it'll show 4096 SHA256:... cardno:... (RSA) in the listing).

When you ssh from a terminal into a bastion (remember to ssh -A for agent forwarding!), it'll prompt in the terminal that you most recently opened for your PIN on initial usage of the key. You'll complete that step, then tap your key to confirm. You're in!


Install these two Chrome apps:

Then open the Secure Shell App (this won't work yet from the Crostini Terminal app because Crostini doesn't have USB pass-through yet, although it's coming in Chrome 75!)

Within the secure shell app's configuration screen for the bastion host:

  • relay server option: --ssh-agent=gsc
  • ssh option: -A

You'll then enter the user PIN when prompted, and tap the security key to confirm when logging into the bastion.

Further reading

The folks at have written some fantastic blogs on securing SSH that go beyond the basic hardening I recommend here.

Hope this helps! Send me a Twitter DM (@lizthegrey) or email ( if you have improvements to suggest!

name "bastion"
description "special hardening for bastions"
version "0.0.1"
depends "aws"
depends "sshd"
package "libpam-oath" do
action :upgrade
aws_ssm_parameter_store 'getOTPsecrets' do
path '/2fa/totp/' # or your own choice of SSM path.
recursive true
with_decryption true
return_key 'totp_secrets'
action :get_parameters_by_path
# No need for aws_access_key and aws_secret_access_key due to implicit EC2 grant.
sensitive true
# Populate the oath file.
template '/etc/users.oath' do
source 'users.oath.erb'
owner 'root'
group 'root'
mode '0600'
:users => lazy { node.run_state['totp_secrets'] }
sensitive true
cookbook_file '/etc/pam.d/sshd' do
source 'sshd'
owner 'root'
group 'root'
mode '0644'
action :create
# Force ssh to consult PAM as well as using SSH keys for primary auth..
include_recipe 'sshd'
auth required
auth optional
# Check a TOTP code, using a time slip of +/- 150 seconds.
auth sufficient usersfile=/etc/users.oath digits=6 window=5
# People without OTPs will need to add an OTP secret to AWS SSM and wait an hour.
auth requisite
account required
@include common-account
session [success=ok ignore=ignore module_unknown=ignore default=bad] close
session required
session optional force revoke
@include common-session
session optional motd=/run/motd.dynamic
session optional noupdate
session optional standard noenv
session required
session required
session required user_readenv=1 envfile=/etc/default/locale
session [success=ok ignore=ignore module_unknown=ignore default=bad] open
@include common-password
# To update this file, generate a SSM parameter in /2fa/totp.
# See this URL for examples:
<% @users.each do |key, value| %>
HOTP/T30 <%= key %> - <%= value %>
<% end %>
Copy link

trknov commented Apr 20, 2019

I see you like Hardening

Copy link

I prefer to use (and have my devs use) ed25519 keys instead of RSA.
I do it both on kryptonco and on laptops

Copy link

This is fantastic, thank you for writing it!

I'd be interested to see how this could be adapted to work with FreeIPA. (Identity management via ldap, Kerberos, sssd and a certificate authority)

Copy link

Thank you for sharing your knowledge and expertise for free. People like you make the world a better place.

Copy link

I would just use gpg hardware keys with SSH.

Then you don't have to mess with 2 factor.

Copy link

Winkster commented Apr 20, 2019

This is awesome, Liz, thanks so much. Very thorough and thoughtful post. All angles considered. You rock.

Copy link

I'm a little annoyed this entire thing assumes we have Chef.

Requirements should be stated at the beginning, or at the least, you should not launch directly into "add blah into Chef" but instead "First you'll need Chef and to add blah into it".

Its presumptuous and rude to assume everybody reading has the requirements and you should instead explitically state them.

Copy link

christian-ehrisman commented Apr 20, 2019

Thank you for this write-up, it's a lot of great information. I don't think it is either presumptuous or rude to document the actual common and normal software you used, though some may be offended by your use of code. I must admit I'm left confused why anyone would have a problem with someone posting on github the thing they did with common software.

Copy link

robertkraig commented Apr 20, 2019

I'd love to see a video posted on youtube or attached in gif in fast-forward mode to see the whole process work / look.

Copy link

sbgoodm commented Apr 21, 2019

Great write-up, thanks for sharing!

Copy link

relaxdiego commented Apr 21, 2019

Area man says it’s presumptuous and rude to share what you know for free on the Interwebz. In other news, research shows that open source is bad for the world because of the multitude of languages and frameworks in use. “It’s just too gosh darn rude and offensive!” according to one bystander.

Copy link

kinduff commented Apr 21, 2019

This is awesome, thanks a lot for sharing!

Copy link

Just wanted to thank you, this is VERY valuable nowadays.

Copy link

Thank you, Liz!

Copy link

It works great! Thanks!

Copy link

This is awesome, and if you disable the Smart Card Connector app you can pass Yubikey devices into Crostini quite nicely these days. There appears to be an "ownership" conflict where it demands exclusive access that prevents sharing into Crostini using vmc usb-attach termina x:yyy if you don't disable the connector.

Copy link

This is awesome, and if you disable the Smart Card Connector app you can pass Yubikey devices into Crostini quite nicely these days. There appears to be an "ownership" conflict where it demands exclusive access that prevents sharing into Crostini using vmc usb-attach termina x:yyy if you don't disable the connector.

@espoelstra OMG thanks! super glad to hear yubikeys can be attached via USB sharing, I thought it was broken, but apparently was due to the app!

Copy link

@espoelstra I've now confirmed this works, so will update the guide accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment