Skip to content

Instantly share code, notes, and snippets.

@jtmoon79
Last active April 19, 2024 06:36
Show Gist options
  • Star 7 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save jtmoon79/745e6df63dd14b9f2d17a662179e953a to your computer and use it in GitHub Desktop.
Save jtmoon79/745e6df63dd14b9f2d17a662179e953a to your computer and use it in GitHub Desktop.
OpenSSH: build and install the latest version

openssh latest

Build, install, and run the latest OpenSSH Server as a systemd service.



About

Running the latest OpenSSH Server is easy and security-wise. This can run side-by-side with the package-manager installed version of OpenSSH (this is recommended).

This is for a Debian-derived Linux System using systemd. This document will use version 9.0p1.

OpenSSH will be built on the system that will also run the ssh service.

Commands are presumed to run as user root.

Prepare /opt

Do this once.

mkdir -vp /opt/

apt install

Do this once.

List the dpkg requirements for OpenSSH server

apt show openssh-server

Most likely, the packages listed under Depends: will be needed to build OpenSSH.

My typical Debian system addtionally needed these

apt install \
    libssl-dev \
    gcc g++ gdb cpp \
    make cmake \
    libtool \
    libc6 \
    autoconf automake pkg-config \
    build-essential \
    gettext \

(I am not sure if all of these packages are needed, but it did the trick)

I suspect you also need these

apt install \
    libzstd1 zlib1g \
    libssh-4 libssh-dev libssl3 \
    libc6-dev libc6 \
    libcrypt-dev

These guesses have not been thoroughly tested.

Other helpful tools for debugging server issues are:

apt install netcat lsof wget diffutils

Download

Do this each update.

  1. Pick a mirror

  2. Download the latest archive.
    This will example will use openssh-9.0p1.tar.gz.

    VER=9.0p1
    wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-${VER}.tar.gz
    

Verify the download using gpg

Import the public signing key

Do this once.

wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/RELEASE_KEY.asc
gpg --import RELEASE_KEY.asc

Verify

Do this each update.

wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-${VER}.tar.gz.asc
gpg --verbose --verify openssh-${VER}.tar.gz.asc

Build

Do this each update.

Preview the next subsection before running these commands.

VER=9.0p1
tar -xvf openssh-${VER}.tar.gz
cd openssh-${VER}
./configure --prefix=/opt/openssh-${VER}
make
make install

If ./configure is missing then the non-portable version was downloaded.

Change the banner (optional) (INCOMPLETE)

This section has not yet successfully tested 😔. Come back later.

Change string that all connecting SSH clients receive (this occurs before authentication). This requires a change before running make from the previous Build section.

The statement that writes the "banner" string to the connected SSH channel is in file ./ssh_api.c.

if ((r = sshbuf_putf(banner, "SSH-2.0-%.100s\r\n", SSH_VERSION)) != 0)

Before running make, change the file ./ssh_api.c

sed -i -Ee 's|(sshbuf_putf\(banner, )("SSH-.*", SSH_VERSION)(\))|\1"please_go_away"\3|' -- ./ssh_api.c

This should change the statement to

if ((r = sshbuf_putf(banner, "please_go_away")) != 0)

Create Symlink for -latest

Do this once.

rm -vf /opt/openssh-latest /etc/ssh-latest
ln -fvs /opt/openssh-${VER} /opt/openssh-latest
ln -fvs /opt/openssh-latest /etc/ssh-latest

Set aside built-in OpenSSH

Do this once.

Very Important! Keep the functioning built-in sshd service in-place or running at a different port.

  1. Change the system built-in SSH Service /etc/ssh/sshd_config, add a non-typical port to the Ports declaration, for example 2222.
  2. Restart the system built-in SSH Service.
  3. Login to the SSH service listening using the non-typical port 2222. ssh -p 2222 user@server
  4. Remove from the Ports declaration the typical port 22.
  5. Restart the system built-in SSH Service.

OR

  1. Have the latest SSH service use a non-standard port(s), like 2222.

Now there is a reliable fallback SSH Service.

Create systemd service -latest

Do this once.

Copy the systemd template service files for the ssh service.

This path may vary among /lib/systemd/system, or /usr/lib/systemd/system, or something else. Be sure not to use the generated systemd files typically found at /etc/systemd.

Find all *ssh* files

find / -xdev -name '*ssh*' 2>/dev/null | sort

Copy the systemd service files. In my case on Debian 11

cd /usr/lib/systemd/system/
cp -av ssh.service ssh-latest.service
cp -av ssh@.service ssh-latest@.service
cp -av ssh.socket ssh-latest.socket
cp -av rescue-ssh.target rescue-ssh-latest.target

MMV: the path to systemd files tends to change often per distribution and and per release.

In the newly copied ssh-latest* systemd files, manually update references from ssh to ssh-latest.

In my case on Debian 11, the changes looked like:

$ cd /usr/lib/systemd/system

$ diff -y --suppress-common-lines ssh-latest@.service ssh@.service
EnvironmentFile=-/opt/openssh-latest/default/ssh              | EnvironmentFile=-/etc/default/ssh
ExecStart=/opt/openssh-latest/sbin/sshd -i $SSHD_OPTS         | ExecStart=-/usr/sbin/sshd -i $SSHD_OPTS
RuntimeDirectory=sshd-latest                                  | RuntimeDirectory=sshd

$ diff -y --suppress-common-lines ssh-latest.service ssh.service
ConditionPathExists=!/opt/openssh-latest/etc/sshd_not_to_be_r | ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
EnvironmentFile=-/opt/openssh-latest/default/ssh              | EnvironmentFile=-/etc/default/ssh
ExecStartPre=/opt/openssh-latest/sbin/sshd -t                 | ExecStartPre=/usr/sbin/sshd -t
ExecStart=/opt/openssh-latest/sbin/sshd -D $SSHD_OPTS         | ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/opt/openssh-latest/sbin/sshd -t                   | ExecReload=/usr/sbin/sshd -t
Type=exec                                                     | Type=notify
RuntimeDirectory=sshd-latest                                  | RuntimeDirectory=sshd
Alias=sshd-latest.service                                     | Alias=sshd.service

$ diff -y --suppress-common-lines ssh-latest.socket ssh.socket
Before=ssh-latest.service                                     | Before=ssh.service
Conflicts=ssh-latest.service                                  | Conflicts=ssh.service
ConditionPathExists=!/opt/openssh-latest/etc/sshd_not_to_be_r | ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
ListenStream=2222                                             | ListenStream=22

$ diff -y --suppress-common-lines rescue-ssh.target rescue-ssh-latest.target
Requires=network-online.target ssh.service                    | Requires=network-online.target ssh-latest.service
After=network-online.target ssh.service                       | After=network-online.target ssh-latest.service

Notice the change of Type from notify to exec.

Copy the default environment file

mkdir -vp /opt/openssh-latest/default/
cp -av /etc/default/ssh /opt/openssh-latest/default/

The /opt/openssh-latest/default/ssh file sets the $SSHD_OPTS which is used by the service file.

Enable and restart

Tell the systemd daemon to enable and start the new services

systemctl enable ssh-latest.service
systemctl enable ssh-latest.socket
systemctl daemon-reload
systemctl start ssh-latest

The command systemctl daemon-reload will generate new files under /etc/systemd.

Watch the service logs

journalctl -f -x -u ssh-latest

Check services status

systemctl status ssh-latest

Verify new ssh service and old ssh service

Check the service replies with the expected latest openssh version. For each port with a listening ssh service

echo | nc localhost 22
echo | nc localhost 2222

This should look like:

$ echo | nc localhost 2222
SSH-2.0-OpenSSH_9.0
Invalid SSH identification string.

Check there are two different sshd -D daemon processes

ps -ef --forest | grep "sshd -D"

Check the ports in-use

lsof -PVn -iTCP | grep sshd | grep LISTEN

openssh updates

Do this each update.

After a new version of OpenSSH Server is released, it's relatively easy to update things.

Repeat section Download, and Build.

Copy the Previous -latest to the New -latest

Set PREV to the appropriate value. In my case it was 8.6p1.

cd /opt/openssh-${VER}/etc/
# set aside default config files
mkdir -vp _original
mv -v ssh_host_* sshd_config ssh_config _original/
PREV=8.6p1
cp -av /opt/openssh-${PREV}/etc/ssh_host_* /opt/openssh-${PREV}/etc/{sshd_config,ssh_config} .

The files in /opt/openssh-latest/etc/ should be similar to:

moduli
_original/
ssh_config
sshd_config
ssh_host_dsa_key
ssh_host_dsa_key.pub
ssh_host_ecdsa_key
ssh_host_ecdsa_key.pub
ssh_host_ed25519_key
ssh_host_ed25519_key.pub
ssh_host_rsa_key
ssh_host_rsa_key.pub

Use default key files (optional)

Optionally, the /opt/openssh-latest/etc/sshd_config can refer to the key files at /etc/ssh. Then there is no need to copy the /opt/openssh-latest/etc/ssh_host_* files. In that case, the prior file listing of /opt/openssh-latest/etc/ should be similar to:

moduli
_original/
ssh_config
sshd_config

The file /opt/openssh-latest/etc/sshd_config should have lines

$ grep -Fe 'HostKey' /opt/openssh-${VER}/etc/sshd_config
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

Review the copied sshd_config with the original default sshd_config.

cd /opt/openssh-${VER}/etc/
diff -y -W $COLUMNS sshd_config _original/sshd_config | less -SR

Update Symlinks for -latest

systemctl stop ssh-latest
cd /opt
rm -v openssh-latest && ln -fvs openssh-${VER} openssh-latest

The directory /opt should look like

openssh-8.6p1/
openssh-9.0p1/
openssh-latest -> openssh-9.0p1

Restart Service

First, watch the service logs

journalctl -f -x -u ssh-latest

Then restart

systemctl restart ssh-latest

Test

For each port with a listening ssh service

echo | nc localhost 22
echo | nc localhost 2222
@c01dkit
Copy link

c01dkit commented Aug 14, 2023

Hi jtmoon79,
In the Change the banner section, you said "This section has not yet successfully tested 😔. Come back later.". Maybe you can directly modify the definition of SSH_VERSION in version.h. I've tested it on v9.4p1, and it works fine for me : )

@vshoshin
Copy link

Hi jtmoon79,
Just a little note - in my case on to start OpenSSH 9.7p1 on Debian 12 a directory /var/empty needs to be manually created.

@da40
Copy link

da40 commented Apr 10, 2024

Hi jtmoon79,

Thank you for this, it was extremely helpful. I encountered a couple of minor issues:

  1. mkdir -vp /opt/openssh-latest/default/
    This failed, because openssh-latest did not exist, so I created it.

  2. I chose to use a non-standard port, so I had to edit that into /opt/openssh-latest/etc/sshd_config

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment