OpenSSH Update Script - Amazon Linux 2

OpenSSH Update Script - Amazon Linux 2

#!/bin/bash
sudo yum install gcc -y
sudo yum install openssl-devel -y
sudo yum install zlib-devel -y
sudo yum install mlocate -y
sudo yum install autoconf -y
wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.1p1.tar.gz
tar zxvf openssh-9.1p1.tar.gz
cd openssh-9.1p1 && ./configure && make && sudo make install

Hey! Good catch! If I can suggest something: force the installation to the /usr folder to match the current place where sshd and ssh is installed on AmazonLinux2 instances.

./configure --prefix=/usr
make
sudo make install

Here are the instructions for updating to 9.0.

You can just copy and paste from line 2-end into the terminal and it should install.

#!/bin/bash
sudo yum install gcc -y
sudo yum install openssl-devel -y
sudo yum install zlib-devel -y
sudo yum install mlocate -y
sudo yum install autoconf -y
sudo wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.0p1.tar.gz
sudo tar zxvf openssh-9.0p1.tar.gz
cd openssh-9.0p1 
sudo ./configure --prefix=/usr
sudo make
sudo make install

Updated the gist for OpenSSH 9.1p1

I did the steps above, then I tried to use the FTP service like this:
$ sftp -vvvv $sftp_user@$my_ip
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017

I got the header OpenSSH_7.4p1
How to make the system use the new version?

@krlinus You likely need to restart your SSH service for the changes to take effect. Try running sudo systemctl restart sshd on your server and then you can try your sftp command again.

Here's an updated script for 9.3 (released March 15, 2023):

sudo yum install -y gcc openssl-devel zlib-devel mlocate autoconf
wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.3p1.tar.gz
tar zxvf openssh-9.3p1.tar.gz
cd openssh-9.3p1
./configure --prefix=/usr
make
sudo make install

I tried the above for both 9.0 and 9.1
But restart of the srervice fails with below

Job for sshd.service failed because a timeout was exceeded. See "systemctl status sshd.service" and "journalctl -xe" for details.

I increase the timeout time still to no avail.

Now I can't ssh into the server.

Anyone ran into this ?

I haven't seen this happen with sshd before, but to get more info on it, you should see what kind of logs / error messages sshd is outputting. Try sudo journalctl -u sshd.service and look for error messages.

@teekennedy,

I'm seeing the timeout/flapping on a fairly clean EC2 instance using your June 1st script:

 systemd[1]: sshd.service start operation timed out. Terminating.
 systemd[1]: Failed to start OpenSSH server daemon.
 systemd[1]: Unit sshd.service entered failed state.
 systemd[1]: sshd.service failed.
 systemd[1]: sshd.service holdoff time over, scheduling restart.
 systemd[1]: Stopped OpenSSH server daemon.
 systemd[1]: Starting OpenSSH server daemon...
 sshd[29999]: Server listening on 0.0.0.0 port 22.
 sshd[29999]: Server listening on :: port 22.
 systemd[1]: sshd.service start operation timed out. Terminating.
 systemd[1]: Failed to start OpenSSH server daemon.
 systemd[1]: Unit sshd.service entered failed state.
 systemd[1]: sshd.service failed.

journalctl -r /usr/sbin/sshd also has sshd[28676]: User ec2-user not allowed because account is locked, which is interesting as that's the account I've used for several weeks since I started the instance. Running sudo passwd ec2-user to set a password and then sudo passwd -u ec2-user to unlock the account lets me login when I catch the service in the middle restart.

Edit: updated info on ec2-user locked acount

Yeah, it's the same error logs I got

It works 👍

@teekennedy,

I'm seeing the timeout/flapping on a fairly clean EC2 instance using your June 1st script:

 systemd[1]: sshd.service start operation timed out. Terminating.
 systemd[1]: Failed to start OpenSSH server daemon.
 systemd[1]: Unit sshd.service entered failed state.
 systemd[1]: sshd.service failed.
 systemd[1]: sshd.service holdoff time over, scheduling restart.
 systemd[1]: Stopped OpenSSH server daemon.
 systemd[1]: Starting OpenSSH server daemon...
 sshd[29999]: Server listening on 0.0.0.0 port 22.
 sshd[29999]: Server listening on :: port 22.
 systemd[1]: sshd.service start operation timed out. Terminating.
 systemd[1]: Failed to start OpenSSH server daemon.
 systemd[1]: Unit sshd.service entered failed state.
 systemd[1]: sshd.service failed.

journalctl -r /usr/sbin/sshd also has sshd[28676]: User ec2-user not allowed because account is locked, which is interesting as that's the account I've used for several weeks since I started the instance. Running sudo passwd ec2-user to set a password and then sudo passwd -u ec2-user to unlock the account lets me login when I catch the service in the middle restart.

Edit: updated info on ec2-user locked acount

followed this to resolve the issue

@samaddico, with that hint I found https://unix.stackexchange.com/a/313159. It all makes sense now. The built from raw source binary does not have systemd support build in, so it never notifies systemd that sshd successfully started causing the flapping/loop. It looks like another option might be to change the unit/service file, but I don't know the (security) ramifications of that trade-off.

@samaddico, with that hint I found https://unix.stackexchange.com/a/313159. It all makes sense now. The built from raw source binary does not have systemd support build in, so it never notifies systemd that sshd successfully started causing the flapping/loop. It looks like another option might be to change the unit/service file, but I don't know the (security) ramifications of that trade-off.

Good catch !

Based on the link provided by @samaddico , this is what I did on Amazon Linux 2

(note I included the line numbers because it may help - they should be similar, if not exactly the same, for other users)

# install pre-reqs and get software
sudo yum install -y gcc openssl-devel zlib-devel mlocate autoconf
wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.3p2.tar.gz
tar zxvf openssh-9.3p2.tar.gz
cd openssh-9.3p2
./configure --prefix=/usr

# edit sshd.c
1. add the following marked line:

127 #include "sk-api.h"
128 #include "srclimit.h"
129 #include "dh.h"
130 #include <systemd/sd-daemon.h>         <---- this line

2. add the following marked lines:

2097                 /* Signal systemd that we are ready to accept connections */     <---- this line
2098                 sd_notify (0, "READY=1");                                        <---- this line
2099
2100                 /* Accept a connection and return in a forked child */
2101                 server_accept_loop(&sock_in, &sock_out,
2102                     &newsock, config_s);

# install devel library
sudo yum install -y systemd-devel

# update the LIBS variable in the Makefile
51 #LIBS=-ldl -lutil  -lresolv
52 LIBS =-lcrypto -ldl -lutil -lz -lcrypt -lresolv -lsystemd

# run make, make install
make
sudo make install

# restart sshd
systemctl restart sshd.service

# check sshd status
systemctl status sshd.service

Thanks for this @sttuartt, following your answer and editing the sshd.c file with the indicated lines made my sshd.service work again, now I can restart it without any problem.

I'm using Amazon Linux 2

Based on the link provided by @samaddico , this is what I did on Amazon Linux 2

(note I included the line numbers because it may help - they should be similar, if not exactly the same, for other users)

# install pre-reqs and get software
sudo yum install -y gcc openssl-devel zlib-devel mlocate autoconf
wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.3p2.tar.gz
tar zxvf openssh-9.3p2.tar.gz
cd openssh-9.3p2
./configure --prefix=/usr

# edit sshd.c
1. add the following marked line:

127 #include "sk-api.h"
128 #include "srclimit.h"
129 #include "dh.h"
130 #include <systemd/sd-daemon.h>         <---- this line

2. add the following marked lines:

2097                 /* Signal systemd that we are ready to accept connections */     <---- this line
2098                 sd_notify (0, "READY=1");                                        <---- this line
2099
2100                 /* Accept a connection and return in a forked child */
2101                 server_accept_loop(&sock_in, &sock_out,
2102                     &newsock, config_s);

# install devel library
sudo yum install -y systemd-devel

# update the LIBS variable in the Makefile
51 #LIBS=-ldl -lutil  -lresolv
52 LIBS =-lcrypto -ldl -lutil -lz -lcrypt -lresolv -lsystemd

# run make, make install
make
sudo make install

# restart sshd
systemctl restart sshd.service

# check sshd status
systemctl status sshd.service

It seems that this works exactly for this openssh-9.3p2 version, because I tried this with a previous version (openssh-9.1p1) and was unsuccessful when compiling.

Thank you, @softlberton ! Just a quick note for others, you need to sudo the systemctl calls.

Based on the link provided by @samaddico , this is what I did on Amazon Linux 2

(note I included the line numbers because it may help - they should be similar, if not exactly the same, for other users)

# install pre-reqs and get software
sudo yum install -y gcc openssl-devel zlib-devel mlocate autoconf
wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.3p2.tar.gz
tar zxvf openssh-9.3p2.tar.gz
cd openssh-9.3p2
./configure --prefix=/usr

# edit sshd.c
1. add the following marked line:

127 #include "sk-api.h"
128 #include "srclimit.h"
129 #include "dh.h"
130 #include <systemd/sd-daemon.h>         <---- this line

2. add the following marked lines:

2097                 /* Signal systemd that we are ready to accept connections */     <---- this line
2098                 sd_notify (0, "READY=1");                                        <---- this line
2099
2100                 /* Accept a connection and return in a forked child */
2101                 server_accept_loop(&sock_in, &sock_out,
2102                     &newsock, config_s);

# install devel library
sudo yum install -y systemd-devel

# update the LIBS variable in the Makefile
51 #LIBS=-ldl -lutil  -lresolv
52 LIBS =-lcrypto -ldl -lutil -lz -lcrypt -lresolv -lsystemd

# run make, make install
make
sudo make install

# restart sshd
systemctl restart sshd.service

# check sshd status
systemctl status sshd.service

So awesome @brianantonelli, many thanks! BTW, this solution works well with OpenSSH v9.5p1 too.
To facilitate subsequent use by others, you can use the following code directly:

sed -i '129a\#include <systemd/sd-daemon.h>' sshd.c
sed -i '2095a\        /* Signal systemd that we are ready to accept connections */' sshd.c
sed -i '2096a\        sd_notify (0, "READY=1");' sshd.c

When I try the configure command, it eventually fails with:

Checking OpenSSL library version... configure: error: OpenSSL >= 1.1.1 required (have "100020bf (OpenSSL 1.0.2k-fips 26 Jan 2017)")

OpenSSL via yum for Amazon Linux 2 only lets me install 1.0.2.

Any ideas?

Use:
yum install -y openssl11 openssl11-devel

I was able to upgrade to 9.6p1 with that.

OK so that installs openssl 11, but does that replace openssl 1.0.2? When I run openssl, it is still returning 1.0.2

…

________________________________ From: andrew1q9o ***@***.***> Sent: Wednesday, March 6, 2024 8:38 AM To: andrew1q9o ***@***.***> Cc: Comment ***@***.***> Subject: Re: roommen/OpenSSH Update Script - Amazon Linux 2 @andrew1q9o commented on this gist.

________________________________ Use: yum install -y openssl11 openssl11-devel I was able to upgrade to 9.6p1 with that. — Reply to this email directly, view it on GitHub<https://gist.github.com/roommen/18cd78d07b0fbc962de4e79c1d468f92#gistcomment-4973015> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACQI2BSARSLXIVZF4F2RND3YW4L63BFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVA4TOMBTHA3TINFHORZGSZ3HMVZKMY3SMVQXIZI>. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

Both versions are installed. Executing this:
openssl11 version

returns 1.1.1

OK so how do I tell the ./configure to use openssl 1.1.1? It still fails with the 1.0.2 version being detected Sorry I do not have much experience with having multiple versions of OpenSSL on a server

…

________________________________ From: andrew1q9o ***@***.***> Sent: Wednesday, March 6, 2024 8:47 AM To: andrew1q9o ***@***.***> Cc: Comment ***@***.***> Subject: Re: roommen/OpenSSH Update Script - Amazon Linux 2 @andrew1q9o commented on this gist.

________________________________ Both versions are installed. Executing this: openssl11 version returns 1.1.1 — Reply to this email directly, view it on GitHub<https://gist.github.com/roommen/18cd78d07b0fbc962de4e79c1d468f92#gistcomment-4973036> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACQI2BXA77DJMCKQCH7CU53YW4M7NBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVA4TOMBTHA3TINFHORZGSZ3HMVZKMY3SMVQXIZI>. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

I literally just did the yum install above and ./configure and everything following worked.
ssh -V afterwards gave me:
OpenSSH_9.6p1, OpenSSL 1.1.1g FIPS 21 Apr 2020

well darn, not working for me

…

________________________________ From: andrew1q9o ***@***.***> Sent: Wednesday, March 6, 2024 8:59 AM To: andrew1q9o ***@***.***> Cc: Comment ***@***.***> Subject: Re: roommen/OpenSSH Update Script - Amazon Linux 2 @andrew1q9o commented on this gist.

________________________________ I literally just did the yum install above and ./configure and everything following worked. ssh -V afterwards gave me OpenSSH_9.6p1. — Reply to this email directly, view it on GitHub<https://gist.github.com/roommen/18cd78d07b0fbc962de4e79c1d468f92#gistcomment-4973073> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACQI2BXQ7NSAQSMX2OAU7NLYW4ONNBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVA4TOMBTHA3TINFHORZGSZ3HMVZKMY3SMVQXIZI>. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

Ah well, I am not having any luck. I even renamed openssl to openssl.orig and openssl11 to openssl so that when openssl is run, it uses the newer version. The configure command still returns:

checking for openssl... /usr/bin/openssl
checking for openssl/opensslv.h... yes
checking OpenSSL header version... 100020bf (OpenSSL 1.0.2k 26 Jan 2017)
checking for OpenSSL_version... no
checking for OpenSSL_version_num... no
checking OpenSSL library version... configure: error: OpenSSL >= 1.1.1 required (have "100020bf (OpenSSL 1.0.2k-fips 26 Jan 2017)")

I did have errors trying to install openssl11-devel as it said that openssl-devel was needed for openssl 1.0.2. I told it to ignore that to get it to install.

After a period of research, here are some conclusions(which may be useful to you).

Checking configure help doc

./configure --help |grep 'ssl'
  --without-openssl       Disable use of OpenSSL; use only limited internal crypto **EXPERIMENTAL**
  --with-ssl-dir=PATH     Specify path to OpenSSL installation
  --without-openssl-header-check Disable OpenSSL version consistency check
  --with-ssl-engine       Enable OpenSSL (hardware) ENGINE support

As you can see, here is a param --with-ssl-dir=PATH that can be used for specify openssl path.

Dive into openssl11

# Install yum-utils
sudo yum install -y yum-utils
# Download the rpm package
sudo yumdownloader openssl11 openssl11-devel
# Check out the package
rpm -qpl openssl11-1.1.1g-12.amzn2.0.20.x86_64.rpm
#> /usr/bin/make-dummy-cert
#> /usr/bin/openssl11
#> /usr/bin/renew-dummy-cert
#> /usr/share/doc/openssl11-1.1.1g
#> /usr/share/doc/openssl11-1.1.1g/FAQ
#> /usr/share/doc/openssl11-1.1.1g/Makefile.certificate
#> /usr/share/doc/openssl11-1.1.1g/NEWS
#> /usr/share/doc/openssl11-1.1.1g/README
#> /usr/share/doc/openssl11-1.1.1g/README.FIPS
#> /usr/share/licenses/openssl11-1.1.1g
#> /usr/share/licenses/openssl11-1.1.1g/LICENSE
#> /usr/share/man/man1/openssl11.1.gz
rpm -qpl openssl11-devel-1.1.1g-12.amzn2.0.20.x86_64.rpm
#> /usr/include/openssl
#> /usr/include/openssl/aes.h
#> /usr/include/openssl/asn1.h
#> /usr/include/openssl/asn1_mac.h
#> /usr/include/openssl/asn1err.h
#> ....

The path /usr/include/openssl is the path that you should fill into above.

Conclusion

Run before: sudo yum install -y gcc openssl11 openssl11-devel zlib-devel mlocate autoconf

Try: ./configure --with-ssl-dir=/usr/include/openssl

BTW, The above method has been verified to work on a brand new AL2 system.

If your colleague also decided to follow these instructions on your day-off, and now finds themselves in a situation where the OpenSSH server is almost inaccessible anymore for new connections (e.g. ssh your-server-ip returns with error "kex_exhcnage_identification: read: Connection reset by peer"), follow my instructions to restore everything back.

Assuming you (or your coworker) still have an open SSH connection and can run commands in the shell:

1. Set a secure password for the root user using sudo passwd (you can use tools like pwgen to generate a strong password).
2. Log into the AWS console, navigate to your EC2 instance, select it, then go to Actions → Monitor and troubleshoot → EC2 serial console → Connect
3. Press Enter a few times in the large black rectangle, and you'll get a login prompt. Type root, press Enter, and then enter the password generated in step 1
4. Stop the SSH server and terminate all connections by running systemctl stop sshd; killall sshd
5. Navigate to the openssh server source code directory, most likely it would be cd /home/ec2-user/openssh-9.1p1
6. Run make uninstall to clean up all the files installed by the previous "make install" command (please avoid doing this on any non-personal machines in the future).
7. After step 6, you'll no longer have SSH client & SSH server on your system, even though the system package manager still thinks OpenSSH is installed. Reinstall the SSH-related packages to restore everything: yum reinstall openssh openssh-server openssh-client
8. Finally, start the SSH server again by running systemctl start sshd, and check if you can connect to the server from the server itself by ssh localhost.

P.S. If upgrading is necessary to make npm work (due to the unsupported option "accept-now"), you can use this workaround by setting an environment variable: GIT_SSH_COMMAND=ssh npm i ... (Source)