As AWS are going to start charging for all public IPv4 endpoints I have started to look at how to rotate out as manu instances as possible but still have the same mechanisms to access and manage them, ie ssh.
I could leave the Bastion and connect to everything on the private network via the Bastion that's much the samed as now except re-rolling all the instances to remove the public interface. There is EC2 Instance Connect which AWS have just beefed up and is available for AWS Linux 2 and Ubuntu so I tried that.
It has all the convolutions of any other soft networking solution and further commits the sin of requiring aws commands to do the usual work of setting up a tunnel. This is presuming that each connection is a new tunnel from the client to the instance, there's no Bastion so the controls in the Security Group need to be tied down.
Everything seemed mostly ok until I tried to actually connect to an instance, this was with using one of the few examples in the documentation. These examples are all command line convolutions there's no use of ssh config files. So I created a copy of my regular connect to an AWS instance with the amendments, and it failed. Sometimes it appeared to connect gave the banner and then rejected me usually with a permissions layer.
I went to my alternate account which had a couple of experiments, cleared everything out and created a new endpoint, checking it was the right subnet and the right availability zone.
I created an entry in ~/.ssh/config
Host EICTest Hostname 172.31.34.205 IdentityFile ~/.ssh/pringle.can@g30 User ec2-user@i-04fb53e42a9d27435 ProxyCommand aws ec2-instance-connect open-tunnel --instance-id i-04fb53e42a9d27435
And it failed in the same way.
Then wondering how I could actually check connectivity using the same routing I thought I could try to push a public key to the instance using
aws ec2-instance-connect send-ssh-public-key --region eu-west-2 --availability-zone eu-west-2b --instance-id i-04fb53e42a9d27435eictest --instance-os-user ec2-user --ssh-public-key file://c:\Users\slucy\.ssh\pringle.can@g30.pub
And that worked so using EIC I could connect to the instance. Thinking more about the error using ssh I thought of the ways that could barf, one was certainly permissions on .ssh and the content of it but I knew that was fine in the existing instance I'd tried and that one of the other causes of failing was the wrong user name.
The example was odd using the instance id as the right hand side of the @, usually I just use the generic default user name ec2-user. So I hacked that off in the config file and tried again and it just worked.
Which was both annoying and pleasant.