Now that we have moved to a mostly Docker-based infrastructure, one of the tricky things is to try and debug things when there is something pear-shaped. It used to be possible to just SSH into the machine with a local port-forward, and then for example access the Elasticsearch server via a handy browser extension to do debugging.
But what to do if your container is running in a Docker Swarm and has no ports forwarded by default? (which is the right thing to do, keep it simple and closed...) Thanks to stirling help from https://github.com/eelkevdbos here is the solution, and I am writing it up here so I can remember it in future, cause I sure am gonna forget the details...
First thing, create a new docker overlay network that you can use for getting to the container in question:
docker network create foobar
Then find the container you want to connect to:
docker ps | grep <somename>
And attach it to the new network you made:
docker network connect foobar <somecontainerid>
Then check the container to see what it's IP address in the new network is:
docker inspect <somecontainerid> | less
The IP address for that network will be somewhere towards the end...
And now you can use that IP to launch a mapped port to "the outside world".
docker run -ti --rm --net management -p 127.0.0.1:9200:9200 bobrik/socat TCP4-LISTEN:9200,fork,reuseaddr TCP4:172.26.0.2:9200
Note, the IP address above, 172.26.0.2 was the internal one assigned to the foobar network, yours will be different.
Be careful if your host does not have a working configured firewall that you don't expose ports unwittingly, but if the above works, log into your machine with a command like:
ssh -L 9200:localhost:9200 yourmachinename
And you will be able to test your ElasticSearch machine from your local machine (or other services)