Using Erlang observer/appmon remotely

observer.md

Using OTP's observer (appmon replacement) remotely

$ ssh remote-host "epmd -names"
epmd: up and running on port 4369 with data:
name some_node at port 58769

Note the running on port for epmd itself and the port of the node you're interested in debugging. Reconnect to the remote host with these ports forwarded:

$ ssh -L 4369:localhost:4369 -L 58769:localhost:58769 remote-host

On your machine, start a hidden Erlang node running the observer app:

$ erl -name debug@127.0.0.1 -setcookie if-server-has-one -hidden -run observer

N.B.: If the server is using -sname, you need to use -sname as well, because Erlang™. You'll also need to change your system's hostname to match the target host, for the same reason.

In observer, go to Nodes - Connect Node and type some_node@127.0.0.1.

You can also get a remote shell:

$ erl -name debug@127.0.0.1 -setcookie if-server-has-one -remsh some_node@127.0.0.1

This technique works if some_node is bound to local interface on remote-host. Any idea how to use ssh tunnelling to connect remote node if it bound to other interface (e.g. eth0)?

Think you have to make sure GatewayPorts set to yes in the remote's /etc/ssh/sshd_config, then find out the address for the other interface so it looks like:
ssh -L 4369:other_interface:4369 -L 58769:other_interface:58769 remote-host

But that's only if you specifically bound the node to the other interface. You may only have bound your application to the other interface.

Note: if you want to observe an exrm packaged app, you have to include :runtime_tools in your applications list.

For the lazy ones:

slogin -v $ERL_HOST $(slogin $ERL_HOST "epmd -names"|grep -wo '[0-9][0-9]*'|sed 's/.*/-L&:localhost:&/')

should get you port forwards to everything that's running on the remote host.

I wrote a step by step blog post based on this gist.
https://sgeos.github.io/elixir/erlang/observer/2016/09/16/elixir_erlang_running_otp_observer_remotely.html

Just threw together a shell script that automatically forwards the necessary ports: https://github.com/Stratus3D/dotfiles/blob/master/scripts/tools/epmd_port_forwarder

Give it the server hostname and it does the rest.

./scripts/tools/epmd_port_forwarder --server=server_fqdn

This doesn't work for me :(

FYI this will not work if your remote nodes are named in the format of:

1. -name some_name@10.10.10.10
2. -name some_name@example.com

If:

• You have one of the above node name formats
• You do not require having multiple nodes connected through the Erlang distribution
• You do not require having a connection into your node through an EPMD port exposed through your firewall

Rename to -name some_name@127.0.0.1, and this guide should work.

This Stack Overflow answer explains why this is the case:

When an Erlang node tries to connect to another Erlang node, it needs to find the IP address of the distant node. If it is the same as itself, it will simply try to connect on the local host. If it is different, it will try to resolve this host name to an IP address.

Then it will connect to the epmd daemon on this host to be told which port Erlang is running. epmd as well as Erlang nodes listen on all interfaces (by default).

EDIT

If you have to have your remote node name in the format of -name some_name@10.10.10.10, in addition to the SSH port forward with -L, you can either do:

• SSH port forward with the argument -R
• Setup iptables or pf rules to redirect traffic from your local machine to 10.10.10.10:EPMD_PORT and 10.10.10.10:APP_PORT back to localhost. Here's a helpful StackOverflow post.

@jozuas do you have an example that's working with -R ?

I have epmd running on and the app on node 4369 28327

So this is the correct forwarding ?

ssh user@host -R4369:localhost:4369 -R28327:localhost:28327 -L4369:localhost:4369 -L:28327:localhost:28327

Silviu

@silviucpp I am not using -R option myself because using -R option requires sshd_config option GatewayPorts yes. This is something I did not want to immediately introduce into the production servers of my company.