make `hydrogen` work with remote kernel

EXECUTABLE_MAGIC

#!/usr/bin/env bash
local_json=$1
remote_ip=$2
remote_json=$3
remote_json_on_local=$(jupyter --runtime-dir)/remote.json

scp $remote_ip:$remote_json $remote_json_on_local
for socket_name in hb_port control_port stdin_port iopub_port shell_port; do
  local_port=$(sed 's/,/\n/g' $local_json | grep $socket_name | grep -o "[0-9]\+")
  remote_port=$(grep $socket_name $remote_json_on_local | grep -o "[0-9]\+")
  ssh $remote_ip -f -N -L $local_port:localhost:$remote_port
done

hydrogen_remote_kernel.md

small example of why atom.io is so freaking awesome...

ok... it could be a lot better, but oh well, it's a good start. Also, note that this might not be safe?! because this is a kernel with --no-secure enabled (someone needs to clarify this for me cause I'm no expert)

context

I have this office computer that I connect to (with VPN, but these are details...) using ssh and do python work usually. Lately I discovered this atom.io text editor that comes with an awesome plugin called hydrogen. Naturally I asked myself (and others) if it's possible to connect hydrogen to a remote kernel... well it turns out that after countless nights spent searching (2 to be exact) I have finally found a solution! It isn't perfect, but hey... it does the job

kernelspec & bash script

The way hydrogen works is... if it doesn't find a running kernel around (on the local machine), then it opens up a process, but the good thing is that it actually looks for kernelspecs before it does that! which is why this method works.

So, the 1st step is to create a folder in $HOME/.local/share/jupyter/kernels (I call it fionn), cause that's the name of the remote machine I'm connecting to. Inside of this folder create a file called kernel.json with the following:

{
  "display_name": "SOME NAME HERE",
  "argv": ["EXECUTABLE_MAGIC", "{connection_file}", "REMOTE_PC_IP_ADDRESS",
           "PATH_TO_JUPYTER/IPYTHON_conn.json"],
  "language": "python"
}

I'll explain PATH_TO_JUPYTER/IPYTHON_conn.json further down in the notes section

2nd step is to create the executable file EXECUTABLE_MAGIC in a folder where the $PATH environment variable points to, for example I keep these things in $HOME/.local/bin and I point my $PATH variable to it (in $HOME/.profile I have: export PATH=$HOME/.local/bin:$PATH). This is the EXECUTABLE_MAGIC file:

#!/usr/bin/env bash
local_json=$1
remote_ip=$2
remote_json=$3
remote_json_on_local=$(jupyter --runtime-dir)/remote.json

scp $remote_ip:$remote_json $remote_json_on_local
for socket_name in hb_port control_port stdin_port iopub_port shell_port; do
  local_port=$(sed 's/,/\n/g' $local_json | grep $socket_name | grep -o "[0-9]\+")
  remote_port=$(grep $socket_name $remote_json_on_local | grep -o "[0-9]\+")
  ssh $remote_ip -f -N -L $local_port:localhost:$remote_port
done

3rd and final step! Connect yourself to the remote machine and execute ipython kernel --no-secure -f conn.json. Now lo and behold, you have a fully fledged atom.io text editor with interactive python that executes on a remote machine! Good job, give yourself a pat on the back. And this is all thanks to hydrogen let's not forget... now do remember that this might not be entirely safe because of the --no-secure option?! (need clarification on this).

Enjoy!, but be careful!

notes

• I have jupyter, ipython and all their goodies, dev. version 4, but I think should work with older versions as well. well... actually, jupyter --runtime-dir only works with jupyter... so that's that, if you have an older version of ipython, then you need to use another command here, or just specify the full path manually...
• this obviously only works on linux cause of the command lines etc. (linux is awesome :D)... probably works on osx, I have no idea how to set this up on windows
• kernelspecs: jupyter stores the kernelspec files in $HOME/.local/share/jupyter/kernels (that is, ipython dev. 4), whereas the other versions of ipython keep their files in $HOME/.ipython/kernels (I only work with local files so don't ask me about system folders!)
• PATH_TO_JUPYTER/IPYTHON_conn.json: when you execute ipython kernel -f connection_file.json, ipython will create a file called connection_file.json in $HOME/.ipython/profile_default/security (in version 3.2, probably others too) and in /run/user/1000/jupyter/ or $HOME/.local/share/jupyter/runtime (in version 4), depending if it has permission to use the /run folder or not. (you need to experiment with this). Well, PATH_TO_JUPYTER/IPYTHON_conn.json is exactly the path to this connection_file.json (on the remote machine), for example in my case it is: $HOME/.local/share/jupyter/runtime/conn.json (because I named it conn.json).

kernel.json

{
  "display_name": "SOME NAME HERE",
  "argv": ["EXECUTABLE_MAGIC", "{connection_file}", "REMOTE_PC_IP_ADDRESS",
           "PATH_TO_JUPYTER/IPYTHON_conn.json"],
  "language": "python"
}