Skypipe is a magical command line tool that lets you easily pipe data across terminal sessions, regardless of whether the sessions are on the same machine, across thousands of machines, or even behind a firewall. It gives you named pipes in the sky and lets you pipe data anywhere.
Skypipe is sort of like named pipes and netcat, but with even more power and a simpler interface. Here is a simple example using skypipe like you would a regular named pipe in order to gzip a file across shells:
$ skypipe | gzip -9 -c > out.gz
Your skypipe is ready to receive some data from another shell process:
$ cat file | skypipe
Unliked named pipes, however, this will work across any machines connected to the Internet. And you didn't have to specify a host address or set up "listen mode" like you would with netcat. In fact, unlike netcat, which is point to point, you could use skypipe for log aggregation. Here we'll used named skypipes. Run this on several hosts:
$ tail -f /var/log/somefile | skypipe logs
And then run this on a single machine:
$ skypipe logs > /var/log/aggregate
This can also broadcast to multiple hosts. With the above, you can "listen in" by running this from your laptop, even while behind a NAT:
$ skypipe logs
You can also temporarily store data "in the pipe" until you pull it out. In fact, you can keep several "files" in the pipe. On one machine load some files into it a named skypipe:
$ cat file_a | skypipe files
$ cat file_b | skypipe files
Now from somewhere else pull them out, in order:
$ skypipe files > new_file_a
$ skypipe files > new_file_b
Lastly, you can use skypipe like the channel primitive in Go for coordinating between shell scripts. As a simple example, here we use skypipe to wait for an event triggered by another script:
#!/bin/bash
echo "I'm going to wait until triggered"
skypipe trigger
echo "I was triggered!"
Triggering is just sending an EOF over the pipe, causing the listening skypipe to terminate. We can do this with a simple idiom:
$ echo | skypipe trigger
What I didn't mention is that you need a free Dotcloud account to use skypipe, but you don't need to know anything about using Dotcloud to use skypipe. The trick to this private pipe in the sky is that when you first use skypipe, behind the scenes it will deploy a very simple messaging server to Dotcloud. Skypipe will use your account to transparently find and use this server, no matter where you are.
This all works without you ever thinking about it because this server is managed automatically and runs on Dotcloud for free. It might as well not exist!
This is a new paradigm of creating tools that transparently leverage the cloud to create magical experiences. It's not quite software as a service, it's software with a service. Nobody is using a shared, central server, and nobody needs to setup or manage their own server. The software deploys and manages its own server for you to use without ever seeing it.
Thanks to platforms like Heroku and Dotcloud, we can now build software leveraging features of software as a service that is packaged and distributed like normal open source software.
I'm excited to see what else can be done with this pattern. I've already got a list of more ideas myself.
Skypipe is still an alpha piece of software. There are some rough edges. For now, you should be able to install skypipe with pip directly from master:
$ pip install -e git+git://github.com/progrium/skypipe.git#egg=skypipe
Skypipe is brand spanking new, so the experience is not entirely optimized. For example, one of the biggest issues I have with it is that it needs to check for the server on every use. This can be done less often and cached, which would make it much snappier and on par with most command line utilities.
This and a few other issues are already tracked in Github Issues, so feel free to take a whack at them. The codebase is intentionally very small, documented, and written to be read, although there are no tests yet.
The project also depends on ZeroMQ, which requires a C extension to be compiled. Even using the pyzmq-static package, you still need certain header files (python.h at the very least) to install skypipe, and not every environment has these. Instead of removing ZeroMQ, I'd much rather find a way to package skypipe in a way that includes all its dependencies. Perhaps PyInstaller can help with this.
Contribution ideas aside, hopefully skypipe is handy to more people than just me, and I hope people see what I mean when I say it's a magical tool. I think a big part of this magic is the use of "software with a service", which I consider a bit novel in itself. What do you think?