Now that I've committed a few of my experiences launching git.ht to a written word, a pattern has emerged in a way I approach DevOps. By happy coincidence it also acquired a name of its own. As I was writing episode 3 in the series, I guess, my brain must've gaffed and so the term TailOps was born, where clearly I meant to write DevOps. Tailscale has become a vital part of GitHoot's infrastructure that ties everything together - only natural that my mind slipped. I'm having trouble resetting my brain, so I guess I'm stuck with it. It got me thinking. It is admittedly fuzzy and ill defined and likely to remain thus. Wouldn't hurt to explore and reflect some. Then, who knows, TailOps might just become a thing and start a life of its own.
Let's make this hoot one, where I indulge in philosophy, make hand-wavy statements and share observations supported by like a single data point. My hoot - my rules. Here goes.
Life after Tailscaling git.ht
Some of these nagging feelings I already expressed here and there in the series. Most prominently in the Ranty preamble to Tailscaling git.ht - which see. Gist of it is that git.ht is a smallish endevour - project conceived, executed and maintained by a single engineer. Yours truly. With kids and other obligations. Out of sheer necessity I had to prioritize and optimize for a single person maintainance.
My hard-earned experience with GitHoot and many contract gigs before it led to a somewhat counter-intuitive conclusion, at least for me: more often than not, you'd be better served by not reaching for off the shelf hosted services of which even AWS itself offers a non-trivial amount - they've been copying and eating competition for years now. Sure, some of it has to do with costs. Cloud solutions are inevitably pricey and we aren't terribly good at bean counting when it comes to the cloud. Part if it is that people aren't (yet) calibrated to think in units and metrics being priced and they can't quite relate to their own experience with computers, where price is usually capped and parameters are few. This price per unit is difficult to attribute to anything - why that much - what will the total be - it may as well be random. Cost aside, my main gripe with ubiquitous move to the cloud regardless of business size is the cognitive burden. Learning new tools and ways of doing things is taxing to say the least. Simply enumerating all the services AWS alone provides will take forever let alone assimilating them into ones head and hands. Getting familiar if not comfortable, gaining a bit of confidence takes deliberate effort, breaking and fixing things - it's a journey. It really is, every time. To what end? Remember these aren't built with whatever problem you face. They can't be. They have to be general(ish), built for the average case and ideally make the most common case easy. In reality, they're often best suited for the cherry picked examples in their respective manuals. By the time you get to use them in anger and notice the impedence mismatch, it is usually too late to switch, reconsider or go back to the drawing board. Looming deadlines, sunken costs, sheer stubborness because you think not seeing it to the end reflects badly on you as a professional. Oftentimes you'll have goal posts move and shadow the actual problem. Tell signs are when people talk more about tools, techniques, patterns and architecture. Oh and meetings - gotta schedule those.
In my experience it takes about as much time to build a solution to the problem you have from first priniciples (please, assume reasonable level of abstraction) as it would trying to fit various off the shelf hosted solutions. Asymptotically - considered over the life of your product or service. Counter-intuitively this is universally due to them trying to aleviate the pains involved in building things. Systematic pains - ones inextricable from the problem - ones that come with the territory. These magical solutions tend to paint over the issues you can't really remove, so they, too, shift goal posts and solve minor annoyances. That'd be nice, were it not for them hiding the very things you ought to consider and pay attention to. You'll discover them eventually. Then it gets worse. Things that promised to just work, say give out of the box replication and failover and what not, need to be reversed - you find yourself in need of understanding exactly how they function. These parts are rarely documented or explained. Your progress slows down, sometimes stops. Ugly hacks begin to pile up.
Consider time it took to build all these magical hosted solutions? More often than people like to admit, devs end up reversing that magic and work around the exact magic that let them move fast early on. Consider this then. It would take mere fraction of that time to document the problem domain carefuly and in detail. Put together a few representative examples but otherwise leave developers to their own devices: here's how DNS failover might work for your load-balancing needs; here's how you may want to replicate your PostgreSQL and the problems you'll likely encounter. Problem, of course, you can't charge money for it. Well, I think we could, but everyone looks to build the next unicorn and can't be satisfied with petty million or two a year.
By now, I think, it is safe to say that DevOps has become synonymous with cloud infrastructure. For better or for worse. Long gone are the days of sysadmins minding the cabinets with servers at the end of the hall. I mean, on-prem remains a thing, but if job ads are any indication, it is all AWS, GCP, Azure, Kubernetes, Docker, etc. Gotta be able to write syntactically correct Yaml by hand on first try. Am I right? Like I mentioned earlier AWS bills can be eyewateringly exorbitant, but even that aside, it is no longer realistic to expect peope to just know how to stitch cloud solutions together when it takes years to master just a few. Not to mention that knowing the latest and greatest doesn't necessarily absolve you from the need to have a working model of what happens under the hood. Unfortunate consequence of what I just mentioned, among other unpleasant things, is the budding wannabe entrepreneur engineer may need a decade of experience minding all of these services in the trenches for someone else, or be funded sufficiently to afford external help. I wonder how many businesses never see the light of day because of it? Not unicorns, but, you know, respectable tiny shops that afford you a comfortable living. I'm being somewhat unfair in drawing such black-n-white picture for you here. Reality, as always, is grey(ish). With some ambition and stubbornness, especially when you are building something you absolutely believe the world can't live without, you'll learn and you'll do it way faster than any curricula could ever hope to teach you. That's just how it is. You could also spend the energy and time required to master these buzzwordy-services on actually solving the exact problem you face and pick up the necessary fundamentals of, say, networking or database-management. Which is better? Looking back I'd say the latter, but - there is always "but" - the fundamentals don't stand out in your CV. Buzzwords on the other hand make HR clerk, sorry, VP of talent aquisition, heart race. Sadly, the first step of landing a job often remains that of passing that first filter. I am, therefore, hesitant to push my worldview onto the young and impressionable. World is unfair and rarely efficient.
There is a point I'm making. It is that cloud has gone off the rails and is no longer, IMO, feasible path for a small guy like me. Not like I couldn't make heads or tails of it, I totally could and I have done, but the payoff is no longer obviously beneficial or sufficiently beneficial to be invested in it. You get a dopamine release by launching some service or other "in the cloud" - get a database running at the click of a button - and it seems like you're going so fast on day one, except that day 1 or month 1 is going to be dwarfed by all of those other countless things you need to figure out and marry to your business. Either you must become a DevOps person yourself or have one (or a team) on hand. If you think either of these options is cheap, well, I have a bridge to sell you.
Bad news - overall complexity of running anything has grown tremendously over the recent years. Good news - you probably don't need as much and what you do need exists and you can often build your own abstractions so the moving parts compose exactly the way you need them to. Stop. Have you noticed how I just tricked you? The good news as stated kinda invalidates the bad news as stated. It's actually gotten a lot easier to run almost anything, but you wouldn't know in all that DevOps noise. Much of said noise had been born out of Google had done some silly shit early on cause they were in a hurry and, you know, webscale and then tried to fix it. You're now being sold a bunch of services and tools that solve earlier services and tools ad infinitum almost. Unless you recognise how little all this crap has to do with your business and engineering problems you face, you'll struggle. It takes good sense and painful experience or incerdible intuition for being scammed and bravery to pave your way.
Also. I don't want to belabor the point but chances are computational resources you have readily available at home right now are superior to whatever you can get in the cloud at reasonable price. That's not to say that cloud is useless and you should avoid it. Far from it. But it may pay (in more sense than one) to think of the likes of AWS as nothing more than utility companies. They provide you with necessary building blocks: EC2 instances, DNS via Route53, S3 object storage, block storage (stupid expensive). Crucially they keep you online 24/7. I wish that latter thing wasn't necessary but it looks as though IPv4 is here to stay and we'll be NATing everything to the end of times.
On another note. AWS Console and its siblings at Google and Microsoft also make for invaluable learning tools - vastly undervalued for that. There is a rarely mentioned hack where a freshly minted company can apply for AWS startup credits and get USD 1000 worth for a year. Not much if you go with hosted everything, but way more than you need to learn and play and even build with just the above mentioned building blocks, where much of compute happens on prem.
Missing piece of the puzzle is the fabric that connects the two worlds: your closet and cloud environment. Tailscale or ZeroTier provide exactly that. You could also challenge yourself and build your own atop Wireguard. Point is, there's little reason to not do it. There is every reason to leverage what you have at home seeing how pulling on-prem and cloud resources together isn't that difficult any more. Now there is one more mighty compelling reason ...
In case that's not obvious - picture on the right (or above if you're on mobile) is TailOps according to DALL-E. SkyNet has ways to go, I guess. Anyway.
I like control. I am also capable of some self-reflection. I therefore realise I maybe a tiny bit attached to the idea of running everything yourself, knowing the inner workings of it all i.e. retaining control. Setting this aside for a moment. I trust you've been keeping tabs if not riding the latest AI hype wave. I guess, the age of ChatGPT is upon us. I'm not going to dwell on it cause it is very much it's own topic, but it does bring about a couple of thoughts in the context of this hoot. First, might just be that GPT assistants get enough smarts to remove much of the complexity or even the need to master cloud services - that is you describe what you want and they, I don't know, spit out a CloudFormation json or Terraform recipe or something. This raises the level of abstraction quite significantly, many of the Cloud services, will inevitably start catering to "the machine" assuming their APIs are to be fed to GPT. Step further and we could be looking at "agents" managed by said GPT "clicking" around in the AWS console (a cartoonish picture but the gist of it is apt). Your task then becomes two-fold (a) provide a coherent prompt with sufficient details, and (b) evaluate the result and confirm it achieves what you asked without going astray - check edge cases, etc. That would be quite cool actually. We kinda suck at programming what with our monkey brains being unable to not f..ck up and pay attention to details. We humans are quite good at quickly judging if something is working as expected or not. It's a little bit like judging a photograph or a painting. We can tell it is a beautiful shot but without training can't explain why. With a bit of machine learning built into your phone camera you no longer need to know why. So it may become with DevOps and such. Ok, now to the second point. With the rise of GPT completions the pull into the cloud and subscription services will increase. It happens even as we speak. I don't want to be the person who brings up the Orwellian nature of what's about to happen, so let me just stick with the monetary side of it. You can and in my opinion absolutely must push towards running inference locally. Get a dedicated machine, just for that, and put in your closet. I'm not suggesting you train multi-billion param models - you won't have compute for that. But running pre-trained models locally is entirely feasible. You may not even need expensive GPUs or a lot of RAM for that - can run on CPUs and those are relatively cheap. Surprisingly, it'll be fast enough especially if you have a multi-core machine with enough memory (quantized you may not even need much). It opens up some interesting oportunities and this leads to, you guessed it, TailOps - something you build and make available for yourself, your business, your family, your fellow hackers. Can be a Mac mini on your desk or a second-hand Dell R730 with 36 cores and 256GB of RAM in your closet - fact is - it is on prem. Fact is, I've had stable 500Mbps downlink and uplink capped at 75Mbps in my house for years and recently had another fibre link installed delivering 3Gbps down and up - I don't have equipment in the house capable of handling this speed. What reason is there to take to the cloud any more? For me. For me.
I'll be honest. Picture being painted right now is looking grim. It is in your hands to brighten it. Get excited and go rogue. Do TailOps.
As always this hoot is nothing more than a GitHub gist. Please, use its respective comments section if you have something to say. Till next time.
I'm running a (mostly) Clojure SWAT team in London at fullmeta.co.uk. We are always on the lookout for interesting contracts or fulltime gigs. Can perform competently in modern Java, TS/ES, Go. Can target JVM, V8 and Node, .Net, Erlang BEAM. Now that git.ht is up, which you should go ahead and check out rigth now, I am back to CTOing and contracting. If you're hiring, check out my CV and shoot an email to vlad at fullmeta.co.uk.
