Regarding this tweet:
Which links to this article:
Which led to some thoughts and questions in my mind about the future of open source systems projects.
Here are some assertions about a class of great open source produced over the past 10 or so years:
It seems to come out largely from developers in big but not usually the biggest companies that have large technology teams (specifically: not necessarily "pure" tech companies). See: Kafka from LinkedIn, Envoy from Lyft, Airflow from airbnb, Mesos from Twitter, and a ton of stuff from Netflix
It was built to meet a specific internal need that developers in the company were facing. It was not built with the intention of being turned into a company, or to sell to customers.
Open sourcing seems initially aimed to: drive brand awareness for hiring at the company in question, give back to the community, reward the developers themselves who worked on it with a degree of industry recognition. Open sourcing is most specifically not done to get more customers for the company itself. (Some will claim that the open sourcing of some of the technologies was done as "trojan horse" exercises to distract other companies with a product that was never actually used internally. I don't believe anyone ever intentionally did something like this, although there are definitely products that caused that outcome for some of their early adopters).
For a lot of these products even to this day, the only way the developers ever got compensated for their innovative work was through the salaries and equity they got from their employers, and the name recognition that made them more desirable to be hired. But a few of the major projects (Kafka and the Hadoop ecosystem being easy examples) decided to go the venture capital route, creating startups (and eventually public companies) built around these technologies. Ideally, this allows the products to be developed more reliably, funds an ecosystem of experts, consultants, trainers who can teach other developers how to effectively use and run these projects, and most importantly provides major rewards for the creators of the projects that are tied to the success of the project itself in a core way, and not just the success of the business that, as a side effect, produced the project.
Along comes the cloud. Now, fewer and fewer people are actually taking this open source software and running it themselves. Sure, the cloud providers employ a lot of engineers, but overall they shrink the demand for widespread knowledge about the operational details of these projects. Furthermore, they capture the lion's share of the value that otherwise might have gone to the creators of the project if they were able to successfully spin the project into a company.
As the article linked above says, the cloud providers are incentivized to hire the people who created these successful projects in the first place, or the major contributors. And they do pay well, perhaps on average a better outcome for many contributors to these projects than any startup-driven outcome might be. Economically, this might be overall good for our open source creators.
But where will the next generation of open source systems come from, in this world? The open source software being created from the ground up by these cloud providers and big company consortiums frequently has complexity commensurate with the scale and resources of its creators. The beauty of many of the successful projects is that they were purpose built to a pretty specific use case, one that just happens to be widely shared in the industry. They started with clear focus and a small customer base they needed to please. We're seeing a ton of projects being built the opposite way, trying to address a perceived need in the industry (or, in my mind, worse: trying to take a piece of a successful ecosystem from a very large and extremely scaled tech company and make it useful for external developers outside of that ecosystem, thus generally violating Gall's Law).
If most companies are completely reliant on the cloud for all of the software that requires operations in some way, I wonder whether we will really see much new open source creation in this space at all. Sure, we'll continue to see new OSS front-end frameworks, and libraries, and languages, and all of that is extremely valuable, but we're going to lose the stuff that made being a distributed systems engineer in the wider world fun over the past 15 years. And I believe that we will lose out in innovation and usability, the kind of work that comes from people who are trying to solve a specific need for a small group of people.
This feels especially dangerous to me as all of the big-3 cloud providers (Google, Amazon, Microsoft) were not initially built on a bedrock of open source contribution. They have all done their fair share of contribution, but for their size, they've been slower than many smaller companies to earnestly give back important pieces of technology, perhaps because their core technologies created a competitive moat that was not the case for smaller companies. I am skeptical of companies who don't really have open source in their founding DNA to do the best thing for the community in the long-run, and I worry that the cloud is just moving us back to a world of proprietary software. Hell, many of these providers are just providing open source API compatibility with custom-built backends! What happens when no new open source comes out of the smaller companies, and the big-3 decide they don't really need or want to play nice anymore?