Questions on the future of Open Source

gistfile1.md

Regarding this tweet:

https://twitter.com/skamille/status/1106955368901550082

Which links to this article:

https://redmonk.com/sogrady/2019/03/15/cloud-open-source-powder-keg/

Which led to some thoughts and questions in my mind about the future of open source systems projects.

Here are some assertions about a class of great open source produced over the past 10 or so years:

1. It seems to come out largely from developers in big but not usually the biggest companies that have large technology teams (specifically: not necessarily "pure" tech companies). See: Kafka from LinkedIn, Envoy from Lyft, Airflow from airbnb, Mesos from Twitter, and a ton of stuff from Netflix

2. It was built to meet a specific internal need that developers in the company were facing. It was not built with the intention of being turned into a company, or to sell to customers. 

3. Open sourcing seems initially aimed to: drive brand awareness for hiring at the company in question, give back to the community, reward the developers themselves who worked on it with a degree of industry recognition. Open sourcing is most specifically not done to get more customers for the company itself. (Some will claim that the open sourcing of some of the technologies was done as "trojan horse" exercises to distract other companies with a product that was never actually used internally. I don't believe anyone ever intentionally did something like this, although there are definitely products that caused that outcome for some of their early adopters).

For a lot of these products even to this day, the only way the developers ever got compensated for their innovative work was through the salaries and equity they got from their employers, and the name recognition that made them more desirable to be hired. But a few of the major projects (Kafka and the Hadoop ecosystem being easy examples) decided to go the venture capital route, creating startups (and eventually public companies) built around these technologies. Ideally, this allows the products to be developed more reliably, funds an ecosystem of experts, consultants, trainers who can teach other developers how to effectively use and run these projects, and most importantly provides major rewards for the creators of the projects that are tied to the success of the project itself in a core way, and not just the success of the business that, as a side effect, produced the project.

Along comes the cloud. Now, fewer and fewer people are actually taking this open source software and running it themselves. Sure, the cloud providers employ a lot of engineers, but overall they shrink the demand for widespread knowledge about the operational details of these projects. Furthermore, they capture the lion's share of the value that otherwise might have gone to the creators of the project if they were able to successfully spin the project into a company. 

As the article linked above says, the cloud providers are incentivized to hire the people who created these successful projects in the first place, or the major contributors. And they do pay well, perhaps on average a better outcome for many contributors to these projects than any startup-driven outcome might be. Economically, this might be overall good for our open source creators.

But where will the next generation of open source systems come from, in this world? The open source software being created from the ground up by these cloud providers and big company consortiums frequently has complexity commensurate with the scale and resources of its creators. The beauty of many of the successful projects is that they were purpose built to a pretty specific use case, one that just happens to be widely shared in the industry. They started with clear focus and a small customer base they needed to please. We're seeing a ton of projects being built the opposite way, trying to address a perceived need in the industry (or, in my mind, worse: trying to take a piece of a successful ecosystem from a very large and extremely scaled tech company and make it useful for external developers outside of that ecosystem, thus generally violating Gall's Law). 

If most companies are completely reliant on the cloud for all of the software that requires operations in some way, I wonder whether we will really see much new open source creation in this space at all. Sure, we'll continue to see new OSS front-end frameworks, and libraries, and languages, and all of that is extremely valuable, but we're going to lose the stuff that made being a distributed systems engineer in the wider world fun over the past 15 years. And I believe that we will lose out in innovation and usability, the kind of work that comes from people who are trying to solve a specific need for a small group of people. 

This feels especially dangerous to me as all of the big-3 cloud providers (Google, Amazon, Microsoft) were not initially built on a bedrock of open source contribution. They have all done their fair share of contribution, but for their size, they've been slower than many smaller companies to earnestly give back important pieces of technology, perhaps because their core technologies created a competitive moat that was not the case for smaller companies. I am skeptical of companies who don't really have open source in their founding DNA to do the best thing for the community in the long-run, and I worry that the cloud is just moving us back to a world of proprietary software. Hell, many of these providers are just providing open source API compatibility with custom-built backends! What happens when no new open source comes out of the smaller companies, and the big-3 decide they don't really need or want to play nice anymore?

Then what's your take on https://aws.amazon.com/blogs/opensource/keeping-open-source-open-open-distro-for-elasticsearch/. Though I do think there's a problem with some "open core" proponents at times pressure from VC can mean these end up restricting anything competing with their business model.

I think it’s great when major Cloud Providers support Open Source, but in reality, their messaging can be somewhat misleading.

The real underlying business case for OSS with cloud providers is to drive cloud consumption (they make all their revenue off the margin between hardware and cloud consumption) free software = more cloud consumption. It seems a little asymetrical, especially when OSS generates all the value that gets captured by the cloud providers.

The case of Elasticsearch is difficult, as, on the one-hand Elastic created the product, and should have some basic right to the intellectual property. On the other hand, I believe the world would be better off (from an innovation and security perspective) if all software was open source.

At the end of the day, it’s best to examine the business incentives, AWS wants elasticsearch open because it drives cloud consumption (hence profits) for Amazon. EC wants a piece of this pie, which is fair, as they built and marketed the initial software. A good middle ground might be if OSS creators could profit off the margin of cloud consumption in the same way that Big Cloud does.

The marriage between Decentralized Cloud swarms and Open Source might be a good pathway towards a better deal for OSS.