An open letter to the Neos development team

openletter.md

An open letter to the Neos development team

Dear members of the development team, CU a few days ago I commented the announcement, that Neos and Flow get rid of "TYPO3" as part of their names, as follows:

Removing "TYPO3" from Neos and Flow makes it easier to quietly abandon these products. \o/

Additionally to this tweet I had a small twitter conversation resulting in this tweet:

@daskitsunet @dfeyer @zdavis I am really in favor of seeing Neos as an experiment. Start over with a worldwide accepted/documented framework

Later this day in the afternoon Christian Müller asked me on Slack where this resentment comes from and I told him that there is none because everything I say in favor of or against any product of the TYPO3 family is not in favor or against persons but decisions and failures. I am pretty much aware of the possible impact of negative comments for the moral of development teams, though. Unfortunately it's not easy to do the splits here and additionally to that fact, twitter is based on written words w/o facial expression, and mostly w/o an historical context as well. So, the meaning of tweets mostly aren't dependent on the author's real meaning, but the recipients understanding. That's the main flaw of written communication but that's the way it is. So each and everyone should avoid taking written words too emotional.

Having that said and as you are currently reading written words as well, this mentioned rule of thumb applies for this text as well. Feel free to get in contact with me if you like to have a discussion.

Let me continue with the conversation between me and Christian because we shortly talked about the essence of my tweets, seeing Neos and an experiment and start over with a worldwide accepted framework. Let me explain, why.

The tweet itself mostly explains why, because Neos and Flow are pretty much undocumented or their documentation state wrong information meanwhile and because the framework is absolutely not accepted. If you have a look at google trends and compare TYPO3 Flow to symfony2, cakePHP, laravel and many more, you see several different facts. See Google Trends:

• The search volume of Flow is limited to Germany. That means, that there are not enough search requests for Flow outside of Germany although its documentation is really bad.
• The search volume of Laravel is incredible. Knowing that it's documentation is up to date and awesome, the search volume impresses even more.
• Even CodeIgniter still finds a decent interest in this world despite that fact that it's really outdated.

I don't find it hard to explain this data to myself as I did projects with CodeIgniter, CakePHP, Ruby on Rails and Flow and it was really easy to develop with all of them, except Flow. Because, whenever I found myself struggling with the concept of a framework, I had a look at the documentation and I could continue my actual work. Whenever I worked with Flow I didn't have any documentation, well, despite the blog example, so I had to debug with xdebug, which gave me the creeps and then, when I understood the concept, hidden in the magic of some proxy class stuff, I often stubled upon bugs in the framework itself. And that's the moment you ask yourself what the actual purpose of a framework was. Making your life easier, getting things done more quickly. But all the time the opposite of that happened which may not be life threatening if you are employed but really can make your whole business fail if you are not into each and every implementation detail of the framework yourself before calculating the workload for a client.

Let me emphasize that I really don't have that much to say against the concepts of Flow and Neos. There are plenty of frameworks in the wild, so it's pretty easy to compare Flow to them and see that most stuff doesn't really differ. For example when it comes to MVC. The pattern is well known, even amongst PHP developers, so I can't say Flow does a bad job here. It's the same for Neos. The content repository approach for unstructured content data is not even new. The java world knows it for years now and even symfony started their CMF with a CR approach. Actually Neos was a pretty early adopter of this idea and I believe it's what content management should look like nowadays and in the future.

So, what a struggle with is the technical implementation of many concepts in Flow and Neos which I'd blame, along with the missing documentation, for the small acceptance in the world. I will not go into detail here – I'll cover that later – but I see huge flaws that do harm the whole system more than bring advantages. To name the elephant in the room. It's the messed up AOP proxy classing concept that nobody wants to work with unless his language of choice is java anyway, and he/she has the right tools for it. As I mentioned above, the debugging is really a pain in the ass. So, if proxy classing is the key feature of a framework to bring well, everything, from DI, over a lot of configuration, to the whole AOP implementation, then, it should better be bullet proof. I remember working with Flow in the end of 2013, finding a severe php class inheritance bug. I could only solve this issue with a lot of time and coffee, xdebug and the help of Christian Müller. This can easily be a major financial harm.

The next problem of a low acceptance of Flow and therefore a low contrbution is the fact, that Flow is not a framework build for the masses, but for Neos. It's just a guess but what is the key case for choosing a framework? Implementing REST-API's! To this day Flow and Neos do not rely on a proper HTTP implementation. There wasn't a need to properly imlement PUT requests for example because there was no need of Neos and no need in actual real life projects. I just used this example to demonstrate the main issues of Flow. Everything works perfectly according to Neos' needs, not to RFCs', resulting in a piece of software that can do a lot but is often not usuable due to missing implemented details.

Additionally to this flaw, Neos is completely build on top of the very unique concepts of Flow. There's no possiblity to port dedicated packages to symfony e.g. because these packages are tightly bound to Flow. It's not just changing a yaml configuration file, it's removing and replacing all annotations. Do you see what you did? You build not just one monolith, it's two, depending on each another. So, if Flow fails, Neos fails as well. That's a mess because if that happens, all of your precious time, that went into the good concepts, is simply lost. The fact alone that Fluid isn't available as a standalone version – well, Claus Due just came up with that –, shows the problems you created in your ecosystem. While the whole PHP world gets more open through composer, PSR's and the senseful decoupling of software, you did exactly the opposite.

Well, this post is not to blame you personally, I said that before. Instead I need to emphasize that in the current situation, blaming people is wrong in general. Over the years a lot of money has been spent, and a lot of time has been used for R&D. That's ok, because that's what you were told to do. Noone managed the process of creating a whole new CMS. You can see this by the fact that the community was surpised by getting a framework first, complaining about you guys and cutting you off the money flow. [UPDATE] There was a 50% cut of the budget application by the TYPO3 Association, a decision, influenced by the community. http://typo3.org/news/article/neos-and-flow-team-waive-typo3-association-budget/ Sure, people blindly put all their money and trust in you. That might be the biggest problem, the lack of control. But that lies in the past. And although we should learn from it, we should not be controlled by it. Yes, people wanted to see Neos 1.0 finally to be released and you did, which caused a lot of disappointment due to missing features. It's really weird, actually. But now, the effect of suprise is gone. Neos slowly follows the release cycle, people get used to bugs or don't use Neos at all. Some still do wait, some left the TYPO3 ecosystem at all, most stick to CMS.

This current situation is a tragedy and as you try to get more contributors, you loose trust because you can't deliver in time. As you loose trust, you loose possible contributors. Well, who wants to ride a possibly dead horse? It's a vicious circle you will most likely not escape. The community wants you to deliver, to make promises and reminds you of all the money you already got but what can you do? Nothing, you need to be relieved. That's why I am so much in favor of seeing Neos as an experiment, stop the development, use an accepted framework and start over, and not just you, but everybody in the community. Mix up the core teams, be more open and let fresh wind blow.

I don't expect anybody of you to follow my view of things. I just needed to explain why I personally see Flow and Neos as a failure and why that results in tweets like the one mentioned above. I am even very impressed you kept up the faith so long and still do. Don't let my words put pressure on you, but relieve you instead.

So, whatever you do, rock on!

Alex, thank you for speaking up. It is the right thing to do but nonetheless a brave thing to do.

In respect of that bravery I've now decided to release the quite similar open letter that I wrote in 2012 but never released. I've added a small introduction to give it some more recent context.

https://gist.github.com/NamelessCoder/e8c5467cfefea4f34df2

Since this was written I have personally made peace with this matter. I mentally treat the situation as I describe it in this letter - and I simply await the day the problem either goes away through fulfilment of requirements or abandonment of the project. The deep wish for the schism to be over is very real. Why else would I spend my Saturday working on this one to prove the point that now is as good a time as any to stop inventing and start using and sharing? All we need to do is be willing to part with some of our inventions; some of our uniqueness even. And to start focusing on the things that are smash hits for the people who use it daily.

Cheers,
Claus

PS: TYPO3 rocks!

I abandoned FLOW3 (and mostly TYPO3) cause of that.

+1 on all points. Also it's a huge pain to get anything committed if you're not a core-team member. We've ultimatively had to fork TYPO3.Flow because of that. This hinders an open development as well.

+1 I will evaluate laravel tonight.

Thanks for your thoughts, let me add mine.

I agree with the statements that

• TYPO3 Flow's coupling to TYPO3 Neos and
• TYPO3 Flow's terrible documentation
  are the cause for many people not being able to see Flow as a serious framework.

I think that Flow can make it though and can still evolve from this experiment status to a world-wide competitor against frameworks like Laravel. There are some additional steps that have to be finished before this journey can begin:

• AOP without bugs
• Github. Pull requests. (really? Why do i have to mention this at all?)
• ReST Support out of the box with ReST Controllers implementing GET PUT POST DELETE with default yaml routes and minimal setup, PATCH HEAD and OPTIONS at least addressable via Framework.
• Fine-graned access to session stuff like defining lifetime for each individual session or querying active sessions
  These features have become so important within the past years that they were a prerequisite today if I had to make a framework choice without specific focus on the TYPO3 universe.

If TYPO3 decides to address these issues i'm definitely looking forward to Flow and future development, being optimistic that it will remain my choice of framwork. If not, other framworks will become so much better one day that natural selection kind of forces me to leave the entire typoverse in favor of laravel or the like.

For the typo devs out there: Finish what you started with flow. You kind of invented a whole new framework from scratch, now make it ready for production.

So I've evaluated Laravel. Apparently I'm not a hipster.

I herewith demand that with every [FEATURE] change, the documentation of said feature must be included in the change, or it shall not be merged.

And yes, that includes core team members and whether Neos wants/needs it or not. No more merge buddies, no more missing documentation. The only disadvantage that TYPO3 always had, was documentation. The only way to solve this is to produce documentation.

I would have been so happy if Neos had been based on an existing framework like Symfony or Laravel, or that the coupling between Flow and Neos would have been less tight. In an existing software landscape of a company, it is very hard to argue for yet another new framework that your devs have to learn and stay up-to-date with.

And as the developers of Flow and Neos, why would you want to put yourself in a situation where you have to build and maintain two monster projects, a CMS and a new application framework? If you're at it , you might just add your own ORM, MySQL fork and custom HVVM. I think it's madness, and either underestimates the effort of the task or the capabilities (time-wise, not competence) of the developers.

Somehow I can't shake the feeling that the devs really wanted to do "their own thing".

Would it be too late to re-evaluate the current framework landscape and think about switching to something established and well-documented?

"I herewith demand that with every [FEATURE] change, the documentation of said feature must be included in the change, or it shall not be merged."
+1

:+1

While Symfony is obviously a framework and component powerhouse these days, of course the Symfony CMF isn't there yet either (aside from the CMF Routing component that has been adopted by Drupal 8 and ezPublish 5+ and several other CMS). I think we have worked harder on keeping our architecture decoupled at the same time we did keep the scope smaller (which is why we call ourselves a CMF and not a CMS, delegating the creation of a CMS to projects like sulu.io, which still call themselves a CMF BTW).

My point being, we (Symfony CMF) have in the past and still believe see a lot of awesomeness in the ideas in Neos especially. We are open to collaborate on the CR level (PHPCR), on the component level (Routing, HTTP caching etc), on the framework level (Symfony), on the UI level (create.js).

It feels like right now the strategy behind Flow and Neos is this rat race where killer features are identified and the hope is if this killer feature appears the world will suddenly jump over to Flow/Neos. This is a high risk high gain strategy. Yet despite Flow/Neos having several such killer features already, it has no panned out.

An alternative strategy could be to simply say we have created a proof of concept full of awesomeness, now lets bring this awesomeness to where the masses already are, rather than waiting for the masses to realize this on their own to then have the masses solve issues around missing features and documentation.

symfony is much more flexible than lavarel, think you should give it a try :)

I think that, what we learned from using Flow in the past, is that it shouldn't be considered and marketed as a product for the very broad audience like Symfony or Laravel. We also never really aimed for positioning it as a component framework which could be used in any kind of application; after all we created Flow as a basis for Neos when there was no Symfony 2 around.

Many people are very happy with Flow and just last week I talked to a developer who re-discovered his passion for PHP by starting with Flow. Personally, I like many of the concepts we introduced there and I don't want to miss them.

If money and other resources would be unlimited or I had a magic wand, what would I wish for today? The features of Flow (the object management, dependency injection, security framework, routing and in general all the auto detection / auto wiring functionality) on top of Symfony.

No magic wand though and very limited resources at the moment. Will that be a problem for Flow or Neos? I don't think so. We should strive for re-using more existing components. But we know from experience, it is always about balancing additional dependencies and labour savings. Any refactoring will also open a new can of bugs and, because we do have a contract with our users through the public API, causes immense costs on the user's side if not done carefully. I think that it is totally possible to replace certain core components by Symfony components step by step (and we do that), but we can only do so if it brings value for our users. A system administrator won't notice if the CLI component is Flow-native or based on Symfony.

@lsmith77 it's still a pity that we didn't manage to collaborate on PHPCR. Considering what I wrote above, it is still only about two things: resources for the refactoring and the feature set we need for content dimensions et al. We see the need of a bigger refactoring going into the direction of Event Sourcing and we actively are seeking for funding to make this possible. As soon as there is a chance of getting things funded, let's talk again – maybe there's a new window of opportunity to throw things together.

Looking at the Neos projects we currently hear of and the new people joining the project, I am optimistic regarding the future of Neos. On the other hand there are lots of valid concerns and problems, a few of them addressed by Alex in his letter. We do discuss these a lot in the Neos team, and are aware of most of them – one of the outcomes for example is that we will stop using Gerrit and move to Github after the Neos 2.0 release. Just being aware of them doesn't mean that we can tackle everything at once.

It's only so much we can do, but we do so in good spirits. As it happens in any community based project, opinions, attitudes and values will differ. When we disagree with values or feel offended, we need to give it a rest sometimes, and don't reply instantly on public channels. You could argue, from a strategical and communicative standpoint, that this is unprofessional. That may be, but we are people, too.

Everybody should support and use the products which helps him or her most and contribute to those they feel passionate about. If I would start contributing code to certain Open Source projects, they would have a really hard time with me, because I disagree with many of their approaches and values. So I better don't, and focus on what I feel is meaningful.

Nice comment @robertlemke and I agree completely.

@lsmith77 Thanks for your comment, if you are in Lausanne at some point in the future, feel free to ping me (best channel is Twitter @dfeyer), that could be a nice opportunity for us to know more about the current status of the PHPCR, and maybe discuss some of our missing feature to support Neos.

In my agency we are currently working a a really big Neos projet (+100'000 nodes, with tons of images, ...) that should go live during the next month. Neos is ready today for production use. Not everything is perfect and a lots can be improved, but be sure that we work hard to improve Neos every day and any contribution is always highly welcome.

@dfeyer doh .. I was in Lausanne today .. but I will ping you next time I am there.

@robertlemke the tricky bit with OSS is always what to do in order to grow the community so that more resources become available. however this is a tricky strategy to do, so the more viable one is to scratch your own itch which is what fundamentally you are doing. so I totally understand the rational. but big things only happen once people actually start collaborating on bigger things and these always tend to slow things down in the short term but they can make a world of difference in the mid/long term. typo3's cms brandname, experience and developers from Neos + symfony's brandname and experience and developers from the CMF imho would quickly be more than the sum of the two.

I cannot agree more. If you ever tried one of the frameworks mentioned above you never look back.

Isn't using a framework better than using none? Which leads me to ask myself if it was so wise by the TYPO3 community to abandon Neos?

I have always used bare bones PHP and TYPO3 CMS more than Wordpress, Drupal and any other (not last because university always used it) until experience gained over decades (that finally led via Windows, Mint, Ubuntu, Gentoo, OS X, BSD to Debian, Void and ArchLinux + Manjaro) showed why Neos is so much more desirable. The concept "convention over configuration" alone says a lot. Though not everyone may share this opinion. But that may change when getting caught in a mess of dependencies or broken configuration files while under great pressure to deliver a product or not to disappoint a person one likes a lot.

Only then when faced with opponents like hopelessly broken package states or totally unrecoverable operating system or CMS updates (e.g. fully breaking a highly frequented production online shop) or when having to maintain some big pieces or hundreds of software alone (which eats all time one once had because entropy and breaking changes or out of support are quicker than a single human being ever can be) one can understand why keeping things simple, small, version controlled and modular helps. "Do one thing and do it well" instead of trying to create a tool that does it all. Consider a tool shipping even its own process scheduler because it not trusts the operating system, great huh? How much fun to create something like that that also works cross-platform - let alone how one shall maintain such monstrous piece of software (this is a hyperbole of course but you get the point).

Of course I still like T3CMS and it improved a lot over the years, e.g. composer instead of Extension manager, Extbase, Fluid, less breaking major updates from TYPO3 v8 onwards and and and.

Still, Neos is more intuitive for people and this is what really matters. Because experience shows individuals and companies alike often are overwhelmed by the backend and how backend and frontend interact. Being able to edit text directly in the frontend gets the majority of those I created webpages for to be able to maintain content themselves.
Good news, because I am not after money, I am after time, because life is all about living beings, and those deserve our time most - and as a plus, clients can now be more relaxed when fine-tuning their content because they can do it themselves whenever they have a small amount of time to spare. And if they have a question they can still ask any time.

I admit having a TYPO3 based on Symfony|Laravel sounds epic (maybe merging Flow, Laravel), but considering the amount of work required to get Extbase et alia to work again, hm, I personally do not want to code it because the benefits may not be worth the time spent, but hey this is only labeled overkill until someone simply makes it happen. ;-)

I agree with @lsmith77, @robertlemke that T3|Neos and Symfony|Laravel could create a very powerful symphony.

I still hope for distributed cross platform package managers like composer (PHP only) or maybe even 0install (all languages, actually any files) to find more appreciation such that more people provide modular packages, but the truth is, time for learning new things is limited. Opinions differ and that is not bad per se. Diversity is good as long as it is peaceful. The same applies for Neos, T3CMS and I think both can benefit from each other. Just like BSD and Linux and all other OS. Finally we all sit in the same boat, planet Earth, and we want it to be great.