Thoughts on a new CMS

README.md

Some thoughts on a new CMS, in different progressions.

v1.md

This is the original idea of a new version of DocPad, taken from: https://discuss.bevry.me/t/docpad-architecture-vision-2015/46?u=balupton

---

Currently, DocPad is doing too many things:

1. It is a static site generator
2. It is a dynamic web server
3. It is a document parser
4. It is a document database
5. It is a document query system
6. It is a rendering system
7. It is a plugin environment
8. Including a home for content management plugins
9. Including a home for content importers

That's the reason why we've struggled defining DocPad for so long. It's time we change this. The biggest change here is to revaluate our goals. For 2013 our goals were:

1. Become a plugin environment for importers and content management systems/interfaces
2. Create a REST API plugin so that content management systems can then interact with docpad which can then interact with the database which can then interact with the file system
3. However, if we are on a remote environment, we also got to create a Commit Plugin that will also listen to the changes, and commit any changes to a git repo, then push that git repo to a web server - and hope that the web server isn't on an ephemeral file system and doesn't reset between changes and their push.
4. Somehow develop content exporters so we can export content to other database systems (including the file system)

This is absurd. It means that DocPad also becomes an ORM on top of all that it was before.

It really should have been this:

1. We utilise a popular external database system
2. We develop a standard for the schema that DocPad will use, other SSGs will use, and Content Plugins can use
3. DocPad then queries the database for changed documents and re-renders them, applying the renderings to the database

That way:

1. Content Management Systems can be developed for the content without depending on DocPad
2. Importers can be written without depending on DocPad
3. Exporters can be written without depending on DocPad

This means there is the following pieces:

1. There is a blackboxed utility for watching the filesystem and importing it into the database
2. There is a blackboxed utility for generation, that comprises of:
3. There is a blackboxed utility for querying the database
4. There is a blackboxed utility for rendering the results of the query
5. There is a blackboxed utility for writing the rendered results back to the database
6. There is a blackboxed utility for writing the rendered results to the file system when desired
7. There is a blackboxed utility for creating a web server on the database
8. There is a blackboxed utility for reading the configuration
9. There is a blackboxed utility for creating an event system
10. There are blackboxed utilities for regenerating changes:
11. There is a blackboxed utility for receiving a notification, updating the database, and telling the generation utility to generate
12. There is a blackboxed utility for watching the database,and on changes, regenerating the changes
13. There is a blackboxed utility for adding a web admin control centre to the above system, instead of just a CLI
14. Tying these individual blackboxed utilities together, will create the dynamic static-site generator we have come to love

Then outside of these core, there will be:

1. There is a blackboxed content management system(s) that interacts with the database directly
2. There are blackboxed importer scripts for a whole bunch of different services that read in a feed, and convert it into the database format, then write it to the database

This makes a lot more sense, as it means rather than a highly coupled system, instead DocPad is comprised of tiny little independent modules (aka blackboxed modules).

While we can utilise a lot of the existing code for this, it will be breaking changes all over the place probably.

Considerations:

1. When the database system is asynchronous, how do we do templating? Perhaps ES6 will have a solution here? Or we could use the way the partials plugin does it, which is inserting a place holder into the template synchronously that is then replaced with the actual data when it arrives asynchronously.
2. This will be 1-2 years of work before we have something stable for everyday use. That's a lot of funding and a lot of time. We need a guarantee that the investment of our time is going to be worthwhile - and that we can still afford to thrive during that time (not just live...).

Thoughts?

v2.md

This is the original idea doing a new project, instead of incrementing DocPad:

---

Idea:

1. People write their content on external services - git repos, prismic, wordpress.com, tumblr.com, medium.com, soundcloud.com
2. The application imports the data from the external sources at specific intervals or realtime
3. The application renders the imported data to a static website and rest api
4. The application deploys/serves the website
5. The user configures the external sources to import, the address of the website, as well as the theme to use

Technology Options:

• PouchDB - works on client and backend, has leveldb support, supports replication, couchdb like interface and functionality - is it isomorphic?
• http://rethinkdb.com/ - same as pouch more or less
• Dat - could be used for pulling in data from external sources then exporting to a pouchdb
• Hoodie - was a consideration, but pouchdb seems like a better fit
• Electron - can be used to wrap the app up locally
• React or Dust - use for interface, allows for server-side rendering too

Manifesto:

• App should render on the backend for SEO, however it should also expose a REST/Sockets API (PouchDB provides this) for client-side magic and modern templates
• App should be frontend/GUI first
• External content import adapters should be able to be written by the community and incorporated into the project easily
• Templating should be easy and shareable - tumblr.com easy
• Plans can be based around frequency of updates (every day, every hour, each request)
• Metadata gets imported locally, but remote data is not imported (by default?)
• No local content? No local content administration? Must use external service for editing content? (like prismic)
• Should each site be a silo? Or should be together like a network?

Questions:

• How big is the market for this?
• Could this be crowd-funded?
• Could I assemble a team for this?
• Could dat be used for all importing? And we create something for rendering, then provide a nice interface to couple the importing with dat and the rendering using blah?

Opinions:

• No local content editing or management
• Content providers should be the content networks
• Our network will be around importers (tumblr, wordpress, keystone, prismic, soundcloud, flickr) and rendering (react widgets, plugins, and themes)

Lame taglines:

• Consolidate your data and render it awesomely
• Your personal website, from your data
• Your website, your data, our rendering

Action items:

• If this sounds interesting, I'm looking for team members and early adopters, contact me
• How to take a 30% cut on stripe? Yes. https://stripe.com/docs/connect/payments-fees

Follow ups:

• Exporting data from dat: dat-ecosystem/dat#300 (comment)
• Suitability of dat: dat-ecosystem/dat#305 (comment)
• Early brainstorm: max-mapper/messages#49

Resources:

• PouchDB API: http://pouchdb.com/api.html
• Dat API: https://github.com/maxogden/dat-core/blob/master/test/streams.js
• Electron: http://electron.atom.io/
• React: http://facebook.github.io/react/

Roadmap:

• Create dat adapter for prismic and tumblr
• Create dat exporter for pouch
• Create isomorphic renderer for pouch data
• Export renderer as plugin
• Export data adapters as plugins
• Create deployment plugins to github pages and surge.sh
• Develop interface to contain app logic and install/configure/find plugins
• Launch app on github
• Document plugin APIs and the app like crazy
• Launch in alpha
• Add dynamic abilities for plugin authors
• Launch website in beta with dynamic abilities
• Add payment plans
• Launch website properly, charge customers

Goals:

• Create a bread and butter lifestyle business
• Allow me to travel and pick my workload
• Be simple enough to delegate and have others run the business
• Have a clear business model from the start
• Build a new eco-system with all my new learnings that is successful
• Build and maintain a great team
• Establish a product for everyday people
• Having the success and bread & butter income, be able to travel and speak about lifestyle changes

v3.md

Here are more recent musings.

---

Notions:

• Content is distributed across the web, on the best content network for it - videos on youtube, articles on medium, memes on tumblr, discussions on discourse, communities on facebook, music on soundcloud/spotify.
• People need a way of showcasing their content from the different networks, there is no way of doing this
• Content websites suck immensely, they are slow, heavy, and shit. Native apps for web content are winning on mobile for various reasons, but the shit web experience is a major reason (hence facebook instant articles, apple news, medium's app).
• While people go to native apps for the networks (medium is great for this), medium isn't a showcase for all content - we still need a way of pulling in all content.
• We need a static web experience by default (so mobile and search engines can work) and only enhance on desktop via progressive enhancement. If we become a network of consolidated content, then we can justify a mobile app, however if we are a tool for consolidating information into silos, a native mobile app doesn't make sense.

So the key-value pitch:

• A platform, you register, it pulls in your content from everywhere - it becomes your business card URL, your URL for all of your online content, you can curate it, adjust it, whatever. Content is static and fast by default (making it responsive and enjoyable on mobile, and accessible to search engines) on the desktop the experience is enhanced.
• You get a url like yourself.com/balupton - but you can change it to balupton.com if you wish - you can customise themes and curate content - you pick the data you want to import by signing into whichever services or providing whichever details are needed

So the up-sell pitch:

• With enough people, an app can be made, and you can simply say - find me on yourself.com - it becomes a congruate like facebook instant articles and apple news - but for all web content. A network of people.
• Perhaps browsing people's public conversations, tweets, and social media can also be a part of this. (similar to interconnectapp's idea of recording and browsing convos)

So the tech pitch:

• Can be built with rethinkdb or pouchdb or meteor
• Client can be built with react or meteor, so we render the content on the server-side and if needed, give them live updating data on the desktop client
• React can also native client too, which facebook have proven scales well on both desktop, mobile and native

So the user pitch:

• Not done as a developer app, done as an app for end-users who want a place to showcase their content, done as a network for all people's content
• You write content, you have video content, you have any content - pull it all in together, and we'll provide an awesome browsing experience for the user to peruse your content - better browsing experience than your devs can offer
• This should provide a larger niche than just SSG alone, and a unique niche