Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Thoughts on yada

Thoughts on yada

Frameworks

Let’s define a framework as any library that contains one or more functions that accept callbacks.

Web frameworks are great for beginner developers who need to get stuff done. But they ultimately force you into a corner. In most applications, experienced developers need to retain control. Libraries are better. Developers can choose to use libraries, and any functions in them, judiciously.

By that definition, yada is a framework.

Since yada provides a handler, it takes control once the router has finished.

Since yada takes control away from the developer, it has to provide everything that a developer might need. This means that yada will never be sufficient, and will have to be endlessly maintained and features continually added.

I have since arrived at the opinion that what yada attempts is too ambitious. But the goal of providing libraries that make it easier to achieve greater conformance to HTTP specfications is still worth pursuing.

To that end, I’ve extracted out some functions from yada to make some supporting libraries. For example, the regular expressions used in the parsing and construction of HTTP headers have now been given a proper formal treatment in reap. Likewise, the fledgling conneg.clj namespace (which yada copied from Liberator) has been subjected to a thorough, comprehensive, re-implementation with pick.

Single thread is simpler

yada is async only. Almost by definition, a single thread per request is less complex.

The async nature of yada does make it more difficult to debug.

I have researched alternative approaches (e.g. Vext), using Ring’s newer async-aware protocols but with Loom very much on the horizon now I feel we should direct efforts towards evaluating the new approach to async that Loom offers.

"Just use a map!"

For me, yada’s most valuable insight yada is that it allows you to think about resources as Clojure maps. However, it would have been a better design if the maps didn’t contain functions (callbacks). The use of embedded functions ties what is otherwise a pure data representation to a particular language.

One of the architectural constraints in REST is the uniform interface. Consequently, it is better to have a single function, uniformly operating on a request (map) and a resource (map), as opposed to a custom individual function developed on a per resource basis. And yet, this is the design of virtually every other web framework today.

Site

In HTTP, web resources combine configuration (which methods the resource allows, the acceptable content types of its representations, and so on) with state, the value associated with a resource at a given time.

Both configuration and state are time varying, indeed, as Roy Fielding writes in his seminal dissertation:

"a resource R is a temporally varying membership function" — 

Given the dynamic nature of a resource, I feel that their configuration and state are best stored in a database, where they can be queried and consistently updated by client applications, rather than statically defined in a codebase.

Site is an experiment based on this idea, and I’m using XTDB because it fits my needs for a database where it is easy to persist time-varying Clojure maps. Though any other database could be used.

Unfortunately, Site isn’t (yet) a library or framework.

Future plans for yada

When I started yada, it was a research experiment to try to produce a Clojure web framework that would prioritise conformance to the HTTP specifications. As yada was adopted by a number of projects, and as these projects began to go to production, I felt it was inappropriate to make incompatible changes. Any new research into different approaches would have to be done elsewhere.

Also, at the time, there was some doubt about the future maintenance of aleph and manifold, which yada was built on. However, since then it looks like manifold is receiving some attention.

Current work

This year my research has been almost exclusively focussed on Access Control. Whereas authentication is mostly a solved problem these days, authorization is definitely not. I think Access Control is important for two reasons:

  • There are many appications which would be improved if they allowed the user greater control of their privacy, or the privacy of the individuals they store records about.

  • Access control is a significant fraction of the 'backend logic' that application developers are still required to write by hand.

Once I’m done with this phase of research I intend to invest further in HTTP conformance.

Current advice to developers building Clojure web backends

In the absence of a better solution, I would encourage senior developers building Clojure web backends to build their own handler, either based on the notes I’ve written up in rest.guide, or by stealing and adapting the code in Site’s juxt.site.alpha.handler namespace.

As I said in my 2021 talk "Building a RESTful Web API in Clojure - a new approach", you only need to build one more Ring handler. That handler can be taken from project to project and tweaked accordingly. This approach allows you to retain control, add features where necessary, focus on areas like the security required for your application and much more. It is more work up-front, but will get you to a much better place. The sacrifices involved in accepting the limitations of a web framework are just not worth it in the long-run.

@anderseknert
Copy link

anderseknert commented Sep 19, 2022

Would love to hear you thoughts on the problem of access control! And whether you've considered a decoupled approach like OPA (which is the project I work on). As a hobbyist Clojurist, I've toyed around with integrations in the past, and more recently, me and a colleague have worked on a project for evaluating compiled policy plans directly in Clojure. Still far from being ready for production, but it's getting there :)

@malcolmsparks
Copy link
Author

malcolmsparks commented Sep 19, 2022

Hi @anderseknert - that's very interesting. I've played a bit with OPA and it's definitely something I'll spend more time with. Thanks for the links.

@malcolmsparks
Copy link
Author

malcolmsparks commented Sep 19, 2022

btw. my work on access control is mostly accessible on Site's 'actions' branch: https://github.com/juxt/site/tree/actions. I'm attempting to get Site to be a fully-fledged OAuth2 Resource Server. The Access Control aspects are part of this of course, and I'm mostly researching how to best provide an authorization layer between the data in the database and applications. Ultimately it would make a lot of sense if this layer were to integrate with OPA.

@anderseknert
Copy link

anderseknert commented Sep 20, 2022

Cool! Delegating the resource server responsibilities to OPA makes a lot of sense, and there's pretty good support for doing most things OAuth2 / OIDC in policy evaluation. This is particularly useful in distributed, heterogeneous environments, like microservice clusters where different teams might use their programming languages and frameworks of choice, but the organization (i.e. platform / architect / security team) wants to provide a coherent system for authorization across services, avoiding the need for each team to roll their own authorization systems and policy decision making, auditing processes, and so on. Pretty much the same way authentication is delegated to a centralized component, storage to a database, etc..

Would love to hear your thoughts if you do integrate OPA! And I'd be happy to help in any way that I can.

Oh, and XTDB as a permissions data source for OPA seems like something I'll need to try out in the near future!

@jackdbd
Copy link

jackdbd commented Sep 21, 2022

I'm not sure I understand this sentence:

Consequently, it is better to have a single function, uniformly operating on a request (map) and a resource (map), as opposed to a custom individual function developed on a per resource basis.

You mean that an application developer should write a single request handler? No more routing at the "framework" level?

Also, when you say:

Since yada takes control away from the developer, it has to provide everything that a developer might need.

Do you think that a framework that provides extension points (e.g. hooks) is a reasonable middle ground from having total control (i.e. no framework) and no control at all (i.e. monolithic framework)?

I saw a couple of typos: specfications (in Frameworks) and appications (in Current work).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment