Skip to content

Instantly share code, notes, and snippets.

@knbk
Last active June 24, 2016 18:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save knbk/cd0d339e1d3fa127cf7a to your computer and use it in GitHub Desktop.
Save knbk/cd0d339e1d3fa127cf7a to your computer and use it in GitHub Desktop.
An URL dispatcher API (GSoC 2015)

Table of Contents

  • 1. Abstract
    • 1.1 Powerful but limited
    • 1.2 Goals
  • 2. Designing the new API
    • 2.1 The URLResolver
    • 2.2 Namespaces
  • 3. Timeline
    • 3.1 Design phase
    • 3.2 Namespaces
    • 3.3 Dispatcher API
  • 4. About me
suffix

.

Abstract

Powerful but limited

The current dispatcher features a powerful, regex-based resolving strategy. This allows to both resolve and reverse urls based on practically any url path. It can provide default view arguments and named url patterns can be used to easily reverse urls. The include function and namespaces allow you to mount multiple instances of an application in your url configuration, or to simply reuse common patterns that map to the same views. It provides a simple public API, and what it does, it does well.

Most of the machinery is internal, so you are stuck with what the API has to offer. The internal classes, specifically RegexURLResolver, are hard to extend without having to copy and alter parts of the _populate() function. This function also relies heavily on the internals of included sub-patterns. This makes it difficult to replace a sub-pattern with a duck-typed URL resolver. Even though it is an internal API, the dispatcher is too complicated for any practical extensions and customizations.

The limitations of this are most apparent in third-party content management systems. The current dispatcher only allows for a static definition of the url configuration. This doesn't work well with the dynamic nature of content managed in the database. The result is that most CMS's change their views into URL routers and put their view logic into their models.

Namespaces, as they currently are, satisfy their goal of distinguishing both apps and app instances, but are overly complicated and, as a results, the documentation can be hard to understand. You can specify both a namespace and an app name either in the included configuration or in the include() function, but not both. This means that if you want to specify an app name for your patterns, you need to facilitate setting the namespace as well. Alternatively, you can use the namespace from include(), but you'll have to trust the user that he provides a compatible app name. You need the app name to successfully reverse an url name, so this pretty much requires the first method and all the boilerplate code for namespaces if an app needs to reverse its own urls.

Goals

The goal of this project is to provide an URL dispatcher with a customizable, extensible public API. A resolver should receive all the information that might be needed to route the request to a view. This will no longer be limited to the url path, but will include the complete request object. Reversing the urls should be possible as well using a simple, consistent API that allows for custom business logic. The url configuration API should provide convenient methods to include customized resolvers, and to configure decorators for multiple views. The dispatcher should keep in mind that urls might change during the life of the application. The new dispatcher should be backwards-compatible with all public API's.

App names and namespaces should be clearly separated in their respective uses: an app name uniquely identifies a group of url patterns as belonging to an app. This should be set by the included module, an specifies which url configuration is included. The namespace identifies an instance of those url patterns. This should be set by the including url patterns, and specifies where the url configuration is included.

Benefits

This project changes the way the URL dispatcher can be used. The configuration is no longer a black-box 3-function API, but a fully featured, extensible dispatching framework. It gives full control to the user with a simple public API. URL routing can now make use of all the available information, and it allows for a more dynamic configuration. Reversing can be made easier by moving the conversion logic to the resolver rather than the calling code. The extensibility allows for any kind of custom routing rules.

Designing the new API

The URLResolver

The current API largely depends on the internal RegexURLPattern and RegexURLResolver classes. An abstraction of these classes will be an excellent starting point for the new public API. To keep the public API as simple and transparent as possible, loading the url configurations, which is currently ingrained in the RegexURLResolver class, will be factored out into a separate, internal class. The remaining abstracted classes will be unified into a single URLResolver class.

All URLResolver classes will be required to provide a resolve() and a reverse() method. resolve() will accept a request and a path. It should either return a complete match, or raise a Resolver404 or an StopResolving exception. reverse() will accept a view name and any number of positional and keyword arguments, and must return a relative or fully qualified url, or raise a NoReverseMatch exception.

The public API should at the very least contain URLResolver subclasses that replace the current RegexURLPattern, RegexURLResolver and LocaleRegexURLResolver classes to maintain backwards compatibility. Other extensions are easily added.

Namespaces

The documentation says the following on namespaces [1]:

URL namespaces allow you to uniquely reverse named URL patterns even if different applications use the same URL names. It’s a good practice for third-party apps to always use namespaced URLs (as we did in the tutorial). Similarly, it also allows you to reverse URLs if multiple instances of an application are deployed. In other words, since multiple instances of a single application will share named URLs, namespaces provide a way to tell these named URLs apart.

The aim is to simplify namespaces while staying true to their purpose. An app name will be specified in the app's url configuration itself. The namespace will be specified when the url configuration is included. This design clears up the purpose of both the app name and namespace, and apps that don't specify a namespace can more reliably reverse their own urls. The app_name argument to include() will be deprecated, but the namespace argument will be allowed even when the included configuration provides a namespace hint. The namespace hint from included configurations will be deprecated, in favour of defaulting to the app_name as a namespace if no explicit namespace is provided.

Reversing a namespaced url should be possible by specifying part of the namespace path or just the view name. An empty namespace (i.e. ':<view_name>' or '<global_namespace>::<view_name>') indicates loose matching, as opposed to exact matching. Priority should be given to the current app. The new API allows resolvers to reverse urls on arbitrary arguments. This include e.g. model instances. In such cases, it is desirable that third-party apps need not know about namespaces, but can instead reverse urls purely based on a model instance and a conventional view name such as 'index' or 'edit'.

Timeline

First things first, I will completely design the URLResolver API, including the details of extensible base classes. Then I will make the namespace changes, this can probably be merged separately from the rest of the project. After that, I will write the public API and the separated url configuration loader. Then, the existing classes and public functions will be reimplemented using the new API. At this point the API will be finalized and I will start on a comprehensive topic guide in the documentation. If there is still time left, I can implement some new functionality into Django core using the new URLResolver API.

I have exams starting on the 22th of June. I don't know my exact schedule yet, but I'll be busy for about 1 or 2 weeks and I won't be able to work full-time. I estimate this costs me about a week of full-time development.

Design phase (pre-GSoC - first 2 weeks)

There are some design decisions to be taken into account. I will start working early so that I have the time to discuss any design decisions where I need additional input. Then, in the first 2 weeks of GSoC, I will concretize the public API of URLResolver and its internal mechanisms. These internals should take into account the concerns raised in tickets #7537 and #14761 about subclassing, and #20757 about an object-oriented approach.

Some additional features to be considered:

  • Multiple url matches (#16774)
  • AbortResolving exception (#22425)
  • Separate access to kwargs and extra context (#16406)

I will also discuss options for the syntax of url configurations.

Namespaces (1 week)

With a good design, the namespace changes won't be a lot of work, and it's fairly separated from the rest of the proposal. Finishing this first allows for an early merge, in time for the proposed 1.9 release planning.

Implement changes, (re)write tests, documentation (1 week)

This includes:

  • Deprecate app_name argument and single 3-tuple argument of include().
  • Add convenient way to specify app name in included url configuration (#11642).
  • Change namespace usage throughout Django and contrib packages.

Dispatcher API (9 weeks)

This is the bulk of the work. I will implement the base class (3.3.1, 3.3.2), implement a separated mechanism to load url configurations (3.3.3, 3.3.4) and then reimplement the existing resolver classes (3.3.5, 3.3.6). To wrap it up, I will include the new classes in the API reference (3.3.7), and expand the URL dispatcher topic guide with a comprehensive description of the new API and how to extend it (3.3.8). Without too many setbacks, I will have 2 weeks left to implement some useful extensions to the resolver (3.3.9).

Implement URLResolver class (1 week)

  • Implement basic resolve() and reverse() methods.
  • Implement mechanism to include sub-patterns.
  • Implement stub methods for implementation-specific methods, e.g. match(request, path).

Write tests for URLResolver (half a week)

  • Test inclusion of sub-patterns.
  • Test resolve() and reverse() with mock implementations of stub methods.

Rewrite URL loader (2 weeks)

(Exams - working at about 50% productivity) * Implement lazy loading of url configurations. * Add lazy loading to include() method. * Replace _populate() method with sane alternative outside URLResolver. * Tie it all together in BaseHandler.

Write tests for URL loader (1 week)

  • Test lazy loading.
  • Test that it works with all currently possible arguments for include().

Rewrite existing regex-based resolvers (half a week)

  • RegexURLResolver and RegexURLPattern for generic regex-based matching.
  • LocaleRegexURLResolver for locale-based matching.
  • Tie into url() and i18n_patterns() functions.

Write tests for regex-based resolvers (half a week)

  • Test new classes.
  • Make sure tests existing tests for url() and i18n_patterns() pass.
  • Additional tests for possible new options in url() and i18n_patterns().

Document URLResolver and subclasses (half a week)

  • API reference that describes classes, methods and attributes.

Expand documentation topic guide on URL dispatcher (1 week)

  • Check current topic guide for anything that must be changed.
  • Provide documentation on how to extend and use any URLResolver classes.
  • Provide examples, e.g.:
    • Model router for a specific model.
    • Dynamic urls for CMS pages.

Provide additional resolvers (2 weeks)

Including tests and documentation.

  • Domain/subdomain resolver.
  • Slug- and pk resolver.
  • Date resolver in various formats, based on strptime().

About me

My name is Marten Kenbeek, I'm 20 years old. My time zone is UTC+01:00. My email is marten.knbk+gsoc <at> gmail.com. My nick on the #django-dev IRC channel is knbk.

I'm a third-year student at Eindhoven University of Technology (The Netherlands), majoring in Industrial & Applied Mathematics. I always took a great interest in programming, and completed my first university-level programming course at age 16. I eventually went another direction, but I was also accepted as a developer at a small Wordpress Hosting & Development company (Biz2Web). This is where I gained most of my experience in Wordpress, Django and webdevelopment in general, and where I was introduced to Django: some of our internal systems run on Django, as well as my pet project at the time, globalsatwork.com.

I've been active on StackOverflow for ~1.5 years (http://stackoverflow.com/users/2615075/knbk), helping out people with Django-related questions. I've started working on Django core at around february this year, most notably optimizing migration graphs (#24366) and speeding up model rendering (#24397).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment