"Modifying Django's core to make arbitrary models compatible with Admin and Forms library" proposal for Google Summer of Code 2018.
Abstract
1.1 Current state of the Admin registration process
1.2 Goals
1.3 Benefits
The new framework
2.1 Overview
2.2 Advantages
- Merging admin-registration API
Schedule and milestones
4.1 New registration framework
4.2 Merging admin-registration framework
4.3 If time permits...
- About me
1.1 Current state of the Admin registration process --------------------------------------------------
The current state of Admin registration process is quite diffuse in nature. Using a new database-backend that is not supported by Django requires a lot of configuration on the part of the developer. Also, there is little uniformity in the process of using the hooks provided by Django for custom-backends. An example of this is the django-mailer (created by Daniel Pyrathon, that demos the use of Meta API for GSOC 2014), which uses customised Admin classes to view your GMail inbox in Django Admin. In stark contrast to this, using Apache HBase in Django's admin would require using a Thrift Server or a REST API to request data from HBase. This essentially means that each datastore performs roughly the same tasks, but uses different endpoints. The Adapter Design Pattern can be used to overcome this problem - by making available the high-level tasks in a structured manner and leaving out the most specific details to the developer. For instance, in an Update Query, fields to value mapping must always occur, but the actual query syntax may differ. For each of the CRUD operations, expression-templates can be provided by the developer, and this process should be made easy.
A fixed and uniform API that can be extended to use any datastore by providing concrete implementation of available high-level tasks would make the task easier for developers using Django and extend the domain of Django to areas hitherto unexplored - Big-Data frameworks and NoSQL data-stores.
The initial part of the work required in this project is to provide a uniform API that lays out concrete guidelines for each of the backend functions that Django can understand. This would require the use of Adapter Design Pattern, where the programmer using a specific backend would have to subclass some of the classes given for this purpose to provide concrete implementations of how the endpoints will utilise the backend-specific API. Here are some examples of what might be present as part of the new API:
- Four classes to be written by the developer, encapsulating each of the CRUD operations. - While registering a new model with Django Admin, kwargs to be used to specify each of the four classes: admin.site.register(Blog, create_class=’A’, read_class=’B’, update_class=’C’, delete_class=’D’) - Models will hold their Meta information to override some of the operations on the Meta class. For instance, get_field(fieldname) operation returns an error if a field-name is not present as part of the schema of the tables. However, this behaviour can be overridden by specifying add_or_update = True in the meta class.
The required documentation and tests shall be completed simultaneously.
The next part of the project would require refactoring the API. This can include providing more hooks for the programmers who look for greater control.
The last part of the project will include bringing this API into Django Core. This will probably require completing a case-study.
Finally, I’d like to provide a proof of concept, such as using the new API to use Apache HBase as the backend for Django.
There are multiple benefits of this feature. This feature extends Django to encompass non-traditional and NoSQL datastores by using minimal code. The Checks Framework can be used to ensure that the API standards are being followed in both letter and spirit. This makes Django compatible with the upcoming technology that tends to handle large amounts of data with great speed. This can potentially extend the usage of Django in web-apps that use distributed database backends.
The API is based on Russel Keith-Magee's idea from his talk at DjangoCon 2015. A developer needs to write the custom functionality by implementing the endpoint methods in derived classes. These methods will then be called as and when needed during the run of the application, triggered by get(), save() and filter() methods. The methods must have the signature as described in the base class.
Filter queries and Joins that are not supported will raise errors of type NotSupportedError. Joins should be disabled by default in the base-class that is inherited by the CustomUpdateClass, for some datastores do not support joins natively. For enabling joins, enable_joins class variable should be set to True. Additionally, a for_join method should be specified as an equivalent to the as_sql method that will return the correct SQL for the join. Ticket #25590 actually plans to deal with something similar.
The functions will use any existing names and signatures for the sake of consistency, but will only differ in how they are implemented for a datastore. The databases for which Django does provide support will continue to work the same way, and the new frameworks will raise the correct exceptions when they do not adhere to the contract of the new admin-registration API.
Communication chain. ~~~~~~~~~~~~~~~~~~~
When a new database-backend is to be used, the developer will name the datastore inside the settings file as <appname>.backends.<backend-name> in the ENGINE section of DATABASES. The username, password, name of the database to be used will also be given inside settings.DATABASES.
The backends directory inside the app shall hold the required class-files which represent the objects that will fulfill the contract of the admin-registration API for seamless integration. This will include a subclass of Admin that will bind the database to Django-Admin, and a subclass of Manager that will take care of how the two sides co-operate.
The new framework will extend the Warning and Error classes to specifically provide details on each of the common-types of problems that might occur, and hints on how to fix them, as and when required. The classes can be extended to allow for errors like Bad Request while using a REST API call.
The use of System Checks Framework can be looked into during the last phase of the project.
The solution makes it easier for developers to begin developing their applications using any arbitrary backend with Django. The uniformity in the API will also allow for change of database-backends (should they be needed) with minimal effort. All the developers need to know before they begin writing their app is the way they want to provide their custom functionality. Everything else, probably along with examples will be given in the documentation of Django itself.
This is a long term solution that is extendable and kind of lives up to the tagline of Django - the framework for perfectionists with deadlines.
As you can see, the solution is consistent between different kinds of objects and it does not assume that only a fixed set of object types can be validated. I believe that good long-term solution should be extendable and the new framework allows us to easily add validation of new type of objects -- just modify the validation chain.
This API will provide numerous opportunities to the developers who need to use non-traditional data-stores and link them to their web-applications, like the applications that deal with a lot of data, and hence use distributed data-stores. An example use-case is described below:
Coffee meets Bagel has a Django setup uses Postgres as main backend, but they also use Cassandra to store events for every single user.
In their admin panel, they would like to be able to view their Cassandra tables in the admin page and connect it with other tables in Postgres, such as auth_user.
The first part of this proposal is to develop a modified API that should be used to connect an arbitrary data-store with Django. This will require writing the base-classes for each of the CRUD operations and providing support for kwargs during Admin-registration process.This shall ensure that all the test of the current framework work fine, and that the endpoints provide a minimal but clear structure for specifying the implementation of the functions they represent.
The final part of the project would require adding documentation for the new API, along with case-study based examples that demonstrate the use of the endpoints along with concrete implementation.
The following tasks need to be done before the actual programming period begins:
- Discussing and writing full API of the new admin-registration framework, i. e., what functions should be left to the developers for implementation, what should be their parameters and what output is expected from them.
- A list of new tests for the new admin-registration framework. This should include checks for the existing database-backends, along with new requirements like testing whether a connection has been established with the datastore and whether the user is authorised to use the data present in a certain database.
- Improving English writing skills, i. e. reading "The elements of style".
My university exams will take place from 20th May to 10th June, and during this period I shall be working for less than 40 hours a week on this project. I also have a short trip planned during mid-July that will take up some time from the 8 hours a day schedule of GSoC. Apart from these periods, I do have time on my hands that I can put into this project whole-heartedly.
Internet connectivity should not be an issue during any period of time other than the ones mentioned above. However, I shall make sure that I keep the community posted in case any glitch over network occurs.
(From May 14 until July 2).
I will provide the code-base in an iterative development fashion, i.e., I shall be writing a class, its API methods and their tests and then move on to other classes.
I need to come up with the design of the classes in a more concrete fashion. With the work done during the community-bonding period, this work shall be of a relatively smaller duration.
(I shall have exams at university -- I shall work a little lesser).
This will require modifying the way the registration process currently works. This work will include adjusting some existing tests and adding new ones for the existing databases.
Documenting the new API. There will be a new topic in The development process. There will be extensive documentation in The admin. Giving out details of new Errors and Warnings.
From July 2 till July 30.
The merging process shall go in two phases.
1. Phase 1 will take up CRUD operations one at a time and will start supporting them in steps. Next, the support for custom functionality of Joins and Filters will be merged in this step.
2. Phase 2 shall be dedicated to fix existing errors, add missing features and improving the API for better UX.
If I finished earlier, I would focus on adding more customization to the API by providing support for REST API based datastores using native HTTP verbs. This would further enhance the use of Django as a web-development framework, and will reduce the amount of work that has to be done by developers using frameworks that do provide a REST API.
My name is Piyansh Saxena and I am currently a student of Indian Institute of Technology Mandi (India). My time zone is UTC+05:30. I work on Ubuntu 17.10 with Sublime as my primary editor.
I have been actively programming in Python for the last 3 years, and I have a fair-bit of experience with C++. I have taken courses on Communicating Distributed Processes, Information and Database Systems and Software Design Patterns. In addition to this, I have developed web-based applications in PHP and Python, including an Online Judge and a Patient Referral System for PGI Chandigarh using Django.
I have started contributing to Django from January 2018. I have submitted some patches, and I am looking at some more tickets, mostly feature-requests, to take up a bigger task.
I have an experience of working with Apache HBase in an industry-level project during my internship. I have a Secondary School Certification in English Language. I prefer written communication by email or IRC.
My e-mail is askpriyansh+gsoc magic_character gmail.com. You can find me also at #django-dev and #gsoc IRC channels. My nick is PriyanshSaxena_.