How to reset migrations.md

Prerequisites

• Install https://networkx.github.io/documentation/latest/install.html

• Install https://django-extensions.readthedocs.io/en/latest/installation_instructions.html

• Follow this: https://django-extensions.readthedocs.io/en/latest/graph_models.html

• Remove all old migrations.

• Remove all entries out of the django_migrations table.

How to

There are a couple of different types of migrations to consider here:

• database model structure
• scripts that add e.g. permissions
• one time migrations

We only care about the first two here as the one time scripts already run and don't do anything on an empty databse.

database model structure

So the idea is to first create migrations for all models without any outgoing relationships, then create all migrations for models that only link to those models from the previous step, then create all migrations for models that only link to those models from the previous two steps and so on. In the code sample below I call them "levels".

This is not a one fits all solution but it gives you a good overview on what migrations to create first which ones second and so on.

First step is to generate a dot file that includes all inheritances and relations between models:

$ python manage.py graph_models -a > relations.dot

Then we load that file into a graph representation so we can use networkx to analyze the incoming and outgoing relations.

import networkx as nx
dot = nx.drawing.nx_agraph.read_dot('relations.dot')

The problems with the dot representation are:

1. abstract models don't have migrations but still are connected with edges so we want to filter them out
2. for abstract models that have relations to other models those relations are not inherited so we need to do that manually

abstract_nodes = []
for edge in dot.edges():
    data = dot.get_edge_data(*edge)
    if ' abstract\\ninheritance' in [i.get('label') for i in data.values()]:
        abstract_nodes.append(edge[1])
        
# move links from abstract to inheriting models
for node in set(abstract_nodes):
    edges = dot.in_edges(node)
    abstract_outgoing = dot.out_edges(node)
    for edge in edges: 
        data = dot.get_edge_data(*edge)
        if ' abstract\\ninheritance' in [i.get('label') for i in data.values()]:
            for outgoing in abstract_outgoing:
                dot.add_edge(edge[0], outgoing[1])

# remove abstract models
for node in set(abstract_nodes):
    dot.remove_node(node)

from pprint import pprint

def split_node(name):
    group, model = name.split('_models_')
    group = group.replace('_', '.')
    return group, model


def get_next_level(current_set):
    level = []
    for node in dot.nodes():
        outgoing = dot.out_edges(node)
            
        if {o[1] for o in outgoing if o[1] != str(node)}.issubset(current_set):
            if node not in current_set:
                level.append(node)

    return set(level)


all_entries = set()
for i in range(0, 10):
    print(f'LEVEL {i}')
    new_entries = get_next_level(all_entries)
    all_entries = all_entries.union(new_entries)
    pprint({split_node(n) for n in new_entries})
    
    # group the apps together as we can only run migrations on app level
    print(' '.join(list({split_node(n)[0].replace('apps.', '').replace('marketplace.', '') for n in new_entries})))
    print()

This produces an output like:

LEVEL 0
{('apps.catalogue', 'AttributeDefinition'),
 ('apps.catalogue', 'Category'),
 ('apps.catalogue', 'Manufacturer'),
 ('apps.catalogue', 'Synonym'),
 ('apps.catalogue', 'Vendor'),
 ('apps.catalogue.portfolios', 'Portfolio'),
 ('apps.catalogue.regional', 'Promotion'),
 ('apps.content.flatpages', 'PrismicPage'),
 ('apps.content.flatpages', 'PrismicRefresh'),
 ('apps.datapush', 'PushLog'),
 ('apps.integrations.erequest', 'ERequestNewPurchaseOrder'),
 ('apps.integrations.xero', 'XeroAccount'),
 ('apps.integrations.xero', 'XeroContact'),
 ('apps.integrations.xero', 'XeroInvoice'),
 ('apps.marketplace.order.handling', 'Task'),
 ('apps.marketplace.orders', 'ConversionRate'),
 ('apps.marketplace.shipment', 'Courier'),
 ('apps.marketplace.ticket.management', 'BlockReason'),
 ('django.contrib.contenttypes', 'ContentType'),
 ('django.contrib.sessions', 'Session'),
 ('django.contrib.sites', 'Site'),
 ('mptt.graph', 'GraphModel'),
 ('mptt.graph', 'TreeNode'),
 ('social.django', 'Association'),
 ('social.django', 'Code'),
 ('social.django', 'Nonce'),
 ('social.django', 'Partial')}
datapush orders catalogue.portfolios ticket.management django.contrib.sessions django.contrib.contenttypes 
order.handling social.django mptt.graph shipment catalogue django.contrib.sites integrations.erequest 
content.flatpages integrations.xero catalogue.regional

LEVEL 1
{('apps.catalogue', 'Brand'),
 ('apps.catalogue', 'CategoryAttributeDefinition'),
 ('apps.catalogue', 'Discount'),
 ('apps.catalogue', 'VendorContact'),
 ('apps.catalogue', 'VendorInvoiceInformation'),
 ('apps.user.organizations', 'Organization'),
 ('django.contrib.auth', 'Permission'),
 ('django.contrib.redirects', 'Redirect')}
django.contrib.redirects user.organizations catalogue django.contrib.auth

LEVEL 2
{('apps.catalogue', 'BrandImage'),
 ('apps.catalogue', 'BrandSupplier'),
 ('apps.catalogue', 'Product'),
 ('apps.catalogue', 'VendorContactHours'),
 ('apps.catalogue.regional', 'Price'),
 ('apps.integrations.mapping', 'CategoryMapping'),
 ('apps.integrations.mapping', 'VendorMapping'),
 ('apps.integrations.punchout', 'PunchoutIdentity'),
 ('apps.marketplace.vendor.communication', 'VendorCommunication'),
 ('apps.marketplace.vendor.communication', 'VendorIdentityMapping'),
 ('apps.user.organizations', 'OrganizationVendorAccount'),
 ('apps.user.organizations', 'Subdivision'),
 ('django.contrib.auth', 'Group')}
integrations.punchout user.organizations vendor.communication catalogue catalogue.regional 
integrations.mapping django.contrib.auth

LEVEL 3
{('apps.catalogue', 'ProductImage'),
 ('apps.catalogue', 'Resource'),
 ('apps.catalogue', 'Variant'),
 ('apps.user.accounts', 'User'),
 ('apps.user.organizations', 'SubdivisionMeta')}
user.accounts user.organizations catalogue

[...]

There are also some nodes that are not shown above e.g. because of weird foreign keys:

pprint(set(dot.nodes()) - all_entries)

What we need to do now is to start at level one and take out the groups that make sense (e.g. not django internal models):

$ python manage.py makemigrations portfolios ticket_management order_handling shipment catalogue erequest 
flatpages xero pricing

As those apps might have models that are in other levels we need to go through those migrations and check in the dependencies for e.g. ('ordering', '__first__').

If we find a line like that this means there is a unresolved model here. So we need to search for ordering and remove all foreign keys to it and then remove the ('ordering', '__first__') from the dependencies.

Those lines will be created in separate migrations afterwards again in later levels.

At the end you might need to run some makemigrations for global only and local admin models with the --settings parameter.

scripts that add e.g. permissions

The second part is about keeping the migrations that create seed data.

How I did it was just searching for all files in migrations folders that contain RunPython or RunSQL. Then I removed all those that were one time only migrations (= changing existing data in the database) as the database is empty in e.g. the case of running tests.

All the other ones I copied the functions together grouped by app and created seed migration files for those.

the end

At the end you can apply all migrations with the manage migrate --fake command.