Skip to content

Instantly share code, notes, and snippets.

@flaf
Created February 13, 2022 02:45
Show Gist options
  • Save flaf/72cdc5fb8a874313b531ed17ba768b16 to your computer and use it in GitHub Desktop.
Save flaf/72cdc5fb8a874313b531ed17ba768b16 to your computer and use it in GitHub Desktop.
Django REST, basic questions

Warning, I'm a Django beginner. I'm using Django 4.0.2 and DRF 3.13.1. My project (very basic) uses the default SQLite DB and I have 2 questions. Here is my code:

### My models.py ###

from django.db import models


class Template(models.Model):
    name = models.CharField(max_length=300, unique=True)
    root_partition = models.CharField(max_length=300)
    lvm = models.BooleanField()

    def __str__(self):
        return self.name


class VM(models.Model):
    fqdn = models.CharField(max_length=300, unique=True)
    created = models.DateTimeField(auto_now_add=True)
    from_template = models.ForeignKey(Template, on_delete=models.CASCADE)

    def __str__(self):
        return self.fqdn

    class Meta:
        ordering = ['fqdn']

#----------------------------------------------------------------

#### My serializers.py ###

from rest_framework import serializers
from .models import VM, Template

class TemplateSerializer(serializers.ModelSerializer):

    class Meta:
      model = Template
      fields = ['name', 'root_partition', 'lvm']


class VMSerializer(serializers.ModelSerializer):

    class Meta:
      model = VM
      fields = ['fqdn', 'created', 'from_template']

    from_template = serializers.StringRelatedField(many=False, read_only=False)
    #from_template = TemplateSerializer(many=False)

#----------------------------------------------------------------

#### My views.py ###

from django.shortcuts import render
from rest_framework import generics
from .models import VM, Template
from .serializers import VMSerializer, TemplateSerializer

class VMList(generics.ListCreateAPIView):
    queryset = VM.objects.all()
    serializer_class = VMSerializer

class TemplateList(generics.ListCreateAPIView):
    queryset = Template.objects.all()
    serializer_class = TemplateSerializer

GET requests are OK, as you can see:

$ curl -s http://127.0.0.1:8000/api/templates | jq
[
  {
    "name": "redhat-8-5",
    "root_partition": "/dev/root-part",
    "lvm": true
  },
  {
    "name": "ubuntu-20-04",
    "root_partition": "/dev/root-part",
    "lvm": false
  },
  {
    "name": "ubuntu-20-04-lvm",
    "root_partition": "/dev/root/part1",
    "lvm": true
  },
  {
    "name": "redhat-8-3",
    "root_partition": "/dev/root/partx",
    "lvm": true
  }
]

$ curl -s http://127.0.0.1:8000/api/vms | jq
[
  {
    "fqdn": "mariadb.in.ac-versailles.fr",
    "created": "2022-02-11T00:42:33.473687Z",
    "from_template": "ubuntu-20-04-lvm"
  },
  {
    "fqdn": "srv-1.flaf.fr",
    "created": "2022-02-11T00:41:14.897579Z",
    "from_template": "redhat-8-5"
  },
  {
    "fqdn": "toto.domain.tld",
    "created": "2022-02-11T00:41:00.931524Z",
    "from_template": "redhat-8-5"
  },
  {
    "fqdn": "www.google.com",
    "created": "2022-02-12T16:16:24.522782Z",
    "from_template": "redhat-8-3"
  },
  {
    "fqdn": "x.toto.fr",
    "created": "2022-02-12T16:14:51.645851Z",
    "from_template": "redhat-8-3"
  }
]

Question 1: During the second curl above, I have these SQL requests:

(0.000) SELECT "vbot_template"."id", "vbot_template"."name", "vbot_template"."root_partition", "vbot_template"."lvm" FROM "vbot_template" WHERE "vbot_template"."id" = 3 LIMIT 21; args=(3,); alias=default
(0.000) SELECT "vbot_template"."id", "vbot_template"."name", "vbot_template"."root_partition", "vbot_template"."lvm" FROM "vbot_template" WHERE "vbot_template"."id" = 1 LIMIT 21; args=(1,); alias=default
(0.000) SELECT "vbot_template"."id", "vbot_template"."name", "vbot_template"."root_partition", "vbot_template"."lvm" FROM "vbot_template" WHERE "vbot_template"."id" = 1 LIMIT 21; args=(1,); alias=default
(0.000) SELECT "vbot_template"."id", "vbot_template"."name", "vbot_template"."root_partition", "vbot_template"."lvm" FROM "vbot_template" WHERE "vbot_template"."id" = 4 LIMIT 21; args=(4,); alias=default
(0.000) SELECT "vbot_template"."id", "vbot_template"."name", "vbot_template"."root_partition", "vbot_template"."lvm" FROM "vbot_template" WHERE "vbot_template"."id" = 4 LIMIT 21; args=(4,); alias=default

Two identical requests are repeated twice. Why? Why doesn't Django keep a first request in memory to avoid repetition?

Question 2: If I want to POST de VM via the HTML form of the API, only the fqdn is settable and the from_template field is absent. So it's impossible to make a POST. I have an exception because the field from_template is not set:

NOT NULL constraint failed: vbot_vm.from_template_id

Why? How to fix it to allow a POST directly via the HTML form?

Thanks for your help.

@flaf
Copy link
Author

flaf commented Feb 13, 2022

Question 1: page given by wez on IRC. Not well read yet.

Question 2: solved by this change on serializers.py in the class VMSerializer:

 class VMSerializer(serializers.ModelSerializer):

     class Meta:
       model = VM
       fields = ['fqdn', 'created', 'from_template']

-from_template = serializers.StringRelatedField(many=False, read_only=False)
+from_template = serializers.SlugRelatedField(queryset=Template.objects.all(), slug_field='name')

serializers.StringRelatedField is necessarily read-only according to the doc. serializers.SlugRelatedField(queryset can be read-only or read-write, and the default is read-write.

@flaf
Copy link
Author

flaf commented Feb 14, 2022

Question 1: page given by wez on IRC. Not well read yet.

Question 1 solved with this change on views.py:

 class VMList(generics.ListCreateAPIView):
-    queryset = VM.objects.all()
+    queryset = VM.objects.all().select_related('from_template')
     serializer_class = VMSerializer

Or this change:

 class VMList(generics.ListCreateAPIView):
-    queryset = VM.objects.all()
+    queryset = VM.objects.all().prefetch_related('from_template')
     serializer_class = VMSerializer

With this request:

$ curl -s http://127.0.0.1:8000/api/vms | jq
[
  {
    "fqdn": "aaa.bbb.tld",
    "created": "2022-02-13T21:26:23.135078Z",
    "from_template": "ubuntu-20-04"
  },
  {
    "fqdn": "flaf.flaf.fr",
    "created": "2022-02-13T21:26:57.595711Z",
    "from_template": "redhat-8-3"
  },
  {
    "fqdn": "mariadb.in.ac-versailles.fr",
    "created": "2022-02-11T00:42:33.473687Z",
    "from_template": "ubuntu-20-04-lvm"
  },
  {
    "fqdn": "srv-1.flaf.fr",
    "created": "2022-02-11T00:41:14.897579Z",
    "from_template": "redhat-8-5"
  },
  {
    "fqdn": "srv-xxxx.flaf.fr",
    "created": "2022-02-13T23:51:43.437511Z",
    "from_template": "ubuntu-20-04"
  },
  {
    "fqdn": "toto.domain.tld",
    "created": "2022-02-11T00:41:00.931524Z",
    "from_template": "redhat-8-5"
  },
  {
    "fqdn": "www.google.com",
    "created": "2022-02-12T16:16:24.522782Z",
    "from_template": "redhat-8-3"
  },
  {
    "fqdn": "x.toto.fr",
    "created": "2022-02-12T16:14:51.645851Z",
    "from_template": "redhat-8-3"
  }
]

Here is the SQL request with the first change (select_related):

(0.001) SELECT "vbot_vm"."id", "vbot_vm"."fqdn", "vbot_vm"."created", "vbot_vm"."from_template_id", "vbot_template"."id", "vbot_template"."name", "vbot_template"."root_partition", "vbot_template"."lvm" FROM "vbot_vm" INNER JOIN "vbot_template" ON ("vbot_vm"."from_template_id" = "vbot_template"."id") ORDER BY "vbot_vm"."fqdn" ASC; args=(); alias=default

Here is the SQL requests with the second change (prefetch_related):

(0.001) SELECT "vbot_vm"."id", "vbot_vm"."fqdn", "vbot_vm"."created", "vbot_vm"."from_template_id" FROM "vbot_vm" ORDER BY "vbot_vm"."fqdn" ASC; args=(); alias=default
(0.000) SELECT "vbot_template"."id", "vbot_template"."name", "vbot_template"."root_partition", "vbot_template"."lvm" FROM "vbot_template" WHERE "vbot_template"."id" IN (1, 2, 3, 4); args=(1, 2, 3, 4); alias=default

Not sure to know which is more optimal between select_related and prefetch_related in my case?

In the doc above:

prefetch_related(*lookups)

Returns a QuerySet that will automatically retrieve, in a single batch, related objects for each of the specified lookups.

This has a similar purpose to select_related, in that both are designed to stop the deluge of database queries that is caused by accessing related objects, but the strategy is quite different.

select_related works by creating an SQL join and including the fields of the related object in the SELECT statement. For this reason, select_related gets the related objects in the same database query. However, to avoid the much larger result set that would result from joining across a ‘many’ relationship, select_related is limited to single-valued relationships - foreign key and one-to-one.

prefetch_related, on the other hand, does a separate lookup for each relationship, and does the ‘joining’ in Python. This allows it to prefetch many-to-many and many-to-one objects, which cannot be done using select_related, in addition to the foreign key and one-to-one relationships that are supported by select_related. It also supports prefetching of GenericRelation and GenericForeignKey, however, it must be restricted to a homogeneous set of results. For example, prefetching objects referenced by a GenericForeignKey is only supported if the query is restricted to one ContentType.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment