Skip to content

Instantly share code, notes, and snippets.

@xanderim

xanderim/high_perfomance_django.md

Forked from icu0755/high_perfomance_django.md
Created October 19, 2022 12:01
Show Gist options
  • Save xanderim/7d7b85f4ff01dae8063fd52d162c85c7 to your computer and use it in GitHub Desktop.
Save xanderim/7d7b85f4ff01dae8063fd52d162c85c7 to your computer and use it in GitHub Desktop.
Download ZIP
High perfomance django notes
Raw
high_perfomance_django.md

Query Caching

init.py:

import johnny.cache
johnny.cache.enable()

Alternate Data Stores

Sharding

Russian Doll Caching

{% cache MIDDLE_TTL "post_list" request.GET.page %}
  {% include "inc/post/header.html" %}
  <div class="post-list">
  {% for post in post_list %}
    {% cache LONG_TTL "post_teaser_" post.id post.last_modified %}
      {% include "inc/post/teaser.html" %}
    {% endcache %}
  {% endfor %}
  </div>
{% endcache %}

Custom cache template tag

class CacheNode(template.Node):
  bust_param = 'flush-the-cache'
  
  def needs_cache_busting(self, request):
    bust = False
    if request.GET and self.bust_param in request.GET:
      bust = True
    return bust
  
  def render(self, context):
    value = cache.get(cache_key)
    if self.needs_cache_busting(request) or value is None:
      value = self.nodelist.render(context)
      cache.set(cache_key, value, expire_time)
    return value
    
 def jitter(num, variance=0.2):
   min_num = num * (1 - variance)
   max_num = num * (1 - variance)
   return randint(min_num, max_num)

Do slow work later

Front-end optimizations

Compress images

Serve assets from a cdn

Avoiding single points of failure

  • chaos monkey released into wild

Database

Database tuning

Postgresql

MySQL

uWSGI tuning

  • processes. start with 2x processor cores and go up. if you have other services like memcache or Varnish start with (number of cores + 1)
  • threads if your app is thread-safe. Use stats option and uwsgitop to determine the optimal number of processes and threads for your workload
  • thunder-lock This option helps balance the load better amongst all processes/threads. http://uwsgi-docs.readthedocs.io/en/latest/articles/SerializingAccept.html
  • harakiri max number of seconds a worker can take to process a single request before it is killed off. It prevents all the workers from getting tied up with long-running requests.
  • max-requests applications can leak memory over time, it tells uWSGI to respawn worker after X requests. set it to a sufficiently high number
  • post-buffering The max size of an HTTP request body in bytes (usually a file upload) that will go into memory. Larger requests will be saved to a temporary file on disk.
  • stats publish statistics about uWSGI process. 127.0.0.1:1717, /tmp/stat.sock. pip install uwsgitop
  • auto-procname A nicer human-readable process name
  • procname-prefix-spaced

Tuning Django

Databases

CONN_MAX_AGE. 300 is a good value to start with if you're unsure

Logging

It is a common source of file permissions issues. Output to STDERR and either have uWSGI log this to file or pick up the output with your process manager (upstart, systemd, supervisor) http://uwsgi-docs.readthedocs.io/en/latest/Logging.html

General security

Web accelerator

Varnish caches responses based on the URL and the contetns of the headers defined by Vary header. A typical Django request may vary on Accept-Encoding and Cookie. For anonymous requests, the cookies rarely matter. Improve your hit rate greatly by stripping them out so the anonymous requests all look the same.

You can define a get parameter to pass through the caching. Pick the same param as in custom template cache tag.

Improving your hitrate

https://varnish-cache.org/docs/4.0/users-guide/increasing-your-hitrate.html On sites there users arelogged in and page content varies for every user, split up pages such that some expensive parts do't vary per user. In some cases the only difference is the user name displayed on the screen. For these sorts of pages, you can use 2 phase rendering process. Django renders anonymized version of the page for Varnish to cache, the use AJAX to make an additional request filling in the personalized bits. The other option is to use Eege Side Include and let Varnish use that information to assemble the page for you. https://varnish-cache.org/docs/4.0/users-guide/esi.html

cache any hardcoded redirects

Security

https://github.com/bitly/oauth2_proxy

Backup

https://github.com/wal-e/wal-e

Monitoring

Instrumentation

  • What is the slowest part of my system? A time breakdown per request Pytho, SQL, cache, etc
  • What is the average response time for a request hitting Django
  • Which views are the slowest and consume the most time
  • Which database queries are the slowest and consume the most time?
  • How are all these numbers changing over time?

Load testing

Launch planning

Monitoring the launch

Server resources

  • htop
  • list of open files for process lsof
  • trace library and syscalls ltrace, strace

What to watch

  • is the load average safe? Not exceed the number of CPU cores
  • Any processes constantly using all of cpu core? split the process up across more workers to take advantage of multiple cores.
  • is the server swapping (swp)? more RAM or reduce number of running processes
  • Are any python processes using excessive memory (> 300 MB RES). Profiler
  • Varnish, cache, database use a lot of memory. If they aren't - check configuration.

Varnish

  • varnishstat
  • varnishhist
  • varnishtop
  • varnishlog

What to watch

  • is your hitrate acceptable
  • are URLs you expect to be cached actually getting served from cache?
  • are URLs that should not be cached, bypassing the cache?
  • what are the top URLs bypassing the cache? Can they be cached?
  • Are the common 404 or permanent redirects caught with Varnish instead of Django?

uWSGI

  • pip install uwsgitop

What to watch

  • is the average response time acceptable (< 1s)
  • all workers busy all the time? if there is still CPU and RAM to spare (htop) - add workers or threads

Celery

What to watch

  • Are all tasks completing successfully?
  • Is the queue growing faster than the workers can process tasks? if the server has free resources - add Celery workers, if not - add another server to process tasks

Memcached

What to watch

  • How is your hitrate? It should be > 90%. If not it could be due to a high eviction rate or poor caching strategy
  • Are connections and usage well balanced across the servers? If not you'll want to investigate efficient hashing algorithm or modify the function that generates the cache keys
  • Is the time spent per operation averaging less than 2ms? If not, you may be maxing out the hardware (swapping, network congestion, etc.)

Database Postgres

Database MySQL

What to watch

  • disks are often the bottleneck, iowait time. Check it via top as X%wa in the CPUs row.
  • number of connections is well under the maximum connections you've configured. bump up the max or investigate if that many connections are actually needed
  • watch out for "idle in transaction" connections. if you do see them, they should go away quickly. if they hang around, one of the application accessing your database might be leaking connections
  • are queries running for more than a second? they could be waiting on a lock or require some optimization.
  • check for query patterns that are frequently displayed. could they be cached or optimized?

Locking the database

Final thoughts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment