This slide was intentionally left blank

Speed is a featureA mystical journey through Django performance optimization techniques, tools and gotchas

@martinblech @polmuz

Roadmap

How to find performance problems

Sneak peak: Front-end performance

How to fix them in Django

Why?

[Google] found that Half a second delay caused a 20% drop in traffic.

[Amazon] found that even very small delays would result in substantial and costly drops in revenue.

Users really respond to speed2006

There’s nothing like data

Don’t start with the code. Profile and gather real usage data.

There’s nothing like real data

Identify bottlenecks

New Relic

Very goodVery paid

Google Analytics

Free of chargeLess detail

Your logs

No data to third partiesHarder to use

Profiling

Complex setup requiredOverhead

New Relic

New Relic

Google Analytics Site Speed

There’s nothing like data

Let’s find the culprit!

django-debug-toolbar

django-debug-toolbar-template-timings

Time all the things!

Typical backend bottlenecks

Database

External Services

CPU Intensive task

Template Rendering

Database

Missing index

Big, unused fields

Excessive # of queries

Order

Missing Index

class Comment(Model):...created_at = DateTimeField(db_index=True)blogpost = ForeignKey(Blogpost)

class Meta:index_together = [

["created_at", "blogpost"],]

select_related()

>>> for c in Comment.objects.all(): print c.user.name# select * from comments;# select * from users where id = 1;# select * from users where id = 2;...

>>> comments = Comment.objects.all();>>> for c in comments.select_related(“user”): print c.user.name

# select comments.*, users.*# from comments, users# where comments.user_id = users.id;

prefech_related()

>>> for u in User.objects.filter(id__lt=10): print len(u.comments.all())# select * from users;# select * from comments where user_id = 1;# select * from comments where user_id = 2;...

>>> users = User.objects.filter(id__lt=10)>>> for u in users.prefetch_related(“comments”): print len(u.comments.all())

# select * from users where id < 10;# select * from comments# where user_id in (1,2,3,4,5,6,7,8,9);## Joins them in python

Demo

Background Jobs

Celery

from celery import task

@taskdef send_confirmation_email(user_id): ...

def signup(req): ... send_confirmation_email.delay(req.user.id) return HttpResponseRedirect(“/home/”)

Template Compilation

Use django.template.loaders.cached.Loader

TEMPLATE_LOADERS = (

('django.template.loaders.cached.Loader', (

'django.template.loaders.filesystem.Loader',

'django.template.loaders.app_directories.Loader',

)),

)

Template Fragment Caching

{% load cache %}

{% cache 500 “last_comments” %}

.. last comments ..

{% endcache %}

{% load cache %}

{% cache 500 “my_recent_comments” user.id %}

.. user’s recent comments ..

{% endcache %}

Caching!

Per view cache

from django.views.decorators.cache import cache_page

@cache_page(60 * 15) # secondsdef my_view(request): ...

Caching!

Per view cache

Gotchas:User-specific content

CSRF token

Caching!

Low level cache API

>>> from django.core.cache import get_cache>>> cache = get_cache('default')>>> cache.set('my_key', 'hello, world!', 30)>>> cache.get('my_key')

>>> from django.core.cache import caches>>> cache = caches['default']

Caching!

from django.core.cache import get_cachecache = get_cache('default')

def last_comments(request):comments_ids = cache.get('last_comments')if comments_ids:

comments = Comment.objects.filter(id__in=comments_ids)else:

comments = fetch_last_comments()comments_ids = [c.id for c in comments]cache.set('last_comments', comments_ids)

...

Low level cache API

Cache Invalidation

“There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors.”

post_save/post_delete are a good place to start, but it doesn’t end there!

@receiver(post_save, sender=Comment)@receiver(post_delete, sender=Comment)def invalidate_last_comments(sender, **kwargs): cache.delete('last_comments')

Caching!

Django Cacheback

from cacheback.decorators import cacheback

@cachebackdef fetch_last_comments_ids():

...

def last_comments(request):comments_ids = fetch_last_comments_ids()comments = Comment.objects.filter(id__in=comments_ids)

...

Don’t forget about the browser

80% or more of the end-user response time is spent in the front end

CombineCompress

CacheLess is more

Load slow things laterFocus on making the

important things faster

Don’t forget about the browser

Google PageSpeed Insights

webpagetest.org

Chrome & Firefox

Assets

Django Compressor{% compress js %}<script src="/static/js/one.js"></script><script>obj.value = "value";</script>{% endcompress %}

Django Assets

{% assets "js_all" %} <script async src="{{ ASSET_URL }}"></script>{% endassets %}

Assets

aload.js

<script data-aload="http://foo.com/foo.js"></script><link data-aload="http://foo.com/foo.css" rel="stylesheet">

Assets

Shameless plug: django-critical

{% critical %}<link rel="stylesheet" href="bootstrap.min.css">{% endcritical %}

Alpha stage, use at your own risk!

Q & Maybe A

ReferencesPerformance is a feature - http://blog.codinghorror.com/performance-is-a-feature/

Marissa Mayer at Web 2.0 - http://glinden.blogspot.com.ar/2006/11/marissa-mayer-at-web-20.html

Psychology of Web Performance - http://www.websiteoptimization.com/speed/tweak/psychology-web-performance/

New Relic - http://newrelic.com

Google Analytics - http://www.google.com/analytics/

Tracking Application Response Time with Nginx - http://lincolnloop.com/blog/tracking-application-response-time-nginx/

Logging Apache response times - http://www.moeding.net/archives/33-Logging-Apache-response-times.html

Django Debug Toolbar - http://django-debug-toolbar.readthedocs.org/

DDT Template Timings - https://github.com/orf/django-debug-toolbar-template-timings

Django Database Optimizations - https://docs.djangoproject.com/en/1.7/topics/db/optimization/

Two Hard Things - http://martinfowler.com/bliki/TwoHardThings.html

Django Cache Docs - https://docs.djangoproject.com/en/1.7/topics/cache/

Django Cacheback - http://django-cacheback.readthedocs.org/en/latest/index.html

Cached Templates - https://docs.djangoproject.com/en/1.7/ref/templates/api/#django.template.loaders.cached.Loader

Google PageSpeed Insights - https://developers.google.com/speed/pagespeed/insights/

Web Page Test - http://www.webpagetest.org/

Django Compressor - http://django-compressor.readthedocs.org/en/1.3/

Django Assets - http://django-assets.readthedocs.org/en/0.8/

aload.js - https://github.com/pazguille/aload

django-critical - https://github.com/martinblech/django-critical

Demo - https://github.com/martinblech/pyconar2014_perfdemo