Efficient Django

David Arcos - @DZPMEfficient Django – #EuroPython 2016

Efficient Django


AbstractTips and best practices for avoiding scalability

issues and performance bottlenecks in Django

● 1) Basic concepts: the theory

● 2) Measuring: how to find bottlenecks

● 3) Tips and tricks

● 4) Conclusion (yes, it scales!)


Hi!

● I'm David Arcos

● Python/Django developer since 2008

● Co-organizer at Python Barcelona

● CTO at Lead Ratings

http://pybcn.org/

http://lead-ratings.com/


● “We improve your sales conversions, using

predictive algorithms to rate the leads”

● Prediction API, “Machine Learning as a Service”

● http://lead-ratings.com

http://lead-ratings.com/


1) Basic concepts


The Pareto Principle"For many events, roughly 80% of the effects

come from 20% of the causes"


Prioritize and focus

Focus on the few tasks that will have the most impact


Basic scalability“Potential to be enlarged to handle a growing amount of work”

● Stateless app servers

– Load balance them, scale horizontally

● Keep the state on the database(s)

– This is the difficult part! Each system is different


Database performance

● Do less requests:

– Less reads

– Less writes

● Do faster requests:

– Indexed fields

– De-normalize


Templates

● Cache them

● Jinja2 is a bit faster than the default engine

– but cache them anyways

● You can do fragment caching (for blocks)

http://jinja.pocoo.org/

https://docs.djangoproject.com/en/1.9/topics/cache/#template-fragment-caching


Cache● Generic approach: cache at each stack level

● The cache documentation is excellent

● Beware of the cache invalidation!

https://docs.djangoproject.com/en/1.9/topics/cache/


Cache● Generic approach: cache at each stack level

● The cache documentation is excellent

● Beware of the cache invalidation!

https://docs.djangoproject.com/en/1.9/topics/cache/


Bottlenecks● Where is your bottleneck?

● CPU bound or I/O bound?

– CPU? Run heavy calculations in async workers

– Memory? Compress objects before caching

– Database? Read from db replicas

● How to find it?


2) Measuring


Can't improve what you don't measure

● Measure your system to find bottlenecks

● Optimize those bottlenecks

● Verify the improvements

● Rinse and repeat!


Monitoring● System: load, CPU, memory...

● Database: q/s, response time, size

● Cache: q/s, hit rate

● Queue: length

● Custom: metrics for your app


Profiling● The cProfile module provides profiling of

Python programs by collecting data:

– Number of calls, running time, time per call...

https://docs.python.org/3/library/profile.html


timeit

● The timeit module is a simple way to time

execution time of small bits of Python code:

https://docs.python.org/3/library/timeit.html


ipdb

● Like pdb, but for ipython

– tab completion, syntax highlighting, better

tracebacks, better introspection…

● Use ipdb.set_trace() to add a breakpoint and

jump in with the debugger


django-debug-toolbar● Display debug information about the current

request/response

● Panels, very modular


django-debug-toolbar-line-profiler

● A toolbar panel for profiling

Django Debug Panel

● Chrome extension

● For AJAX requests and non-HTML responses


3) Tips and tricks


Add db indexes● Single (db_index) or multiple (index_together)

● Be sure to profile and measure!

– Sometimes it’s not obvious (i.e., admin)

– Huge difference, i.e. from 15s to 3 ms (3.5M rows)

● But: uses more space, slower writes


Do bulk operations

● Will greatly reduce the number of SQL queries:

– Model.objects.bulk_create()

– qs.update() <- maybe with F() expressions

– qs.delete()

https://docs.djangoproject.com/en/1.9/ref/models/querysets/#django.db.models.query.QuerySet.bulk_create

https://docs.djangoproject.com/en/1.9/topics/db/queries/#topics-db-queries-update

https://docs.djangoproject.com/en/1.9/ref/models/expressions/#f-expressions

https://docs.djangoproject.com/en/1.9/topics/db/queries/#topics-db-queries-delete


Get related objects

● Return FK fields in same query:

– qs.select_related()

● Return M2M fields, extra query:

– qs.prefetch_related()

https://docs.djangoproject.com/en/1.9/ref/models/querysets/#select-related

https://docs.djangoproject.com/en/1.9/ref/models/querysets/#prefetch-related


Slow admin?● Use list_select_related

● Overwrite get_queryset() with prefetch_related

● Is ordering using an index? Same for search_fields

● readonly_fields will avoid FK/M2M queries

● Use the raw_id_fields widget (or better:

django-salmonella)

● Extend admin/filter.html to show filters as <select>

https://github.com/lincolnloop/django-salmonella/


Cachalot● Caches your Django ORM queries and

automatically invalidates them


Queues and workers● Do slow stuff later

● Some operations can be queued, and executed

asynchronously in workers

● Use Celery

http://www.celeryproject.org/


Cached sessions

● Use SESSION_ENGINE to set cached sessions:

– Non-persistent: don’t hit the DB

– Persistent: don’t hit the DB… so often

https://docs.djangoproject.com/en/1.9/topics/http/sessions/#cached-sessions-backend


Persistent connections

● Use CONN_MAX_AGE to set the lifetime of a

database connection (persistence)


UUIDs

● Use UUID for Primary Keys (instead of

incremental IDs)

– Guaranteed uniqueness, avoid collisions

– UUIDs are well-indexed

● Easier db sharding


Slow tests?● Skip migrations: --keepdb

● Run in parallel: --parallel

● Disable unused middlewares, installed_apps,

password hashers, logging, etc…

● Use mocking whenever possible


4) Conclusions

● Measure first

● Optimize only the bottleneck

● Go for the low-hanging fruit

● Measure again


Good resources

● The official Django documentation

● Book: “High Performance Django”

● Blog: “Instagram Engineering”

● “Latency Numbers Every Programmer Should Know”

https://docs.djangoproject.com/

https://highperformancedjango.com/

https://engineering.instagram.com/

http://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html


Thanks for attending!

- Get the slides at http://slideshare.net/DZPM

- We are looking for engineers and data scientists!

http://slideshare.net/DZPM

Technology

Efficient Django