Embed Size (px)
Citation preview
BACKGROUND PROCESSING TO IMPROVE RESPONSE TIMEby Chew Kok Hoor [email protected]
Python @ SolutionX
Django Web Applications Django Mobile Apps backend Django-based Point-of-Sales system (POS) Device Connectors
Issue
Issue: Webpages / Views that takes time to response Contact Us Page Registration Page
Possible causes: Sending email Slow writes to database, business rules etc.
What happens next
Users / Visitors / Customer complains
Boss ask us to resolve it, ASAP.
Short Term Resolution We spin while we can
What happens next after what happened next earlier
More Users / Visitors / Customers
Timeout errors
Why timeout?
In Linux, we use NGINX – GUNICORN - DJANGO There is a limit of exact concurrent request that
a server can handle. General formula for no. of workers is:
(2 x [number of cores]) + 1 Therefore, 8 cores = (2 x 8) + 1 = 17
Slow response = handle less requests per second
Possible solution
Background Processing
Pros n Cons Disadvantages
New process required On failure unable to notify users thru response to
request. Worse case, lost contact to user who submitted request. Need to monitor tasks not done / failed
Advantages Timeout issue resolved Users get quicker response Can choose to process N tasks at a time Can retry tasks etc., background process can afford
slower response
What should be done?
1. Break down tasks of slow responding requests
2. Process tasks in background
Preparation Stage
Perform slow running task in Background
Submit task for
processing
Tasks BreakdownPreparation Stage record necessary
information for task to execute properly or for user to track status.
activities to prevent inconsistency or breaking of business rules
preferably a means of contacting user when it is done (email, mobile notification, etc.)
Slow Running tasks Email sending Create and setup
new database Setup new user
login roles, etc. Complex
calculations or setup
Choosing Task Queue softwareSoftware Pro Consdjango-background-tasks(https://pypi.python.org/pypi/django-background-tasks)
- Very easy setup- Easy to use- Good for small volume
- Uses Django ORM for backend- Absolutely terrible formedium-to-high volume
RQ(http://python-rq.org/)(https://github.com/ui/django-rq)
- Flexible, powered by Redis- Lower memory footprint than Celery- Great for high volume
- Not as many features as Celery- Medium difficulty setup- Redis only option for storage
Celery(http://www.celeryproject.org/)
- De facto Django standard- Manydifferent storage types- Flexible,- Full-featured- Great for high volume
- Challenging setup- overkill for slower traffic sites
From Two Scoops of Django - Best Practices For Django 1.8 by Daniel Roy Greenfeld & Audrey Roy Greenfeld (http://twoscoopspress.org/)
Choosing Task Queue Software
Django Background Tasks- background processing code
INSTALLATION
pip install django-background-tasks
settings.py
INSTALLED_APPS += ( ‘background_task’,)
Django Background Tasks- background processing code
import time
from background_task import background
@background(schedule=0)def dbt_task(data): print "[START] django-background-tasks for: " + data.get("name","") time.sleep(10) print "[DONE] django-background-tasks for: " + data.get("name","")
Django Background Tasks- initiate background task
from . import taskstasks.dbt_task(request.POST.copy())
Django Background Tasks- start worker (only 1 allowed)
To start background processing process:
python manage.py process_tasks
django-rq- background processing codeINSTALLATION
pip install django-rq
settings.py
INSTALLED_APPS += ( 'django_rq',)
RQ_QUEUES = { 'default': { 'HOST': 'localhost', 'PORT': 6379, 'DB': 0, 'DEFAULT_TIMEOUT': 360, }}
django-rq- background processing code
import time
def rq_task(data): print "[START] rq for: " + data.get("name","") time.sleep(10) print "[DONE] rq for: " + data.get("name","")
django-rq- initiate background task
from . import rq_tasksimport django_rq
django_rq.enqueue(rq_tasks.rq_task, request.POST.copy())
django-rq- start worker(s)
First, start redis (e.g. /usr/local/bin/redis-server)
To start background processing process (supports more than 1 worker(s))
python manage.py rqworker default
Demo
Other Practical Usage of Background ProcessingNot just for response time
Scheduling (require additional components, e.g. RQ Scheduler)
Enforce a separation between front-end logic and back-end logic
Batch processing (daily import / export of data)
Reports generation Maintenance - expiring old sessions, sweeping
caches
