Scale Perf Best Practices

8/14/2019 Scale Perf Best Practices

1/39


2/39

2

Standing in for George


3/39

3

Scalability vs. Performance

Scalability:Ability to gracefully handleadditional traffic while maintainingservice quality.

Performance:Ability to execute a singletask quickly.

Often linked, not the same.


4/39

4

Why are Scalability andPerformance Important?

No hope of growth otherwise.

Scalability means you can handleservice commitments of the future.

Performance means you can handle theservice commitments of today.

Both act symbiotically to mean cost-efficient growth.


5/39

5

Why PHP?

PHP is a completely runtime language

Compiled, statically typed languages arefaster.

BUT: Scalability is (almost) never a factor of the

language you use Most bottlenecks are not in user code

PHPs heavy lifting is done in C PHP is fast to learn PHP is fast to write PHP is easy to extend


6/39

6

When to Start

Premature optimization is the root of all evil DonaldKnuth

Without direction and goals, your codewill only get more obtuse with littlehope of actual improvement in

scalability or speed.Design for refactoring, so that when you

need to make changes, you can.


7/39

7

Knowing When to Stop

Optimizations get exponentially moreexpensive as they are accrued.

Strike a balance between performance,scalability and features.

Unless you ship, all the speed in theworld is meaningless.


8/39

8

No Fast = True

Optimization takes effort.

Some are easier than others, but nosilver bullet.

Be prepared to get your hands dirty.


9/39

9

General Best Practices

1. Profile early, profile often.

2. Dev-ops cooperation is essential.

3. Test on production data.4. Track and trend.

5. Assumptions will burn you.


10/39

10

Scalability Best Practices

1. Decouple.

2. Cache.

3. Federate.4. Replicate.

5. Avoid straining hard-to-scale

resources.


11/39

11

Performance Best Practices

1. Use a compiler cache.

2. Be mindful of using external datasources.

3. Avoid recursive or heavy loopingcode.

4. Dont try to outsmart PHP.

5. Build with caching in mind.


12/39

12

1. Profiling

Pick a profiling tool and learn it in and out. APD, XDebug, Zend Platform

Learn your system profiling tools

strace, dtrace, ltrace

Effective debugging profiling is about spottingdeviations from the norm.

Effective habitual profiling is about makingthe norm better.

Practice, practice, practice.


13/39

13

2. Dev-Ops Cooperation

The most critical difference in organizations thathandles crises well.

Production problems are time-critical and usuallyhard to diagnose.

Build team unity before emergencies happen.Operations staff should provide feedback on behavior

changes when code is pushed live.

Development staff must heed warnings from

operations staff.Established code launch windows, developerescalation procedures, and fallback plans are veryhelpful.


14/39

14

3. Test on Production(-ish) Data

Code behavior (especially performance)is often data driven.

Using data that looks like productiondata will minimize surprises.

Having a QA environment thatsimulates production load onallcomponents will highlight problemsbefore they occur.


15/39

15

4. Track and Trend

Understanding your historicalperformance characteristics isessential for spotting emerging

problems. Access logs (with hi-res timings)

System metrics

Application and query profiling data


16/39

16

Access log timings

Apache 2 natively supports hi-restimings

For Apache 1.3 youll need to patch it

(timings in seconds = not very useful)


17/39

17

5. When you assume

Systems are complex and often break inunexpected ways.

If you knew how your system was

broken, you probably would havedesigned it better in the first place.

Confirming your suspicions is almost

always cheaper than acting on them.

Time is your most precious commodity.


18/39

18

6. Decouple

Isolate performance failures.

Put refactoring time only where needed.

Put hardware only where needed.

Impairs your ability to efficiently jointwo decoupled application data sets.


19/39

19

Example:Static versus dynamic content

Apache + PHP is fast for dynamiccontent

Waste of resources to serve static

content from here: images, CSS,JavaScript

Move static content to a separate faster

solution for static content e.g. lighttpdon a separate box -> on ageographically distributed CDN


20/39

20

Example: Session data

Using the default session store limitsscale out

Decouple session data by putting it

elsewhere: In a database

In a distributed cache

In cookies


21/39

21

7. Cache

Caching is the core of most optimizations.Fundamental question is: how dynamic does this bit

have to be.

Many levels of caching

Algorithmic Data Page/Component

Good technologies out there: APC (local data) Memcache (distributed data) Squid (distributed page/component/data) Bespoke


22/39

22

Caching examples

Compiler cache (APC or Zend)

MySQL query cache (tune and usewhere possible)

Cache generated pages or iframes (diskor memcache)

Cache calculated data, datasets, pagefragments (memcache)

Cache static content (squid)


23/39

23

8. Federate

Data federation is taking a single data set andspreading it across multipledatabase/application servers.

Great technique for scaling data.Does not inherently promote data reliability.

Reduces your ability to join within the dataset.

Increases overall internal connectionestablishment rate.


24/39

24

9. Replicate

Replication is making synchronizedcopies of data available in more thanone place.

Useful scaling technique, very popularin modern PHP architectures.

Mostly usable for read-only data.

High write rates can make it difficult tokeep slaves in sync.


25/39

25

Problems

On the slave, you should see two threadsrunning: an I/O thread, that reads data fromthe master, and an SQL thread, thatupdates the replicated tables.

(You can see these with SHOW PROCESSLIST)Since updates on the master occur in

*multiple* threads, and on the slave in a*single* thread, the updates on the slavetake longer.

Slaves have to use a single SQL thread tomake sure queries are executed in thesame order as on the master


26/39

26

The more writes you do, the more likelythe slaves are to get behind, and thefurther behind they will get.

At a certain point the only solution is tostop the slave and re-image from themaster.

Or use a different solution: multimaster, federation, split architecturesbetween replication and federation,etc


27/39

27

Other uses of replication

Remember replication has other usesthan scale out

Failover

Backups


28/39

28

10. Avoid Straining Hard-to-Scale Resources

Some resources are inherently hard to scale Uncacheable data

Data with a very high read+write rate

Non-federatable data Data in a black-box

Be aware of these limitations and be extracareful with these resources.

Try and poke holes in the assumptions aboutwhy the data is hard to manage.


29/39

29

11. Compiler Cache

PHP natively reparses a script and itsincludes whenever it executes it.

This is wasteful and a huge overhead.

A compiler cache sits inside the engineand caches the parsed optrees.

The closest thing to fast = true

In PHP5 the real alternatives are APCand Zend Platform.


30/39

30

12. Xenodataphobia

External data (RDBMS, App Server, 3rdParty data feeds) are the number onecause of application bottlenecks.

Minimize and optimize your queries.

3rd Party data feeds/transfers areunmanageable. Do what you can to

take them out of the critical path.


31/39

31

Managing external data andservices

Cache it (beware of AUPs for APIs)

Load it dynamically

(iframes/XMLHttpRequest)

Batch writes

Ask how critical the data is to your app.


32/39


33/39

33

Indexing problems

Lack of appropriate indexing

Create relevant indexes. Make sureyour queries use them. (EXPLAIN is

your friend here.)

The order of multi-column indexes isimportant

Remove unused indexes to speed writes


34/39

34

Schema design (MySQL)

Use the smallest data type possible

Use fixed width rows where possible (preferchar over varchar: disk is cheap)

Denormalize where necessaryTake static data out of the database or use

MEMORY tables

Use the appropriate storage engine for eachtable


35/39

35

Queries

Minimizing the number of queries is always a goodstart. Web pages that need to make 70-80 queriesto be rendered need a different strategy: Cache the output Cache part of the output Redesign your schema so you can reduce the number of

queries Decide if you can live without some of these queries.

Confirm that your queries are using the indexes you

think that they areAvoid correlated subqueries where possible

Stored procedures are notably faster


36/39

36

13. Be Lazy

Deeply recursive code is expensive inPHP.

Heavy manual looping usually indicates

that you are doing something wrong.

Learn PHPs idioms for dealing withlarge data sets or parsing/packing

data.


37/39

37

14. Dont Outsmart Yourself

Dont try to work around perceived inefficiencies inPHP (at least not in userspace code!)

Common bad examples here include: Writing parsers in PHP that could be done with a simple

regex. Trying to circumvent connection management in

networking/database libraries. Performing complex serializations that could be done with

internal extensions. Calling out to external executables when a PHP extension

can give you the same information. Reimplementing something that already exists in PHP


38/39

38

15. Caching

Mentioned before, but deserves asecond slide: caching is the mostimportant tool in your tool box.

For frequently accessed information,even a short cache lifespan can beproductive.

Watch your cache hit rates. A non-effective cache is worse than nocache.


39/39

39

Thanks!

There are longer versions of this talk athttp://omniti.com/~george/talks/

There are good books on these topics as well:

Advanced PHP Programming, G. Schlossnagle Building Scalable Web Sites, C. Henderson

Scalable Internet Architectures, T. Schlossnagle

Compulsory plug: OmniTI is hiring for a

number of positions (PHP, Perl, C, UI design)http://omniti.com/careers
http://omniti.com/~george/talks/http://omniti.com/careershttp://omniti.com/careershttp://omniti.com/~george/talks/

Documents

Scale Perf Best Practices