Upload
matteo-moretti
View
645
Download
2
Embed Size (px)
Citation preview
Rome, 28 October 2016
Scaling Symfony apps
WHO AM I?
Matteo Moretti
CTO @
website: madisoft.ittech blog: labs.madisoft.it
Scalability
It’s from experience.
There are no lessons.
Nuvola
● > 3M HTTP requests / day● ~ 1000 databases● ~ 350GB mysql data● ~ 180M query / day● ~ 25M of media files● ~ 4.50TB of medis files● From ~5k to ~200k sessions in 5 minutes
Scalability
Your app is scalable if it can adapt to support an increasing amount of data
or a growing number of users.
“But… I don’t have an increasing load”
(http://www.freepik.com/free-photos-vectors/smile - Smile vector designed by Freepik)
“Scalability doesn’t matter to you.”
(http://www.freepik.com/free-photos-vectors/smile - Smile vector designed by Freepik)
“I do have an increasing load”
(http://www.freepik.com/free-photos-vectors/smile - Smile vector designed by Freepik)
Your app is growing
But… suddenly…
Ok, we need to scale
Scaling… what?PHP code?
Database?
Sessions?
Storage?
Async tasks?
Everything?
Can Node.js scale?
Can Symfony scale?
Can PHP scale?
Scaling is aboutapp architecture
App architecture
How can you scale your web server if you put everything inside?
Database, user files, sessions, ...
App architecture / Decouple
● Decouple services● Service: do one thing and do it well
App architecture / Decouple
4 main areas
1. web server2. sessions3. database4. filesystem
There are some more (http caching, frontend, queue systems, etc): next talk!
Web server
Many small webservers
(scale up vs scale out)
Web server
NGINX + php-fpm
PHP 7
Symfony 3
Web server / Cache
PHP CACHE
SYMFONY CACHE
DOCTRINE CACHE
Web server / PHP Cache
Opcache
Bytecode caching
opcache.enable = On
opcache.validate_timestamps = 0
https://tideways.io/profiler/blog/fine-tune-your-opcache-configuration-to-avoid-caching-suprises
PHP code / Symfony cache
● Put Symfony cache in ram ● Use cache warmers during deploy
releaseN/var/cache -> /var/www/project/cache/releaseN
“/etc/fstab”
tmpfs /var/www/project/cache tmpfs size=512m
PHP code / Doctrine cache
● Configure Doctrine to use cache● Disable Doctrine logging and profiling on prod
doctrine.orm.default_metadata_cache:type: apc
doctrine.orm.default_query_cache:type: apc
doctrine.orm.default_result_cache:type: apc
PHP code / Cache
DISK I/O ~ 0%
Monitor
Measure
Analyze
PHP code / Profiling
XHProf
Blackfire
New Relic
PHP code / Recap
● Easy● No need to change your PHP code● It’s most configuration and tuning● You can do one by one and measure how it affects performance● Need to monitor and profile: New Relic for PHP● Don’t waste time on micro-optimization
Take away: use cache!
Sessions
● Think session management as a service● Use centralized Memcached or Redis (Ec2
or ElasticCache on AWS)● Avoid sticky sessions (load balancer set up)
Session / Memcached
No bundle required
https://labs.madisoft.it/scaling-symfony-sessions-with-memcached/
Session / Redis
https://github.com/snc/SncRedisBundle
https://labs.madisoft.it/scaling-symfony-sessions-with-redis/
Session / Redisconfig.yml
framework: session: handler_id: snc_redis.session.handler
Session / RedisBundle config
snc_redis: clients: session_client: dsn: '%redis_dsn_session%' logging: false # https://github.com/snc/SncRedisBundle/issues/161 type: phpredis session: client: session_client locking: false prefix: session_prefix_ ttl: '%session_ttl%'
Session / Redisparameters.yml
redis_db: 3redis_dsn_session: 'redis://%redis_ip%/%redis_db%'redis_ip: redis-cluster.my-lan.comsession_ttl: 86400
Session / Recap
● Very easy ● No need to change your PHP code● Redis better than Memcached: it has persistence and many other features● Let AWS scale for you and deal with failover and sysadmin stuff
Take away: use Redis
Database
Aka “The bottleneck”
Database
Relational databases
Database
NOSQL db?
Database
If you need data integrity do not replace your SQL db with
NOSQL to scale
Database
How to scale SQL db?
Database
When to scale?
Database
If dbsize < 10 GB
dont_worry();
Database / Big db problems
● Very slow backup. High lock time● If mysql crashes, restart takes time● It takes time to download and restore in dev● You need expensive hardware (mostly RAM)
Database / Short-term solutions
Use a managed db service like AWS RDS
● It scales for you● It handles failover and backup for you
But:
● It’s expensive for big db● Problems are only mitigated but they are still there
Database / Long-term solutions
Sharding
Database / Sharding
Split a single big db into many small dbs
(multi-tenant)
Database / Sharding
● Very fast backuo. Low lock time● If mysql crashes, restart takes little time● Fast to download and restore in dev● No need of expensive hardware● You arrange your dbs on many machines
Database / Sharding
● How can Symfony deal with them?● How to execute a cli command on one of them?● How to apply a migration (ie: add column) to 1000 dbs?● …...
Database / Sharding
Doctrine DBAL & ORM
Database / Sharding
Define a DBAL connection and a ORM entity manager for each db
https://symfony.com/doc/current/doctrine/multiple_entity_managers.html
Database / Sharding
doctrine: orm: entity_managers: global: connection: global shard1: connection: shard1 shard2: connection: shard2
doctrine: dbal: connections: global:
….. shard1: …… shard2: …... default_connection: global
Database / Sharding
This works for few dbs (~ <5)
Database / Sharding
Doctrine shardinghttp://docs.doctrine-project.org/projects/doctrine-dbal/en/latest/reference/sharding.html
Database / Doctrine sharding
● Suited for multi-tenant applications● Global database to store shared data (ie: user data)● Need to use uuid
http://docs.doctrine-project.org/projects/doctrine-dbal/en/latest/reference/sharding.html
Database / Sharding
Configuration
doctrine: dbal: default_connection: global connections: default: shard_choser_service: vendor.app.shard_choser shards: shard1: id: 1 host / user / dbname shard2: id: 2 host / user / dbname
Database / Sharding
ShardManager Interface
$shardManager = new PoolingShardManager();
$currentCustomerId = 3;$shardManager->selectShard($currentCustomerId);// all queries after this call hit the shard where customer // with id 3 is on
$shardManager->selectGlobal();// the global db is selected
Database / Sharding
● It works but it’s complex to be managed● No documentation everywhere● Need to manage shard configuration: adding a new
shard?● Need to parallelize shard migrations: Gearman?● Deal with sharding in test environment
Database / Recap● NOSQL is not used to scale SQL: they have different purposes. You can
use both.● Sharding is difficult to implement● Need to change your code● Short-term solution is to use AWS to leverage some maintenance● Doctrine ORM sharding works well but you need to write code and
wrappers. Best suited for multi-tenant apps● When it’s done, you can scale without any limit
Take away: do sharding if your REALLY need it
Filesystem
Users upload files: documents, media, etc
How to handle them?
Filesystem
● Need of filesystem abstraction● Use external object storage like S3● Avoid using NAS: it’s tricky to be set-up correctly
Filesystem / Abstraction
● FlysystemBundle
● KnpGaufretteBundle
https://github.com/1up-lab/OneupFlysystemBundle
Filesystem / Abstractionhttps://github.com/1up-lab/OneupFlysystemBundle
● AWS S3● Dropbox● FTP● Local filesystem● ...
Filesystem / Abstraction
Configuration
oneup_flysystem: adapters: s3_adapter: awss3v3: client: s3_client bucket: "%s3_bucket%"
oneup_flysystem: adapters: local_adapater: local: directory: ‘myLocalDir’
Filesystem / Abstraction
Configuration
prod.yml
oneup_flysystem: filesystems: my_filesystem: adapter: s3_adapter
dev.yml
oneup_flysystem: filesystems: my_filesystem: adapter: local_adapter
Filesystem / Abstraction
Usage
// League\Flysystem\FilesystemInterface
$filesystem = $container->get(‘oneup_flysystem.my_filesystem’);
$path = ‘myFilePath’;$filesystem->has($path);$filesystem->read($path);$filesystem->write($path, $contents);
Filesystem / Recap
● Easy● Need to change your PHP code ● Ready-made bundles● Avoid local filesystem and NAS
Take away: use FlystemBundle with S3
Scaling / Recap
● Sessions and filesystem: easy. Do it● PHP code: not difficult. Think of it. Save money.● Database: very hard. Think a lot● Queue systems, http cache and some other stuff: next
talk but think of them as services
THANK YOU
WE ARE HIRING!(wanna join? ask us or visit our website)
QUESTIONS?