Upload
dustin-whittle
View
2.566
Download
2
Embed Size (px)
DESCRIPTION
Building a platform from open source
Citation preview
Building a pla+orm from open source
Dustin Whittle
Building a pla+orm from open source
Who am I? • A hacker on the web for the last decade • Worked with symfony since open source • Developed ysymfony at Yahoo! • Consultant helping startups scale web apps
• A speaker at conferences around the world
[email protected] @dusBnwhiDle
hDp://dusBnwhiDle.com/
When I started developing…
It was really simple
1 shared host / simple PHP scripts
Then I started working on lots of consulBng projects
I realized I needed a beDer set of tools
I started exploring web frameworks
I realized reading and using other peoples code made me a
better, more productive developer
I started talking about how frameworks like symfony and
YUI improve quality and productivity at conferences
Then I went to work at Yahoo to help build a frontend platform
based on an open source framework
I realized web application stacks are very similar
Lots of the same problems and
solutions
Users
Load Balancers
Frontend
PHP APC, PEAR, PECL
Linux
symfony / YUI
Apache Custom Modules
Backend
MySQL/Memcache Web Services / YQL Ad API User API
Why a frontend pla+orm?
Build applicaBons to requirements Do exactly what you need: no more, no less
Choose open source or build your own
Everyone uses a framework If you use open source, support/maintenance is free
Understand that frameworks add a lot of overhead (features have a cost)
Needs from a frontend pla+orm Fit server environments (Linux/PHP5/Apache) Development Cycle – How easy to develop, test, and deploy?
Clean separaBon between data, logic, and display Independent model layer to fit service oriented architecture
Extensible and pluggable InternaBonalizaBon and localizaBon support Detailed documentaBon and acBve community of support
Open source and ability to contribute back
Why a framework at all? Another so]ware layer (ysymfony, yphp, yapache)
Factors out common paDerns Code Layout ConfiguraBon URL RouBng AuthenBcaBon / Security Form ValidaBon / RepopulaBon InternaBonalizaBon / LocalizaBon
Encourages good design AbstracBon > Consistency > Maintainability
The choice to adopt symfony? Philosophy
Full-‐stack framework for building complex web applicaBons
Adopt best ideas from anywhere, using exisBng code if available (Mojavi, Prado, Rails, Django)
Design Clean separaBon between Model, View, and Controller
Controller using modules and acBons Views using templates in straight PHP with helpers Easy to reuse view modules to compose a page
Layouts, Components, ParBals, Slots
The choice to use symfony
Configurability / Flexibility Features we do not want are easily disabled Use of factories for easy customizaBon
DocumentaBon / Support Community The DefiniBve Guide to symfony Askeet, Jobeet, Cookbooks, Advents
AcBve community with wiki, mailing lists, forums, irc channel
A look at Yahoo! Answers
hDp://answers.yahoo.com Yahoo! Answers is the largest collecBon of human knowledge on the Web with more than 135 million users and 515 million answers worldwide (Yahoo! Internal Data, March 2008).
Yahoo! Answers is the 2nd ranked educaBon & reference site on the web (comScore)
Available in 26 markets and 12 languages
Yahoo! Answers at the beginning
Started as a small development team on PHP4 from a fork of Yahoo! Taiwan Knowledge+
Launched December 2005 by December 2006 there were 60 million users and 65 million answers
The code base eventually became difficult to maintain and iterate new features
Large distributed development teams
JavaScript Framework UBliBes -‐ YAHOO, Dom, Event, AnimaBon, Browser History Manager, ConnecBon Manager, Cookie, DataSource, Drag and Drop, Element, Get, ImageLoader, JSON, Resize, Selector, Loader
Controls / Widgets -‐ AutoComplete, BuDon, Calendar, Charts, Color Picker, DataTable, ImageCropper, Rich Text Editor, Slider, Uploader
Container (Module, Overlay, Panel, ToolBp, Dialog), Layout Manager, Menu, TabView, TreeView
Debug – Logger, Profiler, Test
Y! Developer Network – YUI Javascript
CSS FoundaBon Reset -‐ Neutralizes browser CSS styles Base -‐ Applies consistent style foundaBon Fonts -‐ FoundaBon for typography and font-‐sizing Grids -‐ Thousands of wireframe layouts
User Interface Design PaDerns Library Proven soluBons to common interfaces hDp://developer.yahoo.com/ypaDerns/ Grade Browser Support / Progressive Enhancement
Y! Developer Network – YUI CSS
More than 275 funcBonal examples hDp://developer.yahoo.com/yui/examples/
YSlow + Performance Rules hDp://developer.yahoo.com/performance
YUI Blog hDp://yuiblog.com/
Mailing List @ Yahoo! Groups hDp://tech.groups.yahoo.com/group/ydn-‐javascript/
Y! Developer Network – DocumentaBon
The big picture
A complete pla+orm for building web applicaBons from frameworks PHP Framework JavaScript Framework CSS Framework UI Design PaDerns + Best PracBces Development Tools (logger, profiler, debugger, docs)
Unit + FuncBonal TesBng Frameworks (LIME / YUI Test)
Deployment Tools
What does Yahoo! change? Minor changes to fit our environment
Most changes are easily implemented via factories/plugins Dropped the ORM and pushed down the stack (SOA)
Added a parallel API Dispatcher (ysfAPIClientPlugin) Added dimensions to configuraBons
(ysfDimensionsPlugin) Integrated R3 translaBon/template management
(ysfR3Plugin) R3 -‐ hDp://developer.yahoo.com/r3/
Created a build and deployment soluBon (ysfBuildPlugin) Uses internal tools for packaging/deployment
Integrate support for Y! User Interface libraries (ysfYUIPlugin)
Propel or Doctrine or ???
No ORM for large projects Doctrine for medium sized projects Service Oriented Architecture
Pla+orms/Data as services Thin Controller/Fat Model Use PHP as the frontend glue
No heavy li]ing in PHP = Push down the stack
Java/C++/Erlang + JSON/XML
Localizing with dimensions Cascading ConfiguraBon based on YAML
Framework -‐> Project -‐> ApplicaBon -‐> Module Extending the cascade to be based on dimensions Dimensions can be anything (and can be chained together) Data Center + Environment for customizing configuraBons
Culture for localizing user interface + data Theme for customizing look and feel User info (is user on corporate intranet?)
Caching
Development Tools
A build and deployment system Aggregate and minify stylesheets and javascripts Rewrite templates, css, js for CDN (Akamai, S3, …) Generate translaBons for configuraBons + templates Generate configuraBon cache Aggregate core classes + remove debug statements Run lint, unit, funcBonal tests Package applicaBons as .tgz Deployment via packages
Performance
Hello world benchmarks are not useful You do not use a framework to write hello world
die(‘hello world’);
Performance is relaBve to the features you use ORM I18N Output Escaping Forms/ValidaBon
Ability to scale != performance
What does it mean to scale?
A system whose performance improves a]er adding hardware, proporBonally to the capacity added, is said to be a scalable system.
High Availability + Scalability + Performance Bigger dataset, more traffic, maintainable Not about performance
PHP is slow, but it is very rarely your boDleneck
Languages do not scale, architectures do
Scaling – The basics in PHP PHP is rarely the boDleneck (even though it can be slow) “Most performance comes not from the language, but from applicaBon
design” – Rasmus Share Nothing Architecture Independent, self-‐sufficient, no single point of contenBon No local storage = No PHP Sessions
Use a database Use a small signed cookie (beDer)
Important data in database Individual expiraBon on session objects Small data items
Use an in-‐memory cache Memcache
Premature opBmizaBon is the root of all evil. “ vs. ‘, echo vs. print
Improve performance by queuing jobs that can take a long Bme and caching data that is expensive to query
Once an app has a well designed architecture, developers can focus on
performance specific to the app
Use Xdebug/Webgrind and XHProf to debug and profile applicaBon
performance
hDp://xdebug.org/docs/profiler
Web applicaBon performance isn’t just about the backend, a lot of open source tools can help improve the frontend.
As the framework matured and gained recogniBon more companies started
choosing symfony
At Yahoo scaling applicaBons was easy: infrastructure/process already exists
As I work on my own startup I realized I need to create the same scalable and fault tolerant environment, except without invesBng millions of dollars.
A quality pla+orm is built from best pracBces
OperaBonal Best PracBces
• Global data centers for performance and fault tolerance
• Automated provisioning of networks, servers, storage, etc.
• Automated builds + deployments • Monitoring + ReporBng (performance/
costs) • Centralized Logging + Management
Development Best PracBces
• Clean / Decoupled / Documented code • Stand on the shoulders of giants • Leverage open source communiBes
• Source Control + Project Management • Unit + FuncBonal TesBng • ConBnuous IntegraBon • Either you are always ready to deploy or you
are never ready to deploy
Do it yourself for cheap
Open source so]ware = Free • Linux • Apache • PHP • MySQL / Mongo • Memcache / Gearman • symfony / Doctrine / Twig / Swi] /
PHPUnit
Today it is possible for anyone to scale a web pla+orm just like a big company.
Welcome to the cloud.
It’s so cheap its free!
Who doesn’t love free?
hDp://aws.amazon.com/free
A load balanced cluster with a relaBonal database and storage configured how ever you want in
mulBple data centers around the world.
For free. And then for relaBvely cheap.
Its not fair! Only for new customers.
A way to monitor and manage your cloud
A free global DNS cloud service
A free source control + issue tracking service that enables public or private
collaboraBon
A complete pla+orm for developing, tesBng, and deploying scalable web
applicaBons.
Free cloud infrastructure: Amazon + Zerigo + Landscape
Free open source code:
Symfony + Zend + YUI + AlloyUI
Free development tools: GitHub + Xdebug + PHPUnit
Built in best pracBces + support
Many lessons have been learned and a beDer pla+orm has evolved.
Symfony2 and YUI3
BeDer Components: lighter, faster, cleaner
BeDer Development Model: ConBnuous IntegraBon BeDer tesBng/documentaBon
Stronger CommuniBes: Github enables contribuBons Galleries for plugins/bundles
Symfony
Symfony 2 is a set of cohesive yet decoupled components Easier to use a component to solve a single problem
Easier to build micro frameworks that solve specific problems
Teams generally prefer soluBons that are specific to their exact problem Selling the full stack can be difficult when a team only wants a few components symfony-reloaded.org
Symfony2 stands on the shoulders of giants and doesn’t reinvent the wheel.
We even use Zend Framework
Symfony2 is so good it is only alpha and there are already major sites live in
producBon.
Take a look at the Symfony2 web profiler
Take a look at the Symfony sandbox!
github.com/symfony/symfony-‐sandbox
QuesBons?
Hiring!
If you are a great developer looking to make your next move, let me introduce you to:
Plancast TripIt Awesm OpenSky Gogobot