Building A Platform From Open Source At Yahoo

Building a platform from open sourceDustin Whittle

Overview

• Why symfony?• symfony vs. ysymfony• What does scaling really mean?• Social Search: Delicious and Answers• International: ShopGenie.co.uk, FoxyTunes, Wretch.cc• Yahoo! Open Strategy

– What is the Yahoo! Open Stack?– Developer Tools

• YUI, Design Patterns, Tutorials

– Data & Social APIs• YQL: Yahoo! Query Language• Profiles, Connections, Updates, …• Geo, Flickr, Delicious, Upcoming

– YOS SDK for PHP

Who am I?

• Working with symfony since open source– symfony Core Team Member

• Responsible for the development and support of ysymfony at Yahoo!– Worked with Y! Answers, Delicious, Y! Widgets, Y!

Bookmarks, Yahoo! Application Platform– Developer advocacy for Yahoo! Developer Network

• Consultant– Commercial symfony support + training (USA)

DEVELOPER.YAHOO.COMEXAMPLES | TUTORIALS | CODE SAMPLES

YAHOO! IS POWERED BY OPEN SOURCE TECHNOLOGIES

FREEBSD | LINUX | APACHE | PHP | MYSQL | BUGZILLA | HADOOP | SYMFONY

YAHOO! EMBRACES OPEN STANDARDSW3C | MICROFORMATS | OAUTH | OPENID | OPENSOCIAL

YAHOO! GIVES BACK TO OPEN SOURCEYUI | BROWSER PLUS | DESIGN PATTERNS | R3 | YSLOW + PERFORMANCE RULES

YAHOO! SHARES ITS DATA THROUGH OPEN APIS AND WEB SERVICES

YQL | PIPES | BOSS | CONTACTS | UPDATES | MAIL | DELICIOUS | FLICKR | UPCOMING | HOTJOBS | MAPS | FIREEAGLE | GEOLOCATION | LOCAL | TRAFFIC | WEATHER |

MUSIC | ANSWERS | SHOPPING | FINANCE | TRAVEL

YAHOO! ENGAGES COMMUNITIES WITH OPEN HACK EVENTS AROUND THE WORLD

Conferences | Hack Days | HackU | Tech Talks | YDN Theater

Users

Load Balancers

Frontend

PHPAPC, PEAR, PECL, Custom Extensions

Linux

ysymfony / YUIApacheCustom Modules

Backend

MySQL/Memcache Web Services Ad API User API

Why a frontend platform?

• Rasmus says “frameworks are not well suited for Yahoo!”– Build applications to requirements

• Do exactly what you need: no more, no less• Understand that frameworks add a lot of overhead• Choosing functional components is a better fit

• Despite choosing open source or building your own– Everyone uses a framework– If you use open source, use only the pieces you need

Y! needs from a frontend platform

• Fit existing environment (RHEL/PHP5/Apache)• Development Cycle – How easy to develop, test, and

deploy?• Clean separation between data, logic, and display (MVC) • Independent model layer to fit service oriented

architecture• Extensible and pluggable• Internationalization and localization support• Detailed documentation and active community of support• Open source and ability to contribute back

Why a framework at all?

• Another software layer (ysymfony, yphp, yapache)• Factors out common patterns

– Code Layout– Configuration– URL Routing– Authentication / Security– Form Validation / Repopulation– Internationalization / Localization

• Encourages good design• Abstraction > Consistency > Maintainability

The choice to adopt symfony?

• Philosophy– Full-stack framework for building complex web

applications– Adopt best ideas from anywhere, using existing code if

available (Mojavi, Prado, Rails, Django)• Design

– Clean separation between Model, View, and Controller– Controller using modules and actions– Views using templates in straight PHP with helpers– Easy to reuse view modules to compose a page

• Layouts, Components, Partials, Slots

The choice to use symfony

• Configurability / Flexibility– Features we do not want are easily disabled– Use of factories for easy customization

• Documentation / Support Community– The Definitive Guide to symfony– Askeet, Jobeet, Cookbooks, Advents

• Active community with wiki, mailing lists, forums, irc channel

Why ysymfony for Yahoo! teams?

• Eliminate common patterns by adding a layer on PHP– Code layout/structure (MVC)– Configuration– Internationalization

• ysymfony is just a toolkit– Learn one set of tools

• Shift between multiple projects• Consistency

– Long term maintainability through platform

How Yahoo! uses symfony

• Full Stack– Yahoo! Bookmarks– Yahoo! Answers– Delicious– Yahoo! Widgets– Yahoo! Application Platform– ShopGenie.co.uk, Foxytunes, Wretch.cc– Internal Dashboards + Reporting Tools

• Components– Dependency Injection– Forms– Caching (APC/Memcache)– Routing

symfony MVC framework

A look at Yahoo! Answers

• http://answers.yahoo.com• Yahoo! Answers is the largest collection of human

knowledge on the Web with more than 135 million users and 515 million answers worldwide (Yahoo! Internal Data, March 2008).

• Yahoo! Answers is the 2nd ranked education & reference site on the web (comScore)

• Available in 26 markets and 12 languages

Yahoo! Answers at the beginning

• Started as a small development team on PHP4 from a fork of Yahoo! Taiwan Knowledge+

• Launched December 2005 by December 2006 there were 60 million users and 65 million answers

• The code base eventually became difficult to maintain and iterate new features

• Large distributed development teams (US / UK)

The big picture

• A complete platform for building web applications from frameworks– PHP Framework– JavaScript Framework– CSS Framework– UI Design Patterns + Best Practices– Development Tools (logger, profiler, debugger, docs)– Unit + Functional Testing Frameworks (LIME / YUI Test)– Deployment Tools (rsync deployment system)

What does Yahoo! change?

• Minor changes to fit our environment– Most changes are easily implemented via factories/plugins

• Dropped the ORM and pushed down the stack (SOA)– Added a parallel API Dispatcher (ysfAPIClientPlugin)

• Added dimensions to configurations (ysfDimensionsPlugin)• Integrated R3 translation/template management

(ysfR3Plugin)– R3 - http://developer.yahoo.com/r3/

• Created a build and deployment solution (ysfBuildPlugin)– Uses internal tools for packaging/deployment

• Integrate support for Y! User Interface libraries (ysfYUIPlugin)

Propel or Doctrine or ???

• No ORM for large projects• Doctrine for medium sized projects• Service Oriented Architecture

– Platforms/Backends as services (reusable to all)– Thin Controller/Fat Model (where model == services)– Use PHP as the frontend glue

• No heavy lifting in PHP = Push down the stack

• Java/C++/Erlang + JSON/XML

Localizing with dimensions

• Cascading Configuration based on YAML– Framework -> Project -> Application -> Module

• Extending the cascade to be based on dimensions– Dimensions can be anything (and can be chained

together)• Data Center + Environment for customizing configurations• Culture for localizing user interface + data• Theme for customizing look and feel• User info (is user on corporate intranet?)

• Caching

symfony I18n + L10n

• Translation + Template Management– Integrated with symfony i18n + view layers

• <r3:trans>hello</r3:trans> vs __(‘hello’)• Dictionary Management (YALA)

– SQLite/XLIFF Backend• Added additional tasks

– Importing/Exporting XLIFF dictionary– Generating sample translations for projects– Find missing strings

A build and deployment system

• Aggregate and minify stylesheets and javascripts• Rewrite templates, css, js for CDN (Akamai, S3, …)• Generate translations for configurations + templates• Generate configuration cache• Aggregate core classes + remove debug statements• Run lint, unit, functional tests• Package applications as .tgz• Deployment via packages• Integrated with Yahoo! internal tools

Hello World Performance

• Hello world benchmarks are generally not a useful measurement– You don’t use a framework to write hello world

• die(‘hello world’);

• Performance is relative to the features you use– ORM– I18N– Output Escaping– Forms/Validation

• Ability to scale != performance

What does it mean to scale?

• A system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system.

• High Availability + Scalability + Performance• Bigger dataset, more traffic, maintainable• Not about performance

– PHP is slow, but it is very rarely your bottleneck• Languages do not scale, architectures do• Planning to grow and planning to fail

– Capacity Planning– Business Continuity Planning

Scaling – Planning

• Planning hardware purchases and hosting options to have as much as you need without breaking your wallet

• Partitioning and distributing databases to support large datasets and simultaneous transactions

• Offline transactions and queuing• Monitoring your applications to find and clear

bottlenecks• Providing service APIs and using services from other

providers to increase your site's reach and capabilities• Think Minimal, Plan to grow, Plan to fail

Scaling – The basics in PHP

• PHP is rarely the bottleneck (even though it can be slow)– “Most performance comes not from the language, but from application design” –

Rasmus• Share Nothing Architecture

– Independent, self-sufficient, no single point of contention– No local storage = No PHP Sessions

• Use a database (works for distributed)• Use a small signed cookie (ideal)

– Important data in database– Individual expiration on session objects– Small data items

– Use a distributed cache• Memcache

• Forget about small efficiencies– Premature optimization is the root of all evil.

• “ vs. ‘, echo vs. print

Scaling Databases – The basics

• Drop the ORM– It’s a choice of convenience over performance

• Master/Slave Replication– First steps– Helps with reads, writes are still bottleneck

• Partitioning– Segmenting data

• Sharding (horizontal partitioning)– Segmenting data onto different physical machines– Make problems smaller, easier to grow

Improving latency with Caching

• Always use PHP opcode cache (APC, Xcache, etc)– Use for routing and i18n cache

• Memcache (distributed cache)– Use for view cache

• Distributed invalidation can be a pain– sfViewCacheManager makes this easy!– Be intelligent about cache_keys (uri, user, state)

• There is a fine line to caching– At what point do you spend more time managing the

cache, than reading from it?

Tweaking Performance

• Do not use .htaccess (move to real apache config)• Set a minimal include path• Increase realpath_cache_size + realpath_cache_ttl• Use apc.stat=0• Don’t use features you do not need

– Disable in settings.yml / factories.yml• Use core_compile to aggregate classes to reduce file i/o• Remove debug statements + optimize file lookups

– sfOptimizerPlugin / project:optimize• Use @routeName and use caching for factories

Symfony (v2)

• Symfony 2 is a set of cohesive yet decoupled components– This makes it much easier to use single component to

solve a single problem– Which makes it easier to build micro frameworks that

solve very specific problems– Yahoo! Teams generally prefer solutions that are

specific to their exact problem• Selling the full stack can be difficult when a team only wants a

few components

– Symfony 2 is the right direction, even if it breaks backwards compatability

Do it yourself for cheap

• Open source software = Free– Apache– PHP– MySQL– Memcache / Perlbal / MogileFS / Squid / Gearman– symfony / Doctrine / Propel / Swift– Nagios / Cacti

• Amazon Shared Infrastructure = Cheap– EC2 Cloud– S3 Storage + Cloudfront– SimpleDB

Y! OS – The Open Stack

What Yahoo! offer developers

• Yahoo! Developer Network• Developer Tools (YUI, etc)

• Social APIs• Profiles• Connections• Updates

• Data APIs• Flickr, Delicious, Search, etc

• Yahoo! Query Language• Yahoo! Application Platform

• OpenSocial

developer.yahoo.com

What is Yahoo! Developer Network?

The Yahoo! Developer Network offers open source tools and open data APIs to make it easy for developers to build applications and

mashups.

• 50+ APIs / Web Services– Developer Dashboard to create/manage Oauth applications– Tutorials + Code Samples on using our apis– Complete API Documentation

• Yahoo! User Interface libraries• ASTRA Flash Components• Design Patterns Library• Evangelism: Conferences / Theater / Blogs / Events

• JavaScript Framework– Utilities - YAHOO, Dom, Event, Animation, Browser

History Manager, Connection Manager, Cookie, DataSource, Drag and Drop, Element, Get, ImageLoader, JSON, Resize, Selector, Loader

– Controls / Widgets - AutoComplete, Button, Calendar, Charts, Color Picker, DataTable, ImageCropper, Rich Text Editor, Slider, Uploader

– Container (Module, Overlay, Panel, Tooltip, Dialog), Layout Manager, Menu, TabView, TreeView

– Debug – Logger, Profiler, Test

Y! Developer Network – YUI Javascript

• CSS Foundation– Reset - Neutralizes browser CSS styles– Base - Applies consistent style foundation– Fonts - Foundation for typography and font-sizing– Grids - Thousands of wireframe layouts

• User Interface Design Patterns Library– Proven solutions to common interfaces– http://developer.yahoo.com/ypatterns/

– Grade Browser Support / Progressive Enhancement

Y! Developer Network – YUI CSS

• More than 275 functional examples– http://developer.yahoo.com/yui/examples/

• YSlow + Performance Rules– http://developer.yahoo.com/performance

• YUI Blog– http://yuiblog.com/

• Mailing List @ Yahoo! Groups– http://tech.groups.yahoo.com/group/ydn-javascript/

Y! Developer Network – Documentation

A Query Language for the Web

SELECT * FROM INTERNET

• Thousands of web services that provide valuable data

• Require developers to read documentation and form URLs/queries.

• Data is isolated and can not be combined• Needs combining, tweaking, shaping even after it

gets to the developer.

Before YQL

• SQL-Like Language– Synonymous with Data access– Familiar to developers– Expressive enough to get the right data

• Self Describing - show, desc table• Allows you to query, filter and join data across Web

Services.

Y! Open Stack – YQL

YQL – Open Tables

• Twitter• Google• Facebook• Friendfeed• Wesabe• Whitepages• Zillow• Search• Weather

• Flickr• Upcoming• Delicious• Dopplr• Github• New York Times• Shopping• …any web service can be as

easy as SQL

Available on github - http://github.com/spullara/yql-tables/

YQL – Open Tables

YQL – Open Tables

YQL – Open Tables

• select * from social.connections• select * from delicious.feeds.popular• select * from flickr.photos.interestingness• select * from friendfeed.status• select * from github.checkins

YQL - Examples

OAuth Endpoint• http://query.yahooapis.com/v1/yql?q=...

Public Endpoint • http://query.yahooapis.com/v1/public/yql?q=

YQL Console • http://developer.yahoo.com/yql/console

YQL – API End Points

Books worth reading

QUESTIONS?(I will be around for the whole event, so please say hello and ask questions)

DEVELOPER.YAHOO.COMEXAMPLES | TUTORIALS | CODE SAMPLES

ENJOY THE REST OF SYMFONYLIVE 2010

WANT TO JOIN YAHOO?WE ARE HIRING!