Paul Degnan Director of Web Development, Marvel Entertainment Also CEO, CFO, CTO, OMG of The...

Paul DegnanDirector of Web Development, Marvel Entertainment

Also CEO, CFO, CTO, OMG of http://www.milermeter.com

The Development and Operations Practices Even Startups

Really Really Can't Afford to Skimp On

Rationale

● Young enterprises have limited resources● Choices must and will be made● Some things aren't optional● This talk will be pragmatic, specific, and

opinonated

Logo SpewLogo Spew

● Load Testing● Performance Testing● Monitoring ● Security● Tech Hiring / Retention● Lightning Round

Topics

● Load testing is the process of putting demand on a system or device and measuring its response.

http://en.wikipedia.org/wiki/Load_testing

● Your application's behavior under load should not be a surprise at launch

● If you experience a load event, then by definition you are interfacing with a large number of customers

● Most testing occurs in unrealistic conditions, with one or two users hitting a server.

Load Testing

Load Testing

Load Testing

Apache Bench Demo

● Load testing is hard to get right

– Apache Bench can only soak the server: you also must execute a sustained test to see if performance characteristics change

– Almost no worthwhile tests can be done without scripting input

– Scripting input will have to be variable; test with different login accounts

– Get training or hire experienced users● Dominant commercial tools are

– SOASTA– LoadRunner

Load Testing

● Performance testing is testing the responsivness of your site from a consumer perspective in the browser

● 57 of shoppers at a travel site waited 3 seconds or less before abandoning

http://www.akamai.com/html/about/press/releases/2010/press_061410.html

● Page speed also affects SEO and ad ranking● Performance testing tells you what resources are

slowing down your page

Performance Testing

● Yslow (can test local dev)

● WebpageTest (data firehose)

● Pagespeed Insights (can test local dev. Also, google. 'nuff said)

Performance Testing

● YSlow

http://yslow.org/– Evaluates against Yahoo's list of best

practices– 23 rules (of varying importance)– Have to review “grades” to see if they're

valid

Performance Testing

Performance Testing (YSlow output)

● WebpageTest

http://www.webpagetest.org/– Detailed waterfall graphs with timing at a

granular level– Can run tests from servers in many

locations– Runs test twice, to show cached/uncached

performance– Useful for seeing performance of each

resource

Performance Testing

Performance Testing (WebpageTest output)

Performance Testing (WebpageTest output)

Performance Testing (WebpageTest output)

● PageSpeed Insights

https://developers.google.com/speed/pagespeed/insights

– Browser extension also available– Similar output to Webpagetest – Descriptions more readable and therefore helpful

Performance Testing

Performance Testing (PageSpeed plugin)

Performance Testing (PageSpeed site)

● Your site will not perform perfectly at all times● Monitor different layers of the stack

– Network– Web Application– Database

● Outages at more than one layer tell you which layers are affected

Monitoring

● Hosting generally comes with monitoring● You should monitor above and beyond that

– Even if host monitoring works, layer of remove introduces possibility of error

● SiteScope: Part of HP family like Loadrunner– You'll find people trained in it– Expensive

● Cheaper small services like Pingdom

Monitoring

● Two types of business

– Those that have been hacked or infiltrated already– Those that haven't been hacked yet

● Most of security industry produces implementations of CYA protocol

– You'll still learn something from a scan– But don't “check the box”

● WAF can be a stopgap

● But secure coding practices are the surest defense

● You can have as much security as you want to pay forhttp://securityreactions.tumblr.com/post/52945594604/when-a-developer-says-their-app-is-secure-because-its

Security

● Learn the OWASP top 10http://owasptop10.googlecode.com/files/OWASP%20Top%2010%20-%202013.pdf

Security

http://xkcd.com/327/

● Technical people must be hired by other technical people

● Programmers must write code in the hiring process

● Programming skills aren't everything● Gillian Welch: “We were gonna do it

anyway”

Tech Hiring

● Programmers must write code in the hiring process

– Code Samples

● Gives you a sense of what a candidate considers worth showing off● Fakeable● If they point you to their github, it's a very good sign. But you still have to

review the code– Whiteboard coding

● Excellent way to observe thought process● Can elevate stress level of interview● Places too much emphasis on syntax knowledge

– Coding exercies

● Allows you to assign a task that represents the business you do● Unfakeable● Weeds out candidates who aren't serious

Tech Hiring

● Programming skills aren't everything

– Must be able to work with other people● Beware: Arrogant techs shut down communication● Must be able to work with your team. Have them

meet other people on the team– Must be pragmatic (Not true for all businesses. IS true

for yours.)● Architecure astronauts don't mean to kill small

businesses. But it is their nature.– Willingness to take ownership is the most important

non-tech quality

Tech Hiring

● Gillian Welch: “We're gonna do it anyway” (From Everything is Free Now. Help her out and buy it.)

– Many (not all) programmers love to code. Hire those.– Ask for their github– Open source projects– Ask them what they're curious about (they should be

curious about something)– Don't hire offshore development. You get inferior work

and insult your own people

Tech Hiring

● The best talent can work wherever they want. If money motivates them, they can go to a large corporation and get stable cash

● Developers respond to the chance to learn– Learn how to build a business (for you)– Learn to use new technologies

(But people must be paid fairly – well enough to get money “off the table”)

Retention

● Dan Pink: The Puzzle of Motivation http://www.ted.com/talks/dan_pink_on_motivation.html

Retention

● Autonomy● Mastery ● Purpose

Retention

● Joel Spolsky's Smart and Gets Things Done: http://www.joelonsoftware.com/items/2007/06/05.html

● Most or all of that content is at the articles below:

– http://www.joelonsoftware.com/articles/fog0000000050.html

– http://www.joelonsoftware.com/articles/HighNotes.html

– http://www.joelonsoftware.com/articles/FindingGreatDevelopers.html

– http://www.joelonsoftware.com/articles/FieldGuidetoDevelopers.html

– http://www.joelonsoftware.com/articles/SortingResumes.html

– http://www.joelonsoftware.com/articles/ThePhoneScreen.html

– http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html

Tech Hiring

● Mobile● Source Control● Database Administration● Caching● Build Small, Grow Organically

Lightning Round

Lightning Round: Mobile

KPCB Internet Trends 2013 by Mary Meeker:http://www.slideshare.net/kleinerperkins/kpcb-internet-trends-2013

Lightning Round: Mobile

Lightning Round: Mobile

● Distributed (new hotness)

– Git – Mercurial

● SVN

● Visual tools:

– Tortoise (Win SVN)– Cornerstone (Mac SVN)– GitHub (Mac/Win Git)– Tower (Mac Git)

Lightning Round: Source Control

Adapted from http://xkcd.com/303/

● Learn to keep your DB happy and healthy– Learn about replication / sharding for your

platform– Yes, you need that index– Learn the instrumentation you need to inspect

long running queries– Queries are easy to screw up– ORM is a double edged sword. You still need

db expertise

Lightning Round: Database Admin

● Caching allows you to save processor time on your webservers and avoid db lookups

● Pretty close to required nowadays● Caching systems

– Redis● Different datatypes● Persistence

– Memcached● Creates operational tight coupling between client and

server– Varnish

● Low complexity, limitations

Lightning Round: Caching

● Build small and iterate

– Every feature you create carries opportunity cost for the others you didn't

– Sometimes you won't know what the real feature is – Larger feature sets mean:

● More code ● More bugs● More testing cost

Lightning Round: Iterate

pdegnan@marvel.com

paul.degnan@gmail.com

@pauldegnan

@gmappedometer

http://marvel.com

http://milermeter.com

cough cough Marvel is hiring and we play with Mongo and Ruby and Backbone and Sass and get millions of visits per

month and you get free previews of the movies and passes to NY Comicon cough cough

Thank You