Startup survival lessons

Engineering: Weekly Tech Talk

Startup Survival Lessons

Martin CozziMay 2014

Developer

Development environment

● Install all the things

Development environment

● Install all the things● Manually

Development environment

● Install all the things● Manually● Don’t document any of the steps

Development environment

● Install all the things● Manually● Don’t document any of the steps● Or the versions

Development environment

● Install all the things● Manually● Don’t document any of the steps● Or the versions

Developersdevelopers developers

Development environment(s)

● Install all the things | What things?

Development environment(s)

● Install all the things | What things?● Manually | Mac? Linux? Windows?

Development environment(s)

● Install all the things | What things?● Manually | Mac? Linux? Windows?● Don’t document any of the steps● Or the versions

○ Postgres 9.1, 9.2, 9.3, 9.4a○ ES 0.90.4, 0.90.11, master

Development environment(s)

● Install all the things | What things?● Manually | Mac? Linux? Windows?● Don’t document any of the steps● Or the versions

○ Postgres 9.1, 9.2, 9.3, 9.4a○ ES 0.90.4, 0.90.11, master○ Ruby 1.9.3, 2.0.0, 2.1.0, master

I have an idea

Development environment

● Install all the things● Manually● Don’t document any of the steps● Or the versions

Development environment

● Install all the things● Manually● Don’t document any of the steps● Or the versions

IN ONE PLACE

Development environment

Development environment

Development environment

Development environment

SPoF

Availability

Expectations

I have an idea

Development environment

Lesson 1

Configuration management

still by developers developers

Chef

chef-solo

Development environment

Development environment

● Install all the things● Manually● Don’t document any of the steps● Or the versions

code >

Configuration management

● Consistent workstation○ developers○ sandbox○ production

● Shared● Code

Lesson 2

Deployments

Versionsmatter.

dev -> sandbox -> prod

dev -> sandbox -> prod

dev -> sandbox -> prod

dev -> sandbox -> prod

dev -> sandbox -> prod

dev -> sandbox -> prod

dev -> sandbox -> prod

dev -> sandbox -> prod

dev -> sandbox -> prod

Blue Green

Blue Green deploys

Blue Green deploys

Blue Green deploys

Blue Green deploys

Blue Green deploys

Canary

Canary deploys

Lesson 3

OwnershipIf you wrote it support it.

Test it

Measure it

Log it

Monitor it

Externally too

Externally too

Watch it

bulletins over time

Look for the signs

Lesson 4

PrioritizationPick your battles

Prioritization

● Priority○ p0, p1, p2, p3○ Critical, High, Medium, Low

Prioritization

● Priority○ p0, p1, p2, p3○ Critical, High, Medium, Low

● Channels○ Email○ Hipchat○ Text○ Phone call

Get some sleep!

Lesson 5

CommunicationEscalate, but don’t panic.

NotifySupport, Bizdev, Engineering

Fixbut don’t take shortcuts

UpdateSupport, Bizdev, Engineering

Post MortemsHelp you learn from your mistakes.

TestsCatch it early on next time

Automate

Lesson 5

Cattle not Cats(don’t be sad)

If someone knows an IPnuke the box

(clouds don’t have feelings)it’s OK

Lesson 6

Time > MoneyYMMV

Blackboxes are great

Instant value

EBS

EBS

● Stop your instance

EBS

● Stop your instance● No data loss on deletion (instance)

EBS

● Stop your instance● No data loss on deletion (instance)● Resize

EBS

● Stop your instance● No data loss on deletion (instance)● Resize● Snapshot

EBS

● Stop your instance● No data loss on deletion (instance)● Resize● Snapshot● IOPS

Blackboxes are greatuntil they break

Monitor Cost

High Level

AWS Breakdown

AWS / service breakdown

Conclusion

● Code > Wiki

Conclusion

● Code > Wiki● Systems fail

Conclusion

● Code > Wiki● Systems fail● Own your stuff

Conclusion

● Code > Wiki● Systems fail● Own your stuff● Communicate

Conclusion

● Code > Wiki● Systems fail● Own your stuff● Communicate● Track your costs

Conclusion

● Code > Wiki● Systems fail● Own your stuff● Communicate● Track your costs● Be proud

Questions?