Embed Size (px)
DESCRIPTION
My user group presentation.
Citation preview
AppDynamics: Making Healthcare Perform
Mike RuangutaiDirector, Platform EngineeringCornerstone Ondemand
Who the hell is this guy?
We’re hiring15 folks in some of the following roles:● API Architect● Lead engineer● Sr. Engineer, Platform● Sr. Engineer, Data
In my prior lives...
Director of Engineering
Technical Director
We’re going to talk about this one primarily.
At Kareo...
Director of Engineering
● Responsible for…
○ Technology architecture, management, strategy, and culture.
○ Scaling team
■ Built out Java team, and stood up offshore teams.
Philosophy
Goals is to get around this circle as
quickly as possible.
Philosophy
Which means deploying more quickly
and frequently.
Diametric oppositionHumans are at the root of all mistakes that happen in software.
Diametric oppositionHumans are the root of the creative process.
Diametric opposition...So let’s put in some gates to increase trust.
Diametric opposition● QA
o Paid to question changes● Operations
o Paid to prevent changes● Software Engineers
o Paid to make as many changes as possible.
Diametric opposition
Oh wait, and we want to do this Agile thing too.
Diametric opposition
Agile or Safeis easy.
Diametric opposition
Agile or Safeis easy.
Diametric opposition
Agile and Safeis hard.
Invest in tools
How do we employ tools at each step
so we can be both agile and safe and
ultimately increase trust?
Invest in toolsDevelopment operations
● Team City, Jenkins● Chef, Puppet● Selenium
Technology metrics● Gomez● Pingdom● AppDynamics● Splunk, Sumologic
Product metrics● Mixpanel● Flurry● Google Analytics● AppDynamics
EUM
Use Product + Technology metrics to get high up Maslow’s Hierarchy of needs.
Hierarchy of Needs
At minimum, get to this point.
Hell yeah! Like a boss!
The Kareo circumstance
What is Kareo?
● Multi-tenant SaaS for small office healthcare providers, .NET / SQL Server
● .NET desktop client for practice management, aka billing, scheduling
● Growing like gangbusters in all measures:
○ Organization
○ Revenue
○ Users
○ Etc.
Bet the farm
● Move from thick desktop client to web platform, aka new architecture.
● Evolve the engineering culture
● Hire, hire, hire!
….And one more thing.
Bet the farm
While you’re doing all that, let’s go ahead and acquire an EHR (electronic health
records) system that:
● Puts us in direct competition with our partners, aka significant source of
revenue
● Is already browser-based and iOS
● Is completely on Java/Oracle stack and a foreign architecture
● ...Needs to work seamlessly with our .NET application in < 9 months with no team
in place.
● Oh and hire them in secret.
.NET Architecture
Java Architecture
Integration Architecture
So why AppDynamics?
AppDynamics was the only solution that gave us the
confidence needed to launch such an ambitious project.
No other solution supported an architecture predicated on
message queues.
Us vs. Them
vs.
Crappy response times
● EHR system was notoriously slow; sometimes 1o seconds to
login
● Significant barrier to adoption and usage
Never climbing the mountain.
We couldn’t even get here.
Improved response times
Using a combination of Gomez, Apache log analyzer, and
AppDynamics…
● Quickly identified clusters of slow business transactions
● We reduce a Gomez six step workflow from > 20 seconds to ~ 6
seconds
● Sales calls were no longer painful
Identify and triage
● Lots of chatty calls
o Several thousand JDBC calls
o Superfluous JMS calls
● Application areas loading unnecessary data
● Correlating product use to performance metrics, removing
functionality
● Identified JRuby bottlenecks
● Uncovered masked exceptions, HTTP errors
o Third party API integration response times and service
failures
Ultimately led to...
Architecture redesign.
Called Project Galvatron
Trust
● Engineering, QA, and Operations using same tools,
looking at the same data.
● Less guessing, less finger pointing
Next...
● Implement AppDynamics End User Monitoring
● OpsCode Chef
● Sumologic
Learnings
● Implementing an APM like AppDynamics is a large commitment
o Application configurations
o Business transaction analysis
Which are most important?
What are the thresholds? Standard deviation? Absolute
value?
o Internal training and SOP
o Identifying appropriate actions, i.e. Pager Duty, running
scripts
o Know your architecture
RecapDevelopment operations
● Team City, Jenkins● Chef, Puppet● Selenium automation
Technology metrics● Gomez● Pingdom● AppDynamics● Splunk, Sumologic
Product metrics● Mixpanel● Flurry● Google Analytics● AppDynamics
EUM
Use Product + Technology metrics to get high up Maslow’s Hierarchy of needs.
Repeat buyer
Could use some AppDynamics at Cornerstone Ondemand.
Thanks!
Q&A
