Want Easier Decision Making Just Listen to Your Users

Healthcare Analytics Best Practices Summit

Want easier decision-making?

Just listen to your users.

© 2015 American Medical Association. All rights reserved.

Introductions

• Jayne Gretz, VP of Market Research & Analytics

• Nicholas Abruzzo, VP of Digital Strategy & Services

2


We know physicians.

Physicians don’t have time

• They are accessing the internet more often, but the amount of time per

session is decreasing

• They do not read, they scan

• They don’t like video as much as consumers because they have to

invest too much time to even evaluate its worth

– They would prefer to scan the transcript first to decide if it’s worth their time

3


We know physicians.

Physicians want to curate their own content• They prefer to use search over navigation

• They ignore suggested results – they prefer to filter

• Even though they have preferred, trusted sites they aren’t the best for every question

• They do not bookmark, they want to select the best sites vis-à-vis their current question

• After they arrive at your site, they’ll give you two clicks then they’re back out to the search results page – too many other options that can deliver what they want

4

© 2015 American Medical Association. All rights reserved. 5

“I use Google mostly.

Or I go directly to the

site I’m interested in, but

I mostly just Google

search everything.”

“If there isn’t a great

search mechanism, it

gets frustrating. I will

just give up and go

somewhere else.”

“I prefer to use search – it’s

just easier. If I use the tabs,

the information under the tabs

is not always intuitive.”

“The suggestions

seems too

commercial to me,

like an

advertisement, as

opposed to a place

for resources.”

“Those featured topics

feel like an ‘editor’s pick.’

You need to get rid of

some of that fluff and find

a way to make it more

targeted toward the

audience looking for

particular resources.”

“I ignore the suggested

results mostly because

I ignore sponsored

links. Those

suggestions look like

ads and sponsored

sites.”


These learnings have implications for how we engage with physicians on our site and social

media platforms. The goal is an outside-in experience where the needs of our stakeholders

define the objectives of our digital ecosystem.

6


How we approach our digital strategy

• Both VoC and digital user behavior metrics are valuable in

understanding what our users want and expect from us

• How do you decide when to use which methodologies to get at your

answer?

– Web analytics deep dives and A/B/n tests are good for quick delivery, but are

limited to the scope of what the user was presented – nothing exploratory

– Usability testing and Market Research are good for thorough user input, but

will require extra time and budget

7


Blended approach

• Since we want the best of both worlds, we frequently partner to tackle

complex digital questions and projects

• Following case studies give a taste of how each team feeds the other to

improve our analysis and recommendations

8

Case 1:

Social Login

9


Case 1 | Social Login

• Offering social login as an option is becoming a best practice

• But will this hold up with our audience?

10

Digital Analytics Approach:

Test it with our users

Market Research Approach:

Ask our users



Market Research Approach

• Deployed email survey to 250 physicians, residents, and students

• Gauge awareness and usage of social media platforms for both

personal and professional use

• Probed into likelihood to use social login on AMA website

11


About half of medical students (55%), residents (55%) and PCPs

(55%) say they are not likely to use social login on the AMA site.

12

26%

29%

14%

24%

8%

36%

19%

10%

26%

10%

28%

27%

11%

21%

13%

Not At All Likely

Not Very Likely

Neither Likely nor Unlikely

Somewhat Likely

Very Likely

Med Students (N=71)

Residents (N=75)

PCPs (N=103)

Likelihood of using social login

Q. If presented as an option, how likely are you to use your social media account(s) to log into or register with the AMA website? N=249Data timeframe: August 2014


“I have a social

media account,

but have not used

it in several

months.”

“I do not like to use social

media, which I use for

pleasure, for any aspect of

my professional life.”

“I don't like giving people

access to my social

media accounts or any

more information about

me than necessary.”

“I never use

social media

accounts to

log into

anywhere.”

“I have no interest in the

AMA site, and am much

more likely to log onto

AAP.org, Sermo, Medscape

or Epocrates.”

“I would not want my

Facebook status to

constantly let

everyone know when

I have signed in or

used the AMA

website.”

Data timeframe: August 2014



• Results: neutral to negative

• Action: determined we should not move forward with investing in social

login

• Update: because of licensing changes, considering deploying as option

in a/b test format to gauge usage/interest among our total audience

population since a large number of our visitors are non-physicians

– This will now incorporate Digital Analytics Approach

14

Case 2:

Corporate Site

Usability

15


Case 2 | Corporate Site Usability

• Tasked with improving overall user experience

• How do we identify biggest pain points and where to start?

16

Digital Analytics Approach:

Web analytics deep dive

Market Research Approach:

Ask our users



Blended Approach

• Used web metrics to identify pain points and areas where there was a

gap between user expectations and content delivered

• Formal usability tests conducted – tasks selected were informed by

web metrics deep dive

• Deployed Site Intercept surveys on our Audience pages

17


Web Behavior Analysis

36,385 pageviews

Top 5 Previous Actions

Home page pageview 49%

Search results pageview 12%

Entered the site 10%

“Is Your Doctor an AMA

Member?” pageview2%

Patient Resources

pageview2%

Top 5 Next Actions

Exit the site 25%

Conduct a search 24%

Home page pageview 14%

Patient Resources

pageview11%

CME pageview 3%

Physician Audience Page Navigation Summary

Among top 5 actions, this is the

only behavior taken from content

within the page Data timeframe: Q2’14


“I can not find how to get to

the code helper.”

“Can't find the rvu medicare

payment rate.”

“I am seeking a summary of

the Figures and Tables

contained within the AMA5

PI Guides for medico legal

reporting purposes. I do

not require the entire

publication but need a

reference list of what page

each of the Figures and

Tables is on for Permanent

Impairment reports.”

“This page is

confusing.”

“Impossible to find

a place to enter a

search.”

“I am still looking for the

way to search and learn

if the organization has

board certified Medical

Doctors.”

Site Intercept Survey

Data timeframe: Q3-Q4’14


Usability Tests

Data timeframe: Q4’14

INSIGHT // AUDIENCE-SPECIFIC CONTENT DOESN’T MATCH EXPECTATIONS

Audience sections do not match that audience’s

career path and lifecycle, e.g. residents have a

very defined career course, and expected to find

content to match their career milestones.

Opening a practice (found in the Medical Students

and Residents sections) was actually expected to

be found in the Physicians section, since users in

that stage in their career are free to consider

opening a practice.

Maybe add more tabs in the residents section to show the different options you have after the residency.

- P5 Resident

“

If I’m looking for information on opening a practice, I would go to ‘physicians’, because they are the ones who would be opening practice. I would not categorize it as a resource for medical students, because right now they are focused on getting through medical school and finding a residency.

- P1 Student

“



Results drove action:

• UX enhancement work streams & projects

– EX: Site navigation re-architecture & Jargon removal

• Further research into content preferences among audiences

– EX: This is drive the Audience page redesign

• Additional web analytics tracking enhancements

– EX: Usage of accordion menus and scroll tracking

• New site intercept surveys

– EX: Are you a physician or a member of the healthcare team?

21

Case 3:

Onsite Search

22


Case 3 | Onsite Search

Key Business Question:

• What content on our website do people search for the most?

23


With web analytics

alone, it is hard to

determine what is

the most commonly

searched content

areas are.

24




Blended Approach

• Pulled 2014 onsite search data from Google Analytics

• Using SAS text miner, we’re working to identify answers to our research

objectives which can now be more complex

25



Research Objectives

1. What categories of content are searched for the most?

2. As changes are made to the site, how do the search trends for content categories change? (ex: As some content is made easier to find, will they be searched less often?)

3. How can we match categories back with authenticated users of the website to provide an indicator of their “interest”?

4. For unauthenticated / “other” users, how can the categories be grouped to provide more insight into who they might be (i.e. content focused on specific practice settings, phase of career, etc)?

26



Hypotheses

1. Since the majority of the traffic to our homepage is from branded organic search, users have been trained to use our homepage as a search engine because they do not find it easy to discover the content they are looking for through our navigation.

2. The top categories among authenticated physicians will vary from the general population of visitors to the site.

3. Searches for consumer/patient focused terms (ex: medical marijuana, obesity) will have the highest level of refinement and exits from the page because of the lack of content on our site.

27



Results are currently being analyzed and will be used

to inform the AMA’s enterprise digital strategy.

28

Q&A

29


Data & Analytics

Want Easier Decision Making Just Listen to Your Users