MEASURING ENGAGEME

NTReggie Bustinza

Director of Alumni RelationsLewis University

Romeoville, IL Private Catholic institution 6,500 students (4,500 undergrad, 2,000

grad) 37,000 alumni – primarily in Chicago area Alumni Relations staff of 3 Database: Raiser’s Edge (“R/E”)

About Lewis

Why try to measure engagement? What are some methods? The Lewis System

◦ Process◦ Details◦ Results

Now what?

Overview

Metrics can guide decision making◦ Spot trends◦ Identify opportunities◦ Abandon dead ends◦ Quantify program success

More efficiency & efficacy Justify our existence

Why Measure Engagement?

Hope College Bradley University Grand Valley State University Lewis University

Metric Systems

Used regression analysis to identify 5 representative variables that drove engagement.

Each item was given a weight (sum of 10) ◦ Phone number 2.970◦ Volunteerism 2.009◦ Valid email address 1.221◦ E-newsletter subscriber 1.208◦ Event attendance 0.681

Method Overview – Hope College

Courtesy Scott TravisDir. Of Alumni & Parent RelationsHope College

Method Overview – Hope CollegeStrengths Weaknesses

Statistically highly accurate

Formula they can run easily

Few factors make it easy to look for correlations

Scale of 1 to 10

Not intuitive Difficult to set up

(unless you’re a statistician and programmer)

Giving not considered Still room for errors of

human judgment Scale of 1 to 10

Stratified alumni engagement levels

◦ The most engaged activity a person is involved in defines which engagement level they are placed in

Method Overview – Bradley University

Non Interest – Universe of BU alumni

Interested – Passive Activity Engaged – Actions that

result in benefit TO alumnus

Active – Actions that provide benefit TO university

Advocate – Actions result in sphere of influence to benefit BU

Method Overview – Bradley University

Advocate

Active

Engaged

Interested

Non-Interest

Method Overview – Bradley UniversityStrengths Weaknesses

Very simple Gives nice overview Clearly defined

stratum

Very simple No number assigned Requires

understanding of the system

Limited applications Cannot be used in a

query

Cumulative point system

◦ Track things like giving, events, student activities, alumni groups, leadership positions, legacy, volunteerism, social media

◦ Points accumulate over alumnus’s lifetime

Method Overview – Grand Valley State

Courtesy Mindy Kalinowski EarlyAssistant Dir. Of Alumni RelationsGVSU

Method Overview – Grand Valley StateStrengths Weaknesses

Works with existing data

Relatively simple setup

Point inflation No accounting for

timing of involvement

Requirements◦ Work with existing data◦ Searchable◦ Valid as aggregate and/or individual data◦ Easy to understand

Not required, but nice to have…◦ Inexpensive to implement◦ Ability to run ourselves, as frequently as we want

OR dynamic◦ Option to exclude data to look for correlations◦ Simplicity

The Lewis System

Liked the way Hope College utilized their process but we wanted to consider more data points

Liked the ease of the Bradley system, but really wanted numerical output

Liked the breadth of GVSU’s categories, but didn’t like the point inflation

The Lewis System

Process1. Make sure database is capable of outputting

what we desire, and we can import results2. Identify what relevant data we track – “What

information do we have that shows some kind of engagement?”

3. Assign relative values4. Test5. Repeat until values are no longer questioned

The Lewis System

Step 1: Can Database Handle It?

Step 2: What do we Track?

Is it indicative of engagement? Is it accurate? Will we keep tracking it?

Step 2: What do we Track?

• Event Attendance• Giving – how much

and how often• Valid email• Open emails• Social media• Valid address• Valid business info• Active volunteer• Board member

• Award winner• Legacy parent• Campus visits• Interested volunteer

Can we categorize?

Step 2: What do we Track?

EventsEvent Attendance

GivingGiving – how much and how often

CommunicationsValid emailOpen emailsSocial mediaValid addressValid business info

VolunteerismActive volunteerBoard memberInterested volunteer

OtherAward winnerLegacy parentCampus visitsEmployeeAffinity Partners

Step 3: Assign Relative Values

Some categories require point decay◦ Giving consecutive years (most recent 3 years)◦ Event attendance over years

Some attributes require point cutoffs◦ Email open rates◦ Giving consecutive years (beyond 3 years)

Some attributes require point differentiation◦ Where gifts come from (event vs. outright)◦ Volunteer type

Step 3: Assign Relative Values

Challenges◦ Not all board members are equally engaged.◦ How stratified should we make giving levels?◦ Free events vs. Paid events

Step 3: Assign Relative Values

Category: Events

Step 3: Assign Relative Values

Indicator Value

FY – 13 4 / event

FY – 12 2 / event

FY – 11 1 / event

Category: Giving

Step 3: Assign Relative Values

Indicator Value

Gift – FY 13 Event – 5Outright – 10

Gift – FY 12 Event – 2Outright – 5

Gift – FY 11 Event – 1Outright – 2

Planned Gift 7

Presidents Circle – FY 13 5

Presidents Circle – FY 12 2

Consecutive Years 2 pts / year (max 10 years)

Category: Communications

Step 3: Assign Relative Values

Indicator Value

Receive Emails 2

Open Emails 1-3 = 2 ---- 4+ = 7

Click Emails 1-3 = 2 ---- 4+ = 7

Business Info 2

Valid Address 2

Valid Email 2

Valid Phone 2

Net Community 2

Social Media 2

Category: Volunteer

Step 3: Assign Relative Values

Indicator Value

Active 5

Interest 5

Affinity Group 3

Affinity Group Board 5

LUAA Board 5

BoT 10

Committee Chair 2

Class Speaker 5

Organization (mock trial, etc)

7

Other 3

Category: Other

Step 3: Assign Relative Values

Indicator Value

Initial Visit Acceptance (dvlp)

2

Voluntary Campus Visit 2

Legacy Parent 5

Award Winner 2

Employee 5

Liberty Mutual 2

6 scores are actually produced◦ 1 for each category◦ Overall Engagement Score (sum of each category)

Step 3: Assign Relative Values

Step 4: Test

Share values with colleagues for feedback Run the numbers, see what results are Spot check results Pull top 10, top 20, top 50, top 100 alumni

◦ Does it add up?◦ If not, why?

Step 4: Test

Tweak values, repeat test

Step 5: Repeat!

Step 5: Repeat!

Run Quarterly (past 12 months)

Exported each category to Excel where values are assigned and coded

SPSS is used to merge data Import integers back into Raiser’s Edge

Final Process

Results

Example 1: Tyler Durden – top 5%

Results - Individual

Item Points

Given 2 consecutive years

4 points (2 per year)

Given 2 most recent years

15 points (10 this year, 5 last)

Attended 1 event 3 years ago

1 point

Valid address & phone # 4 points (2 for each)

TOTAL 24 points

Example 2: Art Vandelay – top 25%

Results - Individual

Item Points

Given once, 7 years ago 0 points

No events, volunteerism 0 points

Read 6 emails, 1 click-through

9 points (7 for more than 4 emails opened, 2 for 1 clickthrough)

Valid email, address, phone #

6 points (2 for each)

TOTAL 15 points

Category Giving

Events

Comm.

Vol. Other TOTAL

All Alumni 1.3 0.152 8.859 0.042 0.227 10.584

Young Alumni 1.087 0.189 9.48 0.038 0.103 10.898

Athletes 2.78 0.48 9.69 0.06 0.35 13.29

Alumni Board 21.2 19.52 15.2 6.64 2.24 64.8

Board of Trustees

35.5 13.82 12.73 10.4 2.77 75.22

Aviation Alumni 0.794 0.065 8.78 0.0299 0.195 9.865

Results - Aggregate

Results - Aggregate

Top X % Point Cutoff

1% 45

5% 23

10% 17

25% 12

50% 8

Results - Aggregate

Lewis System: Strengths & WeaknessesStrengths Weaknesses

Can run in house Values recent activity

over old activity Results are easy to

understand

Some data can be suspect (eg: acquired mailing lists)

Have to export, use two programs, then import for scores

As data points are added, historical scores are distorted

Now What?

Metrics are half of the battle. The real question is: How will you use this tool?◦ Whittle mailing lists◦ Identify prospects that were under-the-radar◦ Look for correlations between giving and

programming◦ Identify potential affinity groups◦ Snapshot of prospect’s history◦ Evaluate programming

Now What?

Questions