FCN Presentation - Measuring Communication - Nov 2012

© 2012 The MITRE Corporation. All rights reserved.

Presentation to the

Federal Communicators Network

Jeffrey Brooke, ABC

Principal Consultant

Organizational Communication & Change Management

[email protected]

Presentat ion approved for Publ ic Release:12-4611. Dist r ibut ion Unl imi ted * A B C – A c c r e d i t e d i n B u s i n e s s C o m m u n i c a t i o n b y t h e I n t e r n a t i o n a l A s s o c i a t i o n o f B u s i n e s s C o m m u n i c a t o r s

Measuring Communication - Simple Approaches That Can Change Your Image

mailto:[email protected]

Agenda

Logistics

– 30 minute presentation

– Q&A – via text box

– Presentation will be forwarded

Illustration - a typical communicator dilemma

What to measure

– Conceptual framework for measurement

How to measure

– Other examples and tools

| 2 |

Communication measurement

is a way of thinking

Measurement is about:

– Gathering data

– To tell a story

– That allows people to make decisions

Not about:

– Complicated math

| 3 |

Start with internal communication

Focusing on internal communication today

An organization’s brand/reputation, comes from the inside out.

Employees treat customers as management treats employees.

– Effective communication improves employee engagement

– Engaged employees deliver better for the customer

Here’s an example.

| 4 |

Scenario - Call center at “the Bureau”

| 5 |

Call center agent, Jill, answers question and

completes service applications for citizens

| 6 |

Goal: communication with Jill will align her

performance with the Bureau’s goals

Communication Desired

interaction with

citizens

| 7 |

Head of call centers needs quality improved (response accuracy)

| 8 |

His e-mails, speeches, newsletters have stressed

quality – but haven’t produced change

| 9 |

He wants your help getting his quality message through

“What other communication should I do? I thought maybe a special newsletter each month focused just on quality, like industry best practices. Then maybe a contest between units and then spotlight the quality superstars in the newsletter each month. What do you think?”

| 10 |

Stepping back for a moment… The communication flow – classic model

Communication Flow

Sender Message Channel Receiver Results

Noise

Feedback

| 11 |

How does the basic model work in our large and complex organizations?

| 12 |

Communicator Perspective

Stakeholder Perspective (e.g. employees)

Client Program Owner Perspective

Sender Message Channel Receiver Results

Organizational roles that tie to steps in the model

Communication Flow

Noise

Feedback

| 13 |

Planned communication in an organization Steps that operationalize the model


Plan Deploy Communication

Activities


Frame of Reference

Communication

Outcomes


Stakeholder Behaviors

Results

Communication Flow

| 14 |

Planning & measures are intertwined Planning starts at the end, works backward


Plan Deploy Communication

Activities


Stakeholder Assessment

(Frame of Reference)

Communication Outcomes

(Knowledge &

Attitude)



Results

Planning Flow

| 15 |

Communication planning Explore measures for all steps, not just the end


Plan

Measures

Deploy

Measures




Measures


(Knowledge &

Attitude)

Measures



Measures

Results

Measures

Planning Flow

| 16 |

How to Measure Communication

Overview of Key Tools

Qualitative

– Interviews

– Focus Groups (and Structured Group Interviews)

– Polls

Quantitative

– Surveys

– Counting

| 17 |

Qualitative Data Gathering

Top tools Best for

understanding

Examples

• Focus groups

• Interviews

• Polls

• Depth—by probing

• To understand the

range of issues on a

topic

• Story-telling nature of

dialogue can uncover

forces around

change—enablers

and barriers to

change

• What people know

• How people feel

about a topic

• Frame of reference

on a topic

| 18 |

Quantitative Data Gathering

Top tools Best for

Examples

Surveys

Prediction

• Breadth—how widely

shared is an opinion

• Precise, for tracking

changes over time.

• Comparison over

time, and between

organizations

• Correlation analysis

for prediction

Counting

Outputs

• Number of articles

• Event attendance

• Percentage of

articles dedicated to

each strategy

| 19 |

Communication planning 1. Results

Communicator Perspective Stakeholder Perspective

(e.g. employees) Client Program Owner

Perspective

Results

Plan What to measure How to measures

Questions to ask:

Outcomes:

• What needs to change and why?

• How will you know if you’ve

accomplished your goal?

Examples:

• Customer satisfaction

• Budget requirements

• Congressional

requirement

Examples:

• Survey

• Financial statements

• GAO/committee

feedback

Outputs:

• What changes in outputs are needed

to achieve the goals?

• How will you measure progress?

• Number

• Quality

• Cost

• Counting

• Sampling

• Cost accounting

| 20 |

Results

Outcomes and Outputs

Will be measured by the program owner

Outcome: Higher satisfaction from

– GAO/Congress

– Citizens

Output: Higher quality calls

| 21 |

Communication planning 2. Targeted stakeholder behavior


(e.g. employees) Client Program Owner

Perspective


Plan What to measure How to measure

Questions to ask

• Who needs to do what differently to

produce the output changes?

• How will you measure the behaviors?

• What will change to enable the new

behaviors?

Examples

Actual behaviors

• Applying new training

• Using new procedures

Behavioral indicators

Examples

Business process

measures

Survey questions

| 22 |

Behaviors

Not clear

Ron assumes the agents have leeway.

Sees his communication as motivation.

| 23 |

What to tell Ron?

One approach

“Yes! I can help you with a quality-focused newsletter and other

ways to get out your message.”

“AND— – how do you feel about first letting me do some feedback sessions

to get a sense of what will and won’t work. I might find better

options for you to consider.”

| 24 |

Communication planning 3. Stakeholder assessment – change readiness


(e.g. employees)


1. Frame of Reference for the change



Questions to ask the program owner

Client hypothesis about reaction to the

change:

• What changes will enable the new

behaviors or remove barriers?

• What is the stakeholder’s frame of

reference on what you are asking

them to do differently?

• What might they see as enabling or

blocking them?

Examples

Validate with the

stakeholder:

• Will you be able to do

this, in light of the

planned support?

Examples

Focus group

Interview

| 25 |

Conduct stakeholder assessment Focus groups at call centers

| 26 |

Finding: Quality measured quarterly, at the office level

| 27 |

Quantity Metric–

Every 30 minutes, Group Level

Finding:

Quantity measured every 30 minutes, at the group level

| 28 |

Frame of reference determines

what Jill will notice, how she will interpret it

Communication

Frame of Reference

(Enablers & Barriers)

Resulting

interaction

with citizens

| 29 |

Stakeholder assessment:

Focus groups determine Jill’s frame of reference

Frame of Reference

(Enablers &

Barriers)

Elements of frame of

reference

• Leadership

• Mission & Vision

• Culture

• Mgt. practices

• Policies &

procedures

• Work unit climate

• Individual needs

• Task & individual

skills

(See Burke-Litwin model)

Resulting

interaction

with citizens

| 30 |

What would you tell Ron?

What would you recommend

to him as a next step?

Might this change how Ron

sees you?

What to tell Ron?

| 31 |

Communication planning

3. Stakeholder assessment – communication fitness


(e.g. employees)


2. Frame of Reference on

communication



During deployment:

How to assess the effectiveness of each

communication activity during

deployment.

Past experience:

Usefulness of communication channels

and approaches with this stakeholder.

“Fitness” of communication

Examples:

- Clarity

- Right channels

- Amount

- Timing

- Exposure

During deployment

• Survey

• Poll

• Focus group

Past experience

• Compilation of

previous data

• Communication audit

“Fitness” measures.

What do they think of

our stuff? Does it “fit”

their needs?”

| 32 |

Example:

Ongoing assessment of "fitness" of messages

Weekly survey of employees and managers

Hottest topic of the week

Statistically reliable sample

| 33 |

Example "Fitness" measures Message penetration and usefulness

| 34 |

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Q7. How would you rate the

amount of communication you've

received on this topic? (Running

average for "About right" is 68%)

Q6. How important is the topic of

"new system procedures" to you?

(Running average for "Important" is

65%)

Q5. Overall, how satisfied were

you with this communication?

(Running average for "Satisfied" is

86%)

Q4. To what extent do you feel

this communication was timely?

(Running average for "Timely" is

82%)

Timely

Satisfied

Important

About right Too much Too little

Unimportant

Dissatisfied

Not timely

Example "Fitness" measures Timeliness, satisfaction, importance, amount

| 35 |

Communication planning 4. Communication outcomes


(e.g. employees)


Knowledge &

Attitude



What knowledge and attitude will

produce the targeted behaviors, as seen

through the stakeholder’s frame of

reference?

Examples

• Knowledge and attitudes

(perceived)

• Actual knowledge

Examples

• Survey

• Poll

• Focus group

| 36 |

Example knowledge measure

| 37 |

Example measure of feelings

9%

23%

17%

20%

31%

13%

38%

25%

0%

25%

0%

5%

10%

15%

20%

25%

30%

35%

40%

A great deal Somewhat A little None I’m not sure yet.

To what extent do you feel the new Performance Plan system will help the agency reach its goals?

Employees

Sups

| 38 |

Example: Measuring knowledge and feelings

for course correction

Situation:

– Executive Committee interactive video town hall

Topic:

– Transformation update – two years into effort

Update on each of five change strategies

Scripts weighted heavily on “why this strategy is important”

Measurement Plan

– Pre-event random survey – two questions for each strategy

Know: “How important do you feel this is to the agency transformation?”

Feel: “How effectively do you think this strategy is being implemented?”

– Recommend possible script changes

– Post-event repeat survey

| 39 |

| 40 |

Measuring knowledge and feelings

Pre-event survey (two days prior)

Several strategies showed:

• good scores for importance

(feeling)

• lower scores for

implementation (knowledge)

Comments indicated lack of

knowledge about early

implementation efforts and

results.

Updated script for these

strategies to weight more heavily

on pilots and early

implementation efforts.

Measuring knowledge and feelings

Post-event survey (one hour after event)

Scripts adjustments were

successful.

| 41 |

Communication planning 5. Deploy communication activities


Deploy communication

activities




What types of communication will best

convey the targeted knowledge and

attitudes?

Communication Outputs

Examples:

• # of articles and events

• % of content supporting

each business strategy

• % of stakeholders who

provide input on a change

• # of awards for a desired

behavior

• # informal leader

conversations

Examples

• Counting

| 42 |

Targeting value change with

messages Primary communication activities

Content, such as intranet

pages, publications and formal

presentations.

Interaction, such as input to

decisions, dialogue with

managers, and collaboration

with colleagues.

Leadership example, such as

how leaders signal what they

value through promotions,

recognition, budget, their time

and informal comments.

| 43 |

Output measures Content analysis – weekly data collection

| 44 |

Themes Totals

% of total

editorial

space E-Bulletin Managers Bulletin Magazine Webbie

Total Length 66586 37702 15582 7289 6013

Vision 222 0% 0 0% 222 1% 0 0% 0

Mission 239 0% 239 1% 0 0% 0 0% 0

Values 12026 18% 2249 6% 498 3% 6165 85% 336

Value: Integrity 208 0% 0 0% 0 0% 0 0% 0

Value: Teamwork 10712 16% 2053 5% 498 3% 6165 85% 336

Value: Commitment 1423 2% 196 1% 0 0% 0 0% 0

Value: Dependability 389 1% 0 0% 0 0% 0 0% 0

Goals 39169 59% 16872 45% 11046 71% 2330 32% 1911

Goal 1: Develop a flexible

digital… 3421 5% 1092 3% 0 0% 1284 18% 0

Goal 2: Prepare and

equip… 6601 10% 3694 10% 1076 7% 1046 14% 0

Goal 3: Continue to

transition the… 784 1% 567 2% 0 0% 0 0% 0

Goal 4: To serve the

public's needs by… 892 1% 342 1% 352 2% 0 0% 23

Goal 5: Develop an

integrated enterprise

approach to… 32496 49% 14603 39% 10694 69% 0 0% 1888

Quotes 16629 25% 5112 14% 3973 25% 7090 97% 315

Photos/Images 497 55 9 24 25

188 63 10 60

Total articles or services 786

Example output measure: Percentage of total editorial space,

by category, for all six channels

1% 0%

6%

28%25%

27%

14%17%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Vision Mission Values Goals Strategic

Priorities

Administrative Recognition Quotes

Example output measure: Percentage of total editorial space,

by category, for E-Bulletin

1% 1%3%

25%

19%

23%

8%

13%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Vision Mission Values Goals Strategic

Priorities

Administrative Recognition Quotes

Example output measure: Percentage of total editorial

space, deep dive in “goals” coverage

28%

3%5%

2% 2%

25%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Goals Total Goal 1: Develop

a…

Goal 2: Prepare

and equip…

Goal 3: Continue

to transition the

…

Goal 4: To serve

the public's

needs by…

Goal 5: Develop

an enterprise

approach to…

Communication Measures Integrated with Communication Planning


Communi-cation plan elements

Measures for each element

5.

Tactics to Deploy

Communication Outputs

Stakeholder Perspective

3.



Change Readiness

Communication Fitness

4.


Knowledge and

Feelings


2.


Specific behaviors needed

Changes that will enable the

behaviors

1.

Results

Outcomes and outputs

| 48 |

Does anyone really care?

Measures can move your “brand” from overhead, toward operations.

Communications Executive Council survey: 2012 vs. 2007 data

Responsibilities of communication function – #1 increase: Social Media 80% (from zero) – #2 increase: Measurement 82% (from 64%)

Top communication strategic priorities for 2013

– #1: Employee engagement – #2: Improving leadership communication – #3: Building company presence in social media

Strategies to improve communication function effectiveness – #1 Improved consultative partnership with business leaders – #2 Deeper business acumen on the team

CEC 2012 report: Taking Your Communications Budgeting and Planning to the Next Level

| 49 |

What next?

Deeper topics

– Design and use tools for valid and reliable data

– Communication audits

Broader topics

– Looking at measures across organizations and time

– What communication function can do to add value

Feedback on today’s presentation

Reach out to me and FCN community

Q&A

| 50 |