EAB Getting the Measure of IT UWMadison...Robert Burton, The Anatomy of Melancholy(1621); EDUCAUSE, “Top 10 IT Issues, 2018,”EDUCAUSE Review, Jan-Feb 2018; EDUCAUSE, Analytics

©2018 EAB Global, Inc. • All Rights Reserved • eab.com

Getting the Measure of ITDeploying Metrics to Create and Demonstrate IT Value

IT Forum


2

IT Supports Analytics for Many Institutional Purposes…

Robert Burton, The Anatomy of Melancholy (1621); EDUCAUSE, “Top 10 IT Issues, 2018,” EDUCAUSE Review, Jan-Feb 2018; EDUCAUSE, Analytics Landscape in Higher Education 2015; Numerify, Visibility to Drive Digital Transformation (2018).

The Shoemaker Goes Barefoot Himself

Analytics-Driven Top 10 CIO Issues

1. Information Security

2. Student Success

3. IT Strategy4. Data-Enabled

Institutional Culture

5. Student-Centered Institution

6. Higher Education Affordability

7. IT Staffing and Organization

8. Data Management & Governance

9. Digital Integrations10. Change Leadership

…But Its Own Capabilities Fall Short

CIOs have full view of all IT processes

35%CIOs agree a full process view is critical to success

80%

Responsibility for Delivering Analytics Services

IT

Shared IT & IR

IR

Other

60%IT

Involvement

https://er.educause.edu/~/media/files/articles/2018/1/er181100.pdf

https://library.educause.edu/resources/2016/1/the-analytics-landscape-in-higher-education-2015

https://ww2.numerify.com/it-digital-transformation-201806.html


3

IT Performance Measurement

IT Brand Communication

Decision Support

Imp

orta

nce

Maturity

Maturity=Importance

IT Functional Diagnostic Finds Performance Measurement Maturity Low

Source: IT Forum Functional Diagnostic benchmark data.

Not Our Strongest Suit

Low High

Low

Hig

h


4

Traditional IT Metrics Abound But Don’t Shed Light On Today’s Real Goals

A Brighter Lamp Won’t Solve the Problem

• Cycles and storage

• Network uptime

• Cost of infrastructure• Major systems availability

• Help desk wait times

Campus IT Monopoly, 1960s – 2000s

• Contribution to mission

• Business value of IT projects

• Cost per service delivered• Impact on student experience

• Health of key relationships

Hybridized Service Provider, 2010 - ??

Goal: Optimize IT ValueGoal: Optimize Operations

Source: Quentin Hardy, “How Cloud Computing Is Changing Management,” Harvard Business Review, 8 February 2018; Richard Katz, ed., The Tower and the Cloud (EDUCAUSE, 2008); EAB interviews and analysis.

https://hbr.org/2018/02/how-cloud-computing-is-changing-management

https://www.educause.edu/research-and-publications/books/tower-and-cloud


5

Business Analysis and Relationship Health Rising in Significance

Source: EDUCAUSE, IT Service Delivery in Higher Education: Current Methods and Future Directions (2015); EAB interviews and analysis.

Softer Skills, Different Problems

39

59

71

2015 2020 2025

Managing Technical Resources

0

100Managing Vendors, Services, Contracts

50

CIO Management Focus

Sof

ter

Har

der

IT S

kills

Tre

nd

Service Metrics• “Business of IT”

• Vendor performance

• Satisfaction & relationships

Operations Metrics• System uptime

• Network speed & capacity

• Data center efficiency

Driver 1: Cloud and Digital Transformation

https://library.educause.edu/~/media/files/library/2015/5/ers1501a.pdf?la=en


6

Strategic Contribution of IT Must Be Actively Demonstrated

Source: SIM IT Trends Study 2018; CHECS CIO Survey 2018; EAB interviews and analysis.

From Speeds and Feeds to Deeds

39%

78%

46%

83%

56%

13%

TechnicalKnowledge

Leadership RelationshipBuilding

…but Leadership May Not

CIOs Leaders

Skills CIOs Should Have Top 5 CIO Personal Performance Measures CIOs Recognize Their Role in Strategy…

• Internal customer satisfaction

• Value of IT to the business

• IT contribution to strategy• Availability/Uptime

• Cybersecurity related

Drip, Drip“One of a CIO’s biggest fears is to be seen as a plumber, or to be seen as a ‘good enough’ utility.”

Robert Grillo, Vice President and CIOFlorida International University

Driver 2: Contested Strategic Role of IT

https://cdn.ymaws.com/www.simnet.org/resource/collection/7A70D436-28BA-4E88-B958-C86941C704C3/2017_SIM_IT_Trends_Study_-_Comprehensive_Report_2018_-_18Jan18.pdf

https://checs.org/reports/


7

CIOs Aren’t Alone in Advocating for Resources

Driver 3: Internal Competition for Resources

Source: Campus Computing Survey 2017; EDUCAUSE Top 10 IT Issues 2015; EAB interviews and analysis.

A Long-Term Competition for Limited Revenue

Declining HS demographics

Increased price sensitivity

No flight to safety with new markets

High fixed costs

Headwinds for Higher Ed

Decreasing state funding

Intensified competition

Cost of funding strategic ambitions

Higher costs to serve

63%IT organizations that have not recovered from recession-era cuts

9%Share of IT budget available for transformative investment

EAB Program Topic %As

Advancement Forum

Quantifying Advancement's Value to the Institution 61%

Enrollment Management Forum

Making Enrollment Everyone’s Job: Empowering University Staff to be Effective Enrollment Ambassadors

55%

Student Affairs Forum

Telling Our Story and Making the Case for Resources: Student Affairs Marketing and Communication Strategies

49%

University Research Forum

Speaking Their Language: Demonstrating the Value and Impacts of Research

45%

Results from EAB Topic Polls, 2016 to 2018


8Value Is the Coin of the Realm

Value as Strategic Alignment“KPIs should go beyond traditional IT metrics involving operational excellence. They need to answer fundamental questions such as, “How well are the IT projects aligned to the business priorities?”

Value as Business Contribution

KPMG

“My vice president cares at some level that our network is up 99.99% of the time, but she cares more about business value.”

Value as Customer Enablement

“We’re not able to report information that causes people to say ‘I got my thing done because of what IT did.’ I’m constantly struggling to demonstrate the value of what we do.”

Value as Good Stewardship

“We operate under resource constraints. We need to be sure we’re getting the best value of the work we’re doing, the staff we’re hiring, the products we buy.”

Many Dimensions, But Common Theme is Showing Impact

CIO, Private Master’s

CIO, Public Doctoral

CIO, Public Master’s

Source: KPMG, Running the Business of IT: Metrics That Matter; EAB interviews and analysis.

https://assets.kpmg.com/content/dam/kpmg/pdf/2016/05/metrics-that-matter-business-of-it.pdf


9

Taking Charge of the Narrative

In the absence of credible information on IT performance, anecdotal evidence about IT failures is likely to dominate the discussion of IT performance in the public sector. With the availability of IT performance metrics, CIOs can both proactively manage performance and demonstrate mission results.”

Kevin Desouza, Arizona State University

Source: Kevin C. Desouza, Creating a Balanced Portfolio of Information Technology Metrics (IBM Center for the Business of Government, 2015).


10

Obstacles to Telling IT’s Story Effectively

Information Rich, Communications Poor

Reaching the Right Audience

Shortage of Skills

• Pull predominates over push

• Limited range of communications channels

• Lack of fluency in customer vernaculars

• Dedicated communications teams are rare

• Jargon often invisible to IT professionals

• No story line connecting discrete events and announcements

Source: EAB interviews and analysis.

Blah, Blah, Blah“We get technical performance data from our IT folks, with best intent, that is just noise to the provost, deans, or students.”

CIO, Public Doctoral


11Getting the Measure of IT

Enhancing IT communications skills

• IT communications director

1 2 3 Communicate IT Value Effectively

Choose the Right Metrics for the IT Organization

Establish IT’s Value to the Institution

Low-Effort, High-Impact Communication

• Strategic initiatives roadmap• IT-at-a-glance infographics• Impact-based project

promotion• IT success stories

Quantifying IT investment

• Activity-based costing

Measurement-Friendly Strategic Alignment

• IT balanced scorecard• Customer unit service

agreements

The Metrics-Savvy IT Organization

• IT metrics tsar • IT metrics selection

framework

Capturing Customer Perspectives

• Fatigue-fighting customer survey tactics

Developing Value Heuristics

• High-impact IT value stories• IT value heuristic

development process


ROAD MAP12

Choose the Right Metrics for the IT Organization1

2 Establish IT’s Value to the Institution

3 Communicate IT Value Effectively


13Finding the Signal Amidst the Noise

# of Technical Tickets

Mobile-Enabled Applications

Availability of Public Computers

# of Software Downloads

Help Desk Overall Satisfaction

# of Unique Visitors to Website

Projects On Budget

Conference Line Support

Research Computing Usage

# of Wireless Routers

Centralized v. Distributed IT Staff

Rate of Ticket First Contact Resolution

Ticket Resolution Time

Student Satisfaction with Wi-Fi

Cost of Device Replacement

# of Accounts Compromised

# of Intercepted Phishing Attempts

Data Storage Capacity

Data Storage Utilization

Systems Update and Maintenance Cost

Network-Connected Devices

Staff Completing Security Training

Project Completion Time

Router Capacity

Campus Internet Coverage

Wireless Download/Upload Speed

# of Password Resets



14

Goals Cascade

IT All Starts with Strategy

Institutional Goals Cascade and Environmental Scan Help Identify IT Goals and Objectives

SWOT Analysis

IT Strategic Objective

IT Strategic

Goal

IT activity categories result from institutional goals cascade.

Goal: A broad outcome or desired state that fulfills a strategic IT need.

Objective: An action that contributes to realizing a goal. SMART: Specific, Measurable, Achievable, Relevant, and Time-bound.



Picture of IT’s Strengths, Weaknesses, Opportunities, and Threats (SWOT)

Source: IT Forum IT Strategic Planning Center of Excellence; EAB interviews and analysis.

https://www.eab.com/research-and-insights/it-forum/resources/2018/it-strategic-planning-center-of-excellence


15

IT Balanced Scorecard

A Framework for Performance at Drake University

Teaching and Learning• Exceptional learning &

living experience• Experiential learning• Inclusive campus

community

Drake’s University-Wide Continuous Improvement Plan (CIP)In lieu of a traditional institutional strategic plan, Drake uses a modified balanced scorecard.

ScorecardBuilt around four key “commitments,” each with objectives and time-trackable performance measures. Progress is reviewed periodically.

Unit AlignmentUnits develop their own CIPs but must follow the basic institutional structure.

Reflection• Data-driven decision

making• Continuous assessment of

learning outcomes

Execution• Vibrant, effective,

efficient facilities• Service excellence• Robust working, learning,

and living technology infrastructure

Stewardship• Financial performance• Environmental

sustainability• Culture of pride and

engagement

Source: Drake University Information Technology Services; EAB interviews and analysis.


16

Institutional Categories Drive IT Objectives & Metrics at Drake


Suit the Metric to the Objective, Not Vice Versa

Commitment: Teaching & Learning

• Overall faculty satisfaction• Faculty satisfaction index• Percent courses in LMS• Classroom quality index

• Ensure stability and performance with key services

• Ensure effective project management

• Ensure all our partners gain tech literacy skills

Commitment: Execution

Measures

• Users of Lynda.com• Satisfaction survey

training question scores• Knowledgebase usage

index

• Robustness index (summary measure of major system availability)

• Project execution index

• Overall project satisfaction

Starting in the Right Place“I’m a strong believer in picking something and starting, but instead of starting with what you can measure, start with the objective you hope to hit.”

Chris Gill, CIO Drake University

IT Objective• Enhance the Drake learning

experience by supporting a limited but configurable set of instructional technologies


17Creative IT Performance Measurement

Lessons from Drake’s IT Continuous Improvement Plan

Begin by assessing how IT can support institutional objectives

Follow the Institution’s Lead

Using standard survey instruments and measurement models saves time and opens the potential for external benchmarking

Don’t Reinvent

Customer satisfaction and behavior information is rich with strategic implications and complements operational data such as system usage

Go to the Customer

Build Summary Metrics

An index derived from multiple measurements can be a powerful tool for capturing and conveying complex situations; e.g., index averaging responses to five faculty satisfaction survey items



18

Outreach and Management• Publish IT-unit alignment

document

• Monitor progress and fulfillment

• Periodic review with unit heads

• Update as needed

IT Strategic Plan Development

• Review institutional strategic plan

• Environmental scan of IT developments

• Draft IT strategic goals and initiatives

University of Denver Customer Agreements Link Unit Strategy to IT Action

PREPARE ALIGN WITH UNITS

1Discuss Unit NeedsMeet with deans and senior staff to identify unit vision and strategic priorities

2Identify IT Supporting ActionsSpecify IT projects and resources that will support the unit

3Establish MeasuresDefine parameters for successful fulfillment of each project

A Sideways Cascade

MONITOR

Customer Unit Service Agreements


19

School of Social Work

College of Education

Strategy Sheets Promote Collaboration Below the Institutional Level

Source: University of Denver Information Technology Strategic Alignment Report; EAB interviews and analysis.

Mutual Strategic Development

“Our approach was simple, to create an overall document which described our priorities with measurable objectives for 2018. We wanted not only to be transparent and accountable, but to encourage discussion on what could be done on a wider scale, or in partnership between units.”

Don Harris, Vice Chancellor and CIO

College of BusinessGoal: Integrate traditional pedagogy with latest technologies to be market-ready and technology-adroit.

IT Projects:

• Provide integration support for Internships Initiative

– Measure: Complete bi-directional integration by end Spring Quarter

• Virtual desktops for academic programs

– Measure: All virtual desktops refreshed no later than July 31, 2018

Appendix maps all IT projects to university, IT, and unit strategic plans

IT strategy sheets document unit goals, IT projects & measures


20

IT Metrics Tsar


Metrics Conflicts Threaten Strategic Progress

Infrastructure

Security User Satisfaction

Usage

Network Speed

Traffic Patterns

System Up-Time

# of Network Users

# of ERP Users

Peak v. Average Load

# of Incidents

# of User Trainings

# of Password Resets First Call Resolution Rate

Satisfaction with Service

Duration of Phone Consultations

Latency


21

University of Nevada Reno Appoints a Metrics and Measurement Director

Source: University of Nevada, Reno Office of Information Technology; EAB interviews and analysis.

An Organizational Home for Metrics

• Select both operational and value metrics in a unified framework that coordinates action

• Identify and implement data-based improvements to business processes, technology utilization, and work practices (e.g., automation)

• Provide evidence of IT performance and needs for high-profile reports and dashboards (e.g., legislative budget request)

• Lead one-on-one and small-group presentations to foster collaborative interaction between OIT and campus stakeholders

• Former project manager reporting to the CIO

Key Functions Fulfilled

Select metrics to align with overall IT strategy

Apply data analysis to process improvement

Communicate IT metrics to campus stakeholders

Duties of the MMD


22

Screening Process Helps Narrow Core Metrics from Long Starting List

IT Metrics Selection Framework


Bringing Metric Selection to Life

Four-Step Metric Selection Filtering Process

100’s of Potential Metrics

15-20 KPIs

Consideration Description

Apply a Collection and ReportingReality Check

Set aside metrics not readily accessible, regularly tracked, supported by reliable data, or easily communicated to others

Map toStrategic Objectives

Identify metrics that most directly measure progress on IT strategic objectives

Ensure Balance of Metric Categories

Avoid over-representation of some areas by sorting metrics by function or strategic perspective

Account for Unit-Specific Imperatives

Add “hot-seat” metrics that shed light on pressing yet temporary areas of concern

1

2

3

4

Metrics are quantifiable measurements collected to track organizational activities and processes

Key performance indicators are metrics that indicate progress toward strategic and operational objectives.


23

Four Pragmatic Screens to Determine Metric Viability


Apply a Reality Check

Suggested Screens

Metric Screen Description Rationale

Accessibilityof Data

Information system must possess the capability to generate data on metrics.

Time-consuming to manually pull and analyze data for each metric.

Frequencyof Tracking

Metrics elevated to unit reports or dashboards should be monitored at regular intervals (e.g., monthly or quarterly).

Infrequent (e.g., annual) data updates hamper ability to assess performance at regular intervals.

Reliability of Data

Data available from information system should be accurate, consistently defined, and measured across the institution.

Absence of trustworthy data results in stakeholder suspicion toward performance, often resulting in inaction.

Communicabilityof Data

Definition and rationale for metrics should be easy to communicate and understand.

Lack of understanding about metric drivers and relevance hinders ability to inflect performance.


24

Ideal Metrics Prompt “Yes” for Every Question in List

Tool: Reality Check Screening for Metrics

Accessibility of Data1. Is the data for this metric collected via an automated system? 2. If not, can someone collect and report the data within a few hours?

3. Is the system capable of calculating and reporting the results for this metric?

Frequency of Tracking4. Can this metric be tracked more than once a year?

5. Can this metric be tracked frequently enough to inform action?

Reliability of Data 6. Do all departments use the same definition for this metric?

7. Is the metric calculated by an automated system?

8. Can you ensure the accuracy of the reported data?9. Do managers trust the data for decision making?

Communicability of Data10. Is this metric easily explained to and understood by leaders outside your

unit?

11. Do managers typically agree on the definition of this metric?12. Are managers aware of the importance of tracking the metric?

13. Do managers understand how performance on this metric impacts institutional goals?


Metric Quality Score

• 10 – 13: Gold standard

• 7 – 9: Use with caution

• 6 or below: Avoid


25

Confirm Metrics Directly Measure Strategic Objectives Rather than Initiatives


Map to IT Strategic Objectives

Strategic Goals Strategic Objectives KPIs Targets Strategic

Initiatives

Des

crip

tion

• Backbone for strategy

• Usually four to eight

• Stem from strategic goals;typically 40 to 60

• Adapted annually to every few years

Indicators that track progress toward objectives

• Indicator goals that motivate performance

• Frequently reset to ensure continuous improvement

Set of actions to raise metrics above target levels

Exam

ple

Foster data-drivendecision making

• Create data management architecture

• Improvereliability of data

• Provide self-service BI

• Percent of institutional data items defined

• Data error rates

• Ad hoc report requests

• 300 new data dictionarydefinitions

• 10% y-to-y decrease in errors

• 20% decrease in staff time for ad hocs

• Start up data governance

• External dataquality review

• Implement self-serve fact book

Framework to Map Metrics to IT Strategic Objectives

Metrics should flow directly from strategic objectives


26

Equitably Distribute Metrics Across IT Functions or Strategic Goals


Ensure Representation From Various Categories

The most straightforward categorization scheme is to group metrics based on IT functions or capabilities, ensuring a balance of metrics across all responsibilities.

A second categorization scheme sorts metrics by institutional strategic pillars, which helps illustrate the link between IT initiatives and overall institution success.

Sample IT Functions

› Network Operations

› Enterprise Systems

› Service Desk› Cybersecurity

Sample Strategic Goals

› Student Success

› Enrollment

› Research and Scholarly Excellence› Financial Strength and Stewardship

Option 2: Strategic or Institutional Perspective

Option 1: Function or Capability


27

Elevate “Hot-Seat” Metrics in Response to External and Internal Pressures


Account for High-Priority Imperatives

2017 KPIs

Metric Status

Help desk satisfaction

Data dictionary definitions

ERP availability

Integrations completed

Student services wait times

Data center electricity consumption

Metric Status



ERP availability


Extent of PII, PHI scanning

Percent of staff completing awareness training

Metric Status



ERP availability


Operating budget execution

Server consolidation

New president appointed; identifies student retention and sustainability as high priorities

State budget cuts force IT to identify operational efficiencies

Data breach incident highlights cybersecurity issues

2016 2017 2018

Example Pressures that Drive New Metrics

2017 KPIs2016 KPIs


ROAD MAP28





29


The Value Demonstration Problem

IT Contribution to Strategic Achievements Often Significant But Indirect

Budgets Don’t Tell the Whole Story

Outcomes Data Hard to Come By

Complex Processes Competition for Credit

Implications for IT Credit Claims

Stovepiped IT budgets obscure total investment in areas of stakeholder concern

IT dependent on business and academic units to collect performance data

Hard to separate IT contribution from other factors

Non-IT stakeholders have more direct relationship to outcomes


30

IT-Centric Budgets Convey Impression Spending is “For” IT


Good for Bean Counting, Bad for Value Demonstration

Promotes view of IT as a cost center

Hard for non-IT people to understand

Obscures IT investment in customer concerns

Shortcomings of the IT-Centric Budget

“What does IT spend on teaching and learning?”

Budgets Built Around IT Functions…

…Can’t Answer Important QuestionsIT Budget

Category Line Item

Enterprise Systems

HR Systems

Student Systems

Reporting and BI

Network andInfrastructure

Network Services

Systems Admin

Database Admin

Client Services Service Desk

Equipment Services

Training Services


31

ABC1 Enables Refined Cost Estimation, But Challenging to Apply

Activity-Based Costing


1) Activity-based costing.

Detailing the Cost of Service Delivery

Fully-Loaded Service Cost Analysis

Activity-Based CostingA cost accounting practice that identifies and assigns costs to overhead activities and then assigns those costs to services

+ =Examples:

• Project FTEs

• Project hardware• Cloud license fees

Examples:

• Administrative overhead

• Network infrastructure• Utilities

Sample analyses informed:

• Cost of delivery at the transactional level

• Cost of service at customer level

• Rationale for budget request

Applying ABC in Higher Ed IT

Difficult to allocate some direct costs accurately

Does not account for service quality or ROI

Quantify and Allocate Direct Costs to Services

Quantify and Allocate Indirect Costs to Services

Elaborate analysis required for complex processes


32

Provided data support for revised RCM IT cost recovery model to replace telecom charges

Informs insource/outsource investment decisions and deferred maintenance impact

University of Illinois at Chicago Makes Multi-Year Commitment to ABC


Better Data, But at a Price

UIC’s Activity-Based Costing Model

ABC Implementation in the IT Organization

Implementation Requires Significant Investment in Process Change and Support

Services broken into cost segments; granular IT staff time allocation starts process

1 IT finance staff FTE dedicated to data collection and analysis

Years to mature implementation including service segmentation and expense tracking4-5

“It was apparent we had to change our funding model. ABC laid the foundation for being able to say, this is what it costs to provide services. Not only do we use it to determine new per FTE funding model, but it also informs where we are investing and areas where we need to invest more.”

Cynthia Herrera Lindstrom, CIO

Cost to deliver 38 services calculated down to the transactional level


33

Developing Value Heuristics


Join the Credit-Claiming Crowd

Strategic Enrollment Initiative

Task Force Report

Admissions Redesign

Brand Refresh

Facilities Upgrade

New CRM System

New Degree Programs

Enrollment Growth!


34


Targeted Claims, Broad Implications

Making the Case for IT Value at Florida International University

Student Success

Automated attendance tracking in 100 classrooms links IT to high-impact success practice

Student Services Efficiency

In first three days of semester, chatbot successfully fields 7000 inquiries with 65% first-call success, freeing up staff for higher-value work

Improved Student Experience

Analytics project reviewing queuing system and call center data leads to 45% year-over-year drop in student walkouts at the One-Stop services center

“When academic units graduate more students in a timely way, the deans will always get credit for that. As a CIO you have to understand where you made the contribution.”

Robert Grillo, CIO

High-Impact IT Value Stories


35

University of Nevada Reno Links HPC Investment to Discovery and Learning

Two Birds With One Stone


“Not all institutional goals have a specific IT mention in them. However, in many cases there is a secondary agenda that has an IT component. We look for where the IT piece is that helps achieve those goals.”

Steve Smith, CIO

Discovery: Create new knowledge…

Learning: Prepare graduates to compete…

HPC cluster bought with central funds

IT measures dollar value of HPC-related grants and

undergrad participation in funded research.

Demonstrates contribution to two strategic goals


36

Where Perfect Isn’t Possible, Aim for Good Enough

Heuristic (n.) - Any approach to problem solving, learning, or discovery that employs a practical method, not guaranteed to be optimal, perfect, logical, or rational, but instead sufficient for reaching an immediate goal.”

Wikipedia



37Structure of a Value Story

Quantifiable

IT Claimable

Impact on Stakeholder Interests

Strategic Relevance

Claim is verified by customer, sponsor, consultant, etc.

Craft IT Value Claims to Eliminate Credit Noise and Subjectivity

ExternalValidation

Can be quantified at least to a rough approximation

IT contribution predominates

Substantially affects a process or success measure meaningful to stakeholder

Clear relevance to institutional strategic objective(s)


Ensure the foundation, aim for the pinnacle

IT Value Heuristic Development Process


38

FILTER

• IT can claim most or all credit

• Contribution can be credibly quantified (within 20%)

• External party will validate claim

Develop Short List of Value Claims

Define Strategic Concern

• Enabled new customer process

• Expanded usage or access

• Made workflow faster, easier, cheaper

• Improved ability to assess or analyze

• Improved security or compliance

Find IT Value Contributions

• Determine area of strategic concern

• Identify stakeholders and their objectives

• List related IT projects and services

IDENTIFY

Rate Value Claims

• Relevance to stakeholder objective

• Magnitude of impact

• Ongoing v. point impact

PRIORITIZE


A Process for Selecting Value Heuristics


39

Refer to IT Value Heuristics Worksheet


Value Heuristics Exercise

1. Choose an area of strategic concern

2. List strategic objectives in that area

3. Consider IT contributions to objectives and list relevant initiatives

4. Identify initiatives that are IT-claimable and quantifiable

5. Use the Value Story Score tool (page 2) to rate each initiative as a potential value story

6. Be ready to share your good stories and how you will communicate them!


40Vox Populi

Strategic SignificanceAdaptable to NeedCapture Outputs, Not Inputs

• Help answer whether investments are effective

• Avoid self-interested judgment of service deliverers and project participants

• Can capture point information or continuous feedback

• Questions and respondent groups can change as needed

• Quantitative or qualitative data

• Always of leadership interest

• Indicates customer relationship strength

• May directly address strategic goals (e.g., student experience)

Customer Satisfaction Surveys a Critical Element of Value Demonstration


Methodology and process affect results

Results subject to interpretation

Requires administrative and analytical resources


41

Higher Education IT Enjoys Diverse Range of Standard Instruments


Customer Satisfaction Survey Options

Survey Core Focus Instrument Customizable

Process Support Service Cost

MISO Library and IT Service Importance, Use, Satisfaction

$2200-$3000

TechQual+ IT Service Expectations and Outcomes

Free

ETRAC (EDUCAUSE)

Student/Faculty Technology Preferences and Attitudes

Free

Homegrown As Desired “Free”


42

Declining Response RatesAccess Channels Increasingly Filtered

• “Too many surveys”• Shift from paper to online processes

reduces response rates

• Increased screening of phone calls

• Low email usage

Rising Desire to SurveyMany Competitors for Student Data

• Retention and engagement initiatives

• Learning outcomes assessments

• Service improvement efforts

25Number of surveys requested of students per year at one UK university

-22% Drop in response rates among first-year students from first to third survey in semester

Inst

itutio

nal D

rive

rsStu

dent

Res

pons

e

Sources: Manja Klemencic and Igor Chirikov, “How Do We Know How Students Experience Higher Education? On the Use of Student Surveys,” The European Higher Education Area(2015); Stephen Porter et al., “Multiple Surveys of Students and Survey Fatigue,” New Directions for Institutional Research (2004); “Student Surveys Are A Waste of Everyone’s Time,” The Guardian, 1 July 2016.

Enough AlreadyRelentless Data Demands Meet Survey Resistance

“Administering multiple surveys in one academic year can significantly suppress response rates.”

Porter et al. (2004)

“Proliferation of student surveys is part of the evidence-based movement in higher education [and] a reflection of growing competition.”

Klemenčič and Chirikov (2015)

Fatigue-Fighting Customer Survey Tactics

https://link.springer.com/chapter/10.1007/%20978-3-319-20877-0_24

https://oia.unm.edu/surveys/survey-fatigue.pdf

https://www.theguardian.com/higher-education-network/2016/jul/01/student-surveys-are-a-waste-of-everyones-time


43


Trading Length for Frequency and Comparability

Fast Turnaround

From survey close to results availability

2 DaysPrivate tuition-dependent institution with strong data-driven culture.

ChallengeSatisfy need for unit performance data without over-burdening campus community.

SolutionSix administrative units jointly conduct one annual customer satisfaction survey:• Information Technology• Facilities• Human Resources• Finance• Communications• Public Safety

Joint Approach Permits Annual Surveying

31%Response rate, format signals respect for respondent time

Structured for Easy Response

Limited to six multi-part questions per unit:• Frequency of unit use• Rate staff• Rate services• Core values alignment• Overall satisfaction • Comments

Top-level questions and response scales are same; detail items differ

Team-Oriented Results Reporting

Leadership team takes turns reporting results in campus newsletter

Common format makes comparisons easier

CIO use in deans’ meetings, etc. sets up feedback loop


44

Four Ways Coppin State University Reaches Students Where They Are

CLASSROOM MINI-SURVEYSFaculty allocate 7 minutes at beginning of class for short trainings and surveys

1

SPECIAL EVENT SURVEYSTechMania and Tech@Night events attract 300-400 students, yield survey responses

2

CAMPUS BOOTHSTechKNOW-staffed tables in high traffic areas provide information about IT services

3

The TechKNOW Squad

ChallengeA commuter HBCU, Coppin State struggles to get satisfactory response rates on student surveys.

SolutionVolunteer team of about 25 students conducts on-the-spot peer trainings and short surveys on IT preferences and service satisfaction.


A Tiger Team for Peer-to-Peer Outreach

FOCUS GROUPSquad members are available for formal and informal consultation

4


45Information Is Power

Work with Faculty and Other Partners for Maximum Survey Impact

IT reviews question pool, creates first draft.

IT and survey provider monitor progress, call on faculty groups for response help as needed.

Pre-briefed partners assist with dissemination and response.

Design Field Analyze Plan

IT consults with faculty to finalize instrument.

Faculty and leadership get sneak peak of early results, help develop action plan.

“It was a heavy lift having faculty engaged in the survey conversation, but that was actually a benefit. We talked about what we needed to know versus what was interesting but not actionable. You need to be transparent and share the results, warts and all. In fact, the problem areas are your friends. Engaged faculty can advocate for improvement.’”

Eric Behrens, CIO, Widener University


Publish

Survey partners help

design projects, seek resources


ROAD MAP46





47

Users and Leaders Fail to Recognize IT’s Contributions to the Campus


No Understanding Without Communication

What the Campus Doesn’t See, the Campus Can’t Appreciate

IT Defensiveness and Isolation Make Matters Worse

A Preponderance of Negative AssociationsThe campus only sees or hears from IT when something is wrong or broken—tempers run high and patience runs low

1

Successes Taken for Granted or Limited in RecognitionBreakdowns are more memorable than reliable services, and good project outcomes only know to a few, even in IT

2

Breaking Down IT’s Image Problem

“Us” versus “Them” Mentality Taking RootIT staff increasingly wary of communicating with campus—perpetuating the vicious cycle of negative perceptions

3

“I couldn’t name a single person in IT!”

“It sure would be nice if IT people knew how to speak plain English”“The only time I see IT is when they’re fixing yet another broken thing”

“We have so much to do that there’s not much time for communication.”

“The only time campus talks to us is to yell at us”

“No one appreciates how hard it is to deliver reliable services to so many different customers”


48

Professional Communications Benefits Are Real But Hard to Quantify

IT Communications Director

Source: EDUCAUSE, The Higher Education IT Workforce Landscape 2016; U.S. Bureau of Labor Statistics; EAB interviews and analysis.

A Dose of Soft Skills for the IT Organization

Important Skills at a Relative Bargain

Importance of communicating effectively among CIO skills

#1Mean salary of writers and editors in higher education

$63,310Cost of writer/editor relative to developer

82%

Skills Contributed

• Ability to communicate complex concepts in accessible language

• Segmented messaging to different groups via appropriate channels

Benefits to IT

• Better customer awareness of services and projects

• Refined communication of IT value and performance

• Listening and learning from a non-IT standpoint

• News judgment: ability to recognize newsworthiness and explain it

• More recognition for IT staff• Lessened workload, greater

effectiveness for IT leadership

https://library.educause.edu/~/media/files/library/2016/4/ers1603.pdf

https://www.bls.gov/oes/current/naics4_611300.htm


49Low-Effort, High Impact Communication

How IT contributes to strategic goals

Scope of IT activities What IT is doing for you

How IT has made a difference

Dense IT strategic plan materials

Diffuse data, narrow user view

Project knowledge limited to a few

Successes stay local

Strategic initiatives roadmapMulti-dimensional representation of strategic initiative impact

IT-at-a-glance infographicsEasily consumed representations of IT activity accessible to a wide range of audiences

Impact-based project promotionTemplated project awareness stories inviting inquiry and participation

IT Success Stories

Systematic efforts to capture and relate IT accomplishments



50

Project Prioritization at Embry-Riddle Assigns Risk and Impact Scores

The Last Mile of IT Strategic Planning

STRATEGIC REVIEW IT REVIEW LEADERSHIP

APPROVALCOLLECT

REQUESTS

IT asks for strategic project requests; requesters provide goals, cost and impact estimates

Appropriate Strategic Implementation Committee reviews for alignment with strategic domain

IT reviews for completeness, estimates cost, assigns risk score and impact score based on:

• Breadth of service

• Potential for revenue increase

• Efficiency/cost reduction

Core leadership team reviews inputs, prioritizes high impact/low effort projects

Project roadmap with timeline, impact, and risk information for all approved projects


Strategic Initiatives Roadmap


51

Embry-Riddle Summarizes IT Direction in Multi-Dimensional Graphic


A University-Wide Technology Road Map


52

Source: Embry-Riddle Aeronautical University Technology Roadmap 2018; EAB interviews and analysis.

Information Density Allows a Richer Story to Be Told

Concentric circles indicate project completion timeframes

Projects grouped by stakeholder categories

Project dot size represents impact, color represents risk

Embry-Riddle Technology Roadmap

Roadmap Identifies Strategic Projects by Category, Time, Impact, and Risk


53

Given Context, Measures of IT Scope and Scale Suggest User Impact


The Case for Volume Metrics

The Virtues of Volume

• Big enough to include everybody

• Inform community of breadth of services

• Show IT is working hard

• Lend themselves to visualization

• Low effort to collect

University of AlbertaTexas A&M

University of Florida

Embry-Riddle Aeronautical University

University of Iowa

IT-at-a-Glance Infographics


54

Embry-Riddle Supplements Roadmap with Performance Facts Infographic

Source: Embry-Riddle Aeronautical University Information Technology; EAB interviews and analysis.

What Have You Done For Me Lately?

Infographic summarizes IT activity touching a wide range of interests, including:

• Project achievements

• Customer satisfaction

• Security interventions

• Budget information

• IT staff development

A Fresh Snapshot of IT Activity


55

DesignPredesigned template built around 9 metrics categories makes twice-annual update easier

Wash, Rinse, Repeat

ProductionMarketing and publications staff update the template to produce new infographic

AssemblyOver a two-week

window, CIO direct reports collect updated

data, consider changing reporting

categories Circulation

Infographic posted to portal, circulated

during meetings and town halls

Quick Production Eases Refresh of Embry-Riddle Performance Facts



56Complementary Quick-Review Perspectives

Strategic Roadmap

Performance Facts Infographic

Relatable Customer Info

“The infographic provides a snapshot of what we have accomplished, and we simultaneously share our roadmap. Most importantly, we put our energy into where we go from here. To me, the infographic and roadmap are conversation tools. Any time I bring them to groups, dialog happens.”

Becky Vazquez, Vice President and CIO

Strategic Vision Plus Ongoing Activity Equals IT Contribution



57

Loyola University Chicago Annual IT Summary Tells a Year’s Story

Source: Loyola University Chicago Information Technology Services; EAB interviews and analysis.

An IT Organization On a Page

The IT Annual Summary• Two sided trifold• Staff collects material• Leadership team writes• 2-3 month cycle time


58


Projects Make It PersonalImpact-Based Project Promotion

Communicate IT Contributions to Influence Campus Perspectives

Showcase Creative Problem Solving

Foster a Narrative of Progress

Highlight Novel Technology Capabilities

Demonstrate Service to Campus Beneficiaries

Publicize IT projects to…


59

University of Iowa Project News Items Raise Cross-Community Awareness

Source: https://oneit.uiowa.edu/news/project-news; EAB interviews and analysis.

IT Projects as IT Promotion

Projects screened for high impact are described in a scrolling panel on the ITS website

Postings are built from a template

filled out by project creator

“Everybody in the IT community gets a chance to see there’s a new project, and can say, ‘I’m interested, can I join your project?”

Steve Fleagle, CIO

https://oneit.uiowa.edu/news/project-news


60

Documentation Reduces Need for Reporting, But Writing Still A Value Add

Project Management Informs Communication

Project Management Info

StoryElement

Sponsor, customer, team

Who

Project description and impact

What

Project plan and status

When

Intended outcomes and success measures

Why

Project management processes capture essentials

Skills and workload issues call for writer

handoff

Quick completion and publication

Source: https://oneit.uiowa.edu/news/project-news; EAB interviews and analysis.

https://oneit.uiowa.edu/news/project-news


61Personalizing Project Impact with Metrics

Headline establishes the need for protection in face of threat

1

Body uses volume metrics to show that IT is on the job

2

Conclusion shows that user still must “prove it’s you” to be safe

3

West Virginia University Infographic Prepares Campus for Two-Factor Authentication

“When we started communicating what IT was doing at WVU, we did a monthly update with two-paragraph summaries and an infographic. People would spend 10 minutes on the infographic but wouldn’t read the paragraphs.”

Vicki Smith, Communications

Director

Source: West Virginia University Information Technology Services; EAB interviews and analysis.


62

Once Established, Success Stories Incentivize Staff Contributions

IT Success Stories


Success May Be Hiding In Plain Sight

Quantifiable

IT Claimable

Impact on Stakeholder Interest

Strategic Relevance

ExternalValidation

“At first, people didn’t get what I was looking for with success stories. Once we wrote a few, the IT staff had an ‘Aha’ moment and started coming to me. It’s a form of recognition. They’re excited to show their work made a difference.”

Nicole Dahya, Communications Manager, University of Iowa

Source Stories

Story Publication

Staff Recognition

Leadership Pump Priming

Washington DC Richmond Birmingham Minneapolis

P 202-747-1000 F 202-747-1010 eab.com

Documents

EAB Getting the Measure of IT UWMadison...Robert Burton, The Anatomy of Melancholy(1621); EDUCAUSE, “Top 10 IT Issues, 2018,”EDUCAUSE Review, Jan-Feb 2018; EDUCAUSE, Analytics