Class 8: Information management, Class 8: Information management, Performance Measurement & Performance Measurement &

Program EvaluationProgram Evaluation

UTA School of Social WorkUTA School of Social Work6371: Community & Administrative Practice6371: Community & Administrative Practice

Dr. Dick SchoechDr. Dick Schoech

Copyright 2009 (permission required before use)Copyright 2009 (permission required before use)

Suggest printing slides for class using: Print | Handouts | 3 slides per page | grayscale optionsSuggest printing slides for class using: Print | Handouts | 3 slides per page | grayscale options

Review of Previous ClassesReview of Previous Classes2.2. Theories, values, perspectivesTheories, values, perspectives

3.3. Problem definition, theories of needProblem definition, theories of need

4.4. Capacities and needs assessmentCapacities and needs assessment

5.5. Evidence based interventions and logic modelsEvidence based interventions and logic models

6.6. Program descriptions, community linkagesProgram descriptions, community linkages

7.7. Mission, Goals and ObjectivesMission, Goals and Objectives

8.8. Information Management, performance Information Management, performance measurement, program evaluationmeasurement, program evaluation

9.9. Budgeting Budgeting §§

Goals of ClassGoals of Class

To examine the basics of program evaluationTo examine the basics of program evaluation To understand the different foci of: To understand the different foci of:

• Process evaluationProcess evaluation• Outcome evaluationOutcome evaluation• Impact evaluationImpact evaluation

To understand the role of measurement & To understand the role of measurement & feedback in managementfeedback in management

To understand organized information concept To understand organized information concept & processes in management& processes in management

Program EvaluationProgram Evaluation

Evaluation OverviewEvaluation Overview

Purposes of evaluationPurposes of evaluation TypesTypes

• Process evaluation (focus on fidelity)Process evaluation (focus on fidelity)• Outcome evaluation (focus on results)Outcome evaluation (focus on results)• Impact evaluation (focus on problem Impact evaluation (focus on problem

reduction)reduction) Evaluation: old vs. new paradigmEvaluation: old vs. new paradigm Issues Issues §§See course pack for PDF on Evaluation

Purposes of EvaluationPurposes of Evaluation

Determination of fidelity Determination of fidelity Provides continuous quality improvement Provides continuous quality improvement

mechanisms (TQM)mechanisms (TQM) Demonstrating accountabilityDemonstrating accountability Gives information for agency decision makingGives information for agency decision making Building stakeholder support with credibilityBuilding stakeholder support with credibility Enhancing future funding potentialEnhancing future funding potential Discovering new knowledge, e.g., cause-effect Discovering new knowledge, e.g., cause-effect

relationshipsrelationships Fits our evidence-informed social work approachFits our evidence-informed social work approach§§

Type: Process/formative/monitoringType: Process/formative/monitoring

Answers: WHO doing HOW MUCH of WHAT to Answers: WHO doing HOW MUCH of WHAT to HOW MANY of WHOM and WHEN are they HOW MANY of WHOM and WHEN are they doing itdoing it

All measures in the process objectives are All measures in the process objectives are covered in the monitoring and trackingcovered in the monitoring and tracking

Measures are:Measures are:• Whether activities are completed as describedWhether activities are completed as described• How much effort things took, e.g., were they How much effort things took, e.g., were they

efficientefficient• Satisfaction with what was doneSatisfaction with what was done• Fidelity, whether intervention implemented Fidelity, whether intervention implemented

correctly, e.g., correct dosage of intervention correctly, e.g., correct dosage of intervention given, services of high quality given, services of high quality §§

Type: Outcome evaluationType: Outcome evaluation

Answers: Who isAnswers: Who is doingdoing how much of whathow much of what to how many of whomto how many of whom, , whenwhen are they are they doing it and withdoing it and with WHAT RESULTSWHAT RESULTS

Measures are:Measures are:• Client change, e.g., Goal Attainment Client change, e.g., Goal Attainment

Scales (GAS)Scales (GAS)• cost/benefitcost/benefit• Effectiveness measuresEffectiveness measures§§

Type: Impact EvaluationType: Impact Evaluation

•Results for Results for person/family/communityperson/family/community

•Unintended side effects on Unintended side effects on person/family/communityperson/family/community

•Measures are:Measures are: If condition changed If condition changed If problems reducedIf problems reduced If need reduced If need reduced §§

Evaluation: Evaluation: Old vs. New Tech ParadigmOld vs. New Tech Paradigm

Imposed from outsideImposed from outside Tedious and onerous Tedious and onerous

activityactivity Results sit unused on Results sit unused on

shelfshelf Fear of what evaluation may Fear of what evaluation may

saysay Complex and expensiveComplex and expensive Takes time from real workTakes time from real work True results are impossible True results are impossible

to quantifyto quantify

Desired from insideDesired from inside Integral to way of providing Integral to way of providing

services so supported by techservices so supported by tech A continuous process of A continuous process of

organizational learningorganizational learning Requires collaboration from all Requires collaboration from all

parties parties Can requires expert Can requires expert

consultation in design and consultation in design and interpretation phasesinterpretation phases

Documents success & how to Documents success & how to improve §improve §

Old New

Evaluation IssuesEvaluation Issues Cost (1-5% of budget)Cost (1-5% of budget) Source (inside or outside)Source (inside or outside) Accuracy or for showAccuracy or for show How results used and disseminated, e.g., public How results used and disseminated, e.g., public

or not publicor not public How to measure change/outcomes (performance)How to measure change/outcomes (performance) Whether client change or lack of change is due to Whether client change or lack of change is due to

program or other causes Public/funders not program or other causes Public/funders not understanding success, 5% change 6 months understanding success, 5% change 6 months after treatment may be very goodafter treatment may be very good

ConclusionConclusion

Evaluation is being taken more Evaluation is being taken more seriouslyseriously

Outside evaluation often requiredOutside evaluation often required

A much less rigorous process if A much less rigorous process if implementing a proven modelimplementing a proven model

Evaluation is changing to support Evaluation is changing to support proactive management & decision proactive management & decision supportsupport

Information Information management and management and Decision SupportDecision Support

Book was as different from the Book was as different from the scroll as computer is from book so scroll as computer is from book so support needed. Click for videosupport needed. Click for video

http://www.milkandcookies.com/link/57946/detail/

Management using Information technologyManagement using Information technology

Information & IT as a basic resourceInformation & IT as a basic resource Information management termsInformation management terms Information system componentsInformation system components Change via 8 step process, 3 approaches, Change via 8 step process, 3 approaches,

several agency structuresseveral agency structures Guidelines for successGuidelines for success Organizational Learning/IntelligenceOrganizational Learning/Intelligence Future of Technology in Social Work Future of Technology in Social Work §§

Information/technology as basic resourceInformation/technology as basic resource

Treat info as other basic resource, e.g., $s, people, Treat info as other basic resource, e.g., $s, people, etc.etc.

Info easy resource to change & manage (compared to Info easy resource to change & manage (compared to people, task, goals, structure)people, task, goals, structure)

Computers add power to information management, Computers add power to information management, but errors can be bigger and more disastrousbut errors can be bigger and more disastrous

Managing info more difficult if shared across Managing info more difficult if shared across departments & outside agenciesdepartments & outside agencies

Information resource is easily sabotaged, e.g., Information resource is easily sabotaged, e.g., garbage in—garbage outgarbage in—garbage out

Technology keeps changing requiring constant Technology keeps changing requiring constant attention attention §§

Information Management TermsInformation Management Terms Database vs. Database Mana SystemDatabase vs. Database Mana System

• Database=a collection of data (records in fiscal, client, Database=a collection of data (records in fiscal, client, services, and staff files) with minimal redundancy and services, and staff files) with minimal redundancy and easy manipulation and access.easy manipulation and access.

• DBMS=software to manage a database (MS-ACCESS)DBMS=software to manage a database (MS-ACCESS)

• Data Warehouse = Department where all data Data Warehouse = Department where all data resides, is managed, analyzed, and disseminatedresides, is managed, analyzed, and disseminated

Computer vs. Information SystemComputer vs. Information System• MIS=system of people, procedures, and equipment for MIS=system of people, procedures, and equipment for

collecting, storing, manipulating, protecting, and collecting, storing, manipulating, protecting, and reporting informationreporting information (can be manual)(can be manual)

ASP, Internet, Intranets, extranets ASP, Internet, Intranets, extranets §§

IT as ChangeIT as Change

Treat as any other changeTreat as any other change

Info mana develops an info model of agency like Info mana develops an info model of agency like financial mana develops a money model of agency. financial mana develops a money model of agency. Agency goals/obj drive info & $, not vice versa.Agency goals/obj drive info & $, not vice versa.

A 2-3 year effort of hard work by allA 2-3 year effort of hard work by all

Social workers are experts in change but have Social workers are experts in change but have problems with technology changeproblems with technology change

All change involves resistance & emotional loss of All change involves resistance & emotional loss of old way of doingold way of doing

Reduce change problems by following a process §Reduce change problems by following a process §

8 Step Development Process8 Step Development Process

1.1. Assess Preparedness & FeasibilityAssess Preparedness & Feasibility

2.2. Analyze Existing SystemAnalyze Existing System

3.3. Conceptual DesignConceptual Design

4.4. Detailed Design & DevelopmentDetailed Design & Development

5.5. System Testing & Agency PreparationSystem Testing & Agency Preparation

6.6. Convert from old to new systemConvert from old to new system

7.7. EvaluationEvaluation

8.8. Operation, maintenance & modificationOperation, maintenance & modification

1: Explore feasibility/preparedness1: Explore feasibility/preparedness

A key step often neglected Communicate about effort to all stakeholders Establish steering committee Estimate resources of money, time, expertise Determine commitment of key individuals Estimate potential impacts (positive & negative) Define IT purpose, objectives, timetables,

responsibilities, evaluation Assess the expectations/reactions of those impacted Circulate preparedness/feasibility report §§

2: 2: Assess Situation (systems Analysis)Assess Situation (systems Analysis)

Identify the major situations/tasks/decisions system will support & what is needed to support them

Define the characteristics of the information needed, its source, and collection methods

Analyze current input forms, data processing, storage files, and output reports

Describe logical routing or flow of information from collection to dissemination

Review similar efforts in other agencies Prepare systems analysis report §§

3: Prepare conceptual Design3: Prepare conceptual Design

Finalize IT scope, goals, and objectives

Develop alternative conceptual designs, i.e., files, data manipulation, forms and reports

Apply restrictions to designs (volume, security, confidentiality, turn around time, statistical capabilities, growth potential, ties to other systems)

Apply resources to designs (money, time, expertise)

Translate designs into software and hardware Detail the advantages, disadvantages, and assumptions of

alternate designs §§

4: Detailed Design & Development4: Detailed Design & Development

Set up controls and technical performance standards for chosen design

Select the software for the chosen design

Select the hardware to match the software

Design and develop input forms, data manipulations & processing operations, file specifications, database structures, indexes, error checks, storage and backup mechanisms and procedures, and output reports

Prepare programming documentation and instruction manuals §§

5: Test System & Prepare Agency5: Test System & Prepare Agency

Prepare system operators, users, and others to receive the system

Develop agency policy and procedural changes necessary for the new system

Develop performance criteria and testing plan

Test programming, forms, operational procedures, instructions, reports, and use of outputs

Educate and train system operators, system users, and others affected §§

6: Implement6: Implement

Develop and approve conversion plan, e.g., stop old system when new system starts or run old and new systems simultaneously for comparison

Incorporate IT into standard operating procedures, e.g., performance appraisals, new employee orientation

Reorganize staff and space if necessary

Convert from old to new equipment, new processing methods, and new proceduresInsure all systems and controls are working §§

7: Monitor and evaluate7: Monitor and evaluate

Compare system performance with initial system objectives, e.g., if system improved client services

Relate benefits and costs to initial estimates

Measure satisfaction with the system

Determine if system outputs are used in decision making §§

8: Operate, maintain & modify8: Operate, maintain & modify

Prepare backup and emergency plans/procedures

Complete documentation e.g., instructions for adding to, deleting from, or modifying system

Assign persons responsible for data integrity, system maintenance, new software appropriateness, virus protection, etc.

Provide continuous training of users Continue to add desired enhancements and to maintain

and debug the system §§

Which Development ApproachWhich Development Approach Top down – specify total system firstTop down – specify total system first Bottom up – Design small systems Bottom up – Design small systems

and then link them into a total and then link them into a total systemsystem

Prototyping – Build as you go and Prototyping – Build as you go and build overall system as needed build overall system as needed §§

Structuring Expertise: Power Balance

Top management

IT Committee

Application Steering Committee

Information Manager/Department

A balance of power exists between the IT committee and the information manager. Disagreements are settled by top management.

Structuring Expertise: Committee Control

Top management

IT Committee

Application Steering Committee

Information Manager/Department

The application steering committee and information manager reports directly to the IT committee (typically the recommended structure).

Structuring Expertise: Management control

Top management

IT Committee

Application Steering Committee

Information Manager/Department

All computing committees are under the information management department. This structure is not usually recommended because it gives too much control to the information manager and makes access to top management by user representatives more difficult.

Guidelines for SuccessGuidelines for Success

IT needs separate high level deptIT needs separate high level dept Planning with stakeholders is essentialPlanning with stakeholders is essential Centralize control, distribute powerCentralize control, distribute power Avoid relying on small # techniciansAvoid relying on small # technicians 10/40/50 rule and 80/20 rule10/40/50 rule and 80/20 rule Expect frustrating and time consuming Expect frustrating and time consuming (2-(2-

3 years for basic system3 years for basic system §§

IT system development issuesIT system development issues

What % of budget to spend on ITWhat % of budget to spend on IT Social workers are experts in change but have Social workers are experts in change but have

problems with technology changeproblems with technology change All change involves loss/resistanceAll change involves loss/resistance IT involves emotional loss of old way of doingIT involves emotional loss of old way of doing How to get the necessary expertise when IT pays How to get the necessary expertise when IT pays

much more than social workmuch more than social work Security, privacy, & confidentiality are big issues Security, privacy, & confidentiality are big issues

when automating and sharing information when automating and sharing information §§

Current vs. FutureCurrent vs. Future

Networked PC on desks Beginning to web enable Basic MIS developed Information as text & #s User passively receives

system outputs High cost info for a few Worker’s info better than

client’s You search for information

Convergence & connectivity (phone, computer, TV)

Information as text, pictures, movies, VR, etc.

Increased interactivity Client’s info better than

worker’s Information finds you Interconnected agencies Infotainment §

Current Future

Organizational Intelligence & DSSOrganizational Intelligence & DSS

Agency has capacity to accumulate expertise Agency has capacity to accumulate expertise

Agency becomes smarter as each case is workedAgency becomes smarter as each case is worked

Intelligence automatically available when and where Intelligence automatically available when and where needed and in the format neededneeded and in the format needed

Intelligent agencies linked to form virtual support Intelligent agencies linked to form virtual support system with global knowledgesystem with global knowledge

Much can be done automatically, e.g., automatic Much can be done automatically, e.g., automatic sharing of information sharing of information

Interactive models (Decision support systems) to Interactive models (Decision support systems) to scenario decision making scenario decision making §§

Examples DP & CAPExamples DP & CAP

DPDP Human services information systemHuman services information system Web Phone System Direct practice—depression Direct practice—panic & anxiety Direct practice—marriage counseling Online therapy Dr. Judith -- consumer beware -- consumer beware

CAPCAP Move on, , Meet up, Kiva, , Kiva,

ResourcesResources

Community Tool boxCommunity Tool box NASW/ASWB Technology StandardsNASW/ASWB Technology Standards Journal of Tech in Human ServicesJournal of Tech in Human Services TechSoup for technology support and TechSoup for technology support and

products for agenciesproducts for agencies HUSITA (HUman Services Information HUSITA (HUman Services Information

Technology ApplicationsTechnology Applications FOSS (free & open source software)FOSS (free & open source software)