Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
NRS Institute onDeveloping State and Local Report
Cards for Adult Education
American Institutes for ResearchMiami, Florida
February 28-March 2, 2007
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Goals of the InstituteGoals of the Institute
By the end of the training, participants will be able to
Discuss the purposes of, and audiences for, report cards for state and local adult ed programs;
Identify essential elements of a report card; Distinguish between effective and ineffective
report cards; Use a template to develop state and local report
cards; Create a rubric for evaluating performance; and Develop a dissemination plan.
02
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Agenda for Wednesday, Feb. Agenda for Wednesday, Feb. 2828
Lessons Learned: States’ Experiences with Report Cards
A Word about Quality: States’ Plans to Improve Quality
Step 1: Define Purpose of and Audience for State and Local Report Cards
Step 2: Select Data Elements/Measures for State and Local Report Cards
Step 3: Select Evaluative Criteria or Rubric Step 4: Design and Format Report Cards State Planning Time and Status Reports Introducing the User Guide and Report Card
Templates Wrap-up and Evaluation for Day 1
03
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Agenda for Thursday, March Agenda for Thursday, March 11
Navigating the Template A Trial Run with a Hypothetical State Q&A about the Template State Teams Build Local Report Cards Progress Checks and Support State Teams Demonstrate Local
Report Cards Parking Lot/Discussion, Wrap-up, and
Evaluation for Day 2
04
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Agenda for Friday, March 2Agenda for Friday, March 2 State Teams Build State Report
Cards Progress Checks and Support State Teams Demonstrate State
Report Cards Step 5: Disseminate and Promote
Program Improvement State Teams Develop Dissemination
Plans Next Steps, Wrap-up, and Evaluation
05
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Why Report Cards?Why Report Cards?
Accountability—Measure Adequate Yearly Progress
Program Improvement
Inform and Advocate for Program
06
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Questions for ConsiderationQuestions for Consideration What would a report card on
hospitals tell you?
What about one on surgeons?
What about one on airlines?
What about one on stocks?
What would you do with the information?
07
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
What is a Report Card?What is a Report Card?
A concise presentation of data and other information about a school or program that assesses performance
The focus on evaluating performance is what makes report cards a unique type of accountability report.
08
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Characteristics of Characteristics of EffectiveEffective Report Cards Report Cards
Focus on outcomes and other data that reflect program quality;
Provide a basis of comparison for evaluating these data; and
Present contextual data or interpretive information that aid interpretation and promote understanding.
09
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Characteristics of Characteristics of IneffectiveIneffective Report Cards Report Cards
Provide poor indicators of quality
Do not support program improvement
Are not informative
10
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Three Primary Factors of Three Primary Factors of IneffectiveIneffective Report Cards Report Cards
1. Wrong measures
2. Insufficient measures
3. Unreliable or low quality data
11
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
A Word about A Word about AggregatedAggregated v. v. DisaggregatedDisaggregated Data Data
We weighed each of you as you entered this room, so we know that the average weight of everyone in this room is 195 lbs. And we posted that average on the wall.
Is this good or bad news?What does this tell us about our entire population?What does it tell us about males v. females?Is this a weight loss group or a steroid users’ support group?
12
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Help us understand patterns of success and failure in the student population;
Help us separate the “whys” from the “whines”
Caution: When we report information on student learning as a mean, it tells us about as much as the weight sign on our wall.
Disaggregated Data…Disaggregated Data…
13
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
NCLB Requirements for NCLB Requirements for State Report CardsState Report Cards
Concise; understandable and uniform format Info on student achievement, both
aggregated and disaggregated Comparison between actual achievement
levels and the state’s annual goals/objectives
Most recent trends in student achievement Performance of local agencies re: AYP with
names of schools identified for school improvement
Professional qualifications of teachers Description of state’s accountability system
14
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
NCLB Requirements for NCLB Requirements for LocalLocal DistrictDistrict Report Cards Report Cards
The same data required for the state report card as applied to the local education agency
Performance of students of local district compared to performance of students in the state as a whole
Performance of local agencies re: AYP with names of schools identified for school improvement
15
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
DifferencesDifferences Between Uses of Between Uses ofState and Local Report CardsState and Local Report Cards
Local: Include a broader range of outcome
measures to evaluate program More suited for program improvement
efforts Use evaluative standard such as program
past performance, performance of similar programs, or local performance standard
State: Use a more limited range of outcome
measures Use evaluative standards such as state past
performance, state performance standards, or national averages16
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
NRS Outcome Measures: NRS Outcome Measures:
The The CenterpieceCenterpiece of the Report of the Report
CardCard
Educational gain Receipt of a secondary credential Entered and retained employment Entry into postsecondary education
17
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Other Measures?Other Measures?
Student attendance and persistence data
Student and employer satisfaction with program
Learner accomplishments Number of hours of instruction
offered Average class size Teacher qualifications
18
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Contextual Data that Contextual Data that Aid InterpretationAid Interpretation
Student demographics
Socioeconomic data about the community
Per student cost
19
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Basis for EvaluationBasis for Evaluation
To evaluate program quality, must have a standard or basis of comparison
Compare students and programs to each other and to fixed standards
Examples: letter grades, past performance, average state performance
20
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Five Steps to Developing a Five Steps to Developing a Report Card for Adult Report Card for Adult
EducationEducation
1.1. DefineDefine purpose and audience
2.2. SelectSelect measures
3.3. SelectSelect evaluative criteria or rubric
4.4. DesignDesign and format
5.5. DisseminateDisseminate and promote program improvement
21
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
1. Define Purpose and 1. Define Purpose and AudienceAudience
Purpose: evaluation, program improvement, or information?
State or local report card? Audience: local program staff, state staff,
funding agencies, legislators, general public? Which information to show/not show? Is this
different for difference audiences? In what areas are your programs doing
well/doing poorly? Why? What messages do you want to send to
audience and what do you want them to do about it?
22
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
2. Select Measures2. Select MeasuresOther Outcome Measures
Instruction Measures
Program and Teacher Measures
Student Measures
State or local test performance measures, such as test score gains
Other employment measures, e.g., better job
Family literacy measures
Amount and type of classes and instruction offered
Average hours of attendance
Budget and average cost per student
Support services provided
Number of teachers and qualifications
Average class sizes
Literacy skills upon program entry
Demographics of service area (e.g., % without H.S. diploma;unemployment rate)
23
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Matrix of Audience, Report Card Matrix of Audience, Report Card Purposes, Measures and Purposes, Measures and
ComparisonsComparisonsRpt Card Purposes
Main Audiences Measures and Comparison
Evaluate Program Quality
State and local staff General public Legislators Funding agencies Partner agencies
Basic outcome measures compared to performance standards, past performances, state or national averages
Program Improvement
TeachersLocal program staff
Broader outcome measures compared to similar programs, local performance standards, or past performance
Inform and Advocate
General publicLegislatorsFundersPartner agencies
Outcomes with comparisons of similar programs’ outcomes & with contextual data about need and community conditions
24
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
3. Select Evaluative Criteria or 3. Select Evaluative Criteria or RubricRubric
Evaluative Criteria Use
Program Past Performance
Demonstrate or monitor continuous improvement
Performance Targets
Uniform standard for programs
State Averages Compare local performance against others in the state
Other Local Program Performance
Compare local performance and account for variations among programs
25
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
What is a Rubric?What is a Rubric?
A scheme for classifying products or behaviors into categories that vary along a continuum
A vehicle for describing varying levels of quality, from excellent to poor, or from meets expectations to unacceptable
Examples of rubrics we are familiar with: letter grades, or the GED essay scoring guide
26
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Advantages of Scoring Advantages of Scoring RubricsRubrics
Clarify expectations about the characteristics of quality programs;
Set standards for program performance; Provide an objective measure for examining
and evaluating programs; Can provide formative feedback to programs; Can be used by programs for self-assessment
and improvement; Can lead to shared standards among staff
and the public about what makes a good program.
27
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Writing Descriptions of Writing Descriptions of RubricsRubrics
Keep criteria or indicators specific, objective, and value neutral;
Describe what the levels of quality look like, without using judgmental language;
Be sure indicators define progress along a continuum from lowest to highest quality.
28
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Seven Steps to Seven Steps to Developing a RubricDeveloping a Rubric
1.1. Identify Identify the measure for the rubric and a possible range of responses.
2.2. IdentifyIdentify the highest possible range of scores to define the top category.
3.3. Define Define an unacceptable level.4.4. DefineDefine the lowest level of acceptable
performance.5.5. DefineDefine an intermediate level, between
the top and the lowest.6.6. LabelLabel the categories.
7.7. Test Test and refine.
29
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
4. Design and Format4. Design and Format
Design PrinciplesDesign Principles Use of high quality graphics and photos Simple, short, succinct
Data PresentationData Presentation Avoid needless data breakdowns and
disaggregation Simplify data for your audience Be careful with percentages, base
numbers, and response rates
30
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Graphic Displays Should…Graphic Displays Should… Show the data; Induce the viewer to think about the
substance, not the methodology or graphic design;
Avoid distorting what the data have to say; Make large data sets coherent; Encourage the eye to compare different pieces
of data; Reveal data at several layers of detail, from
a broad overview to the fine structure; Be closely integrated with the statistical
and verbal descriptions of a data set.
31
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Some Some Do’sDo’s and a and a Don’tDon’t DoDo make report card text short and
easy to read. DoDo make student performance
prominent… but report more than test scores.
DoDo be cautious about assigning labels to programs.
Don’tDon’t overdo displays of demographic data.
32
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Questions to Ask When Questions to Ask When FramingFraming thethe MessageMessage
What does this audience segment need to know?
What are the few key points that best illustrate what you want this audience to know about your program?
What data best supports your message?
Remember: How you say it does matter.
33
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Don’tDon’t Use Use KillerKiller Words WordsDifficult Death Obligation Wrong
Failure Decisions
Fail Bad
Deal Cost Sell Taxes
Liability Worry Lose Hard
Contract Buy Complicated
Dangerous
34
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Disaggregated data Standards-based tests Alternative assessment Performance index Chi-square, p-value, theta,
coefficient Others?
Don’tDon’t Use Use JargonJargon
35
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
The 30-3-30 FormulaThe 30-3-30 Formula
Audience time spent perusing a product 30 seconds: most people. 3 minutes: a smaller segment that will
read headings, subheads, illustrations, opening and summary statements
30 minutes: the smallest segment that will read the whole product
Guess which of the above applies to OVAE staff??
36
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Does Your Message Pass Does Your Message Pass the the Joe Six-Pack TestJoe Six-Pack Test??
(i.e., Does your neighbor understand the message?)
If not, radically oversimplify.
If you don’t, the media will, and will inevitably get it wrong.
37
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
User’s Guide for User’s Guide for Creating Report CardsCreating Report Cards
User’s Guide for Creating and Formatting State and Local Report Cards
38
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
User’s Guide for User’s Guide for Creating Report CardsCreating Report Cards
39
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
PurposePurpose:: What is the purpose of this report card? e.g., training or monitoring?
AudienceAudience:: Who will review this report card? What message should this audience take away from the report card about program quality?
MeasuresMeasures:: What combination of measures will provide locals with a clear message, given the purpose?
Getting Started – Getting Started – Create a Local Report CardCreate a Local Report Card
40
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Getting Started – Getting Started – Create a Local Report CardCreate a Local Report Card
RubricRubric:: What criteria/rubric will indicate the quality of the outcomes or measure progress that programs are achieving?
State Standard
AExemplary
B Meets Standard
C Below Standard
FUnacceptable
Level Completions
ABE = 40% > 40% 37% - 40% 32% - 36% < 32%
ASE = 35% > 35% 32% - 35% 29% - 31% < 29%
ESL = 32% > 32% 29% - 32% 26% - 28% < 26%
41
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
DesignDesign:: Which graphs, tables, text, and pictures, will most effectively communicate the intended message?
DisseminationDissemination:: Begin to consider how the report card will
be disseminated to the audience. How will that affect the design, including how data might be interpreted by various audiences?
Getting Started – Getting Started – Create a Local Report CardCreate a Local Report Card
42
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Begin Begin by selecting a Local Report Card template
Use Use examples as guides: Report Card Templates tab contains sample
NRS report card templates for guidance; Other Report Cards tab contains current
Adult Ed and K-12 examples for ideas. Report outReport out –
Share the Local Report Card draft; Highlight purpose, audience,
measures, rubric, and design choices.
Action!Action!
43
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
5. Disseminate and 5. Disseminate and Promote Program Promote Program
ImprovementImprovement Disseminate
1.1. DevelopDevelop a distribution plan. Vehicle: meetings, mailings, Web site, etc. Audience: legislators, funding agencies, public, etc.
2.2. MakeMake the report cards accessible.3.3. ProvideProvide guidance in interpreting the data.4.4. UseUse credible messengers in the community.5.5. AvoidAvoid defensiveness.6.6. MakeMake the report card easy to find on Web sites or
easy to access by mail or phone.7.7. DistributeDistribute to legislators and funding agencies.8.8. CultivateCultivate relationships with the media, if
permitted.
44
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Questions to Answer Questions to Answer BeforeBefore Disseminating Report CardsDisseminating Report Cards
Who gets the report card (state or local)? By what means (distribution channel)? By what date? How will you evaluate the effectiveness…
Of the report card? Did it serve the purpose you planned or hoped for?
Of the distribution channel? Did the message reach the intended audience effectively and efficiently?
45
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Questions to Answer Questions to Answer BeforeBefore Disseminating Report CardsDisseminating Report Cards
How will the report cards be packaged? How will staff handle incoming
requests for the report cards? Will there be a change for postage?
Will the report cards be made available in pdf format on your Web page?
Who will maintain the Web page?
46
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Questions to Answer Questions to Answer BeforeBefore Disseminating Report CardsDisseminating Report Cards Will you arrange for media coverage
upon the report card’s release? If so, how? What media sources? Will you work through your
system’s/district’s public relations/public affairs officer?
After the report cards are “out there,” will you have a system to accept input from the audience on the quality of the reports and the dissemination system?
47
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Distribution ChannelsDistribution Channels
Direct mail Email Web Pages (Your home page) Newspapers and other print media TV/radio Board meetings and other
appropriate community meetings Credible spokespersons
(representatives of the intended audience segment)
48
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
The The Bottom LineBottom Line in in DisseminationDissemination
Do not rely on any single messenger or distribution method.
The most effective communications strategy uses multiple messengers and channels.
49
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
5. Disseminate and 5. Disseminate and Promote Program Improvement Promote Program Improvement
(Cont.)(Cont.)
Promote Program Improvement Promote use and understanding Opportunities for technical
assistance Recognition for good
performance
50
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Questions for Questions for ConsiderationConsideration
As you look at your data… What do these data seem to tell
you? What do they not tell you? What else do you need/want to
know? What good news is here for you to
celebrate? What needs for program
improvement arise from these data?
51
Developing State and Local Report Cards Developing State and Local Report Cards L. Condelli, M. Corley L. Condelli, M. Corley
Next Steps?Next Steps?
How can we help you?
www.nrsweb.org
52