Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Common Standard 2: Unit and Program

Assessment and Evaluation

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 2

Topics to be Addressed

● Common Standard 2● Understanding the differences

between the Unit and the Program

● Designing a Unit Evaluation and Program Assessment System

#1 Agenda, #2 PowerPoint

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 3

Common Standards 2008

1: Educational Leadership2: Unit and Program Evaluation System3: Resources4: Faculty5: Admissions6: Advice and Assistance7: Field Experiences and Clinical Practice8: District Employed Supervisors9: Assessment of Candidate Competence

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 4

Standard 2: Unit and Program Assessment and Evaluation

The education unit implements an assessment and evaluation system for ongoing program and unit evaluation and improvement. The system collects, analyzes and utilizes data on candidate and program completer performance and unit operations. Assessment in all programs includes ongoing and comprehensive data collection related to candidate qualifications, proficiencies, and competence, as well as program effectiveness, and are used for improvement purposes.

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 5

Standard 2: Unit and Program Assessment and Evaluation

The education unit implements an assessment and evaluation system for ongoing program and unit evaluation and improvement. The system collects, analyzes and utilizes data on candidate and program completer performance and unit operations. Assessment in all programs includes ongoing and comprehensive data collection related to candidate qualifications, proficiencies, and competence, as well as program effectiveness, and are used for improvement purposes.

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 6

Designing the Unit and Program Evaluation System

●Collect Data●Analyze the Data●Utilize the Analysis for

Unit/Program Improvements

Common Standard 2

Medium Institution A

School of Health & Human

Services Dean

School of Hard KnocksDean

ProgramsMS/SS Ed Spec

No credential programs

No credential programs

UNITSchool of Education

ProgramsMS/SS

Rdg Spec

Admin

Ed Spec

HR, Credential Analyst, etc.

Dean

UNITLarge Institution B

School of Health & Human Services Dean

School of Education Dean

School of Hard

KnocksDean

SLP PPSPrograms

ProgramsEd Spec

AdminCTE

MS/SSRdg SpecLibrary

No credential programs

HR, Credential Analyst, etc.

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 10

Small Institution C

Program

Induction

UNIT

District/COE

Induction

HR, Credential Analyst, etc.

Institutional Administration

Small Institution D

Program

Induction

UNITSmall IHE—

1-2 Programs

HR, Credential Analyst, etc.

Education Dept Chair

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 12

Reflection● Did the explanation and graphics

clear up any of your misconceptions about the relationship between the unit and programs?

● Do you have any examples to share that illustrate this relationship—positive or negative?

● Do you have any questions about this relationship?

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 13

Unit and Program Evaluation System

Admin Services Program

CTEL Program

Multiple Subject Program

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 14

LET’S START WITH THE PROGRAM

EVALUATION SYSTEM

Handout #3

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 15

Common Standard 9:Assessment of Candidate

Competence Candidates preparing to serve as teachers

and other professional school personnel know and demonstrate the professional knowledge and skills necessary to educate and support effectively all students in meeting the state-adopted academic standards. Assessments indicate that candidates meet the Commission-adopted competency requirements, as specified in the program standards.

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 16

Designing the Program Evaluation System

● What do we want to know about the candidates and the program(s)?

● Who will be assessed?● What will be evaluated?● Who will provide data on the

program?● When will the data be collected?● How will data be collected and stored?● How will data be analyzed and

reported?

Handout #4

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 17

What do we want to know about candidates and

programs?● About candidates? Common

Standard 9 and the Candidate Competence Standards in the Program Standards

● About programs? Common Standard 1 (leadership), 3 (resources), 4 (faculty), etc.

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 18

Who/what will be assessed? (CS 9)

● Candidates’ competence in areas appropriate for the credential– Performance on the TPA – Performance on class projects– Performance in student teaching/clinical

experiences● Completer’s competence

─ Assessment by supervisor─ Evaluation of a portfolio of work

● Program effectiveness– Evaluation of Instructor/mentor– Satisfaction with program offered

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 19

Who has the information we want?

● Candidates● Program Completers● Faculty/Instructors● Field Supervisors

●program and district ● Employers● Advisory Boards

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 20

Who has the information we want?

● Multiple pathways●Traditional vs. intern●Traditional vs. Early Completion Option● Induction—main program vs. privates

● Multiple locations●Main campus vs. satellite locations●Face to face meetings vs. online

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 21

When to Collect Data?Institution and/or Program must decide:● Annually● Each Term● With each cohort● Beginning, middle and/or end of the

program

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 22

How to collect data?Overall question – What kind of data do we want? What data collection instrument would be most effective in collecting the data? • Surveys• Assignments• Evaluations• Focus groups

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 23

Data CollectionWhat kind of data will each instrument provide?• Open-ended (qualitative)• Multiple choice, scaled scores (quantitative)• A combination of both?

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 24

Data Storage● Collection procedures will impact

storage and analytical procedures● If you use electronic data collection

instruments, the results can be easily stored.

● Other collection procedures might require entry or data manipulation.

● Who will enter the data?● Who will have access to the data?

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 25

Analyze the Data● Who has the skills to analyze the

data, consider the questions?● What tools will be used?● How often will data be analyzed?● How do you want the data

presented? By cohort? By program? By geographic location? All of the above? Other?

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 26

Presentation of Data Section A is the work of each

program and can be individualized or uniform

Benefits of a common template for the program reports– Comparing of apples to apples– Ease of examination by unit personnel

Example: CSULB

Handout #5

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 27

Triangulating the Data Data from

multiple sources helps locate program strengths and areas for growth: candidate competence data, completer and employer surveys, and faculty surveys

Course Evaluations

Employer surveys

Faculty feedback

TPAs

Focus Groups Key

Assessments

Program self-study

MSProgram

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 28

Utilize the Analysis for Improvement

●How will results from each of the analyses be shared?

●With whom will the results be shared?

●When?●Who will ensure that changes

identified by the analyses are undertaken?

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 29

Utilize the Analysis for Improvement (cont’d)

● How will decisions be made if the data indicates that changes should be made?—Will each professor decide? Does the faculty as a whole? Is it an administrative decision?

● How will changes be noted for institutional history and accreditation activities?

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 30

Reflection on Your Program

●What part(s) of the suggested system do you have in place?

●What part(s) are you currently working on?

●What part(s) have you not taken into consideration at all?

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 31

LET’S MOVE TO UNIT

EVALUATION

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 32

Unit Evaluation System

●The unit evaluation system is composed of each program’s evaluation process

●Evaluation of Common Standards

●When, what, who, how…

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 33

Designing the Unit Evaluation System

● What do we want to know about the candidates, program(s), and the unit as a whole?

● Who will be assessed?● What will be evaluated?● Who will provide data on the

program?● When will the data be collected?● How will data be collected and stored?● How will data be analyzed and

reported?

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 34

What will be evaluated?● Program Design (CS 1)● Assessments (CS 2)● Faculty (CS 4)● Admission (CS 5)● Advising (CS 6)● Field Work (CS 7, 8)● Recommendation for credentials

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 35

“The education unit implements an assessment and evaluation system…”Evidence from Multiple Sources● Common Standards document/attachments: - clear graphic of system showing flow of assessment data for decision-making● Meeting minutes: - records of stakeholder involvement in system design/review● Interviews with leadership: - clear descriptions of how data moves through the system and who is responsible for gathering, analyzing and organizing data for decision making

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 36

“The system collects, analyzes and utilizes data on candidate and program completer

performance…”Evidence from Multiple Sources● Interviews with candidates, completers: - clear descriptions of how their performance is/was measured throughout the program - descriptions of how feedback is collected from completers● Interviews with program faculty: - examples of formative and summative data collected from candidates throughout program

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 37

Evidence from Multiple Sources● Interviews with unit leadership: - clear descriptions of how assessment system data informs unit operation decisions

“…and unit operations.”

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 38

“Assessments in all programs includes ongoing and comprehensive data

collection….”Evidence from Multiple Sources● Biennial Reports: - examples of candidate, completer competency data; analysis of data; plans for program improvements based on data analysis● Supporting documentation: - multi-year data summaries; trend analyses; copies of annual data summaries submitted to institution leadership

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 39

“…[data] is used for improvement purposes.”

Evidence from Multiple Sources● Interviews with program faculty: - multiple examples of program improvements made on the basis of data analysis● Interviews with unit leadership: - examples of unit operation improvements made on the basis of data analysis● Interviews with stakeholders, advisory groups: - descriptions of stakeholder/advisory group involvement in program/unit improvement discussions, decisions

40

THE UNIT

Collect‘Gather data’

Analyze‘Organize data’

Utilize‘Drive decision-

making’-Gather data across all of an institution’s approved programs related to the Common Standards: 1) Leader-ship, 3) Resources, 4) Faculty, 5) Admission and 6) Advice & Assistance, 7) Field Experience, 8)District-Employed Supervisors, and 9) Candidate Competence.

-Collect data in ongoing and comprehensive manner.

-Organize the data within the unit and across all of the approved program(s).

-Discuss the data with faculty and others within the unit and all of the approved program(s).

-Draw conclusions from the data to inform decision-making across the unit and all of the approved program(s).

-Use the analysis of the data for unit and program(s) improvement purposes. -Document the cycle of improvement decision-making for the unit and its programs.

-Document actions taken, the basis of those actions and how/when the results will be reviewed next at the unit level.

Handout #6

ALL PROGRAMS

Collect‘Gather data’

Analyze‘Organize data’

Utilize‘Drive decision-

making’-Gather data related to the candidate competencies identified in the Program Standards as well as program effectiveness.

-Collect from candidates, completers, employers, field supervisors and faculty in ongoing and comprehensive manner.

-Organize the data within the program.

-Discuss the data with faculty and others working with the program.

-Draw conclusions from the data to inform decision-making within the program.

-Use the analysis of the data for program improvement purposes.

-Document actions taken, the basis of those actions andhow/when the results will be reviewed next at the program level.

Example of Unit Assessment System

StandardAssess-ment Focus

Evaluation Focus

Evaluation Instrument Interval

Data Analyzed

by1:

Leader-ship

Vision Programs using vision

Action plan Every 5 years

SOE Faculty/ Advisory Groups

2:Assess-ment & Evalua-

tion

Unit & Program

Policies and

practices

Qual/Quant analysis, results,

action plan

Annually

SOE Faculty

3:Resources

Inclusive of all

programs

Adequacy of budget: qualified

personnel, facilities,

etc.

Division chair,

roundtablesurvey

Annually

Dean,Chairs,

Directors

Example of Unit Assessment System

StandardAssess-ment Focus

Evaluation Focus

Evaluation Instrument Interval

Data Analyzed

by4.

Faculty/ Instr. Pers.

Evalua-tion/

Retention

Perform-ance of all personnel

Survey by candidates, completers

End of course, program

Program Directors,

Dean

5.Admission

Well-defined well-run

Diversity/ Qualifica-tions of Cand.

Profile—admitted/ dropped

candidates

Annually

Program Directors,

Dean

6.Advice and Assistance

Quality of

advice, availabil

-ity

Candidateperception

of process, personnel

Survey of candidates,

advisors

Annually, end

of program

Program Directors,

Dean

Example of Unit Assessment System

StandardAssess-ment Focus

Evalua-tion

FocusEvaluation Instrument Interval

Data Analyzed

by7.

Field Exper-iences

Field-based/ clinical

Exp.

Quality of experi-ence

Focus groups, advisory

board

Annually Program Directors,

Dean

8.District-

EmployedSupervisors

Training,evaluation, recognition

Perform-ance of

personnel

Survey by candidates, completers

End of course, program

Program Directors,

Dean

9.Candidate

Assess-ment

Programs Candidate compe-tency,

program effective-

ness

Varies by program

Annually, end of course, end of

program

Program Directors,

Dean, Cred.

Analyst

Guidance for Coming to a Standard Finding on Common Standard 2

Unit Assessment and Evaluation

(unit operations—Common Standards)

Program Assessment and Evaluation

(candidates, completers, and

program effectiveness)Common Standard 2 Finding

Collect Analyze Utilize Collect Analyze Utilize

Yes Yes Yes Yes Yes Yes Met1

Yes Yes No Yes Yes Yes

Met with Concerns2Yes No No Yes Yes Yes

Yes Yes Yes Yes Yes No

Yes Yes No Yes Yes No

Not Met3

No No No Yes Yes Yes

Yes No No Yes Yes No

Yes No No Yes No No

No No No No No No

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 46

Reflection●What part(s) of the suggested

unit system do you have in place?

●What part(s) are you currently working on?

●What part(s) have you not taken into consideration at all?

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 47

ResourcesPSD News To subscribe, send an email to psd-news-subscribe@lists.ctc.ca.gov

Commission Website www.ctc.ca.gov select Program

Sponsors then Accreditation

Com

mis

sion

on

Teac

her C

rede

ntia

ling

Inspire, Educate, and Protect the Students of California 48

ResourcesBiennial Reportshttp://www.ctc.ca.gov/educator-prep/program-accred-biennial-reports.html

Program Assessmenthttp://www.ctc.ca.gov/educator-prep/program-accred-assessment.html

Site Visitshttp://www.ctc.ca.gov/educator-prep/program-accred-site-visits.html