AlabamaProfessionalEducationPersonnelEvaluationProgram

Field Test of Central Office

System

July 2000

Central Office Evaluation System

Alabama Department of Education Central Office Field Test Evaluation

• Currently being field tested

• Purpose of Evaluation is for Improvement

• Criteria Central Office system was developed from a Framework Framework contains Competencies, Tasks, and Knowledge/Skills

• Statewide Performance Standards

• Summative decisions (optional)

Central Office Evaluation System

• 13 Knowledge/Skill Areas

1. Communication 2. Collaboration 3. Assessment/Evaluation 4. Organization

6. Laws/Policies 7. Problem Solving 8. Innovation

10. Program Operations/Management11. Fiscal Management/Leadership

9. Technology Management

13. Leadership of Human Resources12. Professional Responsibilities

5. Planning

Alabama Department of Education Central Office Field Test Evaluation

Central Office Evaluation System

• Scoring 4-point rating scale with rationale statements (rubrics) Holistic scoring cross-cutting 13 knowledge/skill areas

• Evaluators Trained in conducting evaluations Must meet reliability standards

• Central Office Administrators Evaluation content varies with years of experience in position Evaluation content addresses only areas of responsibility Evaluation process is same for all

Alabama Department of Education Central Office Field Test Evaluation

Central Office Evaluation System

1. Evaluation Summary Report (shows scores and score rationales by knowledge/skill area) 2. Professional Development Plan (becomes a data source in next cycle)

• Instrumentation/Data Sources 1. Portfolio developed by central office administrator

2. Surveys of various system administrators, principals, and school staff, when applicable

3. Interviews (written or oral) 4. Supervisor’s Review Form

• Summary Report Sources

• Optional Instrumentation 1. Self-Assessment Form (completed by central office administrator, but not considered in evaluation scoring)

Alabama Department of Education Central Office Field Test Evaluation

Central Office Evaluation System

Central Office Data: evidence of competence produced, assembled and submitted by the central office administrator

Portfolio

Written interview Interpretation of survey responses

Line of Evidence: all data/information supporting the scoreawarded in a knowledge/skill area, whether produced by the central office administrator or the evaluator

Evaluator Data: evidence of central office administrator competence developed by the evaluator

Oral Interview Supervisor's Review Form Clarifying interview

Alabama Department of Education Central Office Field Test Evaluation

• Steps in Evaluation Process

1. Orientation (LEA responsibility)

2. Self-assessment (central office administrator)

3. Prepare portfolio (central office administrator)

4. Distribute and analyze surveys (central office administrator)

5. Supervisor Review Form (evaluator/supervisor)

Central Office Evaluation System

Alabama Department of Education Central Office Field Test Evaluation

• Steps in Evaluation Process (cont.)

7. Conduct interviews (evaluator/central office administrator)

8. Prepare Evaluation Summary Report (evaluator)

9. Summary conference and PDP (evaluator/ central office administrator)

Central Office Evaluation System

6. Review portfolio (evaluator)

Alabama Department of Education Central Office Field Test Evaluation

Self-Assessment

Generates data for all areas of knowledge and skills

Optionally used by the central office administrator and shared with evaluator only if central office administrator agrees

Used by central office administrator to:

1) Identify areas that need improvement

2) Compare the central office administrator’s perceptions ofperformance with the results of theevaluation

3) Determine areas for professional growthactivities to be included in aprofessional development plan.

Central Office Evaluation System

Alabama Department of Education Central Office Field Test Evaluation

Two types of interviews:

A Structured Interview with 6 questions commonly asked of central office administrators (oral or written response format)

A Clarifying Interview with probing questions focused onclarifying or bringing understanding to the portfolio and/orwritten interview (questions differ from central office administrator to central office administrator)

Conducted after the evaluator analyzes central office administrator’s portfolio

Modified scripting is used to record responses

InterviewsCentral Office Evaluation System

Responses can be banked for one year

Alabama Department of Education Central Office Field Test Evaluation

Oral Option: A discussion between the central office administrator and the evaluator (or his/her designee) about particular areas of the central office administrator’s knowledge, skills, and performance.

Structured InterviewsCentral Office Evaluation System

Generates data for seven areas of knowledge/skills: Communication (oral) Collaboration processes and skills Assessment/measurement/evaluation Organizing for results Planning Problem solving Technology management

Alabama Department of Education Central Office Field Test Evaluation

Each of the structured interview question sets is intended to cover one knowledge and skill area (although responses may cut across more than one area). Communication skills are consideredas part of the oral presentation.

The structured interview itself is not scored, but responses are added to other evidence provided through other sources.

Structured Interviews (cont.)Central Office Evaluation System

Written Option: Central office administrator’s written responses to interview questions (will have clarifying discussion).

Alabama Department of Education Central Office Field Test Evaluation

Conducted after the structured interview is completed with a focus on the written interview and, optionally, the central office administrator’s portfolio

No set number of questions, but key questions specific to the written interview and portfolio (optional) are developed by the evaluator to bring clarity and understanding of their contents

Responses are added to other evidence provided through other sources

The Clarifying InterviewCentral Office Evaluation System

Alabama Department of Education Central Office Field Test Evaluation

Provides data where a supervisor should have objective, experiential information

The Supervisor’s Review FormCentral Office Evaluation System

Generates data for six areas of knowledge and skills:

Professional Responsibilities Fiscal Leadership/Management Laws/Policies Innovation Program Operations/Management

Leadership of Human Resources

Alabama Department of Education Central Office Field Test Evaluation

Central office administrator is responsible for providing some evidence of his/her knowledge and skills

The totality of the evidential material (lines of evidence for all

knowledge/skill areas) will constitute central office administrator’s evaluation portfolio

The Central Office PortfolioCentral Office Evaluation System

Generates data for 12 areas of knowledge and skills:

Communication Collaboration Assessment/Evaluation Organizing for Results Planning Laws and Policies Problem Solving Innovation Technology Management Program Operations/Management

Fiscal Leadership Human Resources Leadership

Alabama Department of Education Central Office Field Test Evaluation

There is no one specific form of packaging required for the portfolio, but there are organizational requirements

Each entry (piece of evidence) is accompanied by a Portfolio Entry Form

Central Office Evaluation System

Total portfolio requires approximately 27 pieces of evidence

One piece of evidence might serve 2-3 purposes (e.g., ProgramImprovement Plan)

With multiple purposes, a portfolio may have 10-12 entries (pieces of evidence)

Alabama Department of Education

The Central Office Portfolio

Central Office Field Test Evaluation

Explicit guidelines for developing a line of evidence in each area are provided in the evaluation manual

"Extra" entries will not be considered by the evaluator

Evaluated on the quality of materials rather than the quantity of materials in the portfolio

Central Office Evaluation System

Alabama Department of Education

The Central Office Portfolio

Central Office Field Test Evaluation

Respondents are to remain anonymous and represent a cross-section within the group

Generates data for 12 areas of knowledge and skills: Communication Collaboration Assessment/Evaluation Organizing for Results Laws and Policies Problem Solving Innovation Program Operations/Mgt. Fiscal Leadership Technology Mgt. Professional Responsibilities Human Resources Leadership

SurveysCentral Office Evaluation System

Three groups of respondents:

Other Central Office Administrators Principals School staff, when applicable

Alabama Department of Education Central Office Field Test Evaluation

Analyses and reporting are responsibilities of central office administrator: Information/report will be used with other evidence from other sources

Procedures and methodologies used by the central office administrator will in itself be used as part of evidence for assessing the knowledge/skill area of assessment/evaluation

Analyses and reporting:

Number of respondents by subgroup surveyed

Any problems with data Conclusions by knowledge/skill area

Patterns and interpretation

Surveys (cont.)Central Office Evaluation System

Alabama Department of Education

Numbers for administration:

Total for groups: 20 or all, if fewer in school system

Both system administrators and principals must be in respondent group

Central Office Field Test Evaluation

Scores are not generated by instrument

There will be a total of 13 scores, one for each knowledge/skill area

At end of process, evaluator(s) will: 1. Organize evidence by knowledge/skill area

2. Read through all evidence for a knowledge/skill area

3. Award a score (1, 2, 3, 4) based on the patterns in the evidence in relation to the score level rationales for that particular knowledge/skill area

4. Place score with comments/justification on the Evaluation Summary Report

5. Use the scores and evidence for determining what should be addressed in the Professional Development Plan

ScoringCentral Office Evaluation System

Alabama Department of Education Central Office Field Test Evaluation

Used to present the scores developed from the lines of evidence for each of the 13 knowledge and skill areas

To complete the Evaluation Summary Report, the evaluator:

1. Completes demographic information requested

2. Records the scores developed from the lines of evidence for each knowledge and skill area

3. Writes any comments, explanations, or justifications in the space provided that support the score for the specific area

4. Identifies three areas for focus in planning professional development for the coming year(s)

Evaluation Summary ReportCentral Office Evaluation System

Alabama Department of Education Central Office Field Test Evaluation

Activities focus on areas of improvement as identified in the Evaluation Summary Report

Activities focus on the improvement of student achievementand school improvement

Used to generate portfolio entry for Human Resources Leadership after first year of evaluation

Professional

Development

Plan

Professional Development PlanCentral Office Evaluation System

Alabama Department of Education Central Office Field Test Evaluation

QUESTIONS ?

Administrator Evaluation Systems

Alabama Department of Education Administrator Evaluation

Instrument Superintendent Central Office Principal Asst. Principal

Self Assessment

Portfolio

Surveys

Interviews

Supr. Review

Board Quest

Observation

Eval. Summary

Prof. Devel. X X X X

X X X X

X X X X

X X X X

X X X X

X

X X X

X X X X

X

Administrator Evaluation Systems

Alabama Department of Education Administrator Evaluation

Instrument Superintendent Central Office Principal Asst. Principal

Self Assessment

Portfolio

Surveys

Interviews

Supr. Review

Board Quest

Observation

Eval. Summary

Prof. Devel.

12 know/skills 12 know/skills 11 know/skills 12 know/skills

All know/skills All know/skills All know/skills All know/skills

Cntrl Off/PrinCntrl Off/Prin/

Faculty

7 know/skills 7 know/skills 8 know/skills 7 know/skills

Faculty/ParentsStudents/Comm

Faculty/ParentsStudents

All know/skills

6 know/skills 4 know/skills 6 know/skills

5 know/skills

All know/skills All know/skills All know/skills All know/skills

2-3 know/skills 2-3 know/skills 2-3 know/skills 2-3 know/skills

Central Office Portfolio

Knowledge Skill Area Evidence

Communication 3 pieces of written communication

Collaboration 3 artifacts (documents, tapes, etc.)

Assessment/Evaluation 1 personnel evaluation file (if applicable) 1 document/disk synthesizing and

presenting student achievementresults

1 document related to programevaluation

Organizing for Results

1

2 documents/artifacts representingmarketing knowledge/skills, customer focus and/or change efforts

Planning School improvement planningefforts

1 other document/artifact

Laws/Policies 2 artifacts/documents

Problem Solving 2 artifacts/documents

Innovation 2 artifacts/documents

2

Knowledge Skill Area Evidence

Central Office Portfolio

Technology Management 2 artifacts/documents

Leadership of Human Resources 2 artifacts/documents

3

Knowledge Skill Area Evidence

Fiscal Management/Leadership 2 artifacts/documents

Central Office Portfolio

Program Operations/Management 2 artifacts/documents