Upload
teo
View
25
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Mission Possible : Assessing graduate and professional Programs. Dr. Timothy S. Brophy Director of Institutional Assessment University of Florida Gainesville, FL. Today’s Goals. - PowerPoint PPT Presentation
Citation preview
Dr. Timothy S. BrophyDirector of Institutional Assessment
University of FloridaGainesville, FL
MISSION POSSIBLE: ASSESSING
GRADUATE AND PROFESSIONAL
PROGRAMS
Part 1: To introduce, describe, and explain the basic elements of student learning outcomes and program goals, their development, and measurement
Part 2: To share a structure for assessment planning and reporting for graduate and professional programs and review an example
Part 3: Review and discuss graduate sample academic assessment data reports
TODAY’S GOALS
Size and scope• Multiple colleges/units• Undergraduate, graduate, professional, and certificate programs
Institutional consistency• Outcomes• Assessment reporting• Cycles
Institutional CultureManagement and ToolsHonoring unit autonomy, disciplinary distinctions, and institutional requirementsFaculty comportment
COMMON CHALLENGES
An educational program is a coherent set of courses leading to a credential (degree, diploma, or certificate) awarded by the institution. (SACSCOC, 2011)
HOW ACCREDITORS DEFINE EDUCATIONAL PROGRAMS
There is a clear expectation that an institution be able to demonstrate institutional effectiveness for all its diplomas, certificates, and undergraduate and graduate educational degree programs.The expectation is that the institution will engage in ongoing planning and assessment to ensure that for each academic program, the institution develops and assesses student learning outcomes.
Program and learning outcomes specify the knowledge, skills, values, and attitudes students are expected to attain in courses or in a program.
EXPECTATIONS FOR ACADEMIC ASSESSMENT FOR ACCREDITATION
Methods for assessing the extent to which students achieve these outcomes are appropriate to the nature of the discipline, and consistent over time to enable the institution to evaluate cohorts of students who complete courses or a program. Shared widely within and across programs, the results of this assessment can affirm the institution’s success at achieving its mission and can be used to inform decisions about curricular and programmatic revisions.
At appropriate intervals, program and learning outcomes and assessment methods are evaluated and revised.
EXPECTATIONS FOR ACADEMIC ASSESSMENT
PART 1: STUDENT LEARNING
OUTCOMES, PROGRAM GOALS,
AND OUTPUTS
Student Learning Outcomes (SLOs) are defined generally as “what students are expected to know and be able to do by completion of their degree program”Define this for your faculty and ensure that this definition is consistent across campus and clearly posted
DEFINE AND DISSEMINATE THE TERMS
Undergraduate
Content Knowledge
Critical thinking
Communication
Graduate
Content Knowledge
Professional Behavior Skills
CONSIDER A CATEGORICAL ORGANIZING FRAMEWORK FOR
SLOS
Student Learning Outcomes reflect the curriculum the discipline, and faculty
expectations; as these elements evolve, learning outcomes change.
Recency has to do with the degree to which the outcome reflects current
knowledge and practice in the discipline.Relevance is the degree to which the
outcome relates logically and significantly to the discipline and the degree. Rigor has to do with the degree of
academic precision and thoroughness that the outcome requires to be met
successfully.
THEE CHARACTERISTICS OF SLOS:
RECENCY, RELEVANCE, AND RIGOR
Outputs describe and count what we do and whom we reach, and represent products or services we produce. Processes deliver outputs; what is produced at the end of a process is an output.
An outcome is a level of performance or achievement. It may be associated with a process or its output. Outcomes imply measurement - quantification - of performance.
DISTINGUISH OUTPUTS FROM OUTCOMES
We seek to measure outcomes as well as their associated outputs; however, SLOs focus on outcomes.
For example, while we produce a number of new graduates (the output), it is critical that we have a measure of the quality of the graduates as defined by the college or discipline (the outcome).
Outcomes describe, in measurable terms, these quality characteristics by defining our expectations for students.
OUTCOMES AND OUTPUTS: WHAT IS THE DIFFERENCE?
EXERCISE 1: ARE THESE RESULTS
STATEMENTS OUTPUTS OR OUTCOMES?
OUR PROGRAM GRADUATED 25 STUDENTS IN SPRING 2013.
A. OutputB. Outcome
Output
Outcome
0%0%
75% OF OUR STUDENTS ACHIEVED LEVEL 4 (OUT OF 5) ON OUR PRESENTATION ASSESSMENT RUBRIC.
A. OutputB. Outcome
Output
Outcome
0%0%
WE RECRUITED 10 ADDITIONAL STUDENTS IN 2013-14.
A.OutputB.Outcome
Output
Outcome
0%0%
IN 2013, OUR DOCTORAL STUDENTS PUBLISHED 10 PAPERS IN THE INTERNATIONAL JOURNAL OF PSYCHOLOGY .
A. OutputB. Outcome
Output
Outcome
0%0%
Student Learning Outcomes (SLOs) describe what students should know and be able to do as a result of completing an academic program.
Program faculty set targets for their SLOs
Program Goals describe the unit’s expectations for programmatic elements, such as admission criteria, acceptance and graduation rates, etc.
DISTINGUISH SLOS AND PROGRAM GOALS
EXERCISE 2: STUDENT LEARNING
OUTCOMES OR PROGRAM GOALS?
WE WILL LOWER OUR ATTRITION RATE TO 10%.
A. Program GoalB. Student
Learning Outcome
Program Goal
Student Learning O
utcome
0%0%
MUSIC EDUCATION STUDENTS DISCRIMINATE MUSICAL QUALITY BASED ON SOUND MUSICAL REASONING.
A. Program GoalB. Student
Learning Outcome
Program Goal
Student Learning O
utcome
0%0%
WE WILL REDUCE THE AVERAGE TIME TO DEGREE IN OUR GRADUATE PROGRAM FROM THE 2012-13 RATE OF 6.5 YEARS TO 5 YEARS IN 2014-15.
A. Program GoalB. Student
Learning Outcome
Program Goal
Student Learning O
utcome
0%0%
STUDENTS ANALYZE EXPERIMENTAL DATA AND INTERPRET RESULTS IN THE CELLULAR AND MOLECULAR SCIENCES.
A. Program GoalB. Student
Learning Outcome
Program Goal
Student Learning O
utcome
0%0%
EFFECTIVE SLOS:Focus on what students will know and be able to do.
All disciplines have a body of core knowledge that students must learn to be successful as well as a core set of applications of that knowledge in professional settings.
Describe observable and measurable actions or behaviors. Effective SLOs present a core set of observable, measureable
behaviors. Measurement tools vary from exams to complex tasks graded by rubrics.
The key to measurability: an active verb that describes a observable behavior, process, or product
A framework for developing SLOs: Bloom’s Taxonomy
ENSURE THE OUTCOME IS MEASURABLE
• An internal process that is indicated by demonstrated behaviors – OK for learning goals but not recommended for program or course SLOs
Understand
• Internal processes that are indicated by demonstrated behaviors closely tied to personal choice or preference; OK if the appreciation or valuing is supported by discipline-specific knowledge
Appreciate; value
• Focuses assessment on “becoming familiar,” not familiarity
Become familiar with
• Not observable; demonstrable through communication or other demonstration of learning
Learn about, think about
• Focuses assessment on becoming and/or gaining – not actual awareness
Become aware of, gain an awareness of
• Focuses assessment on ability, not achievement or demonstration of a skill
Demonstrate the ability to
VERBS AND PHRASES THAT COMPLICATE MEASURABILITY
This model connects course-level and program-level SLOs directly to the program learning goals
Course-level Student Learning Outcome these are determined by the faculty and specify course-level, observable products or
demonstrations
Program-level – Student Learning Outcomethese describe what students will do to demonstrate they have met the learning
goals
Program Learning Goal Level – programs establish learning goals for the degree these goals require multiple actions over time to measure
DEVELOPING MEASURABLE SLOS: A THREE-LEVEL MODEL (CARRIVEAU,
2010)
EXERCISE 3: ARE THE FOLLOWING
OUTCOME STATEMENTS
MEASURABLE?
STUDENTS UNDERSTAND GOOD WRITING STYLE.
A. YesB. No
Yes No
0%0%
STUDENTS SIGHT-SING A 16-MEASURE MELODY WITH NO ERRORS.
A. YesB. No
Yes No
0%0%
STUDENTS EXPLORE AND LEARN ABOUT GEOLOGICAL FORMATIONS.
A. YesB. No
Yes No
0%0%
STUDENTS DEFINE THE ETHICAL RESPONSIBILITIES OF BUSINESS ORGANIZATIONS AND IDENTIFY RELEVANT ETHICAL ISSUES.
A. YesB. No
Yes No
0%0%
Direct assessments of student learning are those that provide for direct examination or observation of student knowledge or skills against measurable performance indicators.
Indirect assessments are those that ascertain the opinion or self-report of the extent or value of learning experiences (Rogers, 2011)
BALANCE DIRECT AND INDIRECT ASSESSMENTS
EXERCISE 4: DIRECT OR INDIRECT
ASSESSMENTS?
COURSE FINAL EXAM
A. DirectB. Indirect
Direct
Indirect
0%0%
SERU OR NSSE SURVEY DATA.
A. DirectB. Indirect
Direct
Indirect
0%0%
FINAL PAPER, PERFORMANCE, OR PRESENTATION GRADED BY A FACULTY DEVELOPED RUBRIC.
A. DirectB. Indirect
Direct
Indirect
0%0%
SENIOR EXIT INTERVIEW.
A. DirectB. Indirect
Direct
Indirect
0%0%
PART 2: PLANNING AND REPORTING
Academic Assessment Plans provide a common framework for units to plan how they assess and measure student achievement of the SLOs Plans also present the process for how the data from these assessments are used to enhance the quality of student learning
ACADEMIC ASSESSMENT PLANNING
Provides faculty a focal point for the discussion of the assessment of student learning in the degree programs.
Planning discussions provide an opportunity to revisit the curriculum and its relationship to the SLOs.
Provides a consistent reference resource when faculty and leadership change.
WHY PLAN?
Graduate /Professio
nal program
Assessment Plan
Mission Alignment
Student
Learning
Outcomes
Research
Assessment Timeline
Assessment Cycle
Methods and
Procedures
Assessment Oversight
GRADUATE AND PROFESSIONAL PROGRAM ASSESSMENT PLAN
Call issued by the Office of Institutional Assessment
SACS coordinators collect and
review plans
Plans are submitted to
the online approval system
Director of Institutional Assessment
reviews plans
Approved plans go to the Academic
Assessment Committee
(AAC)
Plans approved by the AAC go
to the University Curriculum Committee
(UCC)
UCC approves plans and
forwards to the Student
Academic Support System
or Graduate School
These are uploaded to the
catalog
ACADEMIC ASSESSMENT PLAN APPROVAL PROCESS
Template: Figure 4 – pp. 3-4 in your handoutRubric: Figure 1, p. 2
SECTIONS OF THE PLAN
Describe briefly the program’s mission, and how the program meets the department, college, and university missions. For example:
The mission of the<enter name> program is to <enter text>. This aligns with the department mission by <enter text.> It also supports the college mission to <enter text>. It supports the university mission by <enter text>.
MISSION
The complete file of graduate and professional program SLOs is on the Institutional Assessment website – http://assessment.aa.ufl.edu
UF’s Graduate SLO categories are Content, Skills, and Professional Behaviors
Online resources at the UF Institutional Assessment website:
“Writing Measurable Student Learning Outcomes” PowerPoint
“Guide to Writing Student Learning Outcomes”
STUDENT LEARNING OUTCOMES
What are the research expectations for students in your program?
Briefly describe the research expectations for students in the degree program.
How does your program prepare the students to become researchers in the discipline?
If the degree is NOT a research degree, briefly state this, and include a brief description of any research-related activities that students complete in the program.
RESEARCH
The Assessment Timeline is a matrix that shows when the SLOs assessed and measured in the program
It should be clear to a student when an assessment occurs, and the type of assessment that is planned (assignment, project, paper, performance, presentation, etc.)
ASSESSMENT TIMELINE
Assessment SLOs
Assessment 1
Assessment 2
Assessment 3
Enter more as needed
Content Knowledge
#1
#2
Skills
#3
#4 Professional Behavior
#5
#6
ASSESSMENT TIMELINEProgram_____________________________________ College ____________________________________
The Assessment Cycle is a matrix that graphically organizes the frequency of SLO assessment
The Assessment Cycle is a multi-year process that is completed in three years
It should be clear to a student when an assessment occurs, and the type of assessment that is planned (assignment, project, paper, performance, presentation, etc.)
ASSESSMENT CYCLE
YearSLOs
11-12 12-13 13-14 14-15 15-16
Content Knowledge #1 #2
Critical Thinking (Undergrad)Skills (Grad/Prof)
#3 #4
Communication (Undergrad)Professional Behavior (Grad/Prof)
#5 #6
ASSESSMENT CYCLEProgram College _Analysis and Interpretation: [Enter date or time frame here]Improvement Actions: Completed by [Enter date here]Dissemination: Completed by [Enter date here]
Each unit employs various methods and procedures to assess and collect data on student learning.
In this section of the plan, units provide information on their specific methods and procedures for the SLO assessments they identify in Assessment Timelines.
They must include a sample grading rubric.
It’s OK to sample the total student population – see the recommended sample size chart on the Institutional Assessment website
METHODS AND PROCEDURES
These are the
individuals who are
responsible for
managing the
assessment work in
your unit
List everyone and their contact
information
The first person on
the list should be the lead contact
Purpose: ongoing
communication
ASSESSMENT OVERSIGHT
DEVELOP A SYSTEM OR CYCLE OF PLANNING AND REPORTING
Sp
ring
A
ssessmen
t P
lans
sub
mitted
for th
e next
AY
Fall Assessment Data, results,
and use of results for
previous AY reported
Planning Reporting
Establish Mission,
Goals, and Outcomes
Assessment Planning
Implement the Plan
and Gather Data
Interpret and
Evaluate the Data
Modify and Improve
The review of Academic Assessment Plans is a constructive exercise designed to guide faculty to consider their assessment work thoughtfully.The 2012-13 PhD in English Academic Assessment Plan is on pp. 8-11 of your handout. Using the rubric on p. 2, discuss in your group the degree to which this plan meets the guidelines.
Does the plan meet or not meet the criteria?
What comments would you provide?
ASSESSMENT PLANNING EXERCISE: PHD IN ENGLISH
REPORTING ASSESSMENT RESULTS – STUDENT LEARNING OUTCOMES
(P. 5)Assessment Method:• List the assignment, exam, project, etc.• If this is a sample, describe the sampling procedure used
Results:• Enter the criterion for success. The “criterion for success” is the minimum
percentage of students who pass the assessment that you consider to be acceptable for your program. If the criterion is less than 70%, provide a rationale.
• State: “X number of students passed the assessment out of a total of Y students, for a percentage of Z%”.
• State: This meets/does not meet the criterion for success.• Attach the data you shared with your faculty (student names redacted). Use of Results:• State who reviewed the results.• Refer to the results that were reviewed.• State actions taken in past tense. For example:
• “Based on our review, we decided to…”• “We modified our SLO #1 because the data indicated that…”• “We changed the course content for ABCXXXX effective fall 20XX because
the data revealed that…”
REPORTING ASSESSMENT RESULTS – ACADEMIC PROGRAM GOALS
Assessment Method:• State the measurement method
Results:• Briefly state your results• Include or attach the data you collected in summary formUse of Results:• State who reviewed the results.• Refer to the results that were reviewed.• State actions taken in past tense. For example:
• “Based on our review, we decided to…”• “Our results led us to modify our goal to…”• “We developed a new measure for this long term goal
based on our review…”
DEVELOP A QUALITY ASSURANCE
PROCESS
Multi-step, institutional review and
approval process
Templates and rubrics for guiding
faculty through the process
Review and evaluate faculty
submissions
Cross-reference plans with data
reported annually
Develop and provide
professional development
Model the process: Modify
and improve quality assurance processes based on the data you
collect
ELEMENTS OF QUALITY ASSURANCE
2012-13 ACADEMIC ASSESSMENT DATA REPORTING REVIEW RESULTSIn early 2014 all of the 2012-13 graduate and professional academic assessment data reports were reviewed.76% were returned for revisions due to failure to follow our guidelinesThe most requested revision was in the Use of Results field. Specifically, programs did not state who reviewed the results. Programs were most likely cited for Program Goal Use of Results statements.Many programs were cited for not stating results and use of results in the past tense
PART 3: EXAMPLES
The 2011-12 Data Analysis Report for the PhD in English is on pp. 6-7 of your handout (Figures 7 and 8).
The 2012-13 Data Analysis Report for this program is on pp. 12-13 of your handout (Figure 10). This report is based on the 2012-13 AAP you have just reviewed. Compare the two data reports. What differences and similarities do you find? Using the Data Entry Guides for Faculty in Figure 5 (p. 5 in your handout) , discuss the degree to which the 2012-13 data report complies with these guidelines. Note: the guidelines were provided beginning with the 2012-13 report; the 2011-12 report was completed without these guidelines.
DATA REPORTING REVIEW EXERCISE – PHD IN ENGLISH
We will review Figure 11 together, on pp. 14-16 of your handout
OTHER EXAMPLES OF DATA REPORTS
1. Define the terms and disseminate them2. Consider an institutional categorical
organizing framework for SLOs3. Recency, Relevance, and Rigor4. Distinguish Outputs from Outcomes 5. Distinguish SLOs from Program Goals6. Ensure the outcome is measurable7. Balance direct and indirect assessments8. Planning Timeline/Cycle9. Templates and Rubrics10.Approval and Management Process11.A system or cycle of planning and
reporting12.Quality Assurance Process
A SUMMARY
Timothy S. Brophy, Ph.D.Director, Institutional Assessment
235 Tigert HallOffice of the Provost
Email: [email protected]: 352-273-4476
QUESTIONS