Self-Assessment Rubric for the Institutionalization of ... · Self-Assessment Rubric for the Institutionalization of Service-Learning in University STEM Departments An Assessment

Self-Assessment Rubric for the Institutionalization of Service-Learning in University STEM Departments An Assessment Tool of the (STEM)2 Initiative

Pretest Results – April 11, 2011

Submitted by :

Rebecca M. Eddy, Ph.D. Todd Ruitman, M.A. & Nancy Hankel

COBBLESTONE APPLIED RESEARCH & EVALUATION, INC.

2120 Foothill Blvd., Ste. 202 | La Verne, CA 91750 | (800) 971-3891

Cobblestone Applied Research & Evaluation, Inc. Page 2

The California State University (CSU) was awarded a grant from Learn and Serve America (LSA) in

July 2010. The goal of the grant is to increase student success and graduation rates in Science,

Technology, Engineering and Mathematics (STEM) through service-learning initiatives. In an effort to

show progress toward increasing the institutionalization of service learning as a result of grant activities,

each campus completed a “Self-Assessment Rubric for the Institutionalization of Service-Learning in

University STEM Departments” to act as a pretest measurement. Each of the 23 CSU campuses will

complete the rubric again at the end of the three-year grant to determine growth as a result of grant

activities.

Description of the Self-Assessment Rubric

The rubric was adapted from the work of Andrew Furco, University of California, Berkeley, 1999

(revised in 2002, 2003, 2006) and based on the Kecskes/Muyllaert Continuums of Service Benchmark

Worksheet. Each campus self-rated twenty-two aspects divided into five dimensions of their service-

learning program specifically related to STEM departments (see Table 1). Dimension 1 measures the

development of a definition, philosophy, and mission of service learning on the campus. Dimension 2

measures STEM faculty support of, and involvement in, service learning. Dimension 3 measures STEM

student support for, and involvement in, service learning. Dimension 4 measures community

participation and partnerships, and finally, Dimension 5 measures departmental support for service

learning.

Table 1. Components of the Five Dimensions

Dimensions Dimension Components

Dimension 1: Philosophy and Mission of Service-Learning

1.1 Definition of Service-Learning in STEM Departments

1.2 Strategic Planning

1.3 Alignment with Departmental Mission

1.4 Alignment with Educational Reform Efforts

Dimension 2: STEM Faculty Support for and Involvement in Service-Learning

2.1 STEM Faculty Knowledge and Awareness

2.2 STEM Faculty Involvement and Support

2.3 STEM Faculty Leadership

2.4 STEM Faculty Incentives and Rewards

Dimension 3: STEM Student Support for and Involvement in Service-Learning

3.1 STEM Student Awareness

3.2 STEM Student Opportunities

3.3 STEM Student Leadership

3.4 STEM Student Incentives and Rewards

Dimension 4: Community Participation and Partnerships

4.1 Community Partner Awareness

4.2 Mutual Understanding

4.3 Community Partner Voice and Leadership

Dimension 5: Departmental Support for Service-Learning

5.1 Coordinating Entity

5.2 Policy-Making Entity

5.3 Staffing

5.4 Funding

5.5 Administrative Support

5.6 Engaged STEM Department Status

5.7 Evaluation and Assessment


For each dimension component, respondents rated their current status on a scale of 1 to 9, with

1 representing the lowest possible rating and 9 representing the highest possible rating. The numerical

ratings were accompanied by a statement to be used by the respondent as a guide for their rating. For

example, one of the components contained the following statement: “Few service-learning

opportunities exist for STEM students; only a handful of service-learning courses are available.” If the

respondents felt that this statement best reflected their STEM departments then they would select one

of three numerical responses that corresponded to this statement. A rating of 1 to 3 represented that a

campus was in the stage of critical mass building meaning that the STEM departments were in the

beginning stages of service-learning institutionalization; a rating of 4 to 6 represented the area of quality

building which suggested that the STEM departments were aware of their service-learning needs and

were working toward service-learning institutionalization; and a rating of 7 to 9 represented the area of

sustained institutionalization meaning that service-learning was fully integrated into the STEM

departments. Following each of the 22 ratings, a space was provided for explanation should the

respondent feel an explanation of the rating was necessary. Figure 1 displays the three stages of the

rubric.

Figure 1. Three Stages of the Self-Assessment Rubric

Method

As a pretest measurement, each of the 23 CSU campuses was provided with a self-assessment

rubric to measure their current institutionalization of STEM service learning on their campus. The rubric

was emailed during the last quarter of 2010 and was to be completed by January 31. Most campuses

were able to return their completed rubrics by this date; however, not all rubrics were completed until

mid-March. Each rubric contained detailed instructions on how to complete the assessment in addition

to a brief introduction. The rubric was primarily completed by each campus Director of Service Learning.

0

1

2

3

4

5

6

7

8

9

Rat

ing

Critical Mass Quality Sustained

Building Building Institutionalization


Many directors included other qualified individuals to assist in the completion of the rubric, such as

STEM faculty or service-learning staff.

Results

Overall Results

Figure 2 below shows the average scores among all campuses for each dimension. The highest-

rated dimension, Dimension 1 (Philosophy and Mission of Service-Learning), had an average score of

4.04 out of 9 while the lowest-rated dimension, Dimension 2 (STEM Faculty Support for and Involvement

in Service-Learning), had an average score of 3.07 out of 9. Overall the scores show that most campuses

are, on average, between the critical mass building and quality building stages in each of the five

dimensions. Specifically, Dimension 1 is the only dimension where the average rating was in the quality

building stage. The other four dimensions all scored above a three but below a four which suggests that

the individual components of each dimension were generally rated in the critical mass building stage

with the exception of some individual components rated in the quality building stage. With such

variance in rating components within dimensions, it was important to analyze the average results of the

individual components.

Figure 2. Average Campus Ratings across Dimensions

While most components within each dimension were fairly consistent, Dimension 3 showed a

surprising result whereas it contained both the highest rated component in addition to one of the

lowest rated components (see Figure 3). This finding illustrates the need to pay particular attention to

individual components in addition to the overall dimensions. The results showed that the highest rated

components, irrespective of dimension, were: 3.4 STEM Student Incentives and Rewards (m = 5.0), 1.2

Strategic Planning (m = 4.3), 1.4 Alignment with Educational Reform Efforts (m = 4.3), 1.3 Alignment with

Departmental Mission (m = 4.2), and 3.1 STEM Student Awareness (m = 4.0). These five components all

were rated on average in the quality building stage of the rubric. All other components had an average

4.043.07

3.72 3.62 3.32

0

1

2

3

4

5

6

7

8

9

1 2 3 4 5

Ave

rage

rat

ing

ou

t o

f 9

Dimension


rating below a four on the rubric with the lowest components being 5.3 Staffing (m = 2.7), 3.3 STEM

Student Leadership (m = 2.8), 2.1 STEM Faculty Knowledge and Awareness (m = 2.9), 2.4 STEM Faculty

Incentives and Rewards (m = 2.9), and 5.4 Funding (m = 2.9).

Figure 3. Average Score by Component

Individual Campus Results

Each campus also received a total score for their self-assessment. The average for each dimension was

added together to obtain the total score. We used the average of each dimension thereby making each

dimension of equal weight in the total score. Out of a total 45 points possible, campuses averaged a

total score of 17.6, with a range of 6.3 to 37.2 points. Figure 4 shows the distribution of scores across all

campuses.

Figure 4. Total Score of Self-Assessment Rubric for all Campuses

0

1

2

3

4

5

6

7

8

9

1.1 1.2 1.3 1.4 2.1 2.2 2.3 2.4 3.1 3.2 3.3 3.4 4.1 4.2 4.3 5.1 5.2 5.3 5.4 5.5 5.6 5.7

Ave

rage

rat

ing

ou

t o

f 9

0

5

10

15

20

25

30

35

40

Cam

pu

s 1

Cam

pu

s 2

Cam

pu

s 3

Cam

pu

s 4

Cam

pu

s 5

Cam

pu

s 6

Cam

pu

s 7

Cam

pu

s 8

Cam

pu

s 9

Cam

pu

s 1

0

Cam

pu

s 1

1

Cam

pu

s 1

2

Cam

pu

s 1

3

Cam

pu

s 1

4

Cam

pu

s 1

5

Cam

pu

s 1

6

Cam

pu

s 1

7

Cam

pu

s 1

8

Cam

pu

s 1

9

Cam

pu

s 2

0

Cam

pu

s 2

1

Cam

pu

s 2

2

Cam

pu

s 2

3

Campus 1

Campus 19

Campus 23

Critical Mass

Building

Quality Building

Sustained

Institutionalization


Figure 4 above also shows the stage (i.e., critical mass building, quality building, sustained

institutionalization) of each campus after averaging out all ratings on the self-assessment rubric. The

total score for the self-assessment rubric shows that eleven campuses are in the critical mass building

stage, eight are in the quality building stage, and four are in the sustained institutionalization stage

overall.

While this information is helpful as an overall view of each campus, this information fails to

show the strengths and weaknesses of each individual campus. For example, after averaging each of the

five dimensions, the overall rating of the campus was in the sustained institutionalization stage on the

rubric; however, a closer look at the individual campus results show a few components in Dimension 2

are rated in the critical mass building stage. While individual campus results are not available in this

report, each of the 23 CSU campuses has either full dimensions or individual components that can be

improved.

Discussion

The self-assessment rubric revealed that most schools are either in the critical mass building stage or

quality building stage. Given the grant activities planned over the next three years, we expect that

participating campuses will see gains in most areas of the self-assessment rubric. Campuses are

expected to participate in varying levels of program activities. For example, some campuses were

awarded funds for Innovation subgrants, while others will have faculty and administration participate in

Engaged Department Institutes. Given the varying levels of program participation, we can investigate

how these activities will impact institutionalization of service learning on all CSU campuses. While the

self-assessment rubric is being used ultimately as a pretest-posttest measure, it has provided important

and useful information about the state of the CSU system as grant activities are just beginning. A copy of

the self-assessment rubric completed by campuses can be found in Appendix A.

Appendix A. Self-Assessment Rubric for the Institutionalization of Service-Learning in University STEM Departments

DIMENSION I: PHILOSOPHY AND MISSION OF SERVICE-LEARNING A primary component of service-learning institutionalization in your Science, Technology, Engineering, and Mathematics (STEM) departments on your campus is the development of a definition for service-learning that provides meaning, focus, and emphasis for the service-learning effort in your STEM departments. How narrowly or broadly service-learning is defined on your campus will effect which campus constituents participate/do not participate, which campus units will provide financial resources and other support, and the degree to which service-learning will become part of the campus’ institutional fabric.

DIRECTIONS: For each of the four categories (rows), highlight or circle the number in the continuum that best represents the CURRENT status of the development of a definition, philosophy, and mission of service-learning in your STEM departments.

Critical Mass Building Quality Building Sustained Institutionalization Explanation

DEFINITION OF SERVICE-LEARNING

IN STEM DEPARTMENTS

(circle one)

There is no definition for service-learning across STEM departments. The term "service-learning" is used inconsistently to describe a variety of service and outreach activities. 1 2 3

There is an operationalized definition for service-learning in the STEM departments, but there is some variance and inconsistency in the application of the term. 4 5 6

The STEM departments have a formal, universally accepted definition for high quality service-learning that is used consistently to operationalize many or most aspects of service-learning. 7 8 9

STRATEGIC PLANNING

(circle one)

The campus does not have an official strategic plan for advancing service-learning.. 1 2 3

Although certain short-range and long-range goals for service-learning have been defined for the campus, these goals have not been formalized into an official strategic plan for STEM departments that will guide the implementation of these goals. 4 5 6

The campus has developed an official strategic plan for advancing service-learning, which includes viable short-range and long-range institutionalization goals for STEM departments. 7 8 9


Please provide any overall comments about your ratings from above.

ALIGNMENT WITH DEPARTMENTAL

MISSION

(circle one)

While service-learning complements many aspects of the campus’s mission, it remains on the periphery. Service-learning is rarely included in larger efforts that focus on the core missions of the campus, particularly within STEM departments. 1 2 3

Service-learning is often mentioned as a primary or important part of the campus’s mission, but service-learning is not included in the campus; official mission or strategic plans, particularly within STEM departments. 4 5 6

Service-learning is part of the primary concern of the campus. Service-learning is included in the campus’s official missions and/or strategic plans, particularly in STEM departments. 7 8 9

ALIGNMENT WITH EDUCATIONAL

REFORM EFFORTS

(circle one)

Service-learning stands alone and is not tied to other important, high profile efforts on campus (e.g., STEM departments/ community partnership efforts, establishment of learning communities, improvement of undergraduate teaching, faculty research). 1 2 3

Service-learning is tied loosely or informally to other important, high profile efforts on campus (e.g., STEM departments/community partnership efforts, establishment of learning communities, improvement of undergraduate teaching, faculty research). 4 5 6

Service-learning is tied formally and purposefully to other important, high profile efforts on campus (e.g., STEM departments/community partnership efforts, establishment of learning communities, improvement of undergraduate teaching, faculty research). 7 8 9


DIMENSION II: STEM FACULTY SUPPORT FOR AND INVOLVEMENT IN SERVICE-LEARNING One of the essential factors for institutionalizing service-learning in higher education is the degree to which faculty members are involved in implementation and advancement of service-learning on a campus (Bell, Furco, Ammon, Sorgen, & Muller, 2000).

DIRECTIONS: For each of the four categories (rows), highlight or circle the number in the continuum that best represents the CURRENT status of STEM faculty involvement in and support for service-learning in STEM departments.


STEM FACULTY

KNOWLEDGE AND

AWARENESS

(circle one)

Very few STEM faculty fully know what service-learning is or understand how service-learning is different from community service, internships, or other experiential learning activities. 1 2 3

An adequate number of STEM faculty members know what service-learning is and understand how service-learning is different from community service, internships, and other experiential learning activities. 4 5 6

A substantial number of STEM faculty members know what service-learning is and can articulate how service-learning is different from community service, internships, or other experiential learning activities. 7 8 9

STEM FACULTY INVOLVEMENT AND

SUPPORT

(circle one)

Very few STEM faculty members are instructors, supporters, or advocates of service-learning. Few support the strong infusion of service-learning into the academic programs or into their own scholarly and professional work. Service-learning activities in STEM are sustained by a few faculty members on campus. 1 2 3

While a satisfactory number of STEM faculty members are supportive of service-learning, few of them are advocates for infusing service-learning into the academic programs or into their own scholarly and professional work. An inadequate or unsatisfactory number of KEY STEM faculty members are involved in service-learning. 4 5 6

A substantial number of influential STEM faculty members participate as instructors, supporters, and advocates of service-learning and support the infusion of service-learning into the academic programs or into their own scholarly and professional work. 7 8 9

STEM FACULTY

LEADERSHIP

None of the most influential STEM faculty members serve as leaders for advancing service-learning in the STEM departments.

There are only one or two influential STEM faculty members who provide leadership to the STEM departments’ service-

A highly respected, influential group of STEM faculty members serves as the STEM departments’ service-learning leaders and/or advocates.



(circle one) 1 2 3

learning effort. 4 5 6

7 8 9

STEM FACULTY

INCENTIVES AND

REWARDS

(circle one)

In general, STEM faculty members are not encouraged to be involved in service-learning activities; few if any incentives are provided (e.g., minigrants, sabbaticals, funds for conferences) to pursue service-learning activities; STEM faculty members' work in service-learning is not usually recognized during their review, tenure, and promotion process. 1 2 3

Although STEM faculty members are encouraged and are provided various incentives (e.g., minigrants, sabbaticals, funds for service-learning conferences) to pursue service-learning activities, their work in service-learning is not always recognized during their review, tenure, and promotion process. 4 5 6

STEM faculty who are involved in service-learning receive recognition for it during the department's review, tenure, and promotion process; STEM faculty are encouraged and are provided various incentives (e.g., minigrants, sabbaticals, funds for service-learning conferences) to pursue service-learning activities. 7 8 9


DIMENSION III: STEM STUDENT SUPPORT FOR AND INVOLVEMENT IN SERVICE-LEARNING An important element of service-learning institutionalization is the degree to which students are aware of service-learning opportunities in the STEM departments and are provided opportunities to play a leadership role in the development of service-learning in the STEM departments.

DIRECTIONS: For each of the four categories (rows), highlight or circle the number in the continuum that best represents the CURRENT status of student support for and involvement in service-learning in your campus STEM departments.


STEM STUDENT AWARENESS

(circle one)

For STEM students, there is no

mechanism for informing students about service-learning courses, resources, and opportunities that are available to them. 1 2 3

While there are some mechanisms for informing STEM students about service-learning courses, resources, and opportunities that are available to them, the mechanisms are sporadic and concentrated in only a few programs (e.g., course flyers). 4 5 6

There are coordinated mechanisms (e.g., service-learning listings in the schedule of classes, course catalogs) that help STEM students become aware of the various service-learning courses, resources, and opportunities that are available to them. 7 8 9

STEM STUDENT OPPORTUNITIES

(circle one)

Few service-learning opportunities exist for STEM students; only a handful of service-learning courses are available. 1 2 3

Service-learning options (in which service is integrated in core academic courses) are limited to only a certain groups of students in the STEM departments (e.g., students in certain majors, honors students, seniors). 4 5 6

Service-learning options and opportunities (in which service in integrated in core academic courses) are available to STEM students in many areas throughout the STEM departments, . 7 8 9

STEM STUDENT LEADERSHIP

(circle one)

Few, if any, opportunities in the STEM departments exist for students to take on leadership roles in advancing service-learning in their department. 1 2 3

There are a limited number of opportunities available for STEM students to take on leadership roles in advancing service-learning in their department. 4 5 6

STEM students are welcomed and encouraged to serve as advocates and ambassadors for institutionalizing service-learning in their department. 7 8 9


STEM STUDENT INCENTIVES AND

REWARDS

(circle one)

The campus has neither formal mechanisms (e.g., catalogued list of service-learning courses, special notation on students’ transcripts) nor informal mechanisms (e.g., news stories in paper, unofficial student certificates of achievement) that encourage or reward STEM students to participate in service-learning activities. 1 2 3

While the campus offers some informal incentives and rewards (e.g., news stories in paper, unofficial student certificates of achievement) that encourage and/or reward STEM students to participate in service-learning activities, the campus offers few or no formal incentives and rewards (e.g., catalogued list of service-learning courses, special notation on students’ transcripts). 4 5 6

The campus has one or more formal mechanisms in place (e.g., catalogued list of service-learning courses, special notation on students’ transcripts) that encourage and/or reward STEM students to participate in service-learning activities. 7 8 9



DIMENSION IV: COMMUNITY PARTICIPATION AND PARTNERSHIPS An important element for service-learning institutionalization is the degree to which the STEM departments nurture community partnerships and encourage community agency representatives to play a role in implementing and advancing service-learning in the STEM departments. DIRECTIONS: For each of the three categories (rows), highlight or circle the number in the continuum that best represents the CURRENT status of community participation and partnerships in your STEM departments.


COMMUNITY PARTNER

AWARENESS

(circle one)

Few, if any, community agencies that partner with the campus are aware of the campus’s goals for service-learning and the full range of service-learning opportunities that are available to STEM students. 1 2 3

Some, but not the majority of community agencies that partner with the campus are aware of the campus’s goals for service-learning and the full range of service-learning opportunities that are available to STEM students. 4 5 6

Most community agencies that partner with the campus are aware of the campus’s goals for service-learning and the full range of service-learning opportunities that are available to STEM students. 7 8 9

MUTUAL UNDERSTANDING

(circle one)

There is little or no understanding between the campus and community representatives regarding each other's needs, timelines, goals, resources, and capacity for developing and implementing service-learning activities in STEM. 1 2 3

There is some understanding between the campus and community representatives regarding each other's needs, timelines, goals, resources, and capacity for developing and implementing service-learning activities, but there are some disparities between community and STEM departments’ goals for service-learning. 4 5 6

Both the campus and community representatives are aware of and sensitive to each other's needs, timelines, goals, resources, and capacity for developing and implementing service-learning activities. There is generally broad agreement between the STEM departments and community on the goals for service-learning. 7 8 9

COMMUNITY PARTNER VOICE &

Few, if any, opportunities exist for community agency

There are a limited number of opportunities available for

Appropriate community agency representatives are formally


LEADERSHIP

(circle one)

representatives to take on leadership roles in advancing service-learning in STEM departments; community agency representatives are not consistently invited or encouraged to express their particular agency needs or recruit student and faculty participation in service-learning. 1 2 3

community agency representatives to take on leadership roles in advancing service-learning in STEM departments; community agency representatives are provided limited opportunities to express their particular agency needs or recruit student and faculty participation in service-learning. 4 5 6

welcomed and consistently encouraged to serve as advocates and ambassadors for institutionalizing service-learning in STEM departments; community agency representatives are provided substantial opportunities to express their particular agency needs or recruit student and faculty participation in service-learning. 7 8 9



DIMENSION V: DEPARTMENTAL SUPPORT FOR SERVICE-LEARNING In order for service-learning to become institutionalized, the STEM departments must provide substantial resources, support, and muscle toward the effort.

DIRECTIONS: For each of the seven categories (rows), highlight or circle the number in the continuum that best represents the CURRENT status of STEM departmental support for STEM service-learning.


COORDINATING ENTITY

(circle one)

There are no STEM faculty and/or STEM department leaders that are devoted to assisting the various campus constituencies in the implementation, advancement, and institutionalization of service-learning. 1 2 3

There are STEM faculty and/or STEM department leaders who utilize service-learning activities, but the entity either does not coordinate them exclusively or provides services only to a certain constituency (e.g., students, faculty) or limited number of STEM departments or programs. 4 5 6

There are STEM faculty and/or STEM department leaders who maintain a coordinating entity (e.g., committee, center or clearinghouse) that are devoted primarily to assisting the various constituencies in the implementation, advancement, and institutionalization of service-learning. 7 8 9

POLICY-MAKING ENTITY

(circle one)

The campus’s official and influential policy-making board(s)/committee(s) do not recognize service-learning as an essential educational goal for the STEM departments. 1 2 3

The campus’s official and influential policy-making board(s)/committee(s) recognize service-learning as an essential educational goal for the STEM departments, but no formal policies have been developed. 4 5 6

The campus’s policy-making board(s)/committee(s) recognize service-learning as an essential educational goal for the STEM departments and formal policies have been developed or implemented. 7 8 9

STAFFING

There are no staff and/or faculty members on campus whose primary paid responsibility is to advance and institutionalize service-learning in STEM.

There is an appropriate number of staff and/or faculty members on campus who understand service-learning fully and/or who hold appropriate titles that can support

The campus houses and funds an appropriate number of permanent staff and/or faculty members who understand service-learning and who hold appropriate titles that


(circle one)

1 2 3

the advancement and institutionalization of service-learning throughout the STEM departments; however their appointments are temporary or paid from soft money or external grant funds. 4 5 6

can support the advancement and institutionalization of service-learning in the STEM departments. 7 8 9

FUNDING

(circle one)

The campus’s service-learning activities in STEM are supported primarily by soft money (e.g., short-term grants) from sources outside the institution. 1 2 3

The campus’s service-learning activities in STEM are supported by both soft money (e.g., short-term grants) from sources outside the campus as well as hard money from within the campus.. 4 5 6

The campus’s service-learning activities in STEM are supported primarily by hard funding from the campus or directly from STEM departments. 7 8 9

ADMINISTRATIVE SUPPORT

(circle one)

The campus’s administrative leaders have little or no understanding of service-learning in STEM, often confusing it with other campus outreach efforts, such as community service or internship programs. 1 2 3

The campus’s administrative leaders have a clear understanding of service-learning in STEM, but they do little to make service-learning a visible and important part of the campus’s work. 4 5 6

The campus’s administrative leaders understand and support service-learning in STEM, and actively cooperate to make service-learning a visible and important part of the campus’s work. 7 8 9

ENGAGED STEM DEPARTMENT

STATUS

(circle one)

The STEM departments are not considered by the broader university community to be engaged departments. 1 2 3

The STEM departments are considered to be emerging engaged departments by the broader university community. 4 5 6

The STEM departments are considered to be advanced engaged departments by the broader university community. 7 8 9

EVALUATION & There is no organized effort An initiative to account for the An ongoing, systematic effort is in


ASSESSMENT

(circle one)

underway for STEM department to account for the number, quality, and impact of service-learning activities taking place. 1 2 3

number, quality, and impact of service-learning activities taking place throughout in the STEM departments has been proposed. 4 5 6

place to account for the number, quality, and impact of service-learning activities that are taking place throughout the STEM departments. 7 8 9


Documents

Self-Assessment Rubric for the Institutionalization of ... · Self-Assessment Rubric for the Institutionalization of Service-Learning in University STEM Departments An Assessment