FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report 1

Final Report Evaluation

Presented toSan Jose Jazz

Progressions Project

Progressions

Final rePort

evaluation

FY 2011-2012Presented by:Community Crime Prevention AssociatesOctober 7, 2012

2 FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report

San Jose Jazz Progressions Project Partners

Partners include:Franklin-McKinley School DistrictCatholic Charities of Santa Clara CountyCORAL After-School ProgramCity of San Jose, Mayor’s Gang Prevention Task ForceFranklin-McKinley Children’s InitiativeKodály Center, Holy Names UniversityHarmony Project, Los Angeles, CA

Funders include:Leo M. Shortino Family FoundationCisco FoundationSanDisk FoundationComerica Bank

Progressions Project StaffHarley Christensen, Director of Education, San Jose JazzJulie Rinard, Progressions Director

FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report 3

Acknowledgments OF Community Crime Prevention Associates (CCPA) Evaluation Team is truly grateful for the as­sistance received during this evaluation report. First and foremost, our gratitude is extended to the staff of San Jose Jazz, Santee Elementary School, Success Academy, the Franklin McKin­ley School District, Catholic Charities of Santa Clara County CORAL Program and the children and parents who provided their assessment of Progressions. Everyone was extremely coop­erative and supportive in assisting the Evaluation Team. The Evaluation Team acknowledges the passion and professionalism of all the partners in this first implementation of the Progres­sions Project. CCPA acknowledges the dedicated caring professionals that make up the Progressions Project Team. Evaluating and describing their work and dedication to serving our children was a gratifying experience. The Evaluators acknowledge all the people involved in the Progressions for their willingness to de sign a comprehensive project to assist our children to be happy, healthy, and productive citizens who are successful at home, at school, and in the community. Dr. Peter Ellis served as the team leader for this evaluation and Dr. Rex Green assisted.

Please address any questions or comments to Dr. Peter Ellis at (510) 814­1844 or e­mail ­ ellis­ccpa@sbcglobal.net.

TABLE OF CONTENTS

Page

Progressions Overview 4Mission and Activities Highlights 5Highlights of Progressions’ First Year Evaluation 8Evaluation Questions Answered 10Challenges This Year 13Challenges Next Year 15 Appendix A Theory of Change 17Appendix B Evaluation Methodology 18

4 FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report

P R O G R E S S I O N S O V E R V I E WPromoting Social Change Through Music

Research informs us that the academic intrinsic motivation of very young students is extremely high. That same research tells us that their intrinsic motivation plummets every year that they remain in school.

The exception? Students who are deeply­engaged in arts programs.

Arts­engaged students exhibit more pro­social behavior, higher affect, greater school retention, and better academic performance – particularly students from low­income families.

Progressions Progressions is San Jose Jazz’s El Sistema­inspired, K­12, youth devel­opment program that uses music to connect with low­income, at­risk students, challenges them with a rigorous music education program, helps them build a pro­social community, and stays with them until they graduate from high school. The program is focused on helping to keep participants in school, out of gangs, and on the road to becom­ing better students and contributing citizens.

Progressions is led by professional musicians/educators/mentors who teach students to read music, master instruments, and collaborate in ensembles, bands, and choirs which help them: develop as individ­uals, learn to set and achieve goals, and understand what it takes to succeed.

The program’s impact is felt by program participants, and also in their homes but in their schools, neighborhoods, and community. Besides music­related activities that help participants develop a positive per­sonal identity, research informs us that school­related performance improves with more frequent attendance, higher grades, fewer dis­cipline problems, and higher standardized test scores than cohorts not participating in programs like Progres­sions.

Effort/Effect ­ Progressions served 103 students in two schools with 14,832 hours of service at a cost of $2.83 an hour. Customer satisfaction of children and their parents surveyed was high. The program met their targeted performance goals for Child Asset Development changes and SJ Jazz Progressions selected changes. Progressions in their first year, demonstrated both efficiency and effectiveness.

Progressions is an El Sistema-inspired pro-gram. According to CBS’ 60 Minutes report-er Bob Simon “el Sistema” – “the system” – is a Venezuelan youth development, music education program that’s all about children, about saving them – hundreds of thousands of children – through music. The children are from the slums of Venezuela. And their orchestra is about the exuberance of youth.

El Sistema was founded in 1975 by econ-omist and musician José Antonio Abreu under the name of Social Action for Music. The organization currently has 31 sympho-ny orchestras, and between 310,000 and 370,000 children attending its music schools around the country. The vast majority of the students come from very poor socio-eco-nomic backgrounds and live in high risk neighborhoods. The success of the program has inspired over 60 programs in the U.S. and around the world.

FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report 5

Progressions Mission

The mission of Progressions is to promote the healthy growth and development of children through the study, practice and performance of music; to build healthier communities by investing in the positive de­velopment of children through music; and to develop children as musical ambassadors amongst people of diverse cultures, backgrounds, and beliefs.

It is a K­12, research­based, youth development pro­gram that uses music to connect with low­income, at­risk students, challenges them with a rigorous music education program, helps them build a pro­social com­munity, and stays with them until they graduate from high school. The program is focused on helping keep participants in school, out of gangs, and on the road to becoming better students and contributing citizens.

Tested as a pilot program with one class at Success Academy in spring 2011, Progressions was launched at Santee Elementary School and Success Academy in fall 2011. In May 2012, Progressions completed its first year as a fully operational program that included program evaluation surveys completed by students and parents.

2011-2012 Highlights of Activities • 103 at­risk, preK­3rd grade students at Santee

Elementary School and Success Academy re­ceived after­school music instruction 4 days per week October 2011­May 2012.

• Students learned about the key fundamen­tals of musicianship, with special emphasis on developing the ability to keep a steady beat, sing in tune, and read music from the staff.

• Students learned to sing and play over 50 folk songs, dances, and games from around the world that are being used for musical skill development.

• The regular incorporation of Spanish lan­guage materials into class curriculum facili­tated a connection to home culture for the predominantly native Spanish­speaking stu­dents. A connection was also made to home culture for students whose primary language at home is Vietnamese.

PROGRESSIONSMission and Activities Highlights

By Progressions Teacher

Promoting Positive Youth Development through Music

6 FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report

• In February, approximately 60 students and parents from Santee Elementary School and Success Academy attended an Information Night to learn more about the goals and future plans for implementation of Progressions. The program was produced in English and Spanish.

• In May, students at Success Academy per­formed as part of a year­end concert and cele­bration; the event was attended by 148 people (all Success Academy CORAL Program students and some Success parents). Songs were sung in English and Spanish. The program was pro­duced in English and Spanish.

• Fifty students at Santee Elementary School performed as part of a year­end concert and celebration; the event was attended by 153 family and friends. Songs were sung in English, and Spanish. The Program was produced in English and Spanish.

• The spring concerts created an opportunity for students to receive positive reinforcement from the school and community at large for participation in a pro­social activity. Parents voiced their support of a program that teaches music to their children.

• Students were given Certificates of Awards for completion of an entire academic year of Progressions.

• Student Assessments: Musical skill develop­ment was monitored regularly through infor­mal class assessments. Goals for skill devel­opment were based on a Kodály1 curriculum for elementary school adapted by the Kodaly Center at Holy Names University in Oakland, CA.

• Program effectiveness was tracked through the following mechanisms: Year­end Parent Surveys

1. Year­end Child Surveys

2. Year­end CORAL Staff Surveys

3. Observation of Progressions Program

4. Year­end Meetings with Santee Elementary and Success Academy Principals

5. School Success Variable Data (attendance, behavior and STAR test results): STAR test results are not available for these students but will be analyzed when scores become available.

1. Zoltan Kodály was a Hungarian musician, composer, pedagogue, anthropologist and philosopher best known for developing a sequential, developmentally­appropriate method of music education for young chil­dren. It has been replicated around the world since it was first created in the mid­1900s and continues to be adapted internationally to meet the needs of the 21st century. The sequential approach encourages chil­dren to master musical concepts as they are developmentally ready and can be especially effective in low­resource settings because it requires little instrumentation other than the human voice. The method is peda­gogically versatile because it reaches children with a variety of learning styles by incorporating aural, visual and kinesthetic teaching strategies. Using folk songs and dances that are native to the culture as the basis for curriculum development, the Kodály method simultaneously teaches music fundamentals and cultivates an appreciation of music that helps weave a cohesive social fabric. Implicit in the method is the belief that music builds community and should be accessible to everyone, not just the wealthy. Kodály­trained children are especially well known for their strong music literacy skills and their beautiful singing intonation.

FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report 7

• Student attrition for the spring semester was kept relatively low, at 11% (11 out of 103). No one left the program due to lack of interest. Rather, all attrition was related to changes in family circumstances, especially moving and the need to take care of younger siblings.2

• Informal music office hours were created to give children additional opportunities to con­nect with the Progressions teacher and work in smaller groups. Average attendance per session was 10 children.

• The Progressions teacher pursued professional development through 1) attendance at the annual national conference for Organization of American Kodaly Educators in Phoenix, AZ, 2) completion of Level II music education certi­fication at the Holy Names University Kodaly Summer Institute and 3) spearheading the cre­ation of a working group to incorporate more Spanish language materials into core Kodaly curriculum in predominantly Latino classrooms in the San Francisco Bay Area.

• Progressions teacher and director Julie Rinard is a classical pianist, vocalist, social activist, and music educator with a specialty in teach­ing music to young children utilizing the Ko­dály Method. She is bi­lingual and has worked with Spanish­speaking populations in the U.S. and Latin America for over a decade. While at University of California, Berkeley, Julie was a member of the University Chorus, the Wom­en’s Chorale, and La Pena Social Justice Com­munity Chorus. Currently, Julie is a member of Silicon Valley Symphony Chorale. Julie has worked as a Development and Program Offi­cer with International Development Exchange

2 The highest attrition rate by class was with the 2nd/3rd grade class; a total of 6 children left the CORAL program between January and May 2012. Two children were transferred out of Progressions and into another CORAL class due to behavioral issues that were preventing the entire class from learning, 2 children left CORAL to help take care of their younger siblings, and 1 child moved back to Mexico where his father lives. The other child quit the CORAL program at the beginning of January for unknown reasons. A total of 2 children in the preK­Kinder class left CORAL for unknown reasons, 2 children in the Santee 1st/2nd grade class left CORAL due to failure to comply with the requirements for participation in CORAL, and 1 child in the Success Academy 1st/2nd grade class left CORAL for unknown reasons.

and as a Program Director at Seva Founda­tion. Julie earned her Bachelor of Arts degree from University of California, Berkeley and Master of Arts degree from American Univer­sity in Washington, DC. Julie is also a graduate of the Kodály Institute at Holy Names Univer­sity.

Results of Evaluation Results of the year one Progressions analysis suggest that overall program outcomes for the first full year of the program are extremely positive. Analysis was conducted by Dr. Peter Ellis and Dr. Rex Green of Community Crime Prevention Associates, evaluation experts and key Progressions evaluation consultants with 80+ years of experience in helping to build resil­ient communities through the practice of continuous quality improvement 3

3 Community Crime Prevention Association is dedicated to assisting communities, families, and public and private agencies to build capacity and practice continuous improvement – so they can foster pro­social and resilient youth, promote healthy families, and build safe neighbor­hoods.

8 FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report

Highlights of SJ Jazz Progressions First Year Evaluation

Effort of Progressions Funded Care for this Year Was Efficient

• Progressions served 103 unduplicated children customers with 10,547 hours of direct service. Each customer received an average of 144 hours of service with an average of $408 spent on each of child customers.

• The average cost per hour of service was $3.98 an hour for services. The cost per hour is the bot­tom line or output of effort. It is calculated by dividing the amount of funding spent by the hours of direct service delivered. This cost per hour is an efficient use of resources.

Progressions Funded Care Was Effective

• Children and youth customers gave Progressions an 82% satisfaction rating and parents gave the care for their child an 90% satisfaction rating. Both are considered to be positive and high satisfac­tion rates.

• Progressions was effective in achieving positive changes in the behaviors and skills of children cus­tomers as measured by two sets of questions. One set of questions for child developmental assets indicated that 59% of targeted child developmental assets showed improvement. The second set of questions asked about Progressions targeted changes. Children customers also indicated that pro­gressions care was effective in producing two out of three (66%) targeted changes selected caused by the Progressions intervention and care. These targeted changes are attitudes, behaviors, skills and knowledge that allow children to develop needed youth assets and skills to ensure a positive future.

This is a summary briefing of the effort, effect, and performance of San Jose Jazz Progression Project for this year.

Progressions Had High Customer Satisfaction• Effectiveness is determined by customer satisfaction and producing change for the better in cus­

tomers as determined by child and parent customers.• Child customers gave Progressions an 84% satisfaction rating and parents gave the same services

for their child an 82% satisfaction rating. Both are considered to be positive and high satisfaction rates.

Chart 2

FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report 9

In addition to satisfaction with services, Progres­sions assessed how much change they produce in their child customers. Green (2003) applied the term “service productivity” to this type of assessment of the effects of services. He fol­lowed the distinction recommended by Heaton (1977): “emphasize measuring the effectiveness of services versus their efficiency when discuss­ing productivity. This distinction seems par­ticularly apt, because services are provided to cause changes in people or their property” (Hill, 1976). Unlike when goods are produced, inven­toried, and valued based on the effort expended to create them, services have no value unless they cause targeted changes in customers.

The assessment of service productivity involves designing questions that relate to service goals for individual customers and phrasing them so that the responder considers whether change

Chart 2

Service Productivity Scores From Child and Parent Customersoccurred due to the services. The amount of productivity for services is calculated by aver­aging the responses. The choices offered must allow the responder to indicate that services made them worse off or caused no change, as well as indicating that there was improvement. Consequently, service productivity ranges from 100% to minus 100%, with zero meaning no change overall. A score of 100% means the responder improved on all items or targeted changes and a score of minus 100% means the responder got worse on all items.

Two types of service productivity are assessed for Progressions–asset development service productivity and Progressions specified service productivity shown below. See Appendix B for additional information on Service Productivity and the Performance Logic Model.

Worse Same Better

Progressions children and parent indi-cated value added from participating in the inter-vention care.

This year the SJ Jazz Progressions services and care was both efficient and effective.

10 FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report

San Jose Jazz Progressions Performance Logic ModelEvaluation Questions AnsweredEffort

Who provided the funding for services?The funding was provided by the following:

What did Progressions spend on services?San Jose Jazz Progressions spent all the above funds.

Who are our 103 child customers?

How much services did we provide?Progressions provided 10,547 hours of service to 103 child customers at two schools.

How much did the services cost to deliver?Cost per hour was $3.98 and the cost per student was $408.

EffectWere our child and parent customers satisfied with our services?Child and their parents gave the program high satisfaction scores.

Were our services effective in producing change for the better for our customers because of the Progres-sion Intervention? Yes, Progressions children and their parents indicated that because of Progressions, child participants im­proved on the target changes measure by responses to surveys completed by child and parent customers.

Male    40%  

Female  60%  

Gender  

La#no  75%  

Asian  21%  

Caucasian  2%  

African  American  

2%  

Ethnicity    

FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report 11

Initial Outcomes

The following are the frequency of responses to customer satisfaction and service productivity questions asked of children and parents about the Progressions Project. Child customers were asked questions for child developmental assets and questions related to targeted skills, knowledge, attitudes and behaviors targeted by the Progressions Project. For each of these variables, a quantitative score was given. The score is the per­cent of targeted changes achieved minus the percent of targeted changes missed. The score could range from ­100% (for worse) to 0% (same) to +100% (for better). Indicated below is the percentage of youth surveyed that indicated they changed for the better because of the services provided by the grantee. Each question began with: “Because of this program…” and has a choice that is positive benefit, no benefit, or negative benefit.

San Jose Jazz Progressions – TOTAL

Child

75% Do you think this program is good, ­19% indicated this program was OK81% Are you learning a lot and having fun ­ yes.88% Are the program people helpful ­ yes.52% Would you tell a schoolmate to come to this program ­yes.

69% Music class makes my school work easier.57% Music class helps me get along with adults better.85% Music class helps me learn new things ­ yes.72% Music class helps me stay safe ­ yes.69% Music class helps me get along with other kids ­ yes.80% Music class makes me feel good about myself ­ yes.81% Music class helps me listen to music in new ways ­ yes.76% Music class helps me express my emotions better.78% Music class helps me feel the beat better.79% Music class increases my interest in things about music more.81% Music class makes me practice to become a better musician more.

San Jose Jazz Progressions – TOTAL

Parent 57% Because of this program, my child does his/her school work better.46% Because of this program, my child gets along with adults better.93% Because of this program, my child learns new things better.86% Because of this program, my child’s confidence in him/herself is better.57% Because of this program, my child gets along with other children better.57% Because of this program, my child’s ability to stay safe is better.

93% Because of this program, my child expresses him/herself through music better.86% Because of this program, my child remembers songs and rhythms better.93% Because of this program, my child my child enjoys learning new songs more.93% Because of this program, my child shares his/her songs with the whole family more.

12 FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report

Intermediate Outcomes

Intermediate results are achieved by the efforts of everyone to improve outcomes for our children. This first year of the program does not have academic test scores available yet as they have not been scored by the California State Testing system. Data was available on attendance and behavior for the school year during which Progressions operated in two schools.

Santee School Intermediate Outcomes Highlights

Santee Progressions Students had a 97% attendance rate. Missing and average of 5.4 days during the year.

Only one Santee Progressions student out of seven K­3 Santee students suspended from school was from the Progressions program.

Success Academy School Intermediate Outcomes Highlights

Success Academy Progressions students had no behavior referrals of the 56 behavior referral registered during the school year.

Eight Success Academy Progressions students had one truancy referral and no Progres­sions students were suspended.

FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report 13

Challenges Identified by Progressions’ Teacher

2011-2012 Challenges

A number of factors (both internal and external) challenged Progressions staff, the most significant of which are cultural shifts that introduction of Pro­gressions set in motion. They relate primarily to: 1) expectations for in­class behavior, 2) expectations for in­class performance, 3) introduction of new materials for music instruction, and 4) introduction to a culture of singing.

1. Expectations for in­class behavior: Building a culture of respect between the students and teacher and establishing high standards for classroom behavior and participation required a significant time commitment throughout the entire year. For example, a set of six “Music Class Expectations” was established at the beginning of the semester, and each group set a “focus for the day” at the beginning of every class. Special emphasis was placed on creating a safe environment where children feel com­fortable to make mistakes and ask questions as part of the process toward musical mastery. A set of logical consequences for classroom behavior was created, and the Progressions teacher worked with CORAL staff throughout the year to ensure consistent enforcement of consequences.

2. Expectations for in­class performance: Setting the bar high while simultaneously creating a culture in which the children have the confi­dence to meet that bar was also critical. As such, recognizable milestones were estab­lished to help scaffold this journey. Finding an appropriate balance between the high bar and the realities of this student population was an ongoing challenge.

3. New materials for music instruction: Prior to the launch of Progressions, children at Santee and Success did not have access to regular, high quality music instruction. This made it necessary to lay an enormous amount of groundwork to establish a culture of respect for and appreciation of a new type of music to which most children had not previously been exposed.

While Progressions is respectful of all types of music, it is designed to use the media of culturally relevant folk and art music as the basis for a solid music education, not popular music. The fact that Progressions is designed this way created some resistance among the children. In particular, one child once asked when they would get to learn some “normal” songs. Once the children begin to under­stand (and hear) the results of this method of instruction and experience the satisfaction of becoming adaptable musicians, it is expected that this friction will dissipate.

4. Introduction to a culture of singing: Most program participants were uncomfortable with the use of their singing voice at the start of the program, making it extremely difficult to introduce new repertoire that serves as the foundation for the introduction of more complex melodic and rhythmic elements. The problem was especially pronounced among the older boys (third grade), who find it diffi­cult to locate the higher register of their voice and who were more self­conscious about the essential process of vocal exploration than the boys in the younger grades.

5. Other Challenges—CORAL Staff Turnover: The 1st/2nd grade CORAL Coordinator at Santee resigned unexpectedly in mid­April.

14 FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report

Challenges for Next Year as Identified by Progressions’ Teacher

Progressions: 2012-2013• Classes for 2012 start the second week of Sep­

tember, and approximately 160 students will be served at Santee Elementary School and Success Academy.

• An outreach campaign will target Santee class­room teachers and CORAL staff as well as the broader Santee parent community to build an understanding of and long­term support for Progressions.4

• Enrollment will be open to CORAL and non­CORAL students at Santee Elementary School and Success Academy.

• Recorder and keyboard classes will be offered to children starting in the third grade.

• Guest performers and music educators will be invited to give lectures and performances.

• Planning will begin to secure donated instru­ments (flute, clarinet, trumpet, and trombone) for the 2013­2014 academic year when in­strument instruction and band rehearsals will begin.

• A Saturday Family Day component is expected to be added at the beginning of 2013.

4 The campaign will include 1) an information session for Santee class­room teachers and CORAL staff to educate them about the academic and social benefits of participation in the program and encourage better cross­program collaboration on behalf of students and 2) marketing materials and information sessions for the broader Santee parent community to make them aware of the existence of the program and encourage them to sign their children up.

Ongoing Planning• Changing CORAL Program Structure: The

CORAL Program lost 50% of its funding for 2012­2013, necessitating a substantial change in program structure at both Santee and Suc­cess. The program, originally open to preK­6th graders, will now be offered primarily to low performing 1st, 2nd , and 3rd graders. Depending on funding, a drop­in homework component may be added for 4th , 5th , and 6th graders.

• Planning for Introduction of Monthly Satur­day Family Day: Planning for Introduction of Monthly Saturday Family Day” – “The timing and method for the launch of the Monthly Sat­urday Family Day (or “nucleo” as it has been named by El Sistema and El Sistema­inspired programs) is important because it will serve as the cornerstone of Progressions by integrating key program components: youth develop­ment, parental involvement, and opportunities to play in group ensemble settings with profes­sional musicians and conductors.

• Stabilizing Existing Program While Projecting Growth: Progressions is designed for hori­zontal and vertical growth in both the short and long term; it will expand into additional elementary schools each year while adding components to existing school programs to ad­dress the advancing music education needs of older students. In 2012­2013, for example, re­corder and keyboard modules will be added to existing vocal musicianship curriculum for 3rd and 4th graders at the same time that planning is underway to add an individual instrumental module (clarinet, flute, trumpet and trom­bone) for 4th and 5th graders starting in 2013­2014. This forward planning is happening in the midst of a shifting program structure and careful reflection about how to refine systems developed during year one of the program.

• Establishing Stable Funding Base to Focus on Long­Term Program Development: Additional funding is needed to ensure that Progressions has the funding to enable staff to focus on building the program as an invaluable com­munity resource creating a deep and lasting impact.

FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report 15

Notes from the Classroom (Teacher’s Corner)Many of the kids I work with at Santee and Success come from unspeakably grim home environments. Amidst these challenges, many of them have developed a resiliency that makes them wise beyond their years. But at the end of the day, they are still kids who want a chance to be kids, to make friends, to be in a positive learning environment, and to have a life dream and a shot at turning it into reality one day.

Vuu is one example of the “progression” that is possible through this program. Vuu is the 3rd grader who sat on the outside of the music circle refusing to sing and mocking the other stu­dents for the first three months of class. He is the kid who told me that music class is stupid. He is the kid I wanted to kick out for consistently creating an unsafe learning environment for the entire class. As it turns out, Vuu also happens to be the kid whose mother (recent­ly remarried) kicked him out of the house one night at 2 am with nothing more than a cell phone in hand and said, “Go live someone else; there is no longer space for you here.”

In April, Progressions received a donation that allowed us to purchase a dozen keyboards. I brought one of the keyboards to school the next day and played Schumann’s “Wild Rider” from the Album for the Young. At the end of class, I noticed Vuu hanging around. As I began to put my things away, he asked if I could help him find middle C. The next day when I came to class without the piano, he asked why I hadn’t brought it. The day after I brought the key­board back, and he asked if I could teach him a song. A week later, when I told the kids that I would be teaching class piano in the coming year, Vuu told me he was reconsidering his decision to quit Progressions.

Vuu knows about middle C because his older cousin taught him about it when Vuu was in town for a visit. Vuu clearly has an innate desire to learn the piano. Yet in his current life configuration, piano lessons are out of the question. I hope he comes back to Santee this fall, because I would love to teach him to play high D and low F and everything in­between.

Other Highlights from the Teacher’s Corner

The day I walked into class and found one of my 2nd grade students writing down the rhythmic notation for one of her favorite songs on the white board (a song we had not studied in class, by the way).

Hearing a boy who had never volunteered to sing alone in class demonstrate that he had a beautiful singing voice and was able to match pitches with ease in a small group.

Seeing all my kids go crazy every time they got to sing and dance to John Kanaka, a sea shanty song dating back to the 1890s that relates the hardships of being a sailor.

Hearing my little ones sing Arroz Con Leche over and over again as I left the class­room at the end of class, day after day after day.

16 FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report

Appendix ATheory of ChangeTiny kids want to learn everything and they want to learn it now!

A robust body of research documents that the academic intrinsic motivation of very young students is extremely high. However, the same body of research shows that stu­dents’ intrinsic motivation plummets for every year they remain in school. The exception? Kids who are deeply­en­gaged in arts programs.

Arts­engaged students tend to remain highly focused and deeply­engaged. They express greater hope and possibility in their use of language – and in their own futures. They see themselves as capable of and responsible for creating positive effects in their own communities. They are more collegial and get along better with others. They exhibit more pro­social behavior, higher affect (mood), greater school retention, and better academic performance – par­ticularly amongst kids from low­income families.

However, RAND Corporation’s ‘Arts & Pro social Impact’ Studies found that while quality arts instruction is neces­sary it is insufficient to produce a powerful and enduring pro social impact. RAND researchers identified a handful of additional factors common to arts programs that pro­duce a powerful and enduring impact on students – par­ticularly those from low­income families.

Extended Time­In­ProgramEmphasis on Performance and PresentationLiberal Use of MentorsTies with other Community­Based OrganizationsComplementary Program Components

These factors are engineered into Progressions Project and Harmony Project. It’s among the factors that differentiate Progressions Project and Harmony Project from other pro­grams that are solely designed as quality arts programs. Progressions and Harmony use research to inform the design of their youth development programs.

1. Shirley Brice Heath’s 10-year longitudinal studyKids were studied in after­school programs over 10 years. Their conversations were transcribed and 2 million words were recorded. Sports programs, community­service programs such as scouting, religious groups, environmen­tal clubs, etc., and arts based programs were all found to

be beneficial to participants. However, sports programs and community service programs didn’t come close to demonstrating the benefits of arts programs. Arts­based programs, hands­down, produced the most powerful and enduring pro social impact – particularly amongst disad­vantaged kids.

The Report: Imaginative Actuality: Learning in the Arts during the Nonschool Hours, Shirley Brice Heath with Adelma Roach. The report was developed at Stanford University and funded by The Spencer Foundation, The GE Fund, and Carnegie Foundation for the Advancement of Teaching.

2. The RAND Corporation’s ‘Arts & Pro social Impact’ StudiesRAND followed up on Heath’s work. Their ‘Arts & Pro social Impact’ Studies asked: Do all arts­based programs produce powerful and enduring pro social benefits? Or, are there specific factors amongst arts­based programs that account for the pro social benefits observed? RAND’s researchers offered several reasons why the arts-based programs were more effective than the other programs they investigated. Viewed broadly, these reasons enter around the idea that arts programs create environments that “put the young on the edge” – situations that com-bine heightened risk, through socially visible performanc-es and peer critiques, with dynamic rules that structure participants’ behavior and encourage personal account-ability.

Many of the study’s results are consistent with such a perspective, which stresses the cognitive and motivation-al contexts created by effective arts interventions. For example, the emphasis on presentations can be viewed as a way in which the arts interventions push participants towards “high stakes” behavior subjected to communal examination; intensive and lengthy participation opportu-nities can reward and shape persistent and dedicated be-havior; and finally, the importance of mentoring appears consistent with the idea of constructive critiques.

The Report: The Arts and Pro social Impact: Program Characteristics and Pro social Effects, Ann Stone, Tora Bikson, Joy Moini, and David McArthur. The report was developed by the RAND Corporation and funded by the City of Los Angeles Cultural Affairs Department.

FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report 17

3. The Champions of Change report was developed for the President’s Committee on the Arts and the HumanitiesChampions of Change researchers found that learners can attain higher levels of achievement through their engage-ment with the arts. Moreover, one of the critical research findings is that the learning in and through the arts can help “level the playing field” for youngsters from disad-vantaged circumstances. Researchers also found clear evidence that sustained involvement in particular art forms – music and theater – are highly correlated with success in mathematics and reading.

The Report: Champions of Change: The Impact of the Arts on Learning. The report was edited by Edward B. Fiske and developed and funded by The GE Fund and The John D. and Catherine T. MacArthur Foundation for the Arts Edu­cation Partnership and the President’s Committee on the Arts and the Humanities.

4. Ryan and Deci’s ‘Self Determination Theory’: Autono-my, Competence and RelatednessHow arts education translates into improvements in student motivation, behavior, mood, school retention and academic performance: In describing ‘Self Deter­mination Theory’, psychologists Ryan and Deci describe three essential psychological needs: individuals’ need for autonomy, for competence, and for relatedness. When those three needs are met, individuals experience high levels of personal resilience, mental health, and well­being. When those three needs are thwarted, mental health and well­being decline dramatically.

Utilizing an experimental design, Ryan and Deci demon­strated that you can increase students’ academic motiva­tion by supporting students’ autonomy, competence, and relatedness. Successful youth arts programs support stu­dents’ competence and autonomy in the exercise of their art, and support students’ relatedness in the performance and presentation of their art to family and peers – and to the larger community. Arts programs represent an effective means of social inclusion.

The Report: Self-Determination Theory and the Facilitation of Intrinsic Motivation, Social Development, and Well-Be-ing was written by Richard M. Ryan and Edward L. Deci. The article was developed at the University of Rochester, published in American Psychologist, and funded by the National Institute of Mental Health.

Other Progressions-Related Supporting Research

A University of Illinois at Chicago study published in August 2012 found that Interest in Art Predicts Social Responsibility. People with an active interest in the arts contribute more to society than those with little or no such interest, the researchers found. They analyzed arts exposure, defined as attendance at museums and dance, music, opera and theater events; and arts expression, de­fined as making or performing art. “Even after controlling for age, race and education, we found that participation in the arts, especially as audience, predicted civic engage­ment, tolerance and altruism,” said Kelly LeRoux, assistant professor of public administration at UIC and principal in­vestigator on the study. “We correlated survey responses to arts­related questions to responses on altruistic actions ­­ like donating blood, donating money, giving directions, or doing favors for a neighbor ­­ that place the interests of others over the interests of self,” LeRoux said. “We looked at ‘norms of civility.’ Previous studies have established norms for volunteering and being active in organizations.” The researchers measured participation in neighborhood associations, church and religious organizations, civic and fraternal organizations, sports groups, charitable organiza­tions, political parties, professional associations and trade unions. Reference: University of Illinois at Chicago (2012, August 16). Interest in arts predicts social responsibility. ScienceDaily.

Northwestern University Research published in 2012 suggests that Early Music Lessons has Long­term Auditory Processing Benefits. “If you get a kid who is maybe 3 or 4 years old and you’re teaching them to attend, they’re not only working on their auditory skills but also working on their attention skills and their memory skills — which can translate into scholastic learning,” say authors. http://www.soc.northwestern.edu/brainvolts/documents/Kraus­Chandrasekeran_NRN10.pdf A study published in 2012 examining the effects of an academic music intervention on conceptual understand­ing of music notation, fraction symbols, fraction size, and equivalency of third. The study suggests that academic music interventions can play a role in helping children understand fractions. Reference: Academic music: music instruction to engage third­grade students in learning ba­sic fraction concepts Educational Studies in Mathematics,

18 FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report

DOI: 10.1007/s10649­012­9395­9. Susan Joan Courey, En­dre Balogh, Jody Rebecca Siker and Jae Paik.University of Cambridge research published in 2012 found that children 8­11 years old who engaged in group musical activity were more likely to develop empathy than those in control groups where music was not included.

Report: http://www.cam.ac.uk/research/news/mu­sic­of­kindness­playing­together­strengthens­empa­thy­in­children/.

2008 Dana Consortium research finds evidence linking training in the arts and improved math and reading skills. The findings add scientific support to the observation that children who participate in the arts also do well academi­cally and suggest that changes in attention networks in the brain may be one reason. Reference: Learning, Arts, and the Brain, The Dana Consortium Report (2008).

Report :http://www.dana.org/news/publications/publica­tion.aspx?id=10760.

FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report 19

Appendix B

PROJECT EVALUATION MethodologyEvaluation is a valuable tool that Progressions Project will utilize on an ongoing basis to make informed decisions. Evaluation will drive program improvements, it will help identify needed enhancements, and it will help program managers by providing them with real­time, decision­mak­ing data. We are committed to using evaluation and data as a critical tool in a process of continual self­examination and improvement. Evaluation is an ally that will help us to build intellectual capital and to continually refine and upgrade our project.

This section was developed and written by Progressions Project’s Co-Directors of Evaluation, Dr. Rex Green and Dr. Peter Ellis.

The Progressions Project Evaluation DesignThe Progressions Project evaluation design is a fusion of the Results and PerformanceAccountability evaluation approach (Friedman, 2007) and the Theory of Change/Logic Model (Connell, Kubisch, Schorr, & Weiss, 1995; Julian, Jones & Deyo, 1995). These complementary approaches to program evaluation yield information which allows policy makers, funders, com­munity leaders, and service providers to make informed decisions about service delivery, budget allocation, and continuous improvement in a timely manner.

Process evaluation will be conducted by a Continuous Quality Improvement (CQI) team with the timing of each component of the evaluation determined by the team’s need for information. The team may employ focus groups, interviews with community stakeholders, and surveys to obtain the information needed about funding require­ments, work processes, service quality, interagency coordi­nation, and overall performance.

Results and Performance Account-abilityMark Friedman has developed training materials for service agency staff members that guide them in identi­fying the results services should produce and the indica­tors to track that will reveal what is being accomplished. Through this approach service agencies can learn how to change service delivery in ways that will turn the curve on the results graph in the desired direction. Training in how to collect appropriate data is also included.

Theory of Change Logic ModelsThe “Theory of Change Logic Model” approach to evaluation is recommended by leading evaluators for programs serving communities in an ongoing manner (Connell, Kubisch, Schorr, &Weiss, 1995). All the Progressions Project service pro­viders have incorporated the United Way of America’s recommended logic model system of evaluation into their evaluations.

Lisbeth Schorr’s Theory of ChangeA thorough description of the “Theory of Change Logic Model” research is contained in LisbethSchorr’s book entitled Common Purpose­­Strengthening Families and Neighborhoods to Rebuild America (Schorr, 1997). Schorr explains the difficulties involved in apply­ing experimental research designs to complex, multiple outcome, and community­based projects. Schorr points out that because experimental designs can only study variables that are easily quantifiable, complex communi­ty­based interventions tend to be less frequently stud­ied.

Schorr calls for theory­based, logic model outcome evaluations. They offer an alternative approach to evaluating programs that do not require random as­signment to different groups (Julian et al.,1995). “By combining outcome measures with an understanding of the process that produced the outcome,” states Schorr, “theory­based evaluations can shed light on both the extent of impact and how the change occurred.” Schorr documents numerous examples of research and evalu­ation studies using new evaluation methods that allow social scientists to observe more complex and promis­ing programs. Schorr challenges evaluators to put less emphasis on elegant and precise statistical manipulation and more emphasis on usable knowledge. This usable knowledge will serve as critical information for the Pro­gressions Project to render thoughtful budget and policy direction, as well as continuous service improvement strategies.

20 FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report

Performance Logic ModelsPerformance Logic Models (PLM) indicate why certain service activities ought to change the behaviors of those receiving services. In that respect, PLMs resemble path diagrams connecting causal variables to effects variables. Performance accountability is divided into three areas: effort, effect, and results. The underlying logic of the PLM is that more effort on the part of staff and customers produces more outputs. More outputs guided by effective strategies produce more change in behaviors and greater satisfaction with services. As more Progressions Project customers are served more effectively, a ripple effect on the larger community will occur, causing long­term pop­ulation outcomes to increase for youth living in the same community.

Progressions Project Performance Logic ModelGraphic 1 shows how the Progressions Project Perfor­mance Logic Model is organized. The left­ hand column shows the Theory of Change/Logic Model term. The next column indicates what must be measured under each of the three areas (effort, effect, results). The Progressions Project evaluation questions further define what data are needed in column 3, followed by the data sources. A specific performance goal for the Progressions Project is noted in column five. The last column links each type of data collected to the underlying theories of changing behavior through service delivery

Emphasizing Continuous Quality ImprovementThe values and concepts described below are embed­ded beliefs and behaviors found in high­ performing organizations. They are the foundation for integrating key performance and operational requirements within a results­oriented framework that creates a basis for action and feedback. The Progressions Project Performance Logic Model Evaluation System is based on the principles and practices of Continuous Quality Improvement. CQI is prac­ticed by many public and private organizations to measure and improve their products and services to their custom­ers. CQI defines quality as meeting or exceeding the needs and expectations of the customer. The Progressions Proj­ect considers the child and their parents as their primary customers whose feedback is important to the continuous improvement of services. CQI was recommended for behavioral health care evaluations by Green and Newman (1999) over traditional program evaluation approaches.

The chief distinctions between CQI and experimental pro­gram evaluation methods are: (1) experimental methods are post­hoc and one­shot; (2) standard evaluation reports often address what happened too late to impact decisions about service delivery; (3) a different evaluation study must be designed to address specific research questions, often stated as a hypothesis; and (4) CQI is an ongoing activity conducted by program staff members to produce timely feedback, sometimes through distinct studies that may lack experimental rigor but provide rapid turnaround of the information.

Quality improvement will be a regular part of each day’s work within every Progressions Project site. The methods employed will be accessible to program staff, thus requir­ing a minimum of training in their application. The Eval­uator sees its role as an evaluation company performing program evaluations in the context of Progressions Project site staff utilizing our reports to improve their services. The Evaluator also will provide technical support to agency staff to assist them in improving the quality of the services.

A CQI team will be formed for the Progressions Project to oversee the collection and analysis of the data, as well as provide reports and other forms of feedback to man­agers and service delivery staff. The statistical analysis of Progressions Project data will extract larger meaning from data, mostly collected in an ongoing manner, and support decision­making and service improvement. The analyses will report trends, projections, and possible cause and effect relationships that would go unnoticed when data are not properly and frequently analyzed. Decision­makers

will utilize these reports for a variety of purposes, such as planning and improving service delivery, reviewing overall performance, improving operations, accomplish­ing change management, and comparing performance across sites with similar organizations, or with “best practices” benchmarks. Indicators must be selected that best represent the factors that lead to improved cus­tomer outcomes and improved operational and financial performance. Through the data collection, tracking, and analysis of Progressions Project data, the measures or indicators themselves may be evaluated and changed to better support Progressions Project goals.

FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report 21

Graphic 1

22 FY 2011-12 San Jose Jazz Society Progressions Final Evaluation Report

Data CollectionCQI requires an ongoing supply of information about staff, customers, effects of services, satisfaction with services and work setting, outcomes, funding, costs, and quality of services. Data must be segmented by, for example, types of service, customer ages, and strategic priorities to facilitate interpretation of the results.

Data collection will occur via an Internet website operated by the Evaluator and accessed by authorized persons only. Excel spreadsheets with macros will be employed to record routine updates to selected databases and questionnaire data collected at appropriate times during the year. The design of the spreadsheets has been largely determined by the Evaluator through extensive experience with the recording of data in a timely and accurate manner. Ad­justments to the spreadsheets will be made in accordance with the specific needs of the Progressions Project on an as needed basis. The databases will include: a) customer master list, b) staff listing, c) class schedules or attendance sheets, d) employee timesheets, and e) questionnaire re­cords. Specific data items will be added or deleted accord­ing to the needs of the Progressions Project.

Examples of data items for each database are: date of birth, gender, and ethnicity for clients and staff mem­bers; date and time spent for each service; date, location, and time spent by each staff member providing services; and questions from the evaluator’s previously employed surveys, including copyrighted questions that reveal how productive services are.

Behavioral Performance Measures and Service ProductivityThe evaluation design requires the collection of service productivity data to determine whether participants are better off as a result of having participated in the Pro­gressions Project. Measurement of service productivity is based on the definition of services as “transformations of people or objects through a service.” An explanation of service productivity was provided by Green (2003). The system used by the Evaluator to obtain and analyze service productivity data is explained in a subsequent article by Green, Ellis, and Lee (2005).

Service productivity data differ from before­after evalu­ation data in the following ways. The respondent to the questionnaire is prompted to consider only the changes they experienced as a result of participating in the service activities. The respondent indicates whether because of the services they are now better, worse, unchanged, or unsure about what happened. The score is calculated by subtracting the “worse” responses from the “better” responses and dividing by the number of all responses minus the “not sure” responses. The score is converted to a percentage score ranging from –100% (all worse) to +100% (all better).

ReferencesConnell, J.P., Kubisch, A.C., Schorr, L.B., & Weiss, C.H. (1995). New approaches to evaluating community initia­tives: Concepts, methods, and contexts. Washington DC: The Aspen Institute.

Friedman, M. (2007). Trying hard is not good enough. Bloomington, IN: Trafford Publishing. Green, R. S., & Newman, F. L. (1999). Total quality management prin­ciples promote increased utilization of client outcome data in behavioral health care. Evaluation and Program Planning, 22, 179­182.

Green, R. S. (2003). Assessing the productivity of human service programs. Evaluation and Program Planning, 26, 21­27.

Green, R. S., Ellis, P. T., & Lee, S. S. (2005). A city initia­tive to improve the quality of life for urban youth: How evaluation contributed to effective social programming. Evaluation and Program Planning, 28, 83­94.

Julian, D. A., Jones, A., & Deyo, D. (1995). Open systems evaluation and the logic model: Program planning and evaluation tools. Evaluation and Program Planning, 18, 333­342.

Schorr, L. B. (1997). Common purpose, strengthening families and neighborhoods to rebuild America. New York: Random House, Anchor Books.