Data-Driven Decision-Making for Quality Control:

The Power of a Relational Database

Vicki L. Cohen, Ed.D.Marlene Rosenbaum, Ed.D.Joshua Cohen

Fairleigh Dickinson UniversityTeaneck, NJ 07666

Presented at the Annual AACTE ConferenceNew Orleans, February 2008

This session will: Describe how using a relational database

becomes the driver of a quality control system;

Describe the development and utilization of a relational database;

Show how data is leveraged to support student learning, program revision, and outcomes based assessment.

The School of Education (SOE) at Fairleigh Dickinson University Comprised of aprox. 1,000 students Multiple programs that apply for state

certification: 5-Year Accelerated QUEST program MAT LD Educational Leadership Reading Specialist

The SOE at a Glance On two campuses (Teaneck and Madison) Located at 3 Community Colleges throughout

the state of New Jersey 15 Full-time faculty members Aprox 35 part-time faculty Place approximately 120 candidates into

student teaching/year Place a total of approximately 700 candidates

into clinical field experiences/year.

FDU at a Glance

FDU has aprox 12,000 students Largest private university in State of NJ SOE is part of University College on Teaneck

campus QUEST program:

45 candidates at Teaneck 200 Madison 75 CC

SOE Is a Complex Program!

Went for TEAC accreditation

Preparation for TEAC School of Education (SOE) needed to collect

accurate information on its claims Started gathering data on programs and student

performance Recognized need to access, organize and

analyze data in meaningful ways Developed a relational database This would become the “driver” of our Quality

Control System

The Quality Control System (QCS)

Every institution and program has a set of procedures and policies to ensure quality in hiring, admissions, curriculum, program design, and student learning.

Together, these procedures and structures function as a Quality Control System (QCS). (TEAC)

Need for Valid and Reliable Data

QCS must yield valid and reliable evidence about the program’s practices and results, which influences its policies and decision making.

What is Valid Evidence? Are we measuring what we intended to

measure? Are we sure that our evidence is pointing

us in the right direction? How confident do we feel about the data

we collected?

“Am I measuring what I think I am measuring?”

What is Reliable Evidence?

Yields results that are accurate and stable

Collected in a consistent way Confident we we are making the right

decision.

SOE TEAC Process

1. Developed a QCS that ensured we are collecting valid and reliable data on our claims and cross-cutting themes

Instruments (rubrics, observation forms, surveys) were validated through external panel of “experts”

Inter-rater reliability is being established

2. Developed infrastructure and system to collect the data

SOE TEAC Process

3. Analyzed data: aggregated and disaggregated

4. Determined strengths and weaknesses with total faculty involvement

5. Analyzed what revisions needed • Programs• Curriculum• Processes and policies

6. Currently making revisions based upon evidence.

SOE Assessment Philosophy

We use multiple sources of data that are designed to assess the performance of teaching candidates as they progress through our program.

Collect data in three areas.

1) Throughout the Program

Continuous assessment of teaching candidates throughout the program from entrance, midpoint, and exit Grades GPA Praxis scores Rubrics Reflections

2) In Clinical Field Experiences

Assessment of pedagogical knowledge and skills that occurs during clinical field experiences; Placement Observation forms

3) Completion of Program

Perceptions of teaching candidates and alumni after they have completed their program, which is used for program improvement Alumni Surveys Exit Surveys Focus Groups

Traditionally….Data Deficient

Schools of Education have not been collecting data systematically

Infrastructure not set up Not able to access multiple

sources of information

Traditionally…..Data Dummies

What data do we want to collect? How can we manage it? What do we do with data? How do we organize it? Access it? Make sense?

Currently….Data-Driven

Systematically collecting data Infusing into our faculty culture Meeting regularly to assess evidence Making decisions

based upon evidence Beneficial process

due to TEAC

Data-Driven Decision-Making

Plan

CollectData

Impl

emen

tA

nalyze

Data

Reflect

& Revise

Example of Data Collection:Praxis Results 2005/2006

Program FDU SOE Pass Rate

NJ Pass Rate

Elementary Ed Content Knowledge 99% 84%

English Lang Lit Content Knowledge 100% 71%

Social Studies Content Knowledge 100% 64%

Secondary: Math Content 100% 92%

Middle School Math 100% 65%

General Science Content Knowledge 100% 78%

Spanish Content Knowledge 100% 87%

Mean Scores of Teaching Candidates on Selected CCI Indicators Related to Pedagogy as Rated by Supervisors (Fall 2005, Spring 2006)

Midpoint Final

Indicator N Mean Standard Deviation

N Mean Standard Deviation

2.2 Creates and implements lessons that are developmentally appropriate.

57 4.47 .630 51 4.75 .483

3.2 Effectively incorporates multicultural information and strategies when appropriate into the lesson; presents issues from a multicultural perspective.

57 2.82 2.010 51 3.59 1.846

4.1 All essential components of a well designed plan for coherent instruction.

57 4.58 .533 51 4.71 .672

4.2 Appropriate instructional objectives that are clearly stated, measurable and aligned with the NJCCCS.

57 4.42 .706 51 4.67 .476

4.3 Effective use of a wide range of resources including technology to enhance instruction and student learning.

56 3.86 1.420 51 4.39 .918

4.4 Demonstrates understanding of curriculum design and implementation, including various curricular approaches (i.e. interdisciplinary, integrated, thematic).

57 3.79 1.278 51 4.12 1.336

Wanted: Database Administrator

New Job Description: full-time professional staff

Resources Support from administrators Ensure candidate had appropriate

knowledge and skills to design database

What Is a Database System? A collection of data organized in tables,

which can be accessed and manipulated, without having to restructure the tables

Elements of a Database System• A storage system • Data structures• Manipulation tools

Database Advantages Analyze sophisticated correlations easier

because relationships are established between data sets

Make decisions based on information derived from data

Streamline business operations Organize data and eliminates:

• Inconsistent data• Missing data• Redundant data

Problem #1: Field Placement Office

Staff was overwhelmed with managing Clinical Placements

In 2007 800+ letters were mailed to 424 schools asking for placements

Previously, these letters were individually prepared in MS Word documents

Problem #1: Field Placement Office (cont’d)

Clinical Placements must be coordinated with• School• Student• Supervisor

Difficult to aggregate data• What districts have most confirmed/declined rates?• What trends are we seeing?• What kind of schools are we sending our candidates to?

Solution #1: Streamlined Field Placement Office Developed a data collection system for

Clinical Placements Data is:

• Entered on 2 campuses• Used to create personalized communications to

schools, students, and supervisors• Used to manage Clinical Placements

• Confirmed/Pending/Declined• Supervisor Assignments

Solution #1: Streamlined Field Placement Office (cont’d)

Gives us the ability to aggregate data, look at trends, and make decisions Confirmed / Declined distribution by district Analyze demographics of cooperating school

districts

Solution #1: Infrastructure

We start with the person record from the university system

Data on the Clinical Placement is entered

Solution #1: Generating letters

Generate standardized reports for “master lists”

Solution #1: Payroll

Problem #2: What Type of Districts Do

We Place Our Candidates Into?

The District Factor Group (DFG) is a socioeconomic indictor used for comparative test reporting of school districts for New Jersey’s statewide programs.

Problem #2: What Type of Districts Do We Place Our Candidates Into? (cont’d)

DFG Factors: % of adult residents failed to complete high school % of adult residents who attended college Occupational status (laborers, service workers, farm

workers, professionals, etc.) Population density Income Unemployment rate Poverty rate

Problem #2: What Type of Districts Do We Place Our Candidates Into? (cont’d)

Eight DFGs have been created based on the 1990 United States Census data

Range from A (lowest socioeconomic district) to J (highest)

A, B, CD, DE, FG, GH, I, J

DFG: State DistributionDistrict Factor Group Distribution

For All Districts In NJ

0

20

40

60

80

100

120

A B CD DE FG GH I J

DFG: Apprenticeship Teaching Distribution

Distribution For Apprenticeship Teaching In Spring 2007

0

5

10

15

20

25

30

35

District Factor Groups

Fie

ld P

lac

emen

ts

A B CD DE FG GH I J

DFG: Alumni DistributionReported Distribution of Working Alumni

0

2

4

6

8

10

12

14

16

18

District Factor Groups

Alu

mn

i

A B CD DE FG GH I J

Solution #2: Share Data, Discuss, Revise Evident discrepancy between where alumni get

jobs and where candidates are placed We share evidence with faculty and key

stakeholders They discuss and make appropriate decisions

DFG: In the future

Will have full record of where candidates performed clinical experience

Will have record of where they are working

Can correlate accordingly

Data Needed

A state database of teacher employment Difficult to track alumni as they move from

school to school Unique teacher & school identifier

State database needs to integrate with University and commercial marketing data systems

Problem #3: Data Systems Not Integrated

SOE’s recordkeeping is not integrated with the University system Student information is entered into SOE

system manually Limits power of reporting Duplicate person records may exist if Student

ID is not entered correctly

Problem #3: Data Systems Not Integrated (cont’d)

A “live” data connection to University system is not possible Technology is not in place Support for integration is needed

More Efficient and Effective Use of Resources

Relational database assists with: Streamlined SOE business operations

Generate mail merge letters Provide reports Automate payroll

Leverages existing data to create information for program improvement

Started Slowly

Started with trying to code and track our students properly

Administrative assistant created rudimentary Access Database

When she left, we hired a consultant to manage database

He totally redesigned and reorganized it

Skill Set for Database Administrator Problem-solving Relational database design skills SQL proficiency Knowledge of structured programming

language Excellent communication skills Work with faculty and technical staff “People-skills”

Conducted Extensive Search Advertised Set up search committee Interviewed many different applicants Required each applicant to take a test Presented problems to candidates to

assess problem-solving abilities Found many could enter data, but not

design or problem solve

Working with the University

Collaborating with the Arts and Sciences Establishing knowledge-base in content

areas and general education Addressing NJCCCS and Professional

Standards in discipline Aligning content courses and standards in

matrices

Working with the University(cont’d)

Meeting with individual departments Establishing long-term relationships Shifting to new paradigm--Learning Outcomes

Assessment Working with A&S to collect data Database Administrator playing key role in

collection of data across college

Leveraging TEAC Across the University Establishing the need for LOA: Middle States Educating the A&S faculty: LOA process Addressing resistance of A&S faculty Establishing a relational database system for

university: program, college Creating the infrastructure to collect data Collecting multiple sources of data for A&S Getting various groups to communicate and

plan.

Conclusion: The Power of a Relational Database

Data-driven Decision-making requires

an integrated system of collecting data from many different sources.

Systemic The whole institution needs to be

vested in the collection of data Data needs to be collected on faculty,

students, courses, grades, scores, rubrics, observations

University and Colleges need to ensure data collection systems are in place early.

Integrated

All data sets need to be connected so relationships can be established

Queries made Reports generated Correlations and relationships analyzed

Benefits for Total University Middle States Accreditation Nursing Engineering College of Business Program improvement Student learning

School of Education Leads the Way

Ultimate goal is to improve teacher quality and impact achievement for all students.

Data provides the means to do this. Relational database is the engine

that makes this possible.

For more information please contact:

Vicki L. Cohen cohen@fdu.eduDirector, School of Education

Marlene Rosenbaum rosenbau@fdu.eduAssociate Dean, University College

Joshua Cohen cohenjp@fdu.eduDatabase Administrator