Statistical Modelingfor Education Planning

randy.raphael@schools.utah.govwww.schools.utah.gov/financeURBPL 5/6020 / April 19, 2007

Who We Are

Utah State Office of Education– Staff to the State Board of Education

Financial and Business Services Division Finance and Statistics Section

What We Deal With

Populations– Students– Staff– Schools

Finance– Minimum School Program (MSP) Budget & NCLB Allocations– Financial Reporting & Auditing– Property Tax

Operations– School Facilities– Student Transportation– Safety

How We Do It

Acquire– Data

Allocate– Money to local education agencies according to

data

Audit– For accuracy of data and appropriateness of

expenditures

Analytics Cycle

Population (Enrollment) Projections– How many people do we need to serve?

Fiscal Impact Analysis– How much will service options cost?

Formula Allocation– How do we get the right amount to the right place?

Compliance Audit– How well did service providers follow the rules?

Program Evaluation– How well did we serve the population?

Enrollment Projections:Institutional Context

Common Data Committee– Legislative Fiscal Analyst– Governor’s Office of Planning and Budget– Utah State Office of Education

Current Work– By county (then allocate to districts and adjust for charter schools)– In October– Single year (to next October)– Agreed upon figures for legislative session

Future Plans– Multiyear project with GOPB using REMI for Baseline 2008

Enrollment Projections:Model

Cohort ProgressionParticipation Ratio

– Kindergarten subset

Enrollment Projections:Data

Historical Variables– Total Enrollment (Current and Prior Years)– Grade 12 Enrollment (Current and Prior Years)– Kindergarten Enrollment (Current Year)– Births (by Month, 4- and 5-Years Prior)

Intermediate Variable– Projected Kindergarten Enrollment

Enrollment Projections:Formula

Formula Element

EY+1

= EY Base Population

+ (BY-4 * (KY / BY-5 )) - GY) Cohort Progress.

+ (EY - EY-1) - (KY - GY-1) Implied Migration

Fiscal Impact Analysis:Example

HB 222 (2002)– “Make recommendation on … the ideal size of

schools districts in this state …”

Optimization Problem Cost Function

– Relates expenditures per student to enrollment (main cost driver) controlling for academic achievement (output measured in quality)

Fiscal Impact Analysis:Design

Sample: Cross section of 40 Utah school districts

Data: Superintendent’s Annual Report, 2000-01

Model: Y = m + (b1X + b2X2) + b3Z + e

Procedure: OLS regression

Fit (adj R2): .29

Predictors: Enroll Enroll2 Lexile

Coeff (b): -.138 .0000016 -665

Sig (p): .01 .05 .03

Fiscal Impact Analysis:Results

Empirical Cost Function– exp = .0000016enr2 - .138enr – 665lex + 6,468

Differentiated and Set Equal to Zero– 0 = .0000032enr - .138

Solution is Optimal Size– enr = .138/.0000032 = 43,125 students

Fiscal Impact Analysis:Politics (TTC, SL Tribune 2/23/04)

Columbia University professor’s critique:– “I’d be happy to go with the [USOE] analysis rather than the

fiscal analyst’s, which is opaque to the point of incomprehensibility”

Fiscal analyst’s defense:– “Anybody’s guess is as good as the next person’s”

Opponents’ critique:– “Foes have long accused the fiscal analyst’s office of

working the numbers to achieve a favorable outcome” Fiscal analyst’s concession:

– “At the outset, the intention is to have it come out in a positive way so there’s not a cost”

Fiscal Impact Analysis:Ethics

Substantive claims must be warranted by evidence

Production of evidence must be based on transparent procedures

Allocation Formulas:Minimum School Program (1)

“The purpose of this chapter [Utah Code 53A-17a] is to provide a minimum school program (MSP) for the state in accordance with the constitutional mandate.”

“It recognizes that all children of the state are entitled to reasonably equal educational opportunities regardless of their place of residence in the state and of the economic situation of their respective school districts or other agencies.”

NOTE: Overriding concern with equity; adequacy is another issue of growing legal importance, but its operationalization is very unclear.

Allocation Formulas:Minimum School Program (2)

“It further recognizes that although the establishment of an educational system is primarily a state function, school districts should be required to participate on a partnership basis in the payment of a reasonable portion of the cost of a minimum program.”

NOTE: Utah sources of revenue (FY 2006):– State 55% from income tax– Local 36% from property tax– Federal 9% from who knows where

Allocation Formulas:Minimum School Program (3)

“Each locality should be empowered to provide educational facilities and opportunities beyond the minimum program and accordingly provide a method whereby that latitude of action is permitted and encouraged.”

NOTE: Local school boards can impose several additional property taxes for specified educational purposes.

Allocation Formulas:Budgeting for Basic Program (1)

Majority of funding is based on “Prior Year + Growth” formula

Prior year is Average Daily Membership (ADM)

Growth is percentage difference between projected Fall Enrollment and current year Fall Enrollment

Hold harmless in case of negative growth

Allocation Formulas:Budgeting for Basic Program (2)

Result is number of Weighted Pupil Units (WPUs), a quantification of the basic service which each local education agency is obligated to provide

Legislature sets monetary value of WPU every year

Total WPUs times WPU $ value determines basic appropriation

Allocation Formulas:Budgeting for Basic Program (3)

If LEA property tax revenue cover its obligation, then buck stops there; otherwise, state pays balance from income tax revenue

In practice, all LEAs need some state assistance and appropriations often fall short, so funds are prorated according to WPUs

Since FY 2001, K-12 funding has approximately kept pace with inflation

Allocation Formulas:Categorical Programs

In addition to the basic program, the Legislature has established dozens of categorical programs to address particular concerns

Cost drivers of categorical programs can be quite complex — “Special Education Add On” is an especially striking example of what can happen when trying to reconcile competing interests through a funding program

Allocation Formulas:Categorical Program Example (1)

Per WPU, which is the greater of the average of Special Education (Self Contained and Resource) ADM over the previous 5 years (which establishes the foundation [hold harmless] below which the current year WPU can never fall) or prior year Special Education ADM plus weighted growth in Special Education ADM.

Allocation Formulas:Categorical Program Example (2)

Weighted growth is determined by multiplying Special Education ADM from two years prior by the percentage difference between Special Education ADM two years prior and Special Education ADM for the year prior to that, subject to two constraints:

Allocation Formulas:Categorical Program Example (3)

Special Education ADM values used in calculating the difference cannot exceed the prevalence limit of 12.18% of total district ADM for their respective years.

If this measure of growth in Special Education exceeds current year growth in Fall Enrollment, growth in Special Education is set equal to growth in Fall Enrollment (incidence limit).

Allocation Formulas:Categorical Program Example (4)

Finally, growth is multiplied by a factor of 1.53.

This weight is intended to account for the additional cost of educating a special education student.

However, the weight is not based on an empirical analysis of the cost of special education relative to "regular" education.

An Australian Approach:Victoria’s Principles (1)

Preeminence of Educational Considerations– Elimination of disparities reflecting historical and

political decisions for which there is no current or future educational rationale

Cost Effectiveness– Relativities among allocations should reflect

knowledge of efficient ways of achieving school and classroom effectiveness

An Australian Approach:Victoria’s Principles (2)

Fairness– Schools with the same mix of learning needs

should receive the same total resources; this requires accurate and comprehensive information on those student characteristics which best predict academic achievement

Transparency– Basis for allocations should be made public and

readily understandable by all with an interest

An Australian Approach:Victoria’s Principles (3)

Subsidiarity– Decisions on resource allocation should be made

centrally only if they cannot be made locally Accountability

– A school that has authority to make decisions on how resources will be allocated should be accountable for the use of the resources, including educational outcomes in relation to learning needs

An Australian Approach: Simple Budget Structure (1)

Core Funding– Grade Level– School Size

Student Disadvantage– Disabilities– Special Learning Needs– English as Second Language– Rurality and Isolation

An Australian Approach: Simple Budget Structure (2)

Facilities (operation & maintenance) Administration Costs outside of control of schools

– e.g., Transportation to and from school

“Priority” Programs– Money for politicians of the day to play with

An Australian Approach:Special Learning Needs

Sample (83 schools; 7,233 students) Hierarchical linear & Structural equation modeling Demographic index to predict achievement:

– Poverty (qualified for education welfare payment)– Parental occupation (skill level)– Language spoken at home (other than English)– Family composition (two parent, one parent, none)– Aboriginality (= Alaska Native or American Indian)– Transience (recently changed schools, = Mobility)

An Australian Approach:Reference

Hill, Peter W. (1996). Building equity and effectiveness into school based funding models: An Australian case study. 18p.

http://nces.ed.gov/pubs97/97535i.pdf

Compliance Audit:Purpose

Provide reasonable assurance that local education agencies are correctly applying State Board of Education rules in accounting for their students

Statistical summaries from individual data files serve as written management assertions

Auditors follow agreed upon procedures

Compliance Audit:Sampling

Efficient auditing depends on selection of sample appropriate to purpose

For example, if you want to adjust statistics based on audit, you need a probability sample

The right sample size then depends on:– Variation in the population– Risk you are willing to take of being wrong

Sample Size: The Price of Precisionpop = 80,000; mean = 154; sd = 25

90% 95% 99%

1% 703 1001 1718

5% 29 41 71

10% 8 11 18

Program Evaluation:Regression with Treatment as Dummy

“In the actual practice of applied social science, the most common mode of causal inference, the most common quasi-experimental design …” (Cook & Campbell)

Crucial to valid interpretation:– Specification of correct theory (of nonrandom

selection process) as represented by equation– (Near) perfect measurement of variables

Program Evaluation:Recommendation

Consider path analysis as extension of regression:– Explicit theory of how program works as a causal

mechanism in form of path diagram Consider multiple indicators of each variable:

– Use factor analysis to obtain composite measure representing only common variance

In short, poor man’s structural equation modeling

Program Evaluation:Path Diagram Example

Program Evaluation:Bibliography

Fitzpatrick, Sanders & Worthen (2004) Program Evaluation: Alternative Approaches and Practical Guidelines

– LB2822.75 .W67 Patton (1997) Utilization Focused Evaluation

– H62.5.U5 P37 Mohr (1995) Impact Analysis for Program Evaluation

– H97 .M64 Cook & Campbell (1979) Quasi Experimentation: Design and

Analysis for Field Settings– H62 .C5857

Scriven (1991) Evaluation Thesaurus– AZ191 .S37

Some Education Data Issues

Is a Navajo living in a hogan homeless? Kanab is on the urban fringe of which city? Who decides the racial identity of a student? When is a person who leaves school without

graduating not a dropout? Is being in a single parent family a reliable

indicator of being at risk of low academic performance?

Highly Impacted Schools Criteria:Factor Analysis

% of Enrollment Median Loading Median Loading

Ethnic Minority .93 .95

Limited English .91 .93

Free Lunch .80 .86

Single Parent -.49 -

Mobile -.60 -

R2 55% 82%

Data Sources

Digest of Education Statisticshttp://nces.ed.gov/programs/digest

NCES Tables & Figureshttp://nces.ed.gov/quicktables

USOE Assessement, Accountability & Divisionhttp://www.schools.utah.gov/eval

Utah State Superintendent’s Reporthttp://www.schools.utah.gov/finance/other/AnnualReport/ar.htm