Measuring confidence to teach statistics to

middle & high school grades:

The development & validation of the SETS instruments

The Research Team

Leigh M. Harrell-Williams M. Alejandra SortoVirginia Tech Texas State Georgia State University University

Rebecca L. Pierce Lawrence M. Lesser Teri J. MurphyBall State University The University of Texas Northern Kentucky

at El Paso University

Supported in part by CAUSE (under NSF DUE #0618790)

Why measure middle and high school grades pre-service teachers’

Self-Efficacy to Teach Statistics?

• Teachers who are prepared to teach mathematics are expected to teach statistics as outlined by the PreK-12 Guidelines for Assessment and Instruction in Statistics Education (GAISE) (Franklin et al., 2007) and the Common Core State Standards for Mathematics (CCSSM) (National Governors’ Association, 2010).

• Self-efficacy is task specific; any instrument that assesses self-efficacy needs to be task specific as well.

Self-Efficacy, Attitudes, and Beliefs

Teacher efficacy affects:- teacher motivation- willingness to use more innovative techniques- student achievement- time spent teaching certain concepts(Czerniak, 1990; Riggs & Enochs, 1990; Wenta, 2000).

Existing Instruments:- attitude towards statistics (SATS, ATS)- efficacy for learning/doing statistics (CSSE, SELS)- statistical knowledge (SCI)

No prior instrument measures self-efficacy for future teachers.

Three Developmental

Levels

A Five

Objectives

B Six

Objectives

C Nine

Objectives

GAISE PreK-12 Curriculum Framework

Process Components

1. Formulate Question2. Collect Data3. Analyze Data4. Interpret Results

“Although these three levels may parallel grade levels, they are based on

development in statistical literacy, not age. Thus, a middle-school student who

has had no prior experience with statistics will need to begin with Level A concepts and activities before moving to

Level B.” (p. 13)

CCSSM Framework

VERSUSGrade-Specific

Standards for Each Grade Level

Topic-Specific Standards

Similar to the 4 processes in the GAISE, there are 8 mathematical practices that are threaded throughout the CCSSM.

Identified representative

behaviors from GAISE items

Determined alignment of GAISE

report to state standards

Draft items created for instrument using

language aligned with GAISE and state

standards

Development Process for Middle Grades SETS Instrument

Spring/Summer 2008

Revised item wording based on input from

practicing elementary & middle school

teachersFall 2008/Spring 2009

Pilot Study 2009

Data Collection Study for Validation Purposes

2010 - 2011

Development Process for Middle Grades SETS Instrument

The Middle Grades SETS Instrument

26 Likert scale items in this format:

Please rate your confidence in teaching middle grades students the skills necessary to complete the following tasks successfully:

Scale of 1 to 6

14 Demographic items

Statistics Standards in the Common Core, Grades 2 - 6

Grades 2 - 5:Represent and interpret dataCollecting measurements and creating line plot and bar graphs

Grade 6: Develop understanding of statistical variability: • Recognize statistical question• Data has distribution with specific

center/variation/shapeSummarize and describe distributions• Create boxplots/histograms• Summarize data numerically

Selected SETS Items based on GAISE Level-A

12 items

• Collect data to answer a posed statistical question in contexts of interest to middle school students.

• Recognize that there will be natural variability between observations for individuals.

• Select appropriate graphical displays and numerical summaries to compare individuals to each other and an individual to a group.

Statistics Standards in the Common Core, Grades 7 - 8

Grades 7: Use random sampling to draw inferences about a population.• Inferences can be made from sample about

population if sample is representative.• Generate multiple samples to gauge accuracy.Draw informal comparative inferences about two populations.• Use graphs to estimate differences.• Use numerical values to assess differences.

Grade 8: Investigate patterns of association in bivariate data. • Construct and interpret scatterplots.• Informally fit, assess and interpret a linear

relationship.• Use contingency table to evaluate relationship.

Selected SETS Items based on GAISE Level-B

15 items

• Recognize the role of sampling error when making conclusions based on a random sample taken from a population.

• Recognize that a sample may or may not be representative of a larger population.

• Recognize sampling variability in summary statistics such as the sample mean and the sample proportion.

• Use interquartile range, five-number summaries, and boxplots for comparing distributions.

• Interpret measures of association.

2010 – 2011 Validation Study

• Four US public institutions of higher education with significant proportion of students pursuing degrees in education

• 309 participants enrolled in either an intro statistics course or a math education course

Validation Study - Methods

• Confirmatory Factor Analysis

• Item Analysis

• Rating Scale Analysis

• Reliability

Validation Study - CFA

• Compared unidimensional and two-dimensional factor structure using Multidimensional Random Coefficient Multinomial Logit Model as implemented in Conquest software

• Two dimensions: (Friel, Curcio, & Bright, 2001)– Efficacy to Teach “Reading the Data”– Efficacy to Teach “Reading Between the Data”

• AIC and BIC confirmed two dimension structure• 0.85 between-dimension correlation

Validation Study – Item AnalysisIndex Statistic Composite Reading the Data Reading Between

The Data

rpoint-polyserial Mean 0.66 0.67 0.69

SD 0.05 0.06 0.05 Min 0.55 0.59 0.62 Max 0.73 0.73 0.77 Nitems 26 11 15MSweighted Mean - 1.00 1.00 SD - 0.13 0.23 Min - 0.71 0.71 Max - 1.17 1.52 NiMS > 1.5 - 0 1

MSunweighted Mean - 1 0.98

SD - 0.12 0.21 Min - 0.78 0.72 Max - 1.18 1.47 NiMS > 1.5 - 0 0

Validation Study – Rating Scale Analysis

Criterion Met? Essential?

N > 10 for each response category Yes Yes

Unimodal distribution for each response category Yes Yes

Average measures increase with category Yes Yes

Outfit MNSQ < 2 for each category Yes Yes

Category thresholds increase with category Yes No

Measure implies Category & Category implies Measure (Coherence) No No

Category thresholds increase by 0.81 logits No No

Category thresholds don’t increase by more than 5 logits Yes No

Validation Study - Reliability

Reliability of Separation - Analogous to Cronbach’s Alpha

Composite Score: 0.94

Subscales:“Reading the Data” (Level A): 0.87 “Reading Between the Data” (Level B): 0.91

The High School Grades SETS Instrument

• Items completed Spring 2012• Based on both GAISE (all levels) and two of the four

strands for CCSSM for High School Statistics & Probability• Interpreting Categorical & Quantitative Data• Making Inferences & Justifying Conclusions

• Data collection• In-service: Summer 2012• Pre-service: Fall 2012 & Spring 2013

• Analysis during Spring/Summer 2013

The High School Grades SETS Instrument

44 Total items:• 26 items from Middle Grades SETS instrument• 18 “new” items based on level C of GAISE and the two

strands of the CCSSM

Item format:• Please rate your confidence in teaching high school students the

skills necessary to complete the following tasks successfully:• Scale of 1 to 6

14 Demographic items

CCSSM “Making Inferences & Justifying Conclusions” Strand

Understand and evaluate random processes underlying statistical experiments• S-IC.1. Understand statistics as a process for making

inferences about population parameters based on a random sample from that population.

• S-IC.2. Decide if a specified model is consistent with results from a given data-generating process, e.g., using simulation.

Make inferences and justify conclusions from sample surveys, experiments, and observational studies• S-IC.3. Recognize the purposes of and differences

among sample surveys, experiments, and observational studies; explain how randomization relates to each.

• S-IC.4. Use data from a sample survey to estimate a population mean or proportion; develop a margin of error through the use of simulation models for random sampling.

• S-IC.5. Use data from a randomized experiment to compare two treatments; use simulations to decide if differences between parameters are significant.

• S-IC.6. Evaluate reports based on data.

Selected SETS High School Items Based on

GAISE Level C and CCSSM

• Evaluate whether a specified model is consistent with data generated from a simulation.

• Explain the role of randomization in surveys, experiments and observational studies.

• Estimate a population mean or proportion using data from a sample survey.

• Evaluate how well the conclusions of a study are supported by the study design and the data collected.

Potential Uses

• Research

• Assessment of Teacher Preparation Programs

• Analysis of Need for In-Service Professional Development Programs

Current Projects Using SETS Instruments

Jean LinnerLassiter High School & GaDOE/GCTM Academy

“Recognizing that low teacher efficacy can inhibit effective teaching as well as student learning, we hope to use the SETS instrument for high school in a longitudinal study to identify and target those professional learning experiences that increase teacher efficacy for teaching statistics.”

Current Projects (continued)

Stephanie Casey & Andrew RossEastern Michigan University

Scores from the SETS for pre-service secondary mathematics teachers enrolled in a statistical methods course were used to assess the pre-service teachers' confidence in teaching statistical concepts in general. Additionally, scores were used to compare the treatment (reform-oriented instruction) section and the control (traditional instruction) section. The results will be used to inform the instruction of the course and assess whether there was a significant difference between the treatment and control groups with respect to their self-efficacy for teaching statistics.

Current Projects (continued)

Salome Martínez & Eugenio Chandía Universidad de Chile & Pontificia Universidad

Católica de Chile

The government of Chile is funding a curriculum project for the preparation of future elementary teachers. The curriculum consists of a series of textbooks for all the content areas, including the Data and Chance strand of the national standards. The SETS instrument will be used to evaluate the impact of the implementation of the textbooks in 10 Chilean institutions of teacher preparation.

Information on Using SETS Instruments

E-mail Rebecca Pierce at rpierce@bsu.edu

to request “Terms of Use” form

Thank you for attending our webinar!

We’d like to open the floor to discuss the following:• How would you envision using one of the SETS

instruments? • What else would you like to know about the

SETS instruments?

Related Papers Hilton, S., Kaplan, J., Hooks, T., Harrell, L. M., Fisher, D., & Sorto, M. A. (2008).

Collaborative projects in statistics education. In Proceedings of the 2008 Joint Statistical Meetings, Section on Statistical Education (pp. 752-756). Alexandria, VA: American Statistical Association.

Harrell, L. M., Pierce, R. L., Sorto, M. A., Murphy, T. J., Lesser, L. M., & Enders, F. B. (2009). On the importance and measurement of pre-service teachers’ efficacy to teach statistics: Results and lessons learned from the development and testing of a GAISE-based instrument. In Proceedings of the 2009 Joint Statistical Meetings, Section on Statistical Education (pp. 3396-3403). Alexandria, VA: American Statistical Association.

Sorto, M. A., Harrell, L. M., Pierce, R. L., Murphy, T. J., Enders, F. B., & Lesser, L. M., (2010). Experts’ perceptions in linking GAISE guidelines to the self-efficacy to teach statistics instrument. In Proceedings of the 2010 Joint Statistical Meetings, Section on Statistical Education (pp. 4289-4294). Alexandria, VA: American Statistical Association.

Harrell-Williams, L. M., Sorto, M. A., Pierce, R. L., Lesser, L. M., & Murphy, T. J. (Under Review). Validation of Scores from a New Measure of Pre-service Teachers’ Self-efficacy to Teach Statistics in the Middle Grades.

REFERENCES

Czerniak, C. M. (1990). A study of self-efficacy, anxiety, and science knowledge in preservice elementary teachers. Paper presented at the National Association for Research in Science Teaching, Atlanta, GA.

Enochs, L.G., Smith, P.L., Huinker, D. (2000). Establishing factorial validity of the mathematics teaching efficacy beliefs instrument. School Science and Mathematics, 100, 194-202.

Finney, S. J., & Schraw, G. (2003). Self-efficacy beliefs in college statistics courses. Contemporary Educational Psychology, 28, 161-186.

Franklin, C., Kader, G., Mewborn, D., Moreno, J., Peck, R., Perry, M., & Scheaffer, R. (2007). Guidelines for Assessment and Instruction in Statistics Education (GAISE) Report: A Pre-K-12 Curriculum Framework. Alexandria, VA: American Statistical Association. (Also available at http://www.amstat.org/education/gaise/)

Friel, S. N., Curcio, F. R., & Bright, G. W. (2001). Making sense of graphs: Critical factors influencing comprehension and instructional implications. Journal for Research in Mathematics Education, 32(2), 124-158.

REFERENCES (continued)

Lutzer, D., Rodi, S., Kirkman, E., & Maxwell, J. Statistical Abstract of Undergraduate Programs in the Mathematical Sciences in the United States, Fall 2005, CBMS Survey, American Mathematical Society, Providence, R.I., 2007.

National Governors Association (2010). Common Core State Standards for Mathematics. http://www.corestandards.org/assets/CCSSI_Math%20Standards.pdf

Riggs, I. M., & Enochs, L. G. (1990). Toward the development of an elementary teacher's science teaching efficacy belief instrument. Science Education, 74(6), 625-637.

Watson, J. (2001). Profiling Teachers’ Competence and Confidence to Teach Particular Mathematics Topics: The Case of Chance and Data. Journal of Mathematics Teacher Education, 4, 305–337.

Wenta, R. G. (2000). Efficacy of preservice elementary mathematics teachers. Unpublished doctoral dissertation, Indiana University.