ReadiStep Summary of Answers and Skills (SOAS) Tutorial SOAS... · The SOAS information is useful to many educational professionals in your building. Certainly, administrators, department

ReadiStep Summary of Answers and Skills (SOAS) Tutorial

Audience

• Administrators

• Department heads

• Curriculum specialists

• Teachers

• Other staff interested in students’ college preparatory skills

Note: the more inclusive the testing within each grade, the more meaningful and applicable the results.

SOAS Introduction

Presenter

Presentation Notes

The SOAS information is useful to many educational professionals in your building. Certainly, administrators, department chairpersons, curriculum specialists, and teachers will be interested in the skill analyses provided in the reports. As integral contributors to the teaching and learning community, counselors are often the professionals who introduce the SOAS to their colleagues. In addition, we can link conversations about school-wide learning goals to student performance gains as demonstrated by ReadiStep. A Tip: Inclusive testing gives the best results. The more student data you provide, the more informative your feedback will be. The SOAS reports are grade specific; therefore the reports are listed as Class of 2016, Class of 2017, etc. You may also view reports by multiple school terms. An important footnote for these reports – they are generated only if you had 10 or more students test within a grade level. Many of the content and rules for math and grammar are taught at the middle school level. The SOAS workshop can be a great opportunity for a vertical teams meeting with middle and high school teachers. Consider waiting until SOAS Year Two if these groups are not used to meeting collaboratively.

Logging In

Scores.collegeboard.org

Presenter

Presentation Notes

If you’ve accessed SOAS reports before, just login with your username and password to see this year’s reports at scores.collegeboard.org.

Your Reports

What do these numbers mean?

Grade_School Code_Test

2014-15

Presenter

Presentation Notes

Once you’ve successfully logged in, you’ll see tabs at the top of the page for each Program you have reports for. The tabs will only appear once you’ve submitted your access code for each of the individual program. You will see a “row” for each of your grade level reports. In the slide, you’ll see reports for 7th grade, 8th grade, and 9th grade. Links to your school’s reports are in the box. Click on the plus sign to show all the reports available to you. Each link is a separate report for individual grade levels. Reports are provided if your school tested 10 or more students in a grade.

SOAS Resources

Supporting Materials

Presenter

Presentation Notes

Before we dive into the report details, notice that there are several resources available to help you maximize use of the SOAS reports. From the SOAS website, there are several online resources available in the Test Questions and Report Tools tabs. The resources include a Skills Reporting Tools video, ‘how-to’ Tutorials, the ReadiStep Skills List, alignment reports linking ReadiStep Skills to Common Core and individual state standards, and information on College and Career Readiness Benchmarks.

Page(s)

• Title Page

• Performance Overview 1

• Skills Analysis 2, 6-7, 11

• Question Analysis 3-4, 8-9, 12-13

• Comparable Group Analysis 5, 10, 14

Report Orientation

Table of Contents

Presenter

Presentation Notes

Let’s go through a SOAS report. Each report is 15 pages long. It is helpful to think of one report as consisting of an introductory Performance Overview, and 3 different presentations of data: Skills Analysis, Question Analysis, and Comparable Group Analysis. The three presentations of data – Skills Analysis, Question Analysis, and Comparable Group Analysis – will be repeated for each section of ReadiStep: Critical Reading, Math, and Writing.

• Number of students in report

• Mean scores and

score distribution

• College and Career Readiness Benchmark

Performance Overview

Page 1

Presenter

Presentation Notes

The Performance Overview shows the number of students included in the report, mean scores and score distribution, and the number of students who have met or exceeded the new College and Career Readiness Benchmark- an indicator of whether students are on track for success in college.

Pages: 2, 6-7, 11

Skills Analysis

• Skills

• Triangles show a comparison to the state and nation

• Each test question is linked to a skill

Presenter

Presentation Notes

The Skills Analysis tables (repeated on pages 2, 6-7, and 11) show how your school performed on each of the skills measured by ReadiStep. These same skills are measured by the PSAT/NMSQT and SAT. You can see how your students’ performed compared to other students in the state and the nation. And, for each test question, you can see the percent of students who answered the question correctly, and whether the question was easy, medium, or hard.

Instructions

• Work within your own school group/academic area, using your school’s SOAS report.

• Identify the skill that your students had the lowest performance on.

• Using the ReadiStep test booklet, look at each question aligned to that skill.

• Things to look for:

Skills furthest below the state/national average.

Skills that are furthest to the left (i.e. those that are the weakest for the group).

Skills that came to your attention after review of Question Analysis.

Activity A Review Skills Analysis

Questions for Consideration

Within your group, please answer the following:

1. Is there any pattern in the skills you identified as problematic?

2. Are there actions or strategies that might help address where students are struggling with certain skills?

3. What skills are relatively strong for your students? Why?

Activity A Review Skills Analysis

Question Analysis

• Student responses and answer patterns

• Comparisons to the state and nation

Presenter

Presentation Notes

The Question Analysis tables (repeated on pages 3-4, 8-9, and 12-14) show student responses and answer patterns for each test question. You can also see how your students’ performed compared to other students in the state and the nation. Correct answers are displayed in gray, and the responses for the incorrect and omitted answers and displayed to the right.

Tips:

Question Analysis

Look for questions your students answered incorrectly that the students in the state and nation answered correctly

Look for commonly wrong answers

Presenter

Presentation Notes

When first looking at the report, there are 2 pieces of information to pay attention to: First, look for questions your students answered incorrectly that the students in the state and nation answered correctly. Second, look for commonly wrong answers. The “eyeball” technique works well because concerns about your students’ performance can readily be seen by the length of the gray bar labeled “School.” When a large percentage of your students answered the question correctly, the gray bar will be quite long. When your students had trouble with a question, it is interesting to note which “incorrect response or responses” attracted them. Notice question 15 where only about 15% of the students selected the correct answer. Also notice for question 15 that many students chose answer C, though A is correct. You can look at the question 15 and the skills associated with it to identify areas for improvement. Another way to identify problem areas is to look for questions your students got wrong but students in the state and national answered correctly. You can identify the skills measured in these questions and target these skills for improvement. There is a lot of information to be curious about on this page. How did my students do? When missing a question, did they gravitate toward the same wrong answer? Or did they not? If a large percentage of students chose the same wrong answer, what was so attractive about that answer? What questions did students omit? This opportunity to observe how students reacted to individual questions offers a unique perspective on your students’ demonstration of college readiness skills.

Instructions

• Work within your own school group/academic area, using your school’s SOAS report.

• Refer to the test booklet to make note of problem questions and how they might be addressed.

• Things to look for:

Easier questions where fewer than 2/3 of the students answered correctly.

A larger proportion of your students answered incorrectly or omitted an answer in comparison to the state/nation.

Commonly wrong answers.

Review any questions that generated interest when you reviewed the test questions in Activity A.

Activity B Review Question Analysis

Activity B Review Question Analysis


Within your group, please answer the following:

1. Are there any patterns in the questions you identified as problematic?

2. Are there actions or strategies that might help students master the skills assessed by these questions?

3. What are the key highlights from your discussion that your group would like to share? Please include both strengths and areas of concern.

Pages: 5, 10, 15

Comparable Group Analysis

Presenter

Presentation Notes

The third section is the Comparable Group Analysis (repeated on pages 5, 10, 15). It allows you to easily ‘eyeball’ the test questions where your students performance diverged from what was expected. The data is produced based on a statistical model called Comparable Group.

What is a comparable group?

• A useful statistical model

• A statistically created group (virtual group)

• Mirrors your group’s performance profile

• This creates an expected performance indicator for your group on each question

• Provides more “actionable” feedback than state or national averages on questions/skills

SOAS Comparable Group

Presenter

Presentation Notes

Let's clarify what the comparable group is. It is a useful model, and is a statistically created group (a virtual group if you will) which mirrors your group’s performance profile. The comparable group is not a real school; rather it is a sample group of students across the entire test-taking population that was chosen because they performed similarly to students at your school. No demographic data is factored in - the comparable group is created solely based on student performance. The comparable group is a statistical model used to create an expected performance indicator on each question providing more actionable feedback than state or national averages on questions and or skills.

Pages: 5, 10, 15

Comparable Group Analysis

Darker blue =

significantly below the comparable group

Lighter blue = significantly above the comparable group

Presenter

Presentation Notes

Back to the report. The table allows you to easily find test questions for further investigation. Look for questions where the dot falls in the darker blue and lighter blue shaded area. These are questions where your students’ performance diverges from what would be expected.

Instructions

1. Work within your own school group/academic area, using your school’s SOAS report.

2. Using the test booklet, look at each question.

3. Things to look for:

Questions that fall into the dark blue area, where your students performed below expectation.

Questions that fall into the light blue area, where your students performed higher than expectation.

Activity C Comparable Group Analysis


Within your group, please answer the following questions:

1. Is there a pattern in the skills aligned with the test questions on which your students performed below expectation?

2. Is there a pattern in the skills aligned with the test questions on which your students performed above expectation?

3. What new information, if any, have you learned from this analysis? Has this changed your thoughts about anything that your group previously discussed? Why or why not?

Activity C Comparable Group Analysis

Now that you’re familiar with the data

the SOAS report provides, we’ll walk through

an example of how to use SOAS reports

using a Critical Reading test question.

20

How does it all add up?

Step 1 of 3: Use reports to find a question to analyze.

Critical Reading Question 17


29% of student s at this school answered correctly, compared to 41% of 8th graders in the state and nation.

Step 2 of 3: Identify the skill measured by this question.



Step 3 of 3: Review the question.



17. The final sentence of the passage is best described as an example of (A) sarcasm (B) simile (C) hyperbole (D) symbolism

Tips for Using Your SOAS Data

• Identify the skills associated with the questions students answered incorrectly.

• Make inferences about what led to errors.

• Identify where in the curriculum the skills are (or are not) taught.

• Work with departments and classrooms to review common concerns.

• What recommendations can you make to your school for increased student success?

Looking Ahead: Check the ReadiStep website for upcoming administration dates and deadlines. readistep.collegeboard.org

Documents

ReadiStep Summary of Answers and Skills (SOAS) Tutorial SOAS... · The SOAS information is useful to many educational professionals in your building. Certainly, administrators, department