Embed Size (px)
Citation preview
Extended Assessment Qualified
Assessor and Trainer Updates
Oregon Department of Education
Lenhardt, Brad
Carrizales, Dianna
Welcome
Introductions
Purpose of Training
Every student should have a fair chance or
equal opportunity to show what he or she
knows and can do. Common training
ensures: Reliability from one assessor to another
Reliability from one occasion to another
Reliability from one district to another
Validity that the results are an accurate reflection of
student knowledge and skill
Validity that the outcome is a reflection of what we
intended to measure
Qualifying to be an Assessor or
Trainer:
Purpose, Process, & Parameters
Qualified Assessor vs. Qualified Trainer
Qualified Assessor (QA)
Prepares materials and
setting for individual
administration of the
assessment
Administers assessments
directly to students
Scores student responses
fairly
Delivers scores to online
data entry system
Interprets results for
student, family, or
educational team
Qualified Trainer (QT) Prepares materials and setting for
individual administration of the assessment
Administers assessments directly to students
Scores student responses fairly
Delivers scores to online data entry system
Interprets results for student, family, or educational team
Serves as the local point person
Provides training and coaching to local Qualified Assessors
Remains fluent in updates and changes
Awards certificates*
Three Key Websites
Oregon Department of Education’s Extended Assessments
Website: http://www.ode.state.or.us/search/results/?id=178
Oregon Extended Assessment Training and Proficiency
Website: http://or.k12test.com/
Oregon Department of Education District Website:
https://district.ode.state.or.us/
Basic Training Options
NEW QTs Attend one of the
five live regional trainings
Returning QTs Participate in one of the three ―Refresher‖
Webinars or attend a live regional training
AND
Review updates and pass the refresher proficiency test on the training and proficiency website after mid-October
NEW QAs
Attend one of the
live trainings
hosted by a local
trainer who has
re-qualified for the
current school
year.
Returning QAs
Review updates and pass the refresher
proficiency test on the training and
proficiency website after mid-October
OPTIONAL: Get district approval to attend
one of the trainings hosted by a local trainer
(who has re-qualified for the current school
year)
State Capacity for Qualified Trainers
Last year: State intent (~target) for QTs was 300
Allocations were committed to districts based on this estimate
Number of QTs: 192 (190 in 2008-09)
Number of QAs: 1187 (1155 in 2008-09)
Number of students assessed: 5400+
Current year: Districts are responsible for ensuring capacity to assess all students
who will be taking the Extended Assessments
Funds allocated according to SECC amounts are similar to last year’s
In addition to assessment training, funds can be used toward the enhancement of assessment for students with disabilities
Qualification Requirements
(as a QT) Complete a regional training or refresh online*
Register on http://or.k12test.com/ Oregon Extended Training Website after October 19th
Practice test administration with a peer or volunteer-student to gain fluency with the assessment, the administration, and the scoring
Complete the proficiency assessments on Oregon Extended Training Website
Submit any retakes as necessary
Review the Oregon Extended Training Website to determine status upgrade to QT status and for updates prior to training
Refresh/update annually
Training info posted at: http://www.ode.state.or.us/search/page/?id=2689
Qualification Requirements
(as a QA) Attend a local training or (those returning) register and update online
http://or.k12test.com/
Practice test administration with a peer or volunteer-student to gain fluency with the assessment, the administration, and the scoring
To qualify, complete the proficiency assessments on Oregon Extended Training Website
Submit any retakes as necessary under ―supervision‖ of a QT (more on this during web-training)
Training info posted at: http://www.ode.state.or.us/search/page/?id=2689
QT & QA Administration
Expectations/Responsibilities
Prepare materials (monitor materials preparation) and
setting for individual administration of the Extended
Assessment
In 2010-11 this will be extremely important because of
the Field Test for science
Administer assessments directly to students
Score student responses
Enter scores in the state’s online data entry system
Interpret results for student, family, or educational team
Maintain security status through District Security
Administrator
Extended Assessment
Alternate Assessment based on Alternate Achievement Standards (AA-AAS)
Allowances for a state to create an alternate assessment for students with the most significant cognitive disabilities.
IEP team decides who participates
Up to 1% of the students who are found proficient (―meets‖ and/or ―Exceeds‖) can count toward state AYP performance reports (see Guidance section on ODE’s Extended Assessments website: http://www.ode.state.or.us/search/page/?id=2690)
Contractor: Behavioral Research and Teaching (BRT) at University of Oregon (Dr. Jerry Tindal)
Decision-Making
Among numerous instructional decisions IEP teams must also decide: Which assessment—General or alternate
If general, what type of supports* to provide during assessment
If alternate, which alternate assessment administration—Standard or Scaffold
Which PSAT option—PSAT or PSSS
If ELL how will they participate in ELPA
*Important Note: A decision to use unapproved accommodations or modifications, is a decision to invalidate the student’s test for all reporting purposes.
Consider General Assessment with or
without accommodations if
Student: Performs at or around grade level
Has academic difficulties that primarily surround reading but may be average or close to average in other subject areas
Has academic difficulties in areas other than reading that are ―mild to moderate‖ and can typically be addressed by using simplified language
Is reading within two to three grades of his or her enrolled level
Instruction: Is primarily general curriculum instruction (but may also use a specialized
curriculum in some areas)
Some Judgment variables: What assessment did he take last year?
How is his attention?
What types of behaviors should be considered?
Consider Standard Administration of the
Extended Assessment if:
Student: Performs well below grade level
Is significantly below grade level in reading
Has academic difficulties that are generalized (to all subject areas) and are significant
Benefits from specialized individual supports
Instruction: Is primarily a specialized curriculum or
From general curriculum must be significantly reduced in breadth, depth, and complexity
Some Judgment variables: What assessment did he take last year?
How is his attention?
What types of behaviors should be considered?
Previous relevant experiences
Consider Scaffold Administration of
Extended Assessment if:
Student: Performance is significantly impacted by a disability
Does not read
Has academic, mobility, and receptive and expressive language difficulties that are generalized and significant
Relies on individual and significant supports to access reduced content materials
Instruction: Is from a specialized curriculum and has functional components and/or
Includes academic goals that are significantly reduced in depth, breadth, and complexity from grade level content
Some Judgment variables: Is the student able to interact with instructional material in a way that
provides meaningful feedback?
Testing Window
February 17th - April 28th.
Materials posted 1 week prior
Data entry window: Extends 2 weeks later
(closes at 11:59 pm, May 19, 2011.
Student Confidentiality and Test
Security
For information regarding student confidentiality and test security
policies and procedures, please review Parts III and IV of the
Oregon Department of Education's 2010-11 Test Administration
Manual at
http://www.ode.state.or.us/wma/teachlearn/testing/admin/2010-
11_tam.pdf or check with your District Test Coordinator.
Please note that all staff who will administer an Extended
Assessment must receive standard test administration and
security training from their district and sign an Assurance of Test
Security form (http://www.ode.state.or.us/search/page/?id=1311)
in addition to qualifying as a QT or QA.
Assessment Format
Description or presentation of Extended
Assessment physical structure 4 subject areas (Reading, Writing, Mathematics, Science)
2 administration options per area (Standard, Scaffold)
3 grade bands (Elementary, Middle, High)
Math is now individual grades 3, 4, 5, 6, 7, 8, and 11
Internal structure (Prerequisite Skills, Independence to Access,
Content Prompts)
In 2010-11 also describe the structure of the science
field test
Prerequisite Codes
Entry
Code
Administration
note
Definition
A Already (has this
skill)
Student already has this skill. (Assessor/teacher
judgment).
I Inappropriate item Item is inappropriate for administration due to the
design of the item and the (sensory) nature of the
student’s disability.
R Refusal Student does not complete or participate.
1 Full Physical Full physical contact is necessary for the student to
respond to this item. E.g. hand over hand.
2 Partial Physical Student requires some physical contact in order to
respond to this item.
3 Verbal, visual, or
gestural
A statement, adjustment, or movement is necessary in
order for the student to respond to this item.
4 Independent Student responds to this item without assistance of any
sort.
Calculating Independence to Access Score
(Support for Access):
To determine the level of support to provide when
moving into the Content Prompts the Assessor will
select the mode (i.e. the most commonly occurring level
of Independence score from the student’s Prerequisite
Skills responses). If there are two modes i.e. two groups
with the same number of scores, select the lower of the
two.
Content Prompt CodesEntry
Code
Administration
note
Definition
D Too Difficult Item is determined too difficult and is not administered to the student
based on assessor/teacher judgment.
I Inappropriate
item
Item is inappropriate for administration due to the design of the item
and the (sensory) nature of the student’s disability.
0 No credit Student response is incorrect.
1 Partial credit Student response is partially correct based on rubric. Student response
demonstrates partial understanding but is incomplete.
2 Full credit Student response is correct based on rubric.
1 Full Physical Full physical supports is necessary for the student to access this item.
E.g. hand over hand.
2 Partial Physical Some physical support is necessary in order for the student to access
this item.
3 Verbal, visual, or
gestural
A statement, adjustment, or movement is necessary in order for the
student to access this item.
4 Independent Student responds to this item without assistance of any sort.
Administration
Physical and logistic Review test and prepare any materials in advance
Arrange substitute time as needed
Consider
reward system as necessary
seating arrangements based on student need
Setting (one on one, no distractions)
Timing (as appropriate)
Assistance (as required)
Administration (continued)
Test All students take Task 1: Prerequisite Skills
To count toward participation, all students must take (in
addition to Task 1) at least 2 Content Prompt tasks.
Maintain awareness of level of support awarded based on
Prerequisite Skills
* NB: Any general assessment submission is
considered the highest score and will override an
Extended Assessment submission for the purposes
of AYP.
Providing Supports During
the Extended Assessments
Levels of Independence
Code 1 Full Physical Contact for response (e.g.
hand over hand)
Code 2 Partial Physical Contact for response (e.g.
gentle repositioning)
Code 3 Visual, Verbal or Gestural supports provided
for response (e.g. move into line of vision, more than
repeating prompt, tapping table, picking up the card)
Code 4 Independent: No contact, no prompting
Scoring the Prerequisite Skills
Score according to the level of support
provided Score ―R‖ if the student refuses to participate in an item
Score ―I‖ if the item is Inappropriate due to nature of the
student’s disability
Note: A code of ―I‖ is a rare category that is used only when
the item as presented is a direct conflict with the sensory
nature of the student’s disability e.g. an item that relies
solely on a student’s sight for a correct response
Determining Level of Independence to
Access
To determine the level of support to provide when moving into the Content Prompts the Assessor will select the mode (i.e. the most commonly occurring level of Independence score from the student’s Prerequisite Skills responses).
If there are two modes i.e. two groups with the same number of scores, select the lower of the two.
Support to Access in Content Prompts
LEVEL OF
SUPPORT
CONTENT PROMPT SUPPORTS
Full Physical
Support*
Based on prolonged hesitation or an indication of student
uncertainty, assessor provides any (or a combination) of the
following:
Moving student to materials
Positioning student to a responding position in the materials
Orienting student to the appropriate response options in the
materials
Moving student’s hand over a series of response options in
the materials
Support to Access in Content Prompts
(continued)LEVEL OF
SUPPORT
CONTENT PROMPT SUPPORTS
Partial Physical
Support
Based on prolonged hesitation or an indication of student
uncertainty, assessor provides any (or a combination) of the
following:
Touch student to direct his/her attention toward the
appropriate materials
Touching student to determine/obtain attention
Support to Access in Content Prompts
(continued)LEVEL OF
SUPPORT
CONTENT PROMPT SUPPORTS
Visual Verbal
Gestural Support
Based on prolonged hesitation or an indication of student
uncertainty, assessor provides any (or a combination) of the
following:
Visual: Maintaining optimal visual placement of assessment
materials for student (i.e. moving materials to ensure they
remain within student gaze)
Verbal: Rephrasing process directions:
―You are choosing from these three‖
―You are putting these in order‖
―You are telling me yes or no‖
Gestural: Pointing to/tapping materials to achieve/maintain
focus on appropriate item
Support to Access in Content Prompts
(continued)LEVEL OF
SUPPORT
CONTENT PROMPT SUPPORTS
Full Independence Student needs no supports to gain access to the item, the
structure of the item, or the associated materials
Minimum Participation
Minimum Participation rule:
An Assessor may consider the minimum participation
option if a student takes all of the Prerequisite Skills
items and also attempts the 10 items of at least two
Content Prompt Tasks. Items on these tasks must be
attempted by the student for participation to be awarded
(i.e. not scored as "D").
All accommodations and appropriate provisions should
be considered thoroughly prior to discontinuing an
Extended Assessment administration. Students taking
any less than this minimum number of items will not
count toward AYP participation.
Data Entry
Data Entry system
Purpose
Where located
How accessed
New format will conform to test structure
―How-to‖ instructions included
For assistance contact ODE and ODE helpdesk
Changes to data entry
Majority of changes to data entry are related
to the field test for Science.
Key points in data entry:
1) Materials for printing are located under ―Resource
Materials‖ link
2) Location for data entry (once tests are
administered) is located under ―Data Entry
Instructions‖
Scoring
In general Correct responses receive full credit (2)
Partially correct responses receive partial credit (1)
Incorrect responses receive no credit (0)
Full credit assigned as listed on scoring protocol (has
the student demonstrated the knowledge and skill
required by the item)
Partial credit responses may not all be listed
More detail on website
Reports and Scores (Student Centered Staging)
See ―Reports‖ tab in Data Entry for ―Individual Student Reports‖ and
―Class Roster Reports‖ that capture students’ frequency of responses.
A quick overview of student centered staging and accessing student
scores can be found at:
1) Staging Web-ex scheduled for Thursday, Nov 4 at 2pm (See
https://district.ode.state.or.us/search/page/?id=198)
2) See ―Student Staging File Format‖ document at
https://district.ode.state.or.us/apps/info/AppDocs.aspx (specifically,
Student Staging User Guide Word doc)
3) If you can’t find it or have questions, then contact your ESD partner
http://www.ode.state.or.us/wma/teachlearn/testing/oaks/esdpartners101
1.pdf
Teachers who are interested in accessing students’ scores are
encouraged to work with your School Test Coordinator.
Reports and Scores (Student Centered Staging)
(cont.)
ESDs will need to contact each of their ESD-supported districts’
District Security Administrator (DSA) to gain access to the Student
Centered Staging application which will show the student data. To
find the DSA for a district, go to the District home page
(https://district.ode.state.or.us/ ) and click on ―Find Security
Administrator‖ under the Quick Links section on the right of the
screen. Here you can search for each and find email/phone contact
information. The good news: once access is granted the individual
should not have to request it again--unless the DSA only wants to do
it for a set period of time.
Discussing Test Results with Parents
When discussing student assessments with parents please note:
Can and should always discuss performance and growth with respect to IEP short term objectives and classroom assignments based on grade level content
Can discuss qualitative skills that are demonstrated at a task or item level (e.g. ―out of 5 items that assessed your son’s knowledge and skills with respect to Interdependence of Organisms in the Environment your son was able to answer 1 correctly‖)
Discussing Test Results with Parents
(continued) Can and should always emphasize that the
Extended Assessment is a summative assessment that is used to evaluate schools and programs and is not an adequate substitute for other forms of classroom progress monitoring
Can and should always emphasize that the Extended Assessment is an alternate assessment and as such performance is not comparable to performance categories on the general assessment (OAKS Online)
Cannot discuss ―performance‖ i.e. meets, does not meet, exceeds etc. until this information is available
General Process
Average length of assessment administration (1.5 hours per subject)
Average length of data entry (~10 minutes)
Average length of time associated with materials preparation (varies)
Manipulation of materials (data entry with student present vs. data entry following assessment)
Participation by Extended Assessment
Subject and Disability Category On average…
5400 students participate in Reading
4900 students participate in Math
2100 students participate in Writing
1600 students participate in Science
The majority of students who participate primarily come from these three disability categories: Mental Retardation (Intellectual Disability)
Specific Learning Disability
Autism Spectrum Disorder
Science Field Test Information
Science Field Test Expectations 2010 - 2011
Each student must take the operational
assessment in science (up to 11 tasks)
Each student will also take one (1) task of the
2010-11 Science Extended Assessment Field
test
Each assessor will submit both field test
information and operational assessment
information via the district secure website
following the same protocol as last year for math
(cf. next two slides)
[Subject Area] Field Test Resource Materials
[Subject Area]Field Test Resource Materials
Science Field Test Process Steps 2010 – 2011:
Printing Materials Print only the task number associated with
your last name
Print tasks needed based on the grade(s) of
your student(s)
If someone else prepares materials for you
First identify which task you are expecting (by last
name) then include this information in your printing
request.
Note: This will be a more difficult task for a central
organizer. Consider printing field test tasks yourself?
Science Field Test Task Selection
Student
Grade
Task 1
QA last name
A-B
Task 2
QA last name
C-D
Task 3
QA last name
E-G
Task 4
QA last name
H-K
Task 5
QA last name
L-M
Task 6
QA last name
N-R
Task 7
QA last name
S
Task 8
QA last name
T-Z
Std Scf Std Scf Std Scf Std Scf Std Scf Std Scf Std Scf Std Scf
3 pdf pdf
4
5
6
7
8
Extended Science Field Test MaterialsSelect ONLY ONE field test Task per student based on the Qualified Assessor’s last name
Science Field Test Process Steps 2010 – 2011:
Preparing to Assess
Consider printing the field test task on a different color
paper
Staple field test task to the operational assessment
Administer only after you have fully exhausted the
operational assessment (i.e. all tasks or task minimum)
If possible, administer the field test during the same test
occasion as the operational test
Administer as any other task.
Science Field Test Process Steps 2010 – 2011:
Data Entry
Accuracy at this stage is just as critical as
during administration and scoring
If someone else is entering data for you
Have them enter and hit the submit button for the
operational data first.
Select the data link for the Science Field test and
Enter data into the fields by task number.
When in doubt—just enter the task by task-
number into the corresponding data entry
fields
Updates and Opportunities
Extended Assessment: 2010-11
Changes this year
Grade of accountability is now 11th grade
For this year only, any students who did not meet the alternate achievement assessment standard in 10th grade will need to take the assessment again this year in their 11th grade year (2010-2011) in order to be counted as participants for state accountability purposes.
Extended Assessment: 2010-11
Changes this year Same structure for Reading, Writing, Science
Math (Grades 3-8, 11)
Science Field Test (Grades 5, 8, 11). Protocol the same as the one used for Math Field Test last year (2009-10)
Similar items
Science Field Test (aligned to new science content standards)
Same difficulty
Math aligned to new content standards
Training Annual re-qualification requirement
3-year update for all Qualified Trainers
Extended Assessment Opportunities 2010-11 Science Standard Setting (July 18 &19, 2011)
For additional information, please contact Brad Lenhardt or Dianna Carrizales.
Online Extended Assessment Item Review - Distributed Item
Review – see below
Slides to Revisit
Slide 6: Three Key Websites
Slides 9 & 10: Qualification Requirements
Slide 18: Student Confidentiality and Test
Security
Slide 38: Reports and Scores (Student
Centered Staging)
Who to go to when
University of Oregon
Behavioral Research
and Teaching
Gerald Tindal
Steve Jonas
Assessment
Administration rules
Web training issues
Oregon Department of
Education
Dianna Carrizales
Brad Lenhardt
Policy
Guidelines
Assessment and
Participation Rules
General complaining
Administration rules
Data Entry issues
Contacts
Brad Lenhardt (Assessment and Monitoring Specialist)
(503) 947-5755
Dianna Carrizales (Director of Monitoring, Systems, and Outcomes)
(503) 947-5634
