Slide 1

Assessment for ArticulationOhios New Assessment SystemMay 14, 2009

Bob MahlmanDirector, Center on Education and Training for EmploymentThe Ohio State Universitymahlman.1@osu.edu614-292-90721Ohios Technical Skills Assessment SystemOnline assessment through www.webxam.orgOnline testing began in 2001About 50 Ohio CTE assessments for SecondarySeveral national certification assessmentsSecure, hierarchical system that tracks results, links to statewide data systemAssessment types(1) Modular, completed over the 2-year program(2) End of programGeneral overview of the systemOCTCA (Ohio Career Technical Competency Assessment) is the statewide testing system for Career-Technical Education in OhioIntroduced in 1993-1994 (OVCA, with OCAPs)Web-based testing (Webxam) began with Information Technology tests in 2001Currently test about 25,000-35,000 students each yearAbout 50 different assessmentsOffer several national certification tests onlineAssessments exist for about 2/3 of the programsSome assessments are modular, some are end of programHierarchical, secure, with logins & passwordsDTC Teacher StudentStudents are activated for tests (proctor required)Students complete and submit tests, results are returned immediatelyMultiple reporting formats availableData is tracked over time, outputted for local, state, and federal reporting

2Assessment Maintenance and RevisionAnnual revisions are done to Enhance psychometric quality (reliability & validity)Keep the test items currentMaintain linkages to current content standardsVast majority of tests receive some level of revision every year.Note, however, that tests in 2008-2009 have NOT been revised. Our focus has been moving toward the new CFTCS.3Test to Content Standards LinkagesMajor Issue: Maintaining linkages to ever-changing content standards.1991 OCAPs1995 OCAPsITACsTCPsCareer Field Technical Content StandardsCareer Clusters4Test to Content Standards LinkagesAnother Issue: How long should content standards be available before tests follow?

Usually 2-3 years to allow standards to become fully implemented.Were pretty much on track for releasing tests a few years after standards are released.

5Moving Toward CFTCS AssessmentsCareer Field Technical Content Standards (similar to Career Clusters)

Perkins IV requires states to have valid and reliable technical skill assessments for ALL CTE programs6Moving Toward CFTCS AssessmentsOhio has assessments for a subset of programs, linked to standards that are being phased out

Now moving in the direction of new CFTCS assessmentsAll modular assessmentsHeavy reliance on scenarios, higher cognate levelPathway-level assessments7Moving Toward CFTCS AssessmentsGeneral ApproachCrosswalk old standards to new standardsPull test items to new standards & verify linkagesCreate test module structure based on postsecondary course deliveryReview & write items to fill gapsReview and validate items and modulesSet benchmark scoresField test and refine 8Old Standards to New Standards2007-2008: Linking old content standards to new CFTCSCrosswalk database applicationInternal staff perform preliminary linkageThree instructors refine the linkagesPhone conferences to resolve discrepanciesCreated linkage product for use by supervisors and instructorsHeres what we did last year, but this is continuing this year as final CFTCS have been just recently released.

The idea here is to pull our current assessment items into the new standards, to give us at least a base to begin creating new assessments.

The linkage work described here is mostly completed, but were continuing it into this year.

9Crosswalk ProductReports show linkage between older content standards and the new Career Field Technical Content StandardsResults are available online: http://www.cete.org/Projects/OCTCA/CrosswalkReview/products.aspxUseful for curriculum planning: shows overlap and gaps

Link HereA product resulting from last years work are the standards linkage results.

10Crosswalk ResultsCrosswalk Homepage

Criminal Justice Page

Criminal Justice Results Old to New

Criminal Justice Results New to Old

Lookup FeatureA product resulting from last years work are the standards linkage results.

11Crosswalk Work 2008-2009Test Item LinkagesLinkage results pull test items into new Career Field Technical Content Standards2007-2008: item to content standard match results are preliminary2008-2009: item to content standard match results are verified by panels of SMEs12Crosswalk Work 2008-2009Test Item LinkagesThree instructors from each content area review linkages between items and new standardsInstructors come to OSU individually to use computerized application to indicate strength of matchFinal results = final gap analysis by Career Field and pathwayItem linkages need to be verified to ensure that the individual test items are indeed measuring what is indicated in the content standards statements.

Scope of work for this year involves bringing in three teachers for each content standard area to review the test items and their associated new standards. Teachers make appointments at our Center online to individually review the item linkages and determine if they match; and if so, the strength of the match. The results of this work will be a more precise gap analysis.

13New Career Field Technical Content Standards Assessments

2009 and Beyond14Development of New CFTCS AssessmentsThe theme for the new CFTCS assessments is assessment for articulation

Goal: create assessments that align with courses being delivered in postsecondary programs

Requires input from postsecondary faculty

THE NEW TESTING SYSTEM

The idea is to develop assessments that are likely to align with courses in programs offering articulated credit.

Input from postsecondary is NEW this year.

Requires buy-in from postsecondary.15Development of New CFTCS AssessmentsAssessments at the pathway levelSignificant implications for curriculum in some areas

All assessments are modularModules completed over the 2-year program

Scenario-based, higher cognate levelsTest specs are that at least 25% are scenario, and 25% are higher level cognate.16Development of New CFTCS AssessmentsModules designed to coincide with content for course articulationBased on the way content is delivered in the community colleges

Create assessments for all pathways over next four years except where viable industry-based certifications exist17Development of New CFTCS AssessmentsAdvantages to the new system

Assessments available to all studentsDevelopment of secondary-postsecondary relationshipsNew utility for technical skill assessments (articulation)Assessments now have added value to the student; greater value to the school/district/state18New CFTCS Assessmentsfor 2009Career Field Technical Content StandardPathwayInformation TechnologyInformation Services and SupportNetwork SystemsProgramming and Software DevelopmentInteractive MediaBusiness and Administrative SciencesBusiness Management FinanceFinancial Services Transportation SystemsGround TransportationMarketingMarketing ManagementOnly doing eight pathways this year because we were unable to get the project off the ground until just this month. Next year, well hopefully have things in place early and hit the ground running with 15 or so pathways.19The Process20Link to PostsecondaryIdentify postsecondary programs with courses being articulated

Link competencies in the targeted pathways to postsecondary courses via surveyPostsecondary faculty indicated competencies addressed by postsecondary program coursesSurveyWith a goal of articulation, the competencies to be tested will be chosen, arranged in test modules, and the number of items distributed across the competencies will be determined based on the degree to which the content standards elements can be used for postsecondary articulation.

CETE conducted a post secondary survey to link the competencies in the pathways to the postsecondary courses. 21Cluster Units of InstructionRun hierarchical cluster analysis (SPSS) to form preliminary modules

Units clustered together based on how they are addressed at the postsecondary levelCluster Analysis22Create Test Module StructureCreate module content structure based on cluster analysis

Enter human judgment into the statistically-based clustering processModule Creation23Create Test ItemsRun item writing workshops

Facilitated groups of secondary and postsecondary faculty (~12 SMEs: 6 secondary, 6 postsecondary)Review/revise current itemsWrite scenarios and new items to fill gaps

CETE staff analyzed the results and developed the preliminary structure of test modules and test specifications for each pathway. During item writing, SMEs adjusted the modules.

Blueprints were applied to item-writing database application, a gap analysis will be run, and materials prepared for item writing workshops.

Item writing will occurred in 3-day workshops. Teams consisted of groups of postsecondary and secondary faculty.

Once written, items will received professional editing to ensure adherence to standard item writing guidelines and to repair any grammar, spelling, and punctuation problems. Banks are cleaned-up and prepared for the next phase.24Validate Items & ModulesRun item & module review workshops using ~12 secondary and postsecondary facultyContent validate itemsItem linkage judgmentsItem essentiality judgmentsItem quality judgments

Content validate modulesJudgments re: Content domain representativeness

Item review includes linkage, essentiality, and quality. Data triangulate among three methods for setting pass/fail benchmark scores.

Angoff with Hofstee and population predictions to moderate.25Validate Items & ModulesSet Benchmark ScoresSet Benchmark ScoresTwo benchmarks = three performance levels; limited, proficient, advancedAdvanced (Expert)ProficientBenchmark 2Limited (Novice)Benchmark 1Item review includes linkage, essentiality, and quality. Data triangulate among three methods for setting pass/fail benchmark scores.

Angoff with Hofstee and population predictions to moderate.26Field TestCreate and publish field test modules Field test the modules (2009-2010)Conduct item and test analysisCreate final forms (Fall, 2010)Repeat the process each year for four years.Field test modules are significantly longer than final modules27Lessons LearnedDifficulty running workshops during the school year (summer better)Postsecondary relationships need to be nurturedConcerns with students abilities to pass scenario-based, higher cognate level testsCurriculum issues28Recommendations for Maintaining The SystemCreate a standing committee for each pathway thatreviews pathway standards annually,reviews tests annually, recommends item replacements, andgenerates new items to replace 1/3 of each test each year.29Recommendations for Maintaining The SystemCommittee members rotate serving 3-year termsInstitute embedded pretest items (12-15 non-scored items in each module each year)Development of new test module structures is done in conjunction with new standards development30Questions / Discussion

31