INtopFORM Assessment Plan

Please sit with other faculty members from your program of study

Goals for today’s session• By the end of today’s session, you should:– Understand basic assessment of student learning;– Tentatively identify when and where assessment of the

six ITF LOs will take place in your program;– Develop a plan for assessment measures to be used (ITF

rubric, major field test, standardized final, etc.);– Develop a plan for collecting baseline data (preferably

during Spring 2014);– Decide when you and your colleagues will discuss

assessment results;– Know what your responsibilities are for reporting; and– Be familiar with the ITF assessment calendar.

Assessment Basics

What is assessment?

1. Collecting data so we can report it to SACS (or THEC, TBR, AACSB, etc.)

2. Something the folks in Dossett Hall came up with to keep faculty busy

3. A useful teaching and learning tool

Assessment is

Deciding what we want our students to learn

Making sure they learn it

From Linda Suskie, courtesy of Jane Wolfson, Towson University

http://img4.wikia.nocookie.net/__cb20100313135124/simpsons/images/4/47/Dabf20.gif

Assessment Cycle

Excerpted from Assessing Student Learning by Linda Suskie, Jossey-Bass, 2009

1)

2)

3)

4)

Program-Level Assessment for INtopFORM

Assessment Template

• Outcomes: Your six discipline-specific ITF LOs• Opportunities: Where will students practice ITF skills?• Measurement Sites: Which courses and assignments?• Measures: Projects, papers, exam questions, clinicals• Sampling (if used)• Baseline Data: When, where, and how collected• Sharing: Who can act on results to improve student

learning?

Step 1: Establish Learning Goals

• Six INtopFORM Learning Outcomes• What do they mean in YOUR

discipline?– See Parts 1 and 2 of your planning

worksheets

QUESTIONINGSEEKING

EVALUATINGUSING

COMMUNICATINGRECOGNIZING

Record draft ITF learning outcomes for your program in Column A of the Assessment Plan Template

Step 2: Provide Learning Opportunities• Must allow students to practice and develop skill in

each of the six INtopFORM Learning Outcomes

• Where will this take place in your curriculum?– See results of Curricular Mapping exercise

• What methods might you use?– See Teaching Ideas (Part 3 of planning worksheets) for

each ITF LO

Record possible learning opportunities for your students in Column B of the Assessment Plan Template; please be specific (course + assignment or activity)

Step 3: Assess Student Learning

• Where will you measure student learning?• Must be measurable or observable• Goal– Collect “reasonably accurate, truthful evidence”• Doesn’t have to be worthy of publication

– Good enough to use with confidence• Possibilities include single assignments,

capstone courses, projects, final exam items

Assessment Measures• Two options:– Modify the ITF assessment rubric to evaluate

papers or projects (preferred);– Study students’ performance on designated items

on major field tests, standardized final exams, or locally designed tests of information fluency

Record where assessment might take place (course, assignment) in your curriculum in Column C of the Assessment Plan Template. Include possible measures (exam items, rubrics) for your program in Column D.

QUESTIONING: Students ask questions that facilitate the solution of problems and the pursuit of opportunities.

Questions define the scope of the area of interest. Questions identify key issues to be resolved. Questions demonstrate understanding of the problem, task or topic. Questions encourage a broad search for information. Questions are relevant to the problem, task or topic. Questions bring needed focus or structure to the problem, task or topic. Questions examine diverse perspectives on the problem, task or topic. Questions reflect attitudes of openness and curiosity. Overall Rating (required item) Other items (subject to approval):

SEEKING: Students locate or create information needed to accomplish their purposes. Sources used are credible. Sources used are appropriate for the purpose. The student uses several sources. The student uses different kinds of sources. Sources and information used indicate a thorough search process. Sources and information used indicate persistence and resourcefulness in seeking information. The student insightfully evaluates the search process. The student conducted an iterative search process. The student conducted an efficient search process. The student correctly identifies limitations of research he or she conducted. Overall Rating (required item) Other items (subject to approval):

EVALUATING: Students apply critical thinking skills in evaluating sources, information and search processes.

The student correctly analyzes the reliability of sources or information. The student correctly analyzes the potential bias of sources. The student insightfully discusses the implications of information. The student demonstrates comprehension of information. The student correctly identifies assumptions underlying information. The student identifies questions raised by information. The student analyzes his or her own and others’ assumptions. Overall Rating (required item) Other items (subject to approval):

USING: Students assemble and synthesize information to accomplish their purposes. Information used is relevant to the purpose. The amount of information provided is sufficient for the purpose. The student draws sound conclusions from information. The student skillfully synthesizes information from multiple sources. The student draws non-obvious insights from information. The analysis of information flows logically from point to point. The student does not oversimplify complex information. The student clearly relates information to the larger purpose. The student draws upon diverse perspectives. Overall Rating (required item) Other items (subject to approval):

COMMUNICATING: Students communicate information effectively. The student aids others’ understanding of information. The student provides useful explanations of information. The student presents information accurately. The student presents information completely. The presentation of information helps others see connections or larger themes. The presentation of information arouses others’ interest. The presentation of information evokes questions or further study by others. The student openly acknowledges when information is inconsistent or inconclusive. Overall Rating (required item) Other items (subject to approval):

RECOGNIZING: Students recognize the responsibilities and consequences related to information ethics and intellectual property.

The student clearly identifies sources of information used. The student respects confidentiality rights associated with information. The student shows fair use in the quotation of sources. Overall Rating (required item) Other items (subject to approval):

INtopFORM Assessment Guidelines• Must evaluate all six ITF LOs• Must occur annually• Must provide data on the culminating skills

and abilities of students in a program of study– Ordinarily in the senior year / 4000-level courses

• Must yield findings applicable to all majors – Can evaluate all majors’ work or that of a

sufficient representative sample

Sampling• More evidence collected means higher

confidence in conclusions, but be practical• If you use a sample, it should adequately

represent your student population– Online vs. on ground– Adult vs. traditional

Record from whom you will collect assessment information in Column E of the Assessment Plan Template; describe sampling (if applicable)

Baseline Data• Must be collected during Year 1

– Spring 2015 preferred• Use standard reporting template (provided) to include:

– Overall assessment for each of the six outcomes• Mathematical average (can be weighted) or separate rating

– Number and % of students rated as:3 = Excellent2 = Satisfactory1 = Unsatisfactory

– Whether each student was rated satisfactory on all outcomes

Record ideas of how your program might collect baseline data in Column F of the Assessment Plan Template.

Student ID A B C Dept. TotalsQuestioning: Students ask questions that facilitate the solution of problems and the pursuit of opportunites

Students raise appropriate questions regarding a particular issue 3.00 3.00 3.00 3.00The line of argument anticipates and adequately responds to relevant objections 3.00 3.00 2.00 2.67Overall assessment of Questioning 3.00 2.00 3.00 2.67Seeking: Students locate or create information needed to accomplish their purposes

Uses evidence-based sources and peer reviewed literature to support all aspects of the paper 3.00 2.00 3.00 2.67Overall assessment of Seeking 3.00 2.00 3.00 2.67Evaluating: Students apply critical thinking skills in evaluating sources, information, and search processes

Identifies appropriate problem for case presented 3.00 2.00 3.00 2.67Demonstrates clear understanding of response to treatment 3.00 3.00 3.00 3.00Correctly determines next steps in process 2.00 2.00 3.00 2.33Overall assessment of Evaluating 3.00 2.50 3.00 2.83Using: Students assemble and synthesize their information to accomplish their purposes

Adequately describes problem using professional terminology 3.00 3.00 3.00 3.00Demonstrates logical progression of relevant data and facts 3.00 3.00 3.00 3.00Develops accurate and appropriate AQI statements 3.00 2.00 3.00 2.67Plans appropriate strategies for monitoring interventions 3.00 3.00 3.00 3.00Overall assessment of Using 3.00 2.75 3.00 2.89Communicating: Students communicate information effectively

Addresses significance of the issue and facts pertinent to the issue 3.00 3.00 3.00 3.00Strong conclusion summarizing data, facts, and information related to issue 3.00 3.00 2.00 2.67Uses appropriate fonts, font sizes, spacing, cover page, bibliography and headings for each section 3.00 3.00 2.00 2.67Uses appropriate word choice, free from grammatical and spelling errors 3.00 2.00 3.00 2.67Overall assessment of Communicating 3.00 2.75 2.50 2.83Recognizing: Students recognize the responsibilities and consequences related to information ethics and intellectual property

Sources cited appropriately using APA citation style 3.00 1.00 3.00 2.33Gives credit when paraphrasing or rewording 3.00 2.00 3.00 2.67Overall assessment of Recognizing 3.00 1.50 3.00 2.50

Example Program of Study QEP Rubric

Step 4: Use the Results

• Assessment is worthless if it isn’t useful– How will results be analyzed? By whom?– How and when will results be shared? – Who can act on your assessment data to make

improvements?

Discuss how your program might use the assessment information you collect and record in Column G of the Assessment Plan Template.

You are not alone• INtopFORM provides support during Faculty

Learning Communities• INtopFORM staff will hold meetings with each

program to discuss assessment and implementation plans

• The Director of Assessment (with assistance from the Director of the QEP) will – Assist faculty in designing assessments– Approve each programs’ assessment protocols

Assessment Goals of INtopFORM

Institutional Assessment Goals

For each program:• Improvement in Years 2-5 compared to baseline data in

Year 1

• >30% of the rubric ratings or test scores on each INtopFORM learning outcome are “excellent”

• >90% of the rubric ratings or test scores on each INtopFORM learning outcome are at least “satisfactory”

• >90% of program majors achieve rubric ratings or test scores demonstrating at least “satisfactory” performance on all INtopFORM learning outcomes

California Critical Thinking Skills Test• ETSU’s Senior Exit Exam, administered since 2005 for

general education assessment• 34 questions, 45 minutes in length• Five areas: Analysis and Interpretation, Inference,

Evaluation and Explanation, Deductive Reasoning, and Inductive Reasoning

• CCTST maps well to EVALUATING and USING INtopFORM learning outcomes

• Office of Assessment will collect CCTST data annually and provide reports to programs of study

CCTST Score Interpretation

CCTST Area Description

Score Interpretation

Weak Moderate Strong

Analysis (maps to EVALUATING)

used when pulling apart arguments and points of view 0 – 2 3 – 4 5 or

more

Inference (maps to USING)

used whenever drawing conclusions based on reasons and evidence

0 – 5 6 – 11 12 or more

Evaluation (maps to EVALUATING)

used when deciding believability of a given statement 0 – 3 4 – 7 8 or

more

Inductionused when deciding that evidence supports truth in given conclusion

0 – 5 6 – 11 12 or more

Deduction

used when deciding a false conclusion is impossible considering all arguments are true

0 – 5 6 – 11 12 or more

CCTST Total Score

• 8-12 (Weak)

• 13-18 (Moderate)

• 19-23 (Strong)

• ≥24 (Superior)

CCTST Trend, 2005-06 to present

2005-06

2006-07

2007-08

2008-09

2009-10

2010-11

2011-12

2012-13

2013-1416

16.2

16.4

16.6

16.8

17

17.2

17.4

17.6

17.8

18

Academic Year

Mea

n CC

TST

Tota

l Sco

re

Year N Mean score Percentile Rank2005-06 1135 17.5 61st – 68th 2006-07 2677 17.4 54th – 61st 2007-08 1815 17.7 54th – 61st 2008-09 1593 17.2 55th 2009-10 1512 17.3 56th 2010-11 1672 17.5 59th 2011-12 1859 17.6 57th 2012-13 1723 16.5 49th 2013-14 2016 17.0 51st

Goals for CCTST PerformanceFor each program:• Mean CCTST Total Scores and subtest* scores should

exceed baseline in Years 2-5

• >30% of scores on each CCTST subtest* at or above “strong”

• >90% of scores on each CCTST subtest* at or above “moderate”

• >30% of CCTST Total Scores are at least 19 (“strong”)• >90% of CCTST Total Scores are at least 15

(“moderate”)*mapped to an INtopFORM learning outcome

Annual Assessment and Improvement Meetings

• Each fall in Years 3-5• Includes Director of Assessment, QEP Director,

program leaders, and other interested faculty• Purpose

– Review previous year’s assessment data– Use results to refine plan for enhancing information fluency– As needed, define improvement actions, responsible

parties, and calendars• Annual assessment and improvement report drafted

by Directors and approved by program faculty

Overview of Assessment Timeline Programs of Study, Cohort 2

Year 1: 2014-15 Year 2: 2015-16Su Fa Sp Su Fa Sp

Develop program assessment plan X X X (refine) Collect baseline data X Collect and analyze results, refine plan, make improvements

X X X

Year 3: 2016-17 Year 4: 2017-18Su Fa Sp Su Fa Sp

Develop program assessment plan

Collect baseline data Collect and analyze results, refine plan, make improvements

X X X X X X

Year 1 Assessment Calendar,Programs of Study

First Year of ParticipationWhen What WhoFall Create a plan to help all students in the

program attain the six INtopFORM learning outcomes

Program of Study Faculty

Design assessment of ITF learning outcomes

Program of Study Faculty

Spring Collect and report baseline assessment data Program of Study Faculty

Fall, Spring, and Summer

Collect and report baseline CCTST data Office of Assessment

Summer Send participating programs of study analysis of baseline data, including CCTST

Office of Assessment

Year 2 Assessment Calendar,Programs of Study

Second Year of ParticipationWhen What WhoFall Meet to review baseline data and discuss

implications for planned improvements in program

Director, QEP; Director, Assessment; Program of Study Faculty

Fall and Spring Implement plans to help students attain learning outcomes

Program of Study Faculty

Collect and report data on ITF learning outcomes

Program of Study Faculty

Fall, Spring, and Summer

Collect and report CCTST data Office of Assessment

Summer Send participating programs of study analysis of data, including CCTST

Office of Assessment

Years 3-5 Assessment Calendar,Programs of Study

Third, Fourth, and Fifth Year of ParticipationWhen What WhoEarly Fall Meet to evaluate prior year assessment

data and refine program’s INtopFORM planProgram of Study Faculty; Director, Assessment; Director, QEP

Fall and Spring Collect and report data on ITF learning outcomes

Program of Study Faculty

Fall, Spring, and Summer

Collect and report CCTST data Office of Assessment

Summer Send participating programs of study analysis of data, including CCTST

Office of Assessment

Decide what you want your students to learn

Make sure they learn it

Contact Information

Cheri ClavierDirector of Assessment

clavier@etsu.edu439-7483

136 Sherrod