Setting the Stage for Program Assessment:A Case Study of the School of Communication

(Part I—9:40-10:10)

Putting on the Play: A Case Study of the School of Communication

(Part II—10:20-10:50)

Brian H. SpitzbergSan Diego State University

PROGRAMMATIC ASSESSMENT

Or,“How I learned

how to stop kvetching and

just do assessment in a way that’s

relatively painless and yet, helpful.”

Not if, but whenIf we must do assessment, we should…

optimize value-added / effort + investment keep it as fair as possible keep it as reliable & valid as possible

All assessment is subjective in one way or another

Building a Communication Assessment Axioms:

All assessment must occur at both: molar/holistic & molecular/microscopic levelsboth self and other perspectives

Assessment should reflect: both inductive & deductive development change over time (multiple assessment times) motivation, knowledge, and skills

Don’t reinvent the wheel

Building a Communication Assessment Axioms (continued):

DISCI-PLINE

PROGRAM

THEORY

MISSION/ S.L.O.s

STAKE-HOLDERS

CURRICULA

FACULTY

MISSION/ S.L.O.s

Ded

uctiv

e InductiveSteps to Program Assessment

Overview of ApproachI. Survey to Produce Content (mission, goals, SLOs)

A. Deductive approachesB. Inductive approaches

II. Translation into Goals & SLOsA. ReductionB. Formalization draft

III. Preliminary ValidityA. Representational/FaceB. Expert review

IV. Formal Articulation:A. Translation into rubricsB. Anchoring the scaling

V. Integration into Turnitin/GrademarkWEAVE

Overview of ApproachI. Survey to Produce Content (mission, goals, SLOs)

A. Deductive approaches: Disciplinary consensus documents Accreditation documents A priori program goals Theoretical models (e.g. Bloom)

B. Inductive approaches: Survey faculty Survey stakeholders (alumni, profession, etc.) Survey curricula and syllabi

C. [For us, this produced ≈ 250 SLO’s]

Adapted Bloom’s Learning Domains

Perceiving phenomena

[SKILLS]: Psychomotor & Behavioral

[MOTIVATION]: Affective attitude

[KNOWLEDGE]: Cognitive/Mental

Responding to phenomena

Organizing

Internalizing values

Interpreting phenomena

Comprehending/Understanding

Analyzing/ Disassembling

Synthesizing/ Reassembling

Evaluating/Creating

Applying/Generalizing

Procedural records review

Origination

Feedback adaptation

Procedural record revision

Threshold routine activation

Overview of Approach: Exemplary Deductive Models

COMMUNICATIONEXEMPLARS:

AssertingCampaigning

Collaborating Comforting

CommentingContributing

Debating Empathizing

ExpressingInterviewing

LeadingListening

Mediating Negotiating NetworkingPersuading

QuestioningReplying

Reviewing Speaking

etc.

[KNOWLEDGE]: Cognitive/Mental

Comprehending/ Understanding

Analyzing/ Disassembling

Synthesizing/ Reassembling

Evaluating/ Creating

Applying/ Generalizing

Designing, constructing, planning, checking, producing, inventing, animating, critiquing, representing, writing, imagining, validating, brainstorming, analogizing, etc.

Integrating, hypothesizing, modeling, linking, organizing, judging, detecting, monitoring, reviewing, moderating, collaborating, networking, etc.

Comparing, outlining, deconstructing, attributing, finding, structuring, reverse engineering, separating, extracting, etc.

Implementing, specifying, planning, using, executing, running, loading, playing, operating, instantiating, exemplifying, testing, experimenting, etc.

Recognizing, listing, describing, identifying, retrieving, naming, locating, finding, highlighting, remembering, recalling, specifying, delineating, etc.

Adapted broadly from: http://edorigami.wikispaces.com/Bloom%27s+Digital+Taxonomy

Overview of Approach: Exemplary Deductive Models

Overview of ApproachII. Translation into goals & SLOs

A. Reduction: Scan for redundancy, excessive abstraction, etc. Edit for consistency in grammar, action verbs,

tense, etc. Print goals/SLOs on cards, and conduct “sort” Multifinality: multiple solutions can be explored Decide on a workable preliminary ‘model’

B. Formalization draft: Write a preliminary… “micro” model (exemplar SLOs) “macro” model (conceptual integration)

Overview of ApproachIII. Preliminary validity & utility

A. Representational/‘Face’: Check for faculty buy-in Run by some colleagues who are also working

on assessmentB. Expert:

Check professional associations Check for consultants Check with those at successful programs

SDSU School of CommunicationInductively-Based Student Learning Objectives

PRO

GRA

M-B

ASED

COM

PETE

NCI

ESCO

LLEGIATE-BASED

COM

PETENCIES

CENTRAL CONCEPTS AND ASSUMPTIONS OF COMMUNICATION

THE COMPETENT ‘CITIZEN’ COMMUNICATOR

COMPETENT ARGUMENT CONSTRUCTION,

ANALYSIS, PRESENTATION

DISCIPLINARY AWARENESS (HISTORY,

BOUNDARIES, ETC.)

KNOWLEDGE OF APPLIED CONTEXTS,

FUNCTIONS

KNOWLEDGE OF THEORY, MODELS,

CONCEPTS

WRITING (APA, LIBRARY RESEARCH)

KNOWLEDGE/APPLICATION RESEARCH METHODS &

APPLICATIONSVIII

VII

VI

V IV

III

II

I

CENTRAL CONCEPTS AND ASSUMPTIONS OF COMMUNICATION

THE COMPETENT ‘CITIZEN’ COMMUNICATOR

COMPETENT ARGUMENT CONSTRUCTION, ANALYSIS,

PRESENTATION

DISCIPLINARY AWARENESS (HISTORY, BOUNDARIES,

ETC.)

KNOWLEDGE OF APPLIED CONTEXTS, FUNCTIONS

KNOWLEDGE OF THEORY, MODELS, CONCEPTS

WRITING (APA, LIBRARY RESEARCH)

KNOWLEDGE/APPLICATION RESEARCH METHODS &

APPLICATIONS

Ability to communicate ethically in a participative societyPerform basic communication competence/skillsDemonstrate awareness of the role of communication in context

Demonstrate knowledge of history/structure of the disciplineDemonstrate understanding of primary contexts of communicationDemonstrate awareness of major communication media & systems

Identify and critique core communication principles & assumptions, Demonstrate understanding of importance of communicationDemonstrate ability to analyze complexity of communication processes

Identification of concepts & principles in applied contextsDemonstration of relevant knowledge in applied contextsDemonstration of relevant skills in applied contexts

Identify, differentiate, compare, contrast theoriesCreate & apply theories with conceptual rigor and contextual relevanceEvaluate theories using established paradigmatic criteria for critique

Identify, differentiate, compare, contrast methodologiesApply methodologies to conduct original paradigmatic researchEvaluate methodologies within and across relevant paradigm standards

Develop comprehensive, relevant, coherent argumentsPresent persuasive arguments orally and in writingEvaluate arguments by established tests of reasoning and evidence

Demonstrate scholarly authorial voice in writingDemonstrate mastery of APA style guideDemonstrate mastery of scholarly research tools/engines VIII

VII

VI

V

IV

III

II

I

Molecu

lar SL

Os

CENTRAL CONCEPTS AND ASSUMPTIONS OF COMMUNICATION

THE COMPETENT ‘CITIZEN’ COMMUNICATOR

COMPETENT ARGUMENT CONSTRUCTION, ANALYSIS,

PRESENTATION

DISCIPLINARY AWARENESS (HISTORY, BOUNDARIES,

ETC.)

KNOWLEDGE OF APPLIED CONTEXTS, FUNCTIONS

KNOWLEDGE OF THEORY, MODELS, CONCEPTS

WRITING (APA, LIBRARY RESEARCH)

KNOWLEDGE/APPLICATION RESEARCH METHODS &

APPLICATIONS

The School of Communication seeks to graduate students who can communicate competently in the major walks of life they will encounter.

The School of Communication seeks to graduate students with an awareness of the history, nature, scope, and evolution of the communication discipline.

The School of Communication seeks to graduate students who have a working knowledge of the core concepts, definitions, and assumptions of the communication discipline.

The School of Communication seeks to graduate students who can diagnose the relevance and implications of paradigms, theories, and models of communication in hypothetical and actual contexts.

The School of Communication seeks to graduate students who can identify, differentiate, and relate core paradigms, theories, and models in the communication discipline.

The School of Communication seeks to graduate students who can identify, differentiate, relate and apply the major methodological paradigms in the generation of original research.

The School of Communication seeks to graduate students who can extemporaneously and proactively generate and competently present sound arguments in communicative performance contexts.

The School of Communication seeks to graduate students who can competently locate, filter, retrieve, evaluate, and cite scholarly research in the composition of well-written textual documents employing APA format.VIII

VII

VI

V

IV

III

II

I

Molar

SLOs

Overview of ApproachIV. Formal Articulation:

A. Translation into rubrics: Review existing rubrics Then compose your own

B. Anchoring the scaling: Review existing rubrics Personally, I recommend a 5-point Then compose your own Consider both elaborated & abbreviated forms

1: 0-20 (F) 2: 21-40% (D) 3: 41-60% (C) 4: 61-80% (B) 5: 81-100% (A)WRITING/APA: FORM:Demonstrates proficiency in grammar, syntax, semantics, academic voice, and application of APA style guidelines.

Form displays: multiple types & instances within type of writing or grammatical errors in expository text, &/or displays inconsistency in rule application; frequent re-editing or rephrasing to achieve more professional voice is suggested.

Form displays: moderately low level of professional voice, composition, and grammatical form with moderate number of errors, inconsistency of rule application, or required editing (intermediate to scales 1 & 3).

Form displays: few types & instances within type of writing or grammatical errors in expository text, &/or inconsistency in rule application; occasional re-editing or rephrasing to achieve more professional voice is suggested.

Form displays: moderately high level of professional voice, composition, and grammatical form with moderately few errors, inconsistency of rule application, or required editing (intermediate to scales 3 & 5).

Writing displays consistent use of professional voice, composition, and grammatical form.

1: 0-20 (F) 2: 21-40% (D) 3: 41-60% (C) 4: 61-80% (B) 5: 81-100% (A)CLAIMS/PROPOSITIONS--CONTENT:Demonstrates ability to articulate researchable claims specifying the interrelationship among variables.

The key claims are not clearly articulated or delineated. Propositions fail by level of scaling, relationship, or syllogistic entailment. “Object lessons” or “list of horrors” are repeated.

Form displays: moderately low level of relationship specification or implications of claims or propositions, or makes several errors in wording (intermediate to scales 1 & 3).

Only minor or one or two claims or propositions need editing for sake of clarity.

Form displays: moderately high level of relationship specification or implications of claims or propositions, or makes few errors in wording (intermediate to scales 3 & 5).

Propositions are both logically sound, and sophisticated in their thematic connection &/or articulation of complex relationships.

Sample Resulting Rubrics:

1: 0-20 (F) 2: 21-40% (D) 3: 41-60% (C) 4: 61-80% (B) 5: 81-100% (A)PRESENTATION PERFORMANCE: Demonstrates the ability to orally express ideas, thoughts, claims, propositions, arguments, and evidence in a competent verbal and nonverbal communication

Demonstrates neither awareness of nor ability to integrate competent verbal and nonverbal communication into an oral presentation of express ideas, thoughts, claims, propositions, arguments, and evidence.

Demonstrates little awareness of and ability to use competent verbal and nonverbal communication to orally express ideas, thoughts, claims, propositions, arguments, and evidence.

Demonstrates an awareness and ability to use competent verbal and nonverbal communication to orally express ideas, thoughts, claims, propositions, arguments, and evidence with a minimal confusion and lack of clarity.

Demonstrates an intermediate awareness of and ability to employ competent verbal and nonverbal communication to orally express ideas, thoughts, claims, propositions, arguments, and evidence.

Demonstrates a mastery of competent verbal and nonverbal communication to orally express ideas, thoughts, claims, propositions, arguments, and evidence.

Sample Resulting Rubrics:

1: 0-20 (F) 2: 21-40% (D) 3: 41-60% (C) 4: 61-80% (B) 5: 81-100% (A)RESEARCH/DATA:CONTENT:Demonstrates ability to locate and appropriately cite and list recent, relevant, and reasonable scholarly research, consisting mostly of peer-reviewed journal sources.

No more than one or two directly or peripherally related external sources were brought to bear on the validity of the selected source claims, &/or those sources applied are distantly relevant to source claims; &/or sources lack recency, relevance, or scholarly imprimatur.

Form displays: moderately low number and quality of location, citation, and listing of recent, relevant, and reasonable scholarly sources (intermediate to scales 1 & 3).

At least 1 to 2 studies relevant to each selected proposition, argument, component, or source claim are provided, &/or some sources applied are distantly relevant to source claims; &/or sources lack recency, relevance, or scholarly imprimatur.

Form displays: moderately high level of development of sound, articulated, and evidential warrants for claims, with high status and appropriate sources (intermediate to scales 3 & 5).

Each major claim is evidenced by sources high in scholarly credibility (i.e., relevance, recency, peer review, etc.)

HALFFULL?

HALFEMPTY?

UNDE-CIDED?

Overview of Approach:Anchoring the scaling?

Overview of ApproachV. Integration into Turnitin/Grademark

A. Re-check faculty buy-in and editing inputB. Every faculty selects:

at least 1 assignment in each course to which at least 1 of the rubrics will be applied

C. A diagnostic grid results of SLOs by courses Which SLO’s are underrepresented? Which courses are under-assessed?

D. Export rubric to all facultyE. End-of-Semester downloads to ExcelF. A program “Assessment Chair” will receive, strip,

aggregate, and summarize across curricula

Assignment inbox GradeMark report

Rubrics Export Excel

Auto-Average each common rubric across curriculaTransfer data to WEAVE

Building a Communication Assessment

TURNITIN/GRADEMARK PROCEDURE:

Oh, what

a tangl

ed web we

Oh, what tangled webs we…

IMPACCT Home Page IMPACCT AdminPage

mpacct: Developing an Online Assessment

“Take your total SAT scores, add your average net income for the last three years, divide by the number of times

you have sex per week, then multiply by your age. That gives us a numerical indicator of your over-all worth as a

human being”

IMPACCT Home Page IMPACCT AdminPage

Description: An online, self-administered communication & critical thinking skills assessment.

Advantages:HeritageBreadth360 peer assessment Longitudinal assessmentRelative convenience (online, self-administered)Potential for national/regional comparative normsPsychometric soundnessPortfolio/feedback resource for students

Building an Online Assessment

Synopsis of impacct:

IMPACCT Home Page IMPACCT AdminPage

Development: Identify relevant skill areasComposition: ≈ 40 constructs, organized as:

Motivation (e.g., public speaking confidence)Knowledge/Critical thinking (e.g., problem-solving)Conversation/interaction (e.g., attentiveness) Interpersonal/Relational (e.g., negotiation)CMC (e.g., medium selection x message content)Group/Leadership (e.g., agenda management)Public (e.g., attention-getting introduction)Outcomes (e.g., appropriateness, effectiveness)

Building a Communication Assessment

IMPACCT Home Page IMPACCT AdminPage

mpacct: Let’s Take a Tour

IMPACCT

What is impacct?

Test Overview

Informed Consent

Students Taking

Student log-in

COMM Depts.

Campaign Creation Preview

Assess-ment

Control Panel

Assessment site architecture:

IMPACCT Home Page IMPACCT AdminPage

mpacct: Let’s Take a Tour

IMPACCT

What is impacct?

Test Overview

Informed Consent

Students Taking

Student log-in

COMM Depts.

Campaign Creation Preview

Assess-ment

Control Panel

Assessment site architecture:

#

IMPACCT Home Page IMPACCT AdminPage

mpacct: Let’s Take a Tour

IMPACCT

What is impacct?

Test Overview

Informed Consent

Students Taking

Student log-in

COMM Depts.

Campaign Creation Preview

Assess-ment

Control Panel

Assessment site architecture:

#

IMPACCT Home Page IMPACCT AdminPage

mpacct: Let’s Take a Tour

IMPACCT

What is impacct?

Test Overview

Informed Consent

Students Taking

Student log-in

COMM Depts.

Campaign Creation Preview

Assess-ment

Control Panel

Assessment site architecture:

#

#

mpacct: So far…

Developments so far:Approximately 2 years of data collectionAll constructs reliableNo apparent sex or ethnicity biasModest ‘Wobegon’ effect (means 4.5-5.5 on 7-point)Significant improvement over time (38 of 40 constructs)R2 = .31 to .73 in predicting communication qualityPeer ratings slightly related to other and to self-ratings Impacct 2.0 expanded to multiple campaigns

mpacct: So far…

Big plans:Develop an “assessment institute” at SDSU

Fund RAs Provide resource for theses Innovation cauldron Research grants/awards for annual proposals

Generate annual “state of the communication nation” report Provide consulting opportunities for institutional reporting needs Prospective innovations:

Video capture for peer assessment stimuli Listening test Emotional intelligence/sensitivity test Media literacy test Brief therapeutic screens

mpacct: So far…

Background Sources:

IMPACCT: Spitzberg, B. H. (2011). The Interactive Media Package for Assessment of Communication and

Critical Thinking (IMPACCT©): Testing a programmatic online communication competence assessment system. Communication Education, 60, 145-173.

Spitzberg, B. H., Lee, C. M., & Lindemann, K. (2011, February). Of dimensions and dementia: Desiderata in determining the composition of communicative competence and skills. Top Four paper presented at the Western States Communication Association, Monterey, CA.

Competence:Spitzberg, B. H. (2003). Methods of skill assessment. In J. O. Greene & B. R. Burleson (Eds.),

Handbook of communication and social interaction skills (pp. 93-134). Mahwah, NJ: Erlbaum.

Spitzberg, B. H. (2009). Axioms for a theory of intercultural communication competence [invited article, Japanese Association of Communication and English Teachers]. Annual Review of English Learning and Teaching, No.14, 69-81.

Spitzberg, B. H., & Cupach, W. R. (2011). Interpersonal skills. In M. L. Knapp & J. A. Daly (Eds.), Handbook of interpersonal communication (4th ed., pp. 481-524). Newbury Park, CA: Sage.

“I think you’ll find my test results are a pretty good indication of your abilities as a teacher.”

“I think you’ll find that my test results are a pretty good indication of your abilities as a

teacher.”

If interested,contact me for

documentsor questions:

spitz@mail.sdsu.edu@ .sdsu.