Upload
vodan
View
219
Download
3
Embed Size (px)
Citation preview
ABET & NCAAA ACCREDITATION
Course Design Workshop
Prof. Ali M. Al-Bahi
Director of the Academic Accreditation Unit
Faculty of Engineering , King Abdulaziz University
Modified and Presented by
Dr. Faisal Iskanderani
UBTVR for Q&D
Sept 11, 2014
Session #1
September 11, 2014 1
Introduction
why Learning Outcomes?
Student Learning Experience
Course Design Workshop
CEIT will be working towards National, and International Accreditation ( NCAAA and ABET)
Time line: end of 2016
Today’s workshop will deal with one component of accreditation, i.e., Students’ Learning Outcomes delay this
Course Design Workshop
Introduction
ABET and NCAAA accreditation is assurance that a college or university program meets the quality standards established by the profession for which it prepares its students.
NCAAA and ABET have similar accreditation criteria
We will work together in parallel to achieve both at the same time, using the same documentation
Course Design Workshop
Introduction
ABET Documents
Criteria for Accrediting Engineering Programs
Accreditation Policy and Procedure Manual
Template for the Engineering Self Study Report
Course binders or files
NB: Documents should be downloaded for the year of the accreditation cycle from:
http://abet.org/
5Course Design WorkshopSeptember 11, 2014
NCAAA Documents
National Qualifications Framework,
NCAAA Standards for the Accreditation and Quality Assurance for Institutions and Programs (11 Standards),
Self Evaluation Scales for NCAAA Standards, and
Forms and Templates for Institutions and Programs as follows: www.ncaaa.gov.sa
Course Design Workshop
NCAAA Forms and Templates
Course Specification
Course Report
Field Experience Specification
Field Experience Report
Program Specification
Annual Program Report
Template for Report on Periodic Program Self-Study
Course Design Workshop
Accreditation Time-Line (a 2-3 year process)
8
1
2
35
4
9
6 8
7
Course Design WorkshopSeptember 11, 2014
Note: In addition to one year for preparations
ABET Criteria for Accrediting Eng. Programs
1. Students2. Program Education objectives (PEOs).3. Student Outcomes (SOs). We will call them
“Program Learning Outcomes”4. Continuous Improvement5. Professional Component (curriculum).6. Faculty.7. Facilities.8. Institutional Support.
+ Program Criteria.
9September 11, 2014 Course Design Workshop
1. Students2. Program Education objectives (PEOs).3. Student Outcomes (SOs). We will call them
“Program Learning Outcomes”4. Continuous Improvement5. Professional Component (curriculum).6. Faculty.7. Facilities.8. Institutional Support.
+ Program Criteria.
10Course Design WorkshopSeptember 11, 2014
ABET Criteria for Accrediting Eng. Programs
NCAAA Standards for Accreditation and
Quality Assurance1. Mission and objectives (ABET C2)
2. Governance and administration (ABET C6 )
3. Management of quality assurance and improvement (ABET C4)
4. Learning and teaching (ABET C3)
5. Student administration and support services (ABET C1)
6. Learning resources (Part of ABET C7)
7. Facilities and Equipment (Part of ABET C7)
8. Financial planning and management (ABET C8)
9. Employment processes (Part of ABET C6)
10. Research (Part of ABET C6)
11. Institutional relationships with the community (ABET C2)
Course Design Workshop
NCAAA Standards for Accreditation and
Quality Assurance1. Mission and objectives (ABET C2)
2. Governance and administration (ABET C6 )
3. Management of quality assurance and improvement (ABET C4)
4. Learning and teaching (ABET C3)
5. Student administration and support services (ABET C1)
6. Learning resources (Part of ABET C7)
7. Facilities and Equipment (Part of ABET C7)
8. Financial planning and management (ABET C8)
9. Employment processes (Part of ABET C6)
10. Research (Part of ABET C6)
11. Institutional relationships with the community (ABET C2)
Course Design Workshop
NCAAA Domains of Learning Outcomes
knowledge, the ability to recall, understand, and
present information including: 1. knowledge of
specific facts, 2. knowledge of concepts, principles
and theories, and 3. knowledge of procedures.
cognitive skills, the ability to apply understanding
of concepts, principles, theories and procedures in
critical thinking and creative problem solving, both
when asked to do so and when faced with
unanticipated new situations.
September 11, 2014 Course Design Workshop 13
September 11, 2014 14
Interpersonal skills and responsibility, the ability to: 1. take
responsibility for their own learning and continuing personal and
professional development, 2. work effectively in groups and exercise
leadership when appropriate, 3. act responsibly in personal and
professional relationships, and 4. act ethically and consistently with high
moral standards in personal and public forums.
Communication, information technology and numerical skills,
including the ability to: 1. communicate effectively in oral and written
form, 2. use information and communications technology, and 3. use
basic mathematical and statistical techniques.
Psychomotor skills involving manual dexterity (this domain applies
only to some programs).
NCAAA Domains of Learning Outcomes
Course Design Workshop
NCAAA Domains of Learning Versus
ABET Outcomes• Domain of Knowledge (Not existing in ABET):
1. Ability to recall, understand, and present knowledge of facts, concepts,
principles and theories of Math and basic sciences.
2. Ability to recall, understand, and present knowledge of discipline-related
practices and procedures.
• Domain of cognitive skills (corresponding to a, b, c, e, h, & j)
3. Ability to identify, formulate and solve real-life discipline related
problems.
4. Ability to design, implement and evaluate a discipline related system or
process to satisfy desired needs.
5. Ability to identify, analyze, and critically evaluate arguments and take
informed criteria-based decisions.
NCAAA Domains of Learning Versus
ABET Outcomes
• Domain of Interpersonal skills (corresponding to i, d, & f)
6. Ability to take responsibility for their own learning and
continuing personal and professional development
7. Ability to work effectively in groups and exercise leadership
when appropriate.
8. Ability to act responsibly in personal and professional
relationships.
9. Ability to act ethically and consistently with high moral
standards in personal and public forums.
NCAAA Domains of Learning Versus
ABET Outcomes
• Domain of Communication, IT, and numerical skills (corresponding to a, g, & k)
10. Ability to communicate effectively in oral and written forms
11. Ability to use information and communications technology.
12. Ability to use basic mathematical and statistical techniques.
• Domain of psychomotor skills (not existing for engineering, IT, or managerial education)
Why Learning Outcomes?ABET Definition of Program Learning Outcomes
Statements that describe what the students are expected to know and be able to do by the time of graduation. These relate to the knowledge, skills, and behaviors that the students acquire as they progress through the program.
19September 11, 2014 Course Design Workshop
Why learning outcomes?
Course Learning Outcomes
Each course should have a set of specific learning outcomes that lead to:
More measurable student outcomes
Better assessment
Higher quality feedback
Improved courses and programs
Improved student learning and achievementSeptember 11, 2014 20Course Design Workshop
IntendedLearning
Outcomes ofthe Lesson
IntendedLearning
Outcomes ofthe Unit
IntendedLearning
Outcomes ofthe Course
Deliver Forward
Design Backward
Infusion of Learning Outcomes into Courses
ProgramOutcomes
September 11, 2014 21Course Design Workshop
22
Criterion 3 Program Learning Outcomes (a-k)
a)an ability to apply knowledge of mathematics, science, and engineering,
b)an ability to design and conduct experiments, as well as to analyze and interpret data,
c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability,
d)an ability to function on multi-disciplinary teams,
e)an ability to identify, formulate, and solve engineering problems,
Course Design WorkshopSeptember 11, 2014
23
f) an understanding of professional and ethical responsibility,
g) an ability to communicate effectively,h) the broad education necessary to understand the
impact of engineering solutions in a global and societal context,
i) a recognition of the need for, and an ability to engage in life-long learning,
j) a knowledge of contemporary issues, andk) an ability to use the techniques, skills, and modern
engineering tools necessary for engineering practice.
Course Design WorkshopSeptember 11, 2014
Criterion 3 Program Learning Outcomes (a-k)
NCAAA and ABET Definitions
for Learning and Teaching
Program Educational Objectives:
Broad statements that describe what graduates are expected to attain within a few years after graduation. Program educational objectives are based on the needs of the program’s constituencies.
Program Learning Outcomes:
Describe what students are expected to know and be able to do by the time of graduation. These relate to the knowledge, skills, and behaviors that students acquire as they progress through the program.
24Course Design WorkshopSeptember 11, 2014
NCAAA and ABET Definitions
for Learning and Teaching
Course Learning Outcomes:
Describe what students are expected to know and be able to do by the end of the course delivery. These relate to the knowledge, skills, and behaviors that students acquire as they progress through the course.
25Course Design WorkshopSeptember 11, 2014
Assessment : one or more processes that identify, collect, and prepare data to evaluate the attainment of student outcomes and program educational objectives. Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the outcome or objective being measured. Appropriate sampling methods may be used as part of an assessment process.
26Course Design WorkshopSeptember 11, 2014
NCAAA and ABET Definitions
for Learning and Teaching (Cont’d)
Evaluation: one or more processes for interpreting the data and evidence accumulated through assessment processes. Evaluation determines the extent to which student outcomes and program educational objectives are being attained. Evaluation results in decisions and actions regarding program improvement
27Course Design WorkshopSeptember 11, 2014
NCAAA and ABET Definitions
for Learning and Teaching (Cont’d)
Mission, Objectives, Outcomes & Learning Experience
Learning
Experience
Curriculum &
Other Activities
Student
Outcomes
of the
Program
Program
Objectives
Mission
Accomplished
Undergraduate
Studies
Upon
Graduation
Work
environmentMission
September 11, 2014 28Course Design Workshop
Mission, Objectives, Outcomes & Learning Experience
Learning
Experience
Curriculum &
Other Activities
Program
Learning
Outcomes
Program
Objectives
Mission
Accomplished
Undergraduate
Studies
Upon
Graduation
Work
environmentMission
September 11, 2014 29Course Design Workshop
Mission, Objectives, Outcomes & Learning Experience
Learning
Experience
Curriculum &
Other Activities
Program
Learning
Outcomes
Program
Objectives
Mission
Accomplished
Undergraduate
Studies
Upon
Graduation
Work
environmentMission
September 11, 2014 30Course Design Workshop
Mission, Objectives, Outcomes & Learning Experience
Learning
Experience
Curriculum &
Other Activities
Program
Learning
Outcomes
Program
Objectives
Mission
Accomplished
Undergraduate
Studies
Upon
Graduation
Work
environmentMission
September 11, 2014 31Course Design Workshop
Student Learning Experience
A path to achieve Program Learning Outcomes and foster the attainment of program educational objectives:
1. Curriculum Design (into courses and other learning activities)
2. Course Design3. Teaching and Learning4. Assessment (course-level and program-level)5. Evaluation and continuous improvement
Teaching versus Learning
• Education has become outcome-based instead of being content-based.
• Educational activities became centered around learning instead of teaching.
INPUTS OUTCOMES
INPUTS OUTCOMESLearning
Teaching versus Learning
• Education has become outcome-based instead of being content-based.
• Educational activities became centered around learning instead of teaching.
INPUTS OUTCOMES
INPUTS OUTCOMESLearning
Domains of Learning
Levels of Learning
Writing Course Learning Outcomes
36September 11, 2014
Session # 2
Course Design Workshop
Domains of the Learning Outcomes
The outcomes of any learning activity, or the change in the learner’s behavior, may fall in one of 3 distinguished domains:
– Cognitive Domain that involves knowledge and the development of intellectual skills
– Affective Domain that includes the manner in which we deal with things emotionally, such as feelings, values, appreciation, enthusiasms, motivations, and attitudes.
– Psychomotor Domain that includes physical movement, coordination, and use of the motor-skill areas. Development of these skills requires practice and is measured in terms of speed, precision, distance, procedures, or techniques in execution.
38September 11, 2014 Course Design Workshop
Cognitive Domain (mental) – Levels of Learning
Knowledge
Comprehension
Application
Analysis
Synthesis (Creative Thinking) Evaluation (Critical Thinking)
40September 11, 2014 Course Design Workshop
Levels of Learning
Classical Terms New Terms• Evaluation
• Synthesis
• Analysis
• Application
• Comprehension
• Knowledge
•Creating
•Evaluating
•Analyzing
•Applying
•Understanding
•Remembering(Based on Pohl, 2000, Learning to Think, Thinking to Learn, p. 8)
The names of six major categories were changed from noun to verb forms. 41September 11, 2014 Course Design Workshop
Affective Domain (Attitudes) – Degrees of
internalizationThis domain includes the manner in which we deal with things emotionally, such as feelings, values, appreciation, enthusiasms, motivations, and attitudes.
1. Receiving: Awareness. Willingness to hear. Selected Action Verbs : chooses, asks,
replies, selects
2. Responding: Active participation. Attends to and demonstrates compliance,
willingness or satisfaction in responding. Selected Action Verbs: complies, conforms, assists
3. Valuing: Is based on the internalization of a set of values, and are expressed in overt and
identifiable behavior. Selected Action Verbs: Demonstrates, proposes, initiates, justifies
4. Organizing: Values are organized into priorities. The emphasis is on comparing, relating,
and synthesizing values. Selected Action Verbs: arranges, integrates, synthesizes, defends
5. Internalizing (characterization): A value system that controls behavior. Behavior
is pervasive, consistent, predictable, and characteristic of the values held. Selected Action Verbs : acts, proposes, discriminates.
Psychomotor Domain (levels of physical skills)
The psychomotor domain includes physical movement, coordination, and use of the motor-skill areas. Development of these skills requires practice and is measured in terms of speed, precision, distance, procedures, or techniques in execution. 1. Set: Readiness to act Includes mental, physical and emotional sets. Selected Action Verbs :
begins, displays, moves, proceeds2. Guided Response: Early stages in learning a complex skill. Involves imitation and trial &
error. Selected Action Verbs: copies, traces, follows, reproduces, responds3. Mechanism: Intermediate skill in learning a complex skill. Includes habituation and
proficiency. Selected Action Verbs: assembles, constructs, dismantles, displays4. Complex Overt Response: Skillful performance that involves complex movement patterns.
Proficient in speed and accuracy. Selected Action Verbs: Same as Mechanism, but will have adverbs that indicate performance is quicker, more accurate and automatic
5. Adaptation: Skills are well-developed and the individual can modify to fit special requirements. Selected Action Verbs: adapts, alters, revises
6. Origination: Creating new patterns to react to particular situations. Emphasis on
creativity based on highly developed skills. Selected Action Verbs: Builds, combines, composes.
A Fluid Mechanics Example
Level Explanation Sample Question
RememberingThe student is expected to recite memorized
information about the concept.What is a fluid?
UnderstandingThe student is expected to explain the concept
in his or her own words.
Why mercury is considered as a
fluid?
ApplyingThe student is expected to apply the concept to
a particular situation.
Calculate the position of the
shock wave in the given nozzle
Analyzing
The student is expected to separate materials or
concepts into component parts so that their
organizational structure may be understood.
Analyse the flow field inside the
test section a high speed wind
tunnel.
EvaluatingThe student is expected to make judgments
about the value of ideas or materials.
Compare the given two nozzle
designs that perform the same
task, which one is better and
why?
Creating
The student is expected to put parts together to
form a whole, with emphasis on creating a new
meaning or structure.
Design a variable area nozzle to
execute the following flight
mission.
A Computer Programming Example
Level Explanation Sample Question
RememberingThe student is expected to recite memorized
information about the concept.What is a program?
UnderstandingThe student is expected to explain the concept
in his or her own words.
How is a program similar to a
recipe?
ApplyingThe student is expected to apply the concept to
a particular situation.
What is the output of the
following program
Analyzing
The student is expected to separate materials or
concepts into component parts so that their
organizational structure may be understood.
Identify the customer high level
requirements based on the
following customer interview.
What additional questions you
may need?
EvaluatingThe student is expected to make judgments
about the value of ideas or materials.
Given two programs that
perform the same task, which
one is better and why?
Creating
The student is expected to put parts together to
form a whole, with emphasis on creating a new
meaning or structure.
Write a program to perform a
given task
Writing Learning Outcomes
• While it is possible to write learning outcomes of all types for each of the three domains, the vast majority are written for the cognitive domain.
Action Verbs (used in writing learning outcomes)
Taxonomy Categories Sample Verbs for Stating Specific Learning Outcomes
Remembering
Understanding
Applying
Analyzing
Evaluating
Creating
Cite, Define, Identify, Label, List, Match, Name, Recognize, Reproduce, Select, State
Classify, Convert, Describe, Distinguish Between, Explain, Extend, Give Examples,
Illustrate, Interpret, Paraphrase, Summarize, Translate
Apply, Arrange, Compute, Construct, Demonstrate, Discover, Modify, Operate,
Predict, Prepare, Produce, Relate, Show, Solve, Use, calculate
Analyze, Associate, Determine, Diagram, Differentiate, Discriminate, Distinguish,
Estimate, Infer, Order, Outline, Point-out, Separate, Subdivide, Model.
Appraise, Assess, Compare, Conclude, Contrast, Criticize, Discriminate, Evaluate,
Judge, Justify, Support, Weigh
Combine, Compile, Compose, Construct, Create, Design, Develop, Devise, Formulate,
Integrate, Modify, Organize, Plan, Propose, Rearrange, Reorganize, Revise, Rewrite,
Tell, Write
48September 11, 2014 Course Design Workshop
September 11, 2014 49
Learning
DomainsMeaning Suggested Verbs
Knowledge
List of basic knowledge that
students should know and
understand when they complete the
course.
list, name, record, define, label,
outline, state, describe, recall,
memorize, reproduce, recognize,
record, tell and write.
Cognitive
Skills
List of the thinking and problem
solving skills the course is intended
to develop. The list should include
the ability to:
(1) Use analytic and predictive
formulae as well as conceptual
tools that students should use when
asked to do so.
(2) Identify and use the tools that
are appropriate for new and
unanticipated problems.
estimate, explain, summarize, write,
compare, contrast, diagram,
subdivide, differentiate, criticize,
calculate, analyze, compose, develop,
create, prepare, reconstruct,
reorganize, summarize, explain,
predict, justify, rate, evaluate, plan,
design, measure, judge, justify,
interpret and appraise.
NCAAA Suggested Guidelines for Learning Outcome Verbs
Course Design Workshop
September 11, 2014 50
Learning
DomainsMeaning Suggested Verbs
Interpersonal
Skills &
Responsibility
List of the outcomes related to
students’ interpersonal skills,
capacity for self directed learning,
and personal and social
responsibility.
demonstrate, judge, choose,
illustrate, modify, show, use,
appraise, evaluate, justify,
analyze, question, and write.
Communication,
Information
Technology,
Numerical
List of outcomes related to
students’ communication, IT and
numerical skills.
demonstrate, calculate,
illustrate, interpret, research,
question, operate, appraise,
evaluate, assess, and criticize.
NCAAA Suggested Guidelines for Learning Outcome Verbs
Course Design Workshop
Elements of Course Design
Step by Step Course Design & Assessment
Course Design Workshop
Session # 3
Elements of Course Design
Project
Course Goals & Course Intended
Learning Outcomes
Instruction
Lectures Labs
Institutional technologies
Student expectations
Syllabus
Assessment
Homework
Exams and Quizzes
StudentPortfolio
Surveys
Process Check
Student outcomes
Instructor's goals
Levels of Learning
Students
Plus & Delta
Modified from Felder and Brent
Learning Methods: Problem-Based, Project-based, Active Cooperative
Feedback from ABET Program Committee
To A
BET
Pro
gram
C
om
mit
tee
STEP BY STEP COURSE DESIGN & Assessment
1. Determine Program Core Courses
2. Apply Course Design Methodology for each Core Course:• Define the goals of the course• Define the Course Intended Learning Outcomes• Construct the Course Mapping Matrix• Construct the Course Articulation Matrix• Prepare the course syllabus
3. Construct Mapping Matrix for all core courses
4. Apply Course Assessment Methodology for each Core Course
5. Evaluate Course Assessment results
Course Design Workshop
Program Core Courses
Program core courses: a set of courses designated to address some or all of the Program outcomes.
Compulsory courses in the Program would be obvious candidates for the core.
Compulsory courses given by other Programs, such as math & physics might be included as long as they consistently address outcomes.
Elective courses or courses whose content varies from one offering to another can't be included.
Course Design Workshop
Step #1: Define the goal or goals of the course
• Two to three lines :“The course is intended to……. ”
• Example (Data Base Management Systems):
Example: The course is intended to:
“Develop the student’s ability to understand, design, and implement Data Base Management tools.”
Course Design Workshop
Step #2: Define The Course Intended Learning Outcomes
• Intended Learning Outcomes: specific observable (measurable) actions that the students should be able to perform if they mastered the course. i.e.
“ by the completion of this course, the students should be able to:1. …2. …………………”
The first word must be one action verb from the process descriptive verbs related to LOL (see the first part of this workshop)
Course Design Workshop
• Look at the items or concepts in detailed course content.
• For each item or concept decide what are the actions or tasks you want the students to do in order to demonstrate their level of mastery. Also think of how you will measure these actions (i.e. your assessment tools) and write down a suitable task statement.
• Each task statement includes one key action verb [such as list, explain, calculate, estimate, derive, model, design, choose, and criticize] along with a definition of the task and possibly a specification of the conditions under which the task is to be performed.
Course Design Workshop
Step #2: Define The Course Intended Learning Outcomes
• Do you want the students to define terms, explain them, apply them to known situations, to new situations, or use them to design a new artifact or process?
• Think of the levels of learning and their associated action verbs and associate a targeted level of learning to each course learning outcome.
• The best choice is to have a maximum of 9 to 12 course learning outcomes (one to 2 per course topic). The more they are detailed, the easier you could design their assessment, but you may find it difficult to have sufficient assessment tools.
Course Design Workshop
Step #2: Define The Course Intended Learning Outcomes
Course Title: Introduction to Database Systems
Current Catalog Description:
Introduction to the database management systems, database administration, data processing, data modeling, database design, development, and implementation. Contrasts alternative modeling approaches. Includes implementation of current DBMS tools and SQL.
Major Topics Covered in the Course:
• Data modeling
• Logical database design
• Physical database design (including performance issues)
• SQL and other database access methods
• Database Administration (including Data Warehousing and Administration)
Course Design Workshop
Step #2: Define The Course Intended LO (EXAMPLE)
Step #2: Define The Course Intended Learning Outcomes
(Good Examples)
Data modeling
1. Model data relevant to a database task, given written description, reports and other information from a systems user.
2. Model data using entity-relationship as well as object-oriented methods.
Logical database design
3. Transform entity-relationship model into a logical design, following the relational approach.
Physical database design (including performance issues)
4. Assess aspects of the physical design, and transform the logical design accordingly.
Course Design Workshop
Step #2: Define The Course Intended Learning Outcomes
(Bad Examples)
• Understand the fundamentals of data base design (not measurable)
• Students demonstrate that they can derive and properly applythe equations of …. (redundant words- what is meant by properly apply? In which situation)
Statements of non-observable actions (such as “learn”, “know”, “understand ”, or “appreciate”) might qualify as educational objectives BUT NOT Intended learning Outcomes
Understanding cannot be directly observed; the student must do something observable to demonstrate his understanding.
Course Design Workshop
Hands-on Exercise
Forming work teams
Define the course goals
Define a set of Intended course learning
outcomes
Presenting
Session # 3
Forming work Teams• Now we will form teams of 3 members each.
• All members of each team belong to one
program.
• Each team will select one core course that
falls in a domain of their common interest.
September 11, 2014 67Course Design Workshop
Step#1: Define the goal or goals of the course
(10 minutes)
• Use the catalogue description of the course to define its goals from the instructors view point
• Take 5 minutes to write 2 - 3 lines :“The course is intended to……. ”
• Remember the following example (Data Base Management Systems):
The course is intended to:“Develop the student’s ability to understand, Design, and implement Data Base Management tools.”
69September 11, 2014 Course Design Workshop
Step#2 Define the Course Intended Learning
Outcomes (30 minutes)
• take 20 minutes to prepare 4 to 5 intended course learning outcomes (1 to
2 CLOs per major topic).
• Notice that each CLO should start by an action verb
• Define the level of learning for each CLO.
• Map each CLO to one or more of ABET Program Learning Outcomes.
Follow the following examples:
Physical database design:
• Assess aspects of the physical design, and transform the logical design accordingly. (Evaluating, level #3, outcome c).
SQL and other database access methods
• Model data relevant to a database task, given written description from a systems user. (Analyzing, level #2, outcome c).
71September 11, 2014 Course Design Workshop
Presenting Your Work
73
• Now one team will come to podium.
• Take 5 minutes to present
September 11, 2014 Course Design Workshop
September 11, 2014 76
Session # 4 Learning Depth and Breadth
Apply Course Design Methodology for each
Core Course
Step #3. Construct the Course Mapping Matrix
Step #4. Construct the Course Articulation Matrix
Documenting Course Design Data
Course Design Workshop
1. The Depth of Learning is measured by level of learning in the cognitive domain
2. The breadth of learning is measured by the number of contact hours allocated to the learning activities targeting an outcome.
3. The 6 levels of learning are grouped into 3 levels of learning depth:• Level #1: low level of learning: Knowledge and Comprehension
or Remembering and Understanding (weight = 1).• Level #2: Medium level of learning: Application and Analysis or
Applying and Analyzing (weight = 2).• Level #3: High level of learning: Synthesis and Evaluation or
Evaluating and Creating (weight = 3).
Learning Depth and Breadth
September 11, 2014 78Course Design Workshop
Student’s Exposure
1. The product of depth of learning of a given student outcome by the corresponding breadth in a course indicates the student’s exposure to this outcome in that course as compared to other courses addressing the same student outcome.
2. Student outcomes that have high exposure in a given course are considered as the critical outcomes of that course and the course is considered as a key course for these outcomes.
3. Add a column to your articulation matrix containing the CLOcoverage hours.
4. Exposure of an outcome = SUM of (the Product of the alignment coefficient of the outcome with each CLO by CLO coverage).
Course Design WorkshopSeptember 11, 2014 79
STEP BY STEP COURSE DESIGN & Assessment
1. Determine Program Core Courses
2. Apply Course Design Methodology for each Core Course:• Define the goals of the course• Define the Course Intended Learning Outcomes• Construct the Course Mapping Matrix• Construct the Course Articulation Matrix• Prepare the course syllabus
3. Construct Program Mapping Matrix for all core courses
4. Apply Course Assessment Methodology for each Core Course
5. Evaluate Course Assessment results
Course Design WorkshopSeptember 11, 2014 81Course Design Workshop
Step #3: Construct the course mapping matrix “Course
Learning Outcomes – Program Learning Outcomes”
1: low levels of learning in Bloom’s Taxonomy (Remembering and understanding)2: medium levels of learning in Bloom’s Taxonomy (Applying and Analyzing)2: high levels of learning in Bloom’s Taxonomy (evaluating and creating)
IE 301 LOL
1 1 1
3 1 1
2 1
Maximum
Attainable1 3 3 2
a b c d e f g h i j kCLO_1
CLO_2
….
ml n
CLO_3
Course Design WorkshopSeptember 11, 2014 82Course Design Workshop
Step #4: Construct the course mapping
matrix (Continued)
• Some schools fill the matrix with an indicator to demonstrate to which degree (High, medium, low) this course learning outcome (CLO) contributes to the achievement of the corresponding student outcome (using letters such as H, M, and L).
• Other schools indicate the corresponding level of learning (LOL).
• A simplest method is to use a binary system (ONE if the Course LO addresses the Program LO and ZERO if it does not).
Course Design WorkshopSeptember 11, 2014 83Course Design Workshop
Documenting Course Design Data• A course articulation matrix assigns 1-2 Intended Course
Learning Outcome (CLOs) to each topic and competency (a total of 9 to 12 CLOs per course).
• Each CLO is associated with a certain number of hours of exposure (contact hours allocated to this CLO).
• In-class & out-of-class learning activities are indicated for each CLO.
• Classical assessment tools are indicated for each CLO.
• CLOs are mapped into Student Learning Outcomes (a-k).
• Key outcomes are identified as those having the highest exposure.
Empty Course Articulation Matrix
Course Design Workshop
Program LO
September 11, 2014 90Course Design Workshop
Why Assessment
• Helps students understand where they should focus their time & energies
• Motivates students to do their best
• Helps students understand their strengths and weaknesses through feedback
• Gives students documentation of what they have learned which can be used to apply for jobs, awards, programs of advanced study, etc.
Course Design Workshop
What Is Assessment? – ABET Definition
One or more processes that identify, collect, and prepare data to evaluate the attainment of student outcomes and program educational objectives. Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the outcome or objective being measured. Appropriate sampling methods may be used as part of an assessment process.
Course Design Workshop
Embedding Assessment With Regular Reflection
Course
Section 1
Teaching &
Learning
Assessment of
Section 1
Topics
Evaluation of
Achievement of
Section 1 CLOs and
Related SLOs Using
HI-CLASS
Revisions
of Course
Plan
Course
Section 2
Teaching &
Learning
Begin
Section 2Begin
Section 1
Embedding Assessment with Regular Reflection Approach Normal teaching Approach
Course Design Workshop
Direct Assessment Methods
(Clear and Compelling Evidence) • “Capstone” experiences (research projects, presentations, theses, oral
defenses, exhibitions, or performances) scored using a rubric • Other written work, performances, or presentations, scored using rubrics • Portfolios of student work • Scores on locally-designed multiple choice and/or essay tests (e.g. final
examinations in key courses, qualifying exams, comprehensive exams) accompanied by test “blueprints” describing what the tests assess
• Score gains between entry and exit on published or local tests or writing samples
• Summaries/analyses of electronic discussion threads • Student reflections on their values, attitudes and beliefs, if developing
those are intended outcomes of the course
Indirect Assessment Methods
(Signs that Students Are Probably Learning, But Exactly What or How Much They Are Learning is Less Clear)
• Course grades (see next slide)
• Assignment grades, if not accompanied by a rubric or scoring guide (See an Example)
• Student ratings of their knowledge and skills and reflections on what they have learned in the course or program
• Questions on end-of-course student evaluation forms that ask about the course rather than the instructor
Course Design Workshop
Assessment and Grading• They do have one common characteristic: both intend to identify what the students
have learned.
• Grades alone do not always give direct evidence: they don’t identify which specific student learning outcomes and at what levels students have learned.
• Some course grades include additional student behaviors that are not related to student learning outcomes (e.g. attendance and participation).
• For example, a grade of B in a mathematics course indicates a student learned an acceptable level of math, but not the specific math knowledge or skills that the student mastered well or what areas need improvement.
• A project or assignment grade would be a better link to the evidence of student learning, but a rubric for the project or assignment would provide the best direct evidence and the level at which the student learned.
• Grades are based on direct evidence of student learning such as the evaluations of tests, papers, and projects, but need to be clearly linked and aligned to learning outcomes and/or rubrics to suffice as direct evidence for assessment purposes.
98September 11, 2014 Course Design Workshop
Proposed Course Level Assessment Tools
• Each semester
– Indirect measures:• Course Learning Outcomes (CLOs) questionnaire (exit and
sometimes entry)
• Course in-class and out-of-class learning activities questionnaire (exit)
• Student Outcomes (SOs) questionnaire (exit)
– Direct measures:• Course Exams, HWs, Design projects, Written reports, Oral
presentations, Course portfolios, Lab Reports evaluated using rubrics or mapped into CLOs
• Assignments for key SOs evaluated using rubrics .
Course Design Workshop
Course Level Indirect Assessment
• Course Learning Outcomes (CLOs) Questionnaire:
– Put your course learning outcomes in the form of a questionnaire to measure the students confidence level in mastering the course learning outcomes. Use the 1 to 5 Eckert scale to cover Excellent-Medium-to-Poor mastery levels.
– Add some questions to evaluate the importance of course prerequisites.
– Distribute the questionnaire twice during the semester: in the 2nd week (entry, if you admit this approach) and/or in the 12th
week (exit) of the course.
– Answer the questionnaire yourself to indicate your estimation of the students average level of mastery. (see an example)
Course Design Workshop
Course Level Indirect Assessment (continued)
• Course learning activities (or instructional tools) Questionnaire:
– Ask the students to evaluate the impact of different in-class and out-of-class learning activities on their level of mastery of course learning outcomes. Use the 1 to 5 Eckert scale to cover Excellent-Medium-to-Poor impact.
– Distribute the questionnaire in 12th week.
– Answer the questionnaire yourself to indicate your estimation of the impact of the learning activities.
(see an example)
Course Design Workshop
Course Level Indirect Assessment (continued)
• Student Outcomes Questionnaire:– ABET Committee should divide each outcome into
outcome elements and to define outcome attributes (measures) or KPIs associated with these outcome elements.
– A questionnaire should be developed for each outcome and used by the instructor who pretends that his course addresses this outcome.
– The questionnaire is to be distributed during the 12th
week of the semester.
(see an example of outcome attributes/indicators and outcome Questionnaire)
Course Design Workshop
Course Level Direct Assessment
• Direct Assessment of:
– HI-CLASS or Similar Assessment of Course Learning Outcomes (CLOs) – Direct Approach
– Direct Rubric-Based Assessment of Key Student Outcomes.
Course Design Workshop
Construct the Articulation Matrix (15 minutes)
Now let us fill the Learning Activities and Assessment tools into the Matrix