Assessment+summary+from+workshop+march26 2012 bonnie

Summary for Group 1 and Group 4: Scored Sample C for CT

WORKSHEET 1SAMPLE C

CT VALUE RUBRICSUMMARY OF SCORING & REASONS FOR SCORES

General Comment about VALUE Rubric: there is lack of fit with respect to sample C and the CT VALUE rubric belief that the student handled the assignment relatively well but scored a bit lower on the

VALUE Rubric than the group thought the student should have scored Important: how well the student performs seems linked to how well the assignment

positions students with respect to demonstrating the criteria in the VALUE rubric Critical thinking is learned across courses and in multiple places; Do assignments actually ask/require/lead students to demonstrate critical thinking

skills? Does course level work make a difference in evaluation? Need to add a zero and need to add a N/A column

A. EXPLANATION OF ISSUES:

General Comments: None

Score:Milestone 2: 9 people Milestone 3: 3 people

Reasons Noted: No thesis; no problem; some terms described, but not well; definition there but not comprehensive;

Handled well a very challenging topic; conclusion could have been the introduction; paragraph 1 was confusing, threw the reader off; ambiguities by statements that were not to be explicitly argued

B. EVIDENCE:


Score:Milestone 3 (or low 4): 4 peopleMilestone 2: 5 people In between Milestone 2 and Benchmark 1 (1.5 score): 1 personBenchmark 1: 1 person

Reasons Noted: summary; not in the analysis; no interpretation or integration of the evidence Selection of evidence was okay but it was not tied together; each selected

1

evidence was independent of ideas; not contemplation of why or contextC. INFLUENCE AND CONTEXT OF ASSUMPTIONS:

General Comments: Not clear what this means; difficult to define context in assignment as student approaches itThere was a lot of variability in our group in scoring and in our discussion; our scoring ranged from 0 3; unclear Score:Milestone 3: 2 peopleMilestone 2: 2 peopleBenchmark 1: 4 peopleZero: 2 peopleUnclear: 2 people

Reasons Noted:Circular arguments; did not question assumptions; did not explore assumptions; did not understand the context of assumptions; did not examine or question assumptions in context of others

D. STUDENT’S POSITION (Perspective, thesis/hypothesis)

General Comments:The student’s position was not there; the group did not score this because they were not sure how to with respect to the assignment. No scores given by group 4 Problems in determining the difference between (3) and (4) in the rubric (Group 1 comment: I (bonnie) am not sure if this refers to (3) or (4) with respect to scoring or with respect to the criteria)

Score:Benchmark 1: 5 peopleZero: 1 person

Reasons Noted: Listing rather than taking a position

E. CONCLUSIONS AND RELATED OUTCOMES (implications and consequences)

General Comments:NoneScore:Milestone 2: 3 peopleBenchmark 1: 9 peopleZero: 1 person

Reasons Noted: Very short with no elaboration or synthesis

2

WORKSHEET 2 SAMPLE C BHCC CT RUBRIC SUMMARY OF SCORING & REASONS FOR SCORES

General Comments about BHCC CT RUBRIC: specificity of rubric: less room for interpreting rubric; more discipline-specific rubric; major specific; doesn’t easily apply to this student artifact; rubric is less helpful Problems with matching the assignment and the rubric Assignment issue: problem with using this rubric to assess this assignment Seemed to be a higher general scoring (when looking at applicable criteria) using the

BHCC rubric than the LEAP VALUE Rubric Assignment is problematic; not much Critical Thinking Requested Bunker Hill rubric seemed to be built upon Blooms

A. IDENTIFIES ISSUES

General Comments: none

Score:Level 1: 1 personLevel 1.5: 3 peopleLevel 2: 4 peopleLevel 3: 5 people

Reasons Noted:Need assistance in understanding; shows minimal understanding of the assignment; says nothing about the assignment; Mix criteria in descriptionClearly addressed the assignment; lacked clarity but had more than a superficial understanding of the assignment and issues

B. UNDERSTANDS CONNECTIONS:General Comments: none

The tools did not line up for usScore:Level 1: 3 peopleLevel 1.5: 2 peopleLevel 2: 6 people

Reasons Noted:No thesis; little connection of points; connects some points – it is not just a summary; the writer has an issue but no thesis; Many in group 4 (4 people) ranked it a level 2 but thought it might be a 2.5 or low 3Three concepts presented but no there was no thesis statement; no connections; no synthesis; no discussion of each concept/characteristic of post-modern ideas

3

C. PROVIDES SUPPORT General Comments: What is evidence: linked to discipline?Group 4 eliminated the bottom bullet

Score:Level 1: 1 personLevel 2: 4 peopleLevel 2.5: 1 personLevel 3: 6 people (all group 4 people)

Reasons Noted:No opposing evidence presented; evidence as examples only; opposing evidence not relevant to assignment; okay support

Very …… on sub points;

Not all arguments/critical thinking require an opposing viewpoint(s); this should be noted on the VALUE rubric as well.

D. CONSIDERS COMPLEXITY

General Comments: The nature of the assignment no opposing perspectives used; “conclusions” should be in next category

Score:Level 1: 3 peopleLevel 2: 3 peopleN/A as defined: group 4 did not think this criterion as defined applied

Reasons group 4 gave: Limited complexity asked for; student was performing conceptual analysis; the student opens the door to introducing complexity/deepening the argument but the student does not actually do so, doesn’t follow through or walk through the door; student did not unpack the context; here we thought it important to note the difference between context versus opposition;Also, we thought it important to distinguish between the complexity in the object of study versus the complexity in the student’s analysis; we discussed that the complexity in the object of study contributed to demonstrating the complexity in student’s thinkingDefinition of complexity is problematic: complexity in student’s thinking versus the complexity in the topic the student is attempting to understand

4

E. DRAWS INFERENCES:

General Comments:Score 2 and Score 3 are too similar to discriminate between

Score:Level 1: 1 personLevel 2: 8 peopleLevel 3: 2 people (2.5 to 3)

Reasons Noted:Superficial Conclusion; No meta-cognitive activityConclusions were based upon evidence but were very superficial

F. UNDERSTANDS ASSUMPTIONS:

General Comments:Inconsistency in the rubric descriptors; rubric is confusingGroup 4 did not score; the group thought that this was not applicable; N/A the group was not sure how to assess; student’s assumptions? Assumption of authors of the readings the student was referring to; student’s state assumptions versus challenge assumptions of othersThere appears to be two separate processes here: stating/recognizing assumptions; challenging assumptions Score:Level 1: 1 personLevel 2: 1 personNot Applicable: 10 people

Reasons Noted:

G. IDENTIFIES IMPLICATIONS:

General Comments:Group 4 noted a lot of variability in scoring here: scores ranged from N/A to level 2 to level 3 to level 4

Score:Level 1: 6 people Level 2: 2 people Level 3: 3 people Level 4: 1 person

Reasons Noted:No thesis

5

WORKSHEET 3: COMPARING RUBRICS

Group 1 completed Worksheet 3Specificity of criteria makes the Bunker Hill rubric harder to apply; BHCC asks about adherence to the assignment – VALUE Rubric does notBHCC rubric seems to be discipline-specific in its criteria; seems like a research paper rubric

Group 4 completed Summary Report (not Worksheet 3)Criteria that aligned:Generally speaking, the BHCC rubric yielded higher ratings (at least in our discussion and general impression)Broad categories align, but the specifics varyRhetorical feature versus CT feature of BHCC rubricBHCC rubric was more suited to the Humanities versus the Social Sciences or Sciences; the BHCC rubric was less institutionally applicable than the VALUE rubricThe BHCC rubric specifically asks if the product/artifact addresses the assignment where the VALUE rubric does not

GENERAL COMMENT:

We should also consider rating the applicability of the assignment with respect to the rubric to be used in assessing the assignment

6

GROUP 6 SCORED UMASS BOSTON SAMPLE WORKSHEET 1

SAMPLE: UMASS BOSTON STUDENT SAMPLE CT VALUE RUBRIC SUMMARY OF SCORING & REASONS FOR SCORES

A. EXPLANATION OF ISSUES:

Score:Milestone 3: 3 people Milestone 4(3.5 -4): 3 people

Reasons Noted: Demonstrated meaning of “comprehensive: and “all”; strong piece

B. EVIDENCE:

Score:Milestone 3: 7 people

Reasons Noted: Use of superlatives; nature of timing and how to fulfill the assignment

C. INFLUENCE AND CONTEXT OF ASSUMPTIONS:

Score:Capstone 4: 3 peopleMilestone 3.5: 2 peopleMilestone 2: 2 people

Reasons Noted:Comparison of authors, put in historical time; interpretation as optimistic

D. STUDENT’S POSITION (Perspective, thesis/hypothesis) Score: Capstone 4: 3 peopleMilestone 3.5: 1 personMilestone 3: 3 people

Reasons Noted: Confidence in perspective; throws in side issue – militarism; because of timing wasn’t able to fully discuss limitations

E. CONCLUSIONS AND RELATED OUTCOMES (implications and consequences) Score:Capstone 4: 1 personMilestone 3.5: 1 person Milestone 3: 5 people

7

Reasons Noted: difficult to know if accurate use of sources

GROUP 6 SCORED UMASS BOSTON SAMPLE WORKSHEET 2UMASS BOSTON SAMPLEBHCC CT RUBRIC SUMMARY OF SCORING & REASONS FOR SCORES


Score:Level 1: 4 people

Reasons Noted:All issues in the assignment or all issues in the topic

B. UNDERSTANDS CONNECTIONS:

Score:Level 1: 5 peopleLevel 2: 3 people

Reasons Noted:Depends on how many times the essay is read for the smaller threads to fit

C. PROVIDES SUPPORT Score:Level 1: 6 peopleLevel 2: 2 people

Reasons Noted:Taking timing into consideration; easier to learn to 4 because of the descriptors in the rubric; more accurate descriptors to decide 3 or 4 A few times wanted more information



Reasons Noted:Difference between 2 and 3 is too great


General Comments:Rubric is constrictive

8

Score:Level 1: 8 peopleF. UNDERSTANDS ASSUMPTIONS:

Score:Level 1: 4 peopleLevel 2: 4 people

Reasons Noted:Nothing seen as misrepresentations; assumptions clearly identified and built into the conclusionSuperlatives were too extreme


Score:Level 1: 7 people High Level 2/Low Level 1: 1 person

Reasons Noted:Use of “most” significant wording in rubric

WORKSHEET 3: COMPARING RUBRICSGroup 6 completed this worksheet, not the summary sheet

Ways in which criteria are similar: BH level 4 is closer to a 3 on the VALUE BH 1 is the zero category Categories are similar

Ways in which criteria differ: Clearly addresses assignment – on BHCC ; not on VALUE Easier to make differences in 3 and 4 by descriptors on the BHCC rubric BHCC distinctions in all categories easier to make VALUE rubric is broader BH language is more useful, VALUE Rubric language is “too nice”

Summary of Groups 2 and 3

9

WORKSHEET 1 SAMPLE C CT VALUE RUBRIC SUMMARY OF SCORING & REASONS FOR SCORES

General Comments about VALUE rubric: Assignment is a big part of the problem. Need professional development for faculty on

how to elicit critical thinking since some assignments do not ask for CT Need N/A

A. EXPLANATION OF ISSUES

Score:

Benchmark 1 person Milestone 2: 9 people Milestone 3: 4 people

Reasons Noted: Issues undefined—gives list of examples—no real analysis

B. EVIDENCE:


Score:Benchmark 1: 5 people Milestone 2: 7 people Milestone 3: 2 people

Reasons Noted: Only asked to reference 1 source; has some interpretation Student did not address either topic but rather bits of each

C. INFLUENCE AND CONTEXT OF ASSUMPTIONS:

Score:Benchmark 1: 5 people Milestone 2: 6 people 2.5: 1 person Zero: 1 person

Reasons Noted: Had relevant quotations that aligned with assertions—didn’t question assumptions

No questioning of assumptions

D. STUDENT’S POSITION (Perspective, thesis/hypothesis)

10

Score:Benchmark 1: 9 people Milestone 2: 3 peopleMilestone 3: 2 people

Reasons Noted:

Didn’t find a coherent starting off point, so can’t get where he needs to be. Assignment is poorly designed

This is basically a description of postmodernism with examples

E. CONCLUSIONS AND RELATED OUTCOMES (implications and consequences)

Score:Level 1: 7 peopleMilestone 2: 8 people Zero: 3 people

Reasons Noted: Clear conclusion but no opposing viewpoints—simplistic conclusion Also assignment issue

If we were looking at a sample of papers that are representative of this sample, then students are not demonstrating critical thinking skillsQuestion arose from group regarding appropriateness of assignment itself for critical thinking—i.e., lack of outside evidenceDescribe knowledge necessary to make evaluations based on some characteristicsWe don’t know if sample reflects thinking on behalf of the student or a regurgitation of what was said in class“Zero” rating vs. “unable to rate” because it’s not applicable, etc.

WORKSHEET 2SAMPLE C

BHCC CT RUBRIC SUMMARY OF SCORING & REASONS FOR SCORES


General Comments: none


11

Level 2: 9 people

Reasons Noted: Didn’t show understanding of assignment

B. UNDERSTANDS CONNECTIONS:General Comments: none

The tools did not line up for usScore:Level 1: 6 people Level 2: 6 people

Reasons Noted:No real thesis; however, did connect key points

C. PROVIDES SUPPORT General Comments: Score:Level 1: 4 peopleLevel 2: 1 personLevel 2.5: 1 personLevel 3: 2 peopleN/A: 2 people

Reasons Noted:N/A. This is not an assignment permitting opposing viewpoints


Score:Level 1: 5 people Level 2: 7 people

Reasons Noted:No multiple perspectives—student does not understand notion of complexity and does understand postmodernismNo focus on how examples do or don’t illustrate


12

Score:Level 1: 1 person Level 2: 7 people Level 3: 3 people Level 4: 1 person Reasons Noted:Conclusions are consistently drawn even if somewhat simplistic

F. UNDERSTANDS ASSUMPTIONS:

Score:Level 1: 1 person Level 2: 3 people Level 3: 7 people

Reasons Noted:Good coherence; set groundwork for conclusions

Logical and consistent argument


Score:Level 1: 2 people Level 2: 5 people Level 3: 5 people

Reasons Noted:

We felt the assignment didn’t ask for thisDo we need more than the assignment—maybe a checklist form of sorts or narrative that “instructor” has identified appropriateness of assignment for particular assessmentIf we were doing this assessment “for real,” we would disqualify the assignment because it is not appropriate for critical thinkingOr if critical thinking is the outcome, should we even consider the original assignment?Not knowing what was covered in lecture does not allow for some sense of exit thinking or repeating lectureBullets within ratings for each characteristic ask if both or if just one achieved—what does that mean for the rating assigned to it?

Worksheet 3 (from groups 2 and 3)

13

Identify Ways in Which Criteria are Similar (AAC&U and BHCC rubric)

o Both look at identification of issueso Both include student perspective/positiono Both ask about evidence/supporto Elements of AAC&U are included in Bunker Hill rubrico Different scales in eacho Disciplines should be considered if appropriate vs. Joe/Jane vs. exit-level

considerations need to be taken

Identify Ways in Which Criteria Differ

o AAC&U’s does not allow for evaluation of whether or not artifact meets assignment requirements

Summary Worksheet (from Group 3)

Identify criteria that alignThe institutional rubric had more distinctions. However, when reviewing the AAC&U

rubric criteria, there are distinctions. Albeit slightly more hidden

Identify criteria that do not align and reasons why there is lack of alignment Assignment directions completely included on institutional rubricDiscipline influence is strong—or can be especially strong—particularly with higher levels on AAC&U rubric

List recommended ways to achieve greater alignment between the two sets of rubrics or identify ways in which institutional rubrics capture the criteria in the AAC &U rubricThe institutional rubric did not capture the criteria in the AAC&U rubric—although the separation of bullet points within the cells created confusion in our group

The type of assignment seemed key to the use of rubrics

While some of us were less comfortable with the institutional rubric, others were more comfortable with the outlined format

14

Technology

Assessment+summary+from+workshop+march26 2012 bonnie