Upload
prosper-oconnor
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Portfolio Assessment: A literature review
Philip Smyth
English Centre
The University of Hong Kong
Overview
Historical perspectives Defining the portfolio Purposes of portfolios Issues in portfolio assessment Research in portfolio assessment My own possible research avenues
Historical perspective
Portfolios widely used for many years Late 80s interest in portfolios for
assessment (Belanoff and Dickson 1991)
90s saw advent of eportfolios A shift in emphasis away from
assessment to learning?
Definitions
“collection of student work that demonstrates achievement or improvement” (Stiggins 1994)
“a portfolio is a collection of evidence that is gathered together to show a person’s learning journey over time and to demonstrate their abilities” (Butler
2006)
Definitions
“…student writing over time, which contains exhibits showing the stages in the writing processes a text has gone through and the stages of the writer’s growth as a writer, and evidence of the writer’s self-reflection on her/his identity and progress as a writer” (Hamp-Lyons 1996)
Definitions
portfolios are “…prepared with a particular audience in mind”, “…are selective” and “call for judgments” (Calfee
and Freedman 1996)
Definitions
“…a purposeful collection of student work that illustrates efforts, progress, and achievement in one or more areas [over time]. The collection must include: student participation in selecting contents, the criteria for selection, the criteria for judging merit, and evidence of self-reflection” (The Northwest Evaluation Association cited in Barret 2005)
Definitions – main characteristics They are collections of work, different from
a single timed impromptu essay or a class essay carried out over a semester.
They are purposeful in that they “demonstrate”, “exhibit” or provide “evidence” of “achievement”, “improvement”, “the writer’s self reflection”, “the writing process” and “the writer’s growth”.
The degree to which these characteristics are evidenced in portfolios largely depends on their purpose.
Types of portfolio
a process portfolio a showcase portfolio an assessment
portfolio A dossier portfolio A reflective portfolio A classroom portfolio
A positivist portfolio A constructivist
portfolio A personal portfolio A structured portfolio An employment
portfolio A working portfolio
Where are portfolios used?
Primary and secondary classrooms In tertiary settings:
Teacher education Medicine Nursing Engineering Dentistry Psychology ESP/EAP writing classes
How are portfolios used?
In a class Across more than one class Statewide Across a university curriculum
Purposes
Accountability; evaluating program or curriculum effectiveness
Evaluating individual student progress; grading, certifying student accomplishment
Diagnosing students’ needs; informing classroom instructional planning
Purposes
Encouraging teacher efficacy; encouraging reflective practice at the school and classroom levels; supporting teachers’ professional development
Encouraging student efficacy; promoting student self-assessment; motivating student performance
(Herman, Gearhart and Acshbacher, 1996)
Issues
Different audiences (who is the portfolio for?)
Grading – who grades? (fairness) Learning and reflection get lost in
drive to measure competency (Herman and Winter 1994)
Issues
Gap between psychometrics and collaborative nature of the revision process (Song and August 2002,Hamp-Lyons and Condon 2000 )
Time needed for both teachers and students (Callahan 1995, Herman and Winters 1994)
Common problems
Purposes – clear to teacher and student? (Callahan 1995)Mismatch between assessment
criteria and goals of programme of study
Student anxiety and confusion (Butler 2006)
How do portfolios function best?
Research in portfolio assessment
A slim collection?
Research in portfolio assessment
Validity and reliability Fairness Impact
Validity
Vermont programCorrelation ranging from .47 to .58
between writing portfolio scores and direct writing assessments
Similar correlation between portfolio scores and multiple-choice maths test scores
(Koretz 1993 cited in Herman and Winter 1994)
Validity
Gearhart and others (1993) cited in Herman and Winter (1994) found: No relationship when comparing writing
portfolios with standard writing assessments Two thirds of students classified as
“masters” on the portfolio assessment would not have been so classified on the standard assessment.
Validity
Gearhart and others (1993) cited in Herman and Winter (1994) also found: When portfolios were scored in two different
ways (holistic and individual pieces scored) correlations were in the .6 range
Half the students who would have been classified as masters on the single portfolio score would not have been so classified when individual pieces were averaged
Which assessment best represents an enduring capability?
Validity
CUNY (Song and August 2002) 2 groups, 1 assessed by portfolio and writing
test, the other only a writing test Students twice as likely to move to the next
course when evaluated by portfolio At the end of the next course the pass rate
and grade distribution for the two groups were nearly identical
Reliability
Vermont interrater reliability of .28 to .60
Pittsburgh portfolio system ranged from .6 to .7
Herman et al. (1993)found correlations of .82 in an elementary school portfolio containing final drafts of writing
Reliability
Little work on other sources of portfolio reliabilityScore stability over timeStability across different rater groupsThe portfolio set in which a particular
portfolio is rated(Herman and Winter 1994, pg. 51)
Reliability
Heller, Sheingold and Myford (1998) think-aloud protocol on portfolio raters to see if they fit process model of portfolio rating
Found score validity was threatened when a major process was omitted or extraneous assessment criteria were applied
Reliability
Nystrand, Cohen and Dowling (1993) found reliability could be significantly improved if:Raters scored each task in response
to a prompt before moving to the next task and
Raters read several examples together to decide how they were to be rated
Impact
Herman and Winter (1994) based on self-reports from teachers and others implementing portfolios appears to have positive effects on instruction
Vermont principals affirmed that the portfolio assessment program had beneficial effects on curriculum and instruction
Impact
Aschbacher’s (1993) action research cited in Herman and Winter 1994 suggests teacher’s instructional practices and their attitudes towards students changed. Reported ways they thought about their own
teaching Two-thirds of teachers expected higher level
of performance from students
Impact
Hirvela and Sweetland (2005) used 2 case studies showing the 2 students did not strongly endorse the portfolios as used in 2 different courses. Seemed to need more explanations of what
portfolio approaches were meant to achieve Even with a 5% final course grade students
saw the portfolio as essentially summative in nature
Impact
Richardson (2000) study involved classroom observations teacher and student interviews and examination of student writing and teacher response. Found that students regard teacher
responses as directives. Were not prepared to make independent judgments largely because of the threat of grades
The future?
More technical quality Issue of fairness needs to be
addressed More research on impact A move away from psychometric
measurement?
Potential research
Equity of portfolio assessment Validity v reliability Nature of feedback Portfolios for employers
References
Abrami, P. C., & Barrett, H. (2005). Directions for Research and Development on Electronic Portfolios. Canadian Journal of Learning and Technology, 31(3).
Belanoff, P., & Dickson, M. (Eds.). (1991). Portfolios : process and product. Portsmouth, N.H.: Boynton/Cook Publishers.
Calfee, R. C., & Freedman, S. W. (1996). Classroom Writing portfolios:Old, New, Borrowed, Blue. In R. C. Calfee & P. Perfumo (Eds.), Writing Portfolios in the Classroom. MahWah, N. J.: L. Erlbaum Associates.
Callahan, S. (1995). Portfolio expectations: Possibilities and limits. Assessing writing, 2(2), 117-151.
Case, S. H. (1994). Will mandating portfolios undermine their value? Educational Leadership, 52(2), 46-47.
References
Desmet, C., & Cummings, R. (2004). Negotiating the Teaching-Assessment Cycle in Writing Programs with XML. Paper presented at the World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2004 Washington DC.
Frederiksen, J. R., Sipusic, M., Sherin, M., & Wolfe, E. W. (1998). Video Portfolio Assessment: Creating a Framework for Viewing the Functions of Teaching. Educational Assessment, 5(4), 225-297.
Hamilton, S. J. (2006). A Principle-Based ePort Goes Public (and Almost Loses its Principles). In A. Jafari & C. Kaufman (Eds.), Handbook of Research on ePortfolios (pp. 434-446). Hershey: Idea Group Reference.
Hamp-Lyons, L. (1990). Second language writing: assessment issues. In B. Kroll (Ed.), Second language writing : research insights for the classroom. Cambridge: Cambridge University Press.
References
Hamp-Lyons, L., & Condon, W. (2000). Assessing the Portfolio: Principles for Practice Theory and Research. Cresskill, NJ: Hampton Press.
Heller, J. I., Sheingold, K., & Myford, C. M. (1998). Reasoning about Evidence in Portfolios: Cognitive Foundations for Valid and Reliable Assessment. Educational Assessment, 5(1), 5-40.
Herman, J. L., Gearhart, M., & Aschbacher, R. (1996). Writing portfolios in the classroom : policy and practice, promise and peril. In R. C. Calfee & P. Perfumo (Eds.), Writing portfolios in the classroom : policy and practice, promise and peril
(pp. x, 374 p.). Mahwah, N.J.: L. Erlbaum Associates. Herman, J. L., Gearhart, M., & Baker, E. L. (1993). Assessing
writing portfolios: Issues in the validity and meaning of scores. Educational Assessment, 1(3), 201-224.
References
Herman, J. L., & Winters, L. (1994). Portfolio research: A slim collection. Educational Leadership, 52(2), 48-55.
Hirvela, A., & Sweetland, Y. L. (2005). Two case studies of L2 writers' experiences across learning-directed portfolio contexts. Assessing writing, 10, 192-213.
Holt, D., & Baker, N. W. (1991). Portfolios as a follow-up option in a proficiency-testing program. In P. Belanoff & M. Dickson (Eds.), Portfolios: process and product. Portsmouth NH: Boynton/Cook.
Jafari, A., & Kaufman, C. (Eds.). (2006). Handbook of Research on ePortfolios. Hershey: Idea Group Reference.
Murphy, S., & Camp, R. (1996). Moving towards systemic coherence: A discussion of conflicting perspectives on portfolio assessment. In R. C. Calfee & P. Perfumo (Eds.), Writing portfolios in the classroom: policy and practice, promise and peril. Mahwah, N.J.: L. Erlbaum Associates.
References
Nystrand, M., Cohen, A. S., & Dowling, N. M. (1993). Addressing reliability problems in the portfolio assessment of college writing. Educational Assessment, 1(1), 53-70.
O'Brien, K. (2006). ePortfolios as Learning Construction Zones. In A. Jafari & C. Kaufman (Eds.), Handbook of research on ePortfolios (pp. 74-82). Hershey: Idea Group Reference.
Ostheimer, M. W., & White, E. M. (2005). Portfolio assessment in an American college. Assessing writing, 10, 61-73.
Pullman, G. (2002). Electronic Portfolios Revisited: The efolios Project. Computers and Composition, 19, 151-169.
Richardson, S. (2000). Students' conditioned response to teachers' response: Portfolio proponents, take note! Assessing Writing, 7, 117-141.
References
Song, B., & August, B. (2002). Using Portfolios to Assess the Writing of ESL Students: A Powerful Alternative? Journal of Second Language Writing, 11, 49-72.
Spalding, E., & Cummins, G. (1998). It was the best of times. It was a waste of time: University of Kentucky students' view of writing under KERA. Assessing Writing, 5(2), 167-199.
Stiggins, R. J. (1994). Student-centred classroom assessment. New York: Merrill.
Wagner, M., & Lamoureaux, E. (2006). Implementing an Outcome-Based Assessment ePortfolio. In A. Jafari & C. Kaufman (Eds.), Handbook of Research on ePortfolios (pp. 539-550). Hershey: Idea Group Reference.
Weigle, S. C. (2002). Assessing writing. Cambridge, UK Cambridge University Press.
References
Yancey, K. B. (1996). Portfolio as genre, rhetoric as reflection: Situating selves, literacies, and knowledge. WPA 19(3), 55-69.
Yao, Y., Thomas, M., Nickens, N., Downing, J. A., Burkett, R. S., & Lamson, S. (2008). Validity evidence of an electronic portfolio for preservice teachers. Educational Measurement: Issues and Practice, Spring 2008, 10-24.