Evaluating Digital Scholarship

Alison Byerly, College Professor, Middlebury College

NITLE Seminar, October 10, 2012

Evaluating Digital ScholarshipNITLE Seminar, Oct. 10, 2012

Alison Byerly , Middlebury College http://blogs.middlebury.edu/alisonbyerly/

Challenges of evaluating digital scholarship

May differ from traditional scholarship in:

(1) analytic approach or content and/or(2) format, medium, method of publication or dissemination

Digital humanities: not really a field or discipline - an approach, an array of practices. These practices are often more typical of sciences and social sciences than of traditional humanities

• data-focused or object-focused• visualization and spatial analysis• collaborative; -open-access and non-proprietary• process-oriented rather than product-oriented

Evaluation

• Good evaluation depends on shared context: clear expectations, defined roles, recognized measures of success

• Both parties – candidate and evaluator(s) –have a role to play in establishing the appropriate context

• Ideally, this work begins the moment a new colleague is hired – if not sooner

Essential tasks for digital scholars

Educate your colleagues

• Define the audience of peers addressed by your work, and actively engage that audience

• Document your specific role and effort, particularly if it is collaborative

• Explain the significance of specific achievements, recognition, or prestige markers

• Look for opportunities to create narratives describing your work

Essential tasks for evaluators

Educate yourselves

• Review and assess any project in the medium in which it was created.

• Recognize the intrinsically collaborative nature of digital projects.

• Consult specialists in relevant disciplines regarding the various components of a candidate’s work

• Assess candidate’s work in relation to overall institutional expectations of quality, recognizing that specific metrics of success will differ in new areas

Guidelines from professional organizations

• MLA (Modern Language Association)– Guidelines for Evaluation of Work in Digital Humanities and Digital Media http://www.mla.org/guidelines_evaluation_digital

• AHA (American Historical Association) – Suggested Guidelines for Evaluating Digital Media Activities in Tenure, Review, and Promotion http://www.historians.org/perspectives/issues/2001/0110/0110pro1.cfm

• 2011 volume of MLA's Profession contains some excellent essays (

http://www.mlajournals.org/toc/prof/2011/1),• -NINES ‘ Evaluating Digital Scholarship site, "Guidelines for Promotion and

Tenure Committees in Judging Digital Work" (http://institutes.nines.org/docs/2011-documents/guidelines-for-promotion-and-tenure-committees-in-judging-digital-work/).

New publication and presentation formats websites, online manuscripts, databases, apps…..

(1) Website NINES: Nineteenth-Century Scholarship Online http://www.nines.org/ 992,065 peer-reviewed

digital objects from 119 federated sites: The Old Bailey Online is a fully searchable edition of the largest body of texts detailing the lives

of non-elite people ever published, containing 197,745 criminal trials held at London's central criminal court.

The Yellow Nineties Online publishes digitized facsimile editions of a select collection of periodicals…a rich historical archive of paratextual materials related to the production and reception of these periodicals…also publish peer-reviewed scholarship: biographies of the writers, authors, publishers, and others associated with the period; scholarly introductions and commentary; and essays on our process of building the site and encoding its digital objects. All documents are marked-up and fully searchable.

(2) In-process online manuscript

Complex TV: The Poetics of Contemporary Television Storytelling by Jason Mittell

. http://mediacommons.futureofthebook.org/mcpress/complextelevision/‘This site hosts the peer-to-peer review of the in-progress manuscript Complex TV: The Poetics of Contemporary Television Storytelling by Jason Mittell (copyright 2012). The book proposal was posted in March 2011, with individual chapters released in serialized installments starting in Spring 2012. The project is currently under review contract with NYU Press, who has allowed me to post the pieces here for pre-publication and open-review. The draft manuscript with comments will continue to live online here, even after the book has been published.’

See also JM’s blog post, “Thoughts on Blogging for Tenure” http://justtv.wordpress.com/2012/01/04/thoughts-on-blogging-for-tenure/

(3) Database or program

Docuscope: a text analysis environment with a suite of interactive visualization tools for corpus-based rhetorical analysis. http://www.cmu.edu/hss/english/research/docuscope.html

Department of English|

DocuScope: Computer-aided Rhetorical Analysis

                                                                                                                                                                                    

Peer review – identifying the relevant scholarly audience or community

• HASTAC: Humanities, Arts, Sciences, and Technology Advanced Collaboratory http://hastac.org/ “HASTAC ("haystack") -- a network of individuals and institutions inspired by the possibilities that new technologies offer for shaping how we learn, teach, communicate, create… the digital era provides rich opportunities for informal and formal learning and for collaborative, networked research that extends across traditional disciplines, across the boundaries of academe and community, across the "two cultures" of humanism and technology, across the divide of thinking versus making….”)

• JDH: Journal of Digital Humanities http://journalofdigitalhumanities.org/“A comprehensive, peer-reviewed, open access journal that features the best scholarship, tools, and conversations produced by the digital humanities community in the previous quarter. JDH offers expanded coverage of the digital humanities ..by publishing scholarly work beyond the traditional research article…by selecting content from open and public discussions in the field…[and] by encouraging continued discussion through peer-to-peer review.”

• DHQ: Digital Humanities Quarterly http://www.digitalhumanities.org/dhq/vol/001/1/000007/000007.html

• Anvil Academic – A new all-digital, open-access publisher for the digital humanities. http://chronicle.com/blogs/profhacker/digital-killed-the-analog-star-an-interview-with-fred-moody-of-anvil-academic/42936

Digital Humanities – major outlets

--there are some printed books, too—

•A Companion to Digital Literary Studies, ed. Susan Schreibman and Ray Siemans. Blackwell, 2008.

•Debates in the Digital Humanities. Ed. Matthew Gold. U of Minnesota, 2012.

Other resources • MLA 2012 Workshop on Digital Scholarship

http://wiki.mla.org/index.php/WORKSHOP_2012

“The Challenges of Digital Scholarship,” Chronicle of Higher Ed, January 25, 2012 http://chronicle.com/blogs/profhacker/the-challenges-of-digital-scholarship/38103

• See 10/5/12 #anvil twitterchat re: evaluation of digital scholarship: http://storify.com/adelinekoh/anvil-academic-publishing-peer-reviewed-digital-wo#publicize