1. Analytics insights into Epistemic Commitments Simon Knight http://sjgknight.com @sjgknight These slides are available at: http://www.slideshare.net/sjgknight/ And the video at: https://www.youtube.com/user/sjgknight

2. Presentation Background/motivation Epistemic commitments as behavioural trace Epistemic Network Analysis as analytic tool Pilot analysis on existing data 3. epistemological beliefs are a lens for a learner's views on what is to be learnt (Bromme, 2009) Certainty, simplicity, source, justification for knowing The Lens of Epistemic Beliefs Commitments 4. exploring students thought processes during online searching allows examination of personal epistemology not as a decontextualized set of beliefs, but as an activated, situated aspect of cognition that influences the knowledge construction process (Hofer, 2004, p. 43). The Lens of Epistemic Beliefs Commitments 5. What do students do, what do they make? Analytics give unprecedented(?) access Marks a shift, from standardised assessments to knowledge in action Psychometrics to situations Performance Assessment 6. Explicitly or implicitly learners: select sources of information have standards by which to judge credibility collate information in meaningful ways build arguments and make claims decide when to start and stop looking for information decide (implicitly or explicitly) the breadth and depth of information required Examples of epistemic behaviour 7. Epistemic commitments Operationalisation centered on connections between: Source selection Links between concepts Use of information to state claims, or reason points Tokens of information Within a search session By Pko (Own work) [Public domain], via Wikimedia Commons https://commons.wikimedia.org/wiki/File%3AAdditive_color_mixing_simulated.png 8. Epistemic commitments BBC thats a good site Answe rs.com lets see what another site says Nobelp rize.or g Its importa nt because It tells us about her It answers the question She won 2 prizes 9. Epistemic commitments BBC thats a good site Answe rs.com lets see what another site says Nobelp rize.or g Its importa nt because It tells us about her It answers the question She won 2 prizes 10. Epistemic commitments BBC thats a good site Answe rs.com lets see what another site says Nobelp rize.or g Its importa nt because It tells us about her It answers the question She won 2 prizes 11. Epistemic Frames for Epistemic Commitments 12. David Shaffer (2005, 2008) Epistemic Frames Discourse Oriented. Connections between (Community of Practice) elements usually described as: Skills, knowledge, values, identities, and epistemological rules. 13. Epistemic Frames and Epistemic Commitments we should try looking on Wikipedia for that 14. Use/justificationConnectionSourcingSearchClaims 15. Use/justificationConnectionSourcingSearchClaims 16. Epistemic Network Analysis (ENA) 17. 2005, 2008 David Williamson Shaffer A B D F C 18. Use/justificationConnectionSourcingSearchClaims 19. Use/justificationConnectionSourcingSearchClaims Model frames nodes are frame elements (keywords/codes), connections represent co-occurrence Novice v Expert network diagrams 20. Characterising groups For any stanza, code co-occurrence of frame elements coded 1 Singular Value Decomposition (like Principal Component Analysis) run across stanzas Creates a space to explore distances between groups of networks 21. Player 1: Good Morning Player 2: Hello everyone, is this chat with our new group now? Player 1: I think so Player 3: Probably Player 1: Good morning new team. I excited to work with with everyone. Player 4: hey Player 1: We are meeting and currently discussing our various materials. 22. Player 1: Good Morning Player 2: Hello everyone, is this chat with our new group now? Player 1: I think so Player 3: Probably Player 1: Good morning new team. I excited to work with with everyone. Player 4: hey Player 1: We are meeting and currently discussing our various materials. 23. Player 1: We ranked the different materials based on the device specifications and our experimental data. Player 2: And we just need to decide on the 5th team experimental device. Player 3: We looked at how the cost changed with each matieral while controling everything else Player 4: hey team, I am going to post the rankings of the membranes as a public document so everyone can see it. 24. Product 25. Product Professionalism Design Materials 26. Product Professionalism Design Materials 27. Product Professionalism Design Materials 28. Product Professionalism Design Materials 29. Professionalism Product Design Materials 30. Data used for this paper 31. The Data Previously analyzed dataset Applied ENA for epistemic commitments 32. The Collaboration Pair and two trios of female 11 year old English secondary school students Three reasons: We need the data Suggestion it is related to better seeking Suggestion certain modes of dialogue (exploratory) may be epistemic in nature 33. The Analysis 1 hour of assigned tasks on role models Closed (How many women have won the Nobel Prize?) & more open (Why do some people think Nelson Mandela is a good role model?). Asked to justify choices & state sources Analysis for epistemic & exploratory dialogue w/reference to screencast 34. Group differences Group 1 Quite successful, characterised: its got all the important information its a good site (quality) Group 2 Quite successful, characterised: I didnt know that (novelty), authority Group 3 Least successful, characterised: Theres loads there (quantity) 35. Applying ENA to prior data 36. Code Definition Source Authority Make reference to authorities when selecting information Source Corroboration Make reference to repetition of information when selecting information Justification Matching Little attempt to sensemake in question; targeted matching of source information to questions. Justification Understanding Attempt to make sense of information, use more accountable/exploratory dialogue key phrases Simplicity Simple Make few connections between information, look for individual tokens of knowledge Simplicity Complex Make connections between tokens of information, possibly across questions, contextualize information (e.g. account for temporal aspect) Search Referring to search or webpage specific aspects of the task General References to general knowledge required References to specific pieces of 37. ENA Group 1 Group 2 Group 3 (nodes have been combined to aid interpretation) 67% - x axis accounts, groups with low values making stronger connections with Simplicity Complex, and groups with high x values making stronger connections with Simplicity Simple and Source Corroboration. 32% - y axis largely distinguishes between Justification Match, and Source Authority with the use of question specific terminology. 38. ENA Group 1 Group 2 Group 3 (nodes have been combined to aid interpretation) y, representing stronger connections to matching, connecting to: understanding, making of Question General and Specific claims, and Simplicity Complex. X, focused on task requirements & linking information to these we have to say why it matters, so [fact] is important because Group 1 39. ENA Group 1 Group 2 Group 3 (nodes have been combined to aid interpretation) y, representing greater use of Authority making the same connections otherwise (the cluster in the middle). X, take info from authorities relevant to them without link to q.s the answers [fact], because its a good site. Group 2 40. ENA Group 1 Group 2 Group 3 (nodes have been combined to aid interpretation) X, perspective that simple knowledge obtained through corroboration is used to match general aims (and justified as such) although not actually targeting question specific knowledge claims Group 3 41. Epistemic Commitment Conclusions Epistemic commitments are seen in collaborative web search Connections can be seen in discourse and interactions with sources of information 42. ENA Conclusions ENA may provide methodological-conceptual resource for analysis Scope for expert v novice comparison Manual analysis comparison 43. Pre-empting a criticism Circularity? Preliminary validation that ENA offers a representational tool for scalable interpretation of epistemic commitment 44. Potential approach Hypotheses: Analysis of sets of connections will yield more insight than particular modes of working (e.g. looking for authority) E.g. Connection of sourcing- authority/corroboration to understanding will give greater insight 45. Thank you Simon Knight http://sjgknight.com @sjgknight Knight, S., Arastoopour, G., Williamson Shaffer, D., Buckingham Shum, S., & Littleton, K. (2014). Epistemic Networks for Epistemic Commitments. In International Conference of the Learning Sciences. Boulder, CO: International Society of the Learning Sciences. http://oro.open.ac.uk/39254/ See also Knight, S., & Mercer, N. (Forthcoming). The role of exploratory talk in classroom search engine tasks. Technology, Pedagogy and Education. http://oro.open.ac.uk/39181/ 46. Code Definition Example Source Authority Make reference to authorities when selecting information "use BBC, that's a good site", just use the top site, its a reliable site Source Corroboration Make reference to repetition of information when selecting information "we said x cuz it was on lots of sites", well everyone agreed Justification Matching Little attempt to sensemake in question; targeted matching of source information to questions. its an answer, look, it says it there, whats it asking? Justification Understanding Attempt to make sense of information, use more accountable/exploratory dialogue key phrases because, so, I think Simplicity Simple Make few connections between information, look for individual tokens of knowledge all the information, need more facts, listing of claims without connections Simplicity Complex Make connections between tokens of information, possibly across questions, contextualize information (e.g. account for temporal aspect) important information, making connections between claims (co-occurrence) Search Referring to search or webpage specific aspects of the task google that, click there, try searching for General References to general knowledge required Keywords selected for general task relevance; e.g. role models Specific References to specific pieces of information in each question Keywords selected for relevance to the specific questions asked e.g. 43 women 47. Some maths happens For the purposes of this analysis, I just presented the final ENA representation, without the trajectories There is interesting scope for the future to look at the trajectories of experts and novices in information seeking for example, do experts evaluate more throughout? Do novices emphasise authoritativeness only at the search stage? 48. Some maths happens For any stanza, if a connection exists ->1 we dont care how many times a connection is made, just that learners are making it PCA across stanzas