Upload
simon-buckingham-shum
View
386
Download
2
Embed Size (px)
Citation preview
Simon Buckingham ShumConnected Intelligence Centre • University of Technology Sydney@sbuckshum • http://utscic.edu.au • http://Simon.BuckinghamShum.net
Towards Contested Collective Intelligence
or… A tour of the CI design space for Hypermedia Discourse
UniversityofMelbourne•SWARMProject,12th Sept.2017
Contested Collective Intelligence...
In wicked problems, there is no master worldview, ontology or logic
So disagreement is a necessary process and vital ingredient
We can disagree well or badly
CI tools should scaffold and improve this proess(e.g. amplify awareness of how stakeholders are framing the problem, reading
the signals, seeing connections, and judging success)
2De Liddo, A., Sándor, Á. and Buckingham Shum, S. (2012). Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, (4-5), pp. 417-448. http://doi.org/10.1007/s10606-011-9155-x
© Simon Buckingham Shum 5
Hypermedia Discourse
the modelling of discourse / the discourse of modelling
…reading and writing networks of documents, concepts, issues, ideas and arguments
Buckingham Shum, S. (2006). Sensemaking on the Pragmatic Web: A Hypermedia Discourse Perspective. In: 1st International Conference on the Pragmatic Web, 21-22 Sept 2006, Stuttgart, Germany. ePrint: http://oro.open.ac.uk/6442
© Simon Buckingham Shum 6
Discourse§ Dialogue§ Deliberation§ Argumentation § Reflection(Online & F-F Meetings)
© Simon Buckingham Shum 7
Hypermedia§ Modelling discourse relations§ Expressing different perspectives on a conceptual space§ Supporting the incremental formalization of ideas § Rendering structural visualizations§ Connecting heterogeneous content
© Simon Buckingham Shum 9
Notation /Visualisation
DiscourseModel
Key ingredients of a Hypermedia Discourse approach
© Simon Buckingham Shum 10
Notation /Visualisation
UserInterface
DiscourseModel
Key ingredients of a Hypermedia Discourse approach
© Simon Buckingham Shum 11
Notation /Visualisation
UserInterface
ComputationalServices
DiscourseModel
Key ingredients of a Hypermedia Discourse approach
© Simon Buckingham Shum 12
Notation /Visualisation
UserInterface
ComputationalServices
Literacy/Fluency
DiscourseModel
Key ingredients of a Hypermedia Discourse approach
Dilemma
If users are required to structure their contributions to a CI repository, the effort must
provide tangible benefit (not just potential benefits to future stakeholders)
Solution(in small synchronous settings)
A skilled mapper resolves the cost-benefit tradeoff, adding
immediate value to the sensemaking
Issue Mapping (or in a meeting real-time: Dialogue Mapping) based on Horst Rittel’s IBIS scheme
Buckingham Shum, S. (2003). The roots of computer supported argument visualization. In P. Kirschner, S. Buckingham Shum, & C. Carr (Eds.), Visualizing Argumentation (pp. 3–24). London: Springer. ePrint: http://bit.ly/VizArgRoots
http://compendiuminstitute.net
Issue Mapping (or in a meeting real-time: Dialogue Mapping) based on Horst Rittel’s IBIS scheme
https://www.youtube.com/watch?v=pxS5wUljfjE
Issue Mapping (or in a meeting real-time: Dialogue Mapping) based on Horst Rittel’s IBIS scheme
this simple set of moves — combined with hypertext,
and mapping fluency —goes a long way…
UKResearchExcellenceFramework(REF)2014ImpactCase
Compendium software (open source)visual hypermedia for managing the connections between ideas flexibly
Deep acknowledgements:
Jeff Conklin CogNexus Institute
Al Selvin & Maarten Sierhuis NYNEX Science & Technology —> Bell Atlantic —> Verizon—> NASA
http://compendiuminstitute.net
20
Structure management in Compendium
§ Associative linkingnodes in a shared context connected by graphical Map links
§ Categorical membership nodes in different contexts connected by common attributes via metadata Tags
§ Hypertextual Transclusionreuse of the same node in different views
§ Templates reuse of the same structure in different views
§ HTML, XML and RDF data exports for interoperability
§ Java and SQL interfaces to add services
Seven Sigma consulting: Issue/Dialogue Mappinghttp://www.sevensigma.com.au/what-we-do/sensemaking.html
“Knowledge Artistry” (Al Selvin)
Selvin, S. & Buckingham Shum, S. (2015). Constructing Knowledge Art: An Experiential Perspective on Crafting Participatory Representations. Morgan Claypool. http://doi.org/10.2200/S00593ED1V01Y201408HCI023
HypermediaDiscoursefluencyatahighlevel
27
Mapping with IBIS Issue-templates to harvest the firm’s collective
intelligence on Y2K contingencies
Selvin, A.M. and Buckingham Shum, S.J. (2002). Rapid Knowledge Construction: A Case Study in Corporate Contingency Planning Using Collaborative Hypermedia. Knowledge and Process Management, 9, (2), pp.119-128.
30
Generating Custom Documents and Diagrams from Compendium Templates
Selvin, A.M. and Buckingham Shum, S.J. (2002). Rapid Knowledge Construction: A Case Study in Corporate Contingency Planning Using Collaborative Hypermedia. Knowledge and Process Management, 9, (2), pp.119-128.
31
Using Compendium for personnel recovery operations planning
Conversational Modelling: real time dialogue mapping combined with model driven templates (AI+IA)
DARPA Co-OPR Project (PI: Austin Tate, AIAI, U. Edinburgh)http://www.aiai.ed.ac.uk/project/co-opr
© Simon Buckingham Shum 32
Mission Briefing: Intent template
Answers to template issues provided in the JTFC Briefing. Answers may be constrained
by predefined options, as specified in the XML schema
© Simon Buckingham Shum 33
Capturing political deliberation/rationale
Dialogue Map capturing the
planners’ discussion of this
option
© Simon Buckingham Shum 34
Planning Engine input to Compendium
Issues on which the I-X planning engine provided candidate Options
35
Mapping with IBIS to build a NASA science team’s collective intelligence for planetary geological exploration
Clancey, William J.; Sierhuis, Maarten; Alena, Richard L.; Graham, Jeffrey S.; Tyree, Kim S.; Hirsh, Robert L.; Garry, W. Brent; Semple, Abigail; Buckingham Shum, Simon J.; Shadbolt, Nigel and Rupert, Shannon M. (2007). Automating CapCom Using Mobile Agents and Robotic Assistants. In: 1st Space Exploration Conference: Continuing the Voyage ofDiscovery, 30 Jan-1 Feb 2005 , Orlando, FL, US. http://eprints.aktors.org/375
NASA remote science team tools
Scientist (Mars)
Scientist (Earth)
Scientist (Earth)
Scientist (Mars)
Scientist (Earth)
Software Agent Architecture
(Mars)
Compendium used as a collaboration medium at all intersections: humans+agents reading+writing IBIS maps
Geology dialogue map between Earth-based scientists and ‘Mars’
Copyright, 2004, RIACS/NASA Ames, Open University, Southampton UniversityNot to be used without permission
Compendium activity plans for surface exploration, constructed by scientists, interpreted by software agents
Compendium science data map, generated by software agents, for interpretation by Mars+Earth scientists
Meeting Replay tool: Earth scientists can browse a (simulated) Mars crew’splanning meeting using Compendium
Dilemma
While co-located mapping is fine for ‘micro-CI’, can we scale this to support asynch. ‘macro-CI’?
Numerous IBIS-based web apps
http://oystr.cohttp://debatemapper.net
http://evidence-hub.net
http://litemap.net
http://cci.mit.edu/klein/deliberatorium.html
50
interpretation
interpretation
interpretation
interpretation
Where our tools fit: we need ways to express interpretations
51
interpretation
interpretationinterpretation
interpretation
interpretation
(a hunch – no grounding
evidence yet)
interpretation
Where our tools fit: we need ways to express interpretations
…and optionally make meaningful connections
52
predictscauses
interpretation
interpretationinterpretation
interpretation
interpretation
(a hunch – no grounding
evidence yet)
interpretation
Is pre-requisite for
53
prevents
predictscauses
interpretation
interpretationinterpretation
interpretation
interpretation
(a hunch – no grounding
evidence yet)Is inconsistent with
interpretation
challenges
Is pre-requisite for
…and optionally make meaningful connections
Potentially moving towards stories that make sense of the evidence… i.e. plausible narratives / arguments
54
Question
Answer
Supporting Argument… Challenging
Argument…
challengessupports
responds to
Assumption
motivates
Potentially moving towards stories that make sense of the evidence… i.e. plausible narratives / arguments
55
Question
Answer
Supporting Argument… Challenging
Argument…
challengessupports
responds to
Hunch
motivates
56
Question
Answer
Supporting Argument… Challenging
Argument…
challengessupports
responds to
Data
motivates
Potentially moving towards stories that make sense of the evidence… i.e. plausible narratives / arguments
57
Convergence of…web annotationsocial bookmarkingconcept mappingstructured debate
a prototype platform for collective intelligence
Opening demo 2:30-10:30:https://www.youtube.com/watch?v=hxI5jPGScoU
Structured deliberation and debate in which Questions, Evidence and Connections are first class entities (linkable, addressable, embeddable, contestable…)
59
60
Structured deliberation and debate in which Questions, Evidence and Connections are first class entities (linkable, addressable, embeddable, contestable…)
63
Structured deliberation and debate in which Questions, Evidence and Connections are first class entities (linkable, addressable, embeddable, contestable…)
Comparison of one’s own ideas to others
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L.(2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
Does the learner compare his/her own ideas to that of peers, and if so, in what ways?
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1st Int. Conf. Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: http://oro.open.ac.uk/25829
What epistemic contributions are learners making in the community?
65
Rebecca is playing the role of broker,
connecting different peers’ contributions in
meaningful ways We now have the basis for recommending that
you engage with people NOT like you…
Evidence Hub: structured storytelling for students, practitioners and researchers
Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
A wizard guides the user through the submission of a structured story:• What’s the Issue?• What claim are you
making/addressing?• What kind of evidence
supports/challenges this?• Link it to papers/data• Index it against the core
themes
Evidence Hub: Argument Maps
Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
The wizard then generates a structured IBIS tree showing evidence-based claims (and disagreements)
Evidence Hub: professional developmenthttp://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning
Issue
PotentialSolution
SupportingEvidence
(practitionerstory)
Dilemma:
Unstructured deliberation platforms provide no scaleable assistance in making sense of
the collective’s progress
PainPointsinSocialInnovationPlatforms
Catalyst Project Deliverable:
http://catalyst-fp7.eu/wp-content/uploads/2014/02/CATALYST-Analysis-of-pain-points-and-user-feedback.pdf
PainPointsprioritised byorgs whorunsocialinnovationplatforms
Hardtovisualise thedebatePoorsummarisationPoorcommitmenttoactionSustainingparticipationShallowcontributionsandunsystematiccoveragePoorideaevaluation
PainPointsprioritised byorgs whorunsocialinnovationplatforms
Hardtovisualise thedebatePoorsummarisationPoorcommitmenttoactionSustainingparticipationShallowcontributionsandunsystematiccoveragePoorideaevaluation
Effectivevisualisation ofconcepts,newideasanddeliberationsisessentialforsharedunderstanding,butsuffersbothfromalackofefficienttoolstocreatethemandfromalackofwaystoreusethemacrossplatformsanddebates
“Asauser,visualisation ismybiggestproblem.Itisoftendifficulttogetintothediscussionatthebeginning.Asamanageroftheseplatforms,showingpeoplewhatisgoingonisthebiggestpainpoint.”
PainPointsprioritised byorgs whorunsocialinnovationplatforms
Hardtovisualise thedebatePoorsummarisationPoorcommitmenttoactionSustainingparticipationShallowcontributionsandunsystematiccoveragePoorideaevaluation
Participantsstruggletogetagoodoverviewofwhatisunfoldinginanonlinecommunitydebate.Onlythemostmotivatedparticipantswillcommitalotoftimetoreadingthedebateinordertoidentifythekeymembers,themostrelevantdiscussions,etc.
Themajorityofparticipantstendtorespondunsystematicallytostimulusmessages,anddonotdigestearliercontributionsbeforetheymaketheirowncontributiontothedebate,suchisthecognitiveoverheadandlimitedtime.
PainPointsprioritised byorgs whorunsocialinnovationplatforms
Hardtovisualise thedebatePoorsummarisationPoorcommitmenttoactionSustainingparticipationShallowcontributionsandunsystematiccoveragePoorideaevaluation
Bringingmotivatedaudiencestocommittoactionisdifficult.Enthusiasts,thosewhohaveaninterestinasubjectbuthaveyettocommittotakingaction,areleftbehind.
Needtopromptactionincommunitymembers
Reachingaconsensuswasconsideredlessimportantthanbeingenabledtoact.
PainPointsprioritised byorgs whorunsocialinnovationplatforms
Hardtovisualise thedebatePoorsummarisationPoorcommitmenttoactionSustainingparticipationShallowcontributionsandunsystematiccoveragePoorideaevaluation
Motivatingparticipantswithwidelydifferinglevelsofcommitment,expertiseandavailabilitytocontributetoanonlinedebateischallengingandoftenunproductive.
Sustainingparticipationismoreimportantthanenlargingparticipation.
“Itisbettertohavequalityinputfromasmallgroupthanalotofmembersbutverylittlecontent”.
PainPointsprioritised byorgs whorunsocialinnovationplatforms
Hardtovisualise thedebatePoorsummarisationPoorcommitmenttoactionSustainingparticipationShallowcontributionsandunsystematiccoveragePoorideaevaluation
Openinnovationsystemstendtogeneratealargenumberofrelativelyshallowideas.
Poorcollaborativerefinementofideasthatcouldallowthedevelopmentofmorerefined,deeplyconsideredcontributions.
Noeasywaytoseewhichproblemfacetsremainunder-covered.
Verypartialcoverageofthesolutionspace.
PainPointsprioritised byorgs whorunsocialinnovationplatforms
Hardtovisualise thedebatePoorsummarisationPoorcommitmenttoactionSustainingparticipationShallowcontributionsandunsystematiccoveragePoorideaevaluation
Patchyevaluationofideas
Poorqualityjustificationforideas.
Hardtoseewhyratingshavebeengiven.
Unclearwhichrationalesareevidencebased.
Solution
Activity analytics + IBIS semantics permit automated checking of the ‘health’ of a
conversation
Crowd-scale deliberation quality metrics + alertsLead: Mark Klein (MIT/Zurich)
https://www.youtube.com/watch?v=UZMJ9mti8h0
Problem-Goal-Exception (PGE) analysis using IBIS syntax checking for potential weaknesses in reasoning
http://catalyst-fp7.eu/wp-content/uploads/2016/01/CATALYST_WP4_D4.2b.pdf
Integrating deliberation metrics in the CI-dashboard
http://catalyst-fp7.eu/wp-content/uploads/2016/01/CATALYST_WP4_D4.2b.pdf
Integrating deliberation metrics in DebateHub
http://catalyst-fp7.eu/wp-content/uploads/2016/01/CATALYST_WP4_D4.2b.pdf
87
“Semantic Google Scholar” — ClaimFinder
Victoria Uren, Simon Buckingham Shum, Michelle Bachler, Gary Li, (2006) Sensemaking Tools for Understanding Research Literatures: Design, Implementation and User Evaluation. International Journal of Human Computer Studies, Vol.64, 5, (420-445).
Solution
Addition of social channels in an IBIS mapping web app can
restore a sense of connectedness
L. Iandoli, I. Quinto, S. Buckingham Shum, A. De Liddo (2015), On Online Collaboration and Construction of Shared Knowledge: Assessing Mediation Capability in Computer Supported Argument Visualization Tools, Journal of the Association for Information Science and Technology, 75 (5), pp.1052-1067
Async online IBIS Mapping + Social Cues is better than IBIS alone in some respects
Solution
Addition of social channels in an IBIS mapping web app can restore a sense of connectedness
BUT…
But the group using a Ning discussion forum still outperforms Social-IBIS and Plain-IBIS
MutualUnderstanding PerceivedEffectivenessofCommunication
L. Iandoli, I. Quinto, S. Buckingham Shum, A. De Liddo (2015), On Online Collaboration and Construction of Shared Knowledge: Assessing Mediation Capability in Computer Supported Argument Visualization Tools, Journal of the Association for Information Science and Technology, 75 (5), pp.1052-1067
DebateDashboard
sociallyaugmentedCoheremapping
Ningdiscussionforum Cohere
But the group using a Ning discussion forum still outperforms Social-IBIS and Plain-IBIS
AccuracyofPrediction(commodityprices)PerceivedEaseofUse
L. Iandoli, I. Quinto, S. Buckingham Shum, A. De Liddo (2015), On Online Collaboration and Construction of Shared Knowledge: Assessing Mediation Capability in Computer Supported Argument Visualization Tools, Journal of the Association for Information Science and Technology, 75 (5), pp.1052-1067
Writing is endlessly expressive and hard to improve on as a
medium for collective reflection/argumentation
(also a social process)
Dilemma:
But we would still like the machine to do some work for us in making sense of the state of
the CI process or product
Solution
NLP could move us beyond simple forum metrics, and help make sense of
the quality of contribution
Academic Writing Analytics: feedback on analytical/argumentative or reflective writing
Infohttps://utscic.edu.au/tools/awa
101
Highlighted sentences are colour-coded according to their broad type
Sentences with Function Keys have more precise functions (e.g. Novelty)
CIC’s automated feedback tool: analytical writing
CIC’s automated feedback tool: reflective writing
Aconcludingparagraphmovingintoprofessionalreflection:
1
CIC’s Text Analytics Pipeline (TAP) A set of linguistic analysis modules + AWA UI —> OSS release
Preparation of texts:text cleaning –> de-identification –> indexing –> metadata management
Analysis of texts:• Metrics: lengths of words, sentences, paragraphs, and statistics of these• Syllables: metrics at the word level based on syllables• Named Entities: e.g. names of People, Places• Statistics: e.g. noun-verb ratio• Vocabulary: compound words, occurrences at sentence, paragraph and document level• Expressions: epistemic, self-critique and affective compound words• Spelling: feedback on spelling and basic grammar• Rhetorical moves: in analytical and reflective writing• Complexity: measures of the complexity of words, sentences and paragraphs
Disputational talkcharacterised bydisagreementandindividualised decisionmaking.Fewattemptstopoolresources,toofferconstructivecriticismormakesuggestions.Disputational talkalsohassomecharacteristicdiscoursefeatures- shortexchangesconsistingofassertionsandchallengesorcounterassertions('Yes,itis.''Noit'snot!').
Cumulativetalkinwhichspeakersbuildpositivelybutuncriticallyonwhattheothershavesaid.Partnersusetalktoconstructa'commonknowledge'byaccumulation.Cumulativediscourseischaracterised byrepetitions,confirmationsandelaborations.
Mercer,N.(2004).Socioculturaldiscourseanalysis:analysing classroomtalkasasocialmodeofthinking.JournalofAppliedLinguistics,1(2),137-168.
Disputational/Cumulative/Exploratorytalk
Exploratorytalk• Partnersengagecriticallybutconstructivelywitheachother'sideas.
• Statementsandsuggestionsareofferedforjointconsideration.
• Thesemaybechallengedandcounter-challenged,butchallengesarejustifiedandalternativehypothesesareoffered.
• Partnersallactivelyparticipateandopinionsaresoughtandconsideredbeforedecisionsarejointlymade.
• Comparedwiththeothertwotypes,inExploratorytalkknowledgeismademorepubliclyaccountableandreasoning ismorevisibleinthetalk.
Disputational/Cumulative/Exploratorytalk
Mercer,N.(2004).Socioculturaldiscourseanalysis:analysing classroomtalkasasocialmodeofthinking.JournalofAppliedLinguistics,1(2),137-168.
-60
-40
-20
0
20
40
60
80
9:28
9:32
9:36
9:
40
9:41
9:
46
9:50
9:
53
9:56
10
:00
10:0
5 10
:07
10:0
7 10
:09
10:1
310
:17
10:2
3 10
:27
10:3
1 10
:35
10:4
0 10
:45
10:5
2 10
:55
11:0
4 11
:08
11:1
1 11
:17
11:2
0 11
:24
11:2
6 11
:28
11:3
1 11
:32
11:3
5 11
:36
11:3
8 11
:39
11:4
1 11
:44
11:4
6 11
:48
11:5
2 11
:54
12:0
012
:03
12:0
412
:05
Average Exploratory …
Discourse analytics on webinar textchat
Sheffield, UK not as sunny as yesterday - still warm
Greetings from Hong Kong
Morning from Wiltshire, sunny here!
See you!
bye for now!
bye, and thank you
Bye all for now
Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations?
Not at the start and end of a webinar…
Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
-60
-40
-20
0
20
40
60
80
9:28
9:32
9:36
9:
40
9:41
9:
46
9:50
9:
53
9:56
10
:00
10:0
5 10
:07
10:0
7 10
:09
10:1
310
:17
10:2
3 10
:27
10:3
1 10
:35
10:4
0 10
:45
10:5
2 10
:55
11:0
4 11
:08
11:1
1 11
:17
11:2
0 11
:24
11:2
6 11
:28
11:3
1 11
:32
11:3
5 11
:36
11:3
8 11
:39
11:4
1 11
:44
11:4
6 11
:48
11:5
2 11
:54
12:0
012
:03
12:0
412
:05
Average Exploratory …
Discourse analytics on webinar textchat
…but if we zoom in on a peak…
Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
Discourse analytics on webinar textchat
-100-50
050
100
9:28
9:40
9:
50
10:0
0 10
:07
10:1
7 10
:31
10:4
5 11
:04
11:1
7 11
:26
11:3
2 11
:38
11:4
4 11
:52
12:0
3 Classified as “exploratory
talk”
(more substantive for learning)
“non-exploratory”
…language is used in a manner more akin to “Exploratory Talk” (Neil Mercer)
Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
© Simon Buckingham Shum 110
Notation /Visualisation
UserInterface
ComputationalServices
Literacy/Fluency
DiscourseModel
So, this is the Hypermedia Discourse design space…
Practitioner Fluency
ModellingFrameworks
ComputingPlatform
LearningCurve
Mastery
Domain
Services Interoperability
Discourse
Interaction Design
EffectivenessExperience
Helpful evaluation criteria for CI platforms?
Consolidation of the previous elements into 3 classes of evaluation criteria
How does the Hypermedia Discourse design space and its tradeoffs compare to the SWARM platform?
What can we learn from each other?