Upload
stefan-trausan-matu
View
1.402
Download
0
Tags:
Embed Size (px)
Citation preview
Stefan Trausan-Matu
PolyCAFe and Social Learning
Support for CSCL in LTfLL
9th December 2010
Stefan Trausan-Matu
Traian Rebedea
Mihai Dascalu
Vlad Posea
“Politehnica" University of Bucharest
Computer Science Department
General dataGeneral data
9 December 20102
The LTfLL - EU FP7 STREP Project (2008-2011)
• Open Universiteit Nederland (coordinator)
• The University of Manchester
• Open University UK
• Universiteit Utrecht
• Eberhard Karls Universität Tübingen• Eberhard Karls Universität Tübingen
• Wirtschaftsuniversität Wien
• Université Pierre-Mendès France, Grenoble
• Politehnica University of Bucharest (PUB)
• Institute for parallel processing of the Bulgarian Academy
• Aurus Kennis- en Trainingssystemen BV
• BIT MEDIA E-learning solution GMBH and CO KG
9 December 20103
Past International Research Projects
Related to LTfLL
� The cognitive paradigm (e.g. Web semantic)� Portable AI Lab (PAIL) – IDSIA Lugano (1994-1995)� PeKADS - EU Copernicus (1994-1996)� LARFLAST – EU Copernicus (1998-2001)� IKF – EU EUREKA (2002-2003)� Towntology – EU COST Action (2005-2008)
� The socio-cultural paradigm (e.g. Web2.0)� VMT –USA NSF (2005-2007)� EU-NCIT – EU FP6 (2005-2008)� Cooper – EU FP6 (2005-2007)
9 December 20104
The Outcomes of the LTfLL Project
Prototypes of next-generation services built on
advanced research on the application of
language technologies in education.
9 December 20105
http://www.ltfll-project.org
LTfLL Services
9 December 20106
LTfLL Development Cycles
9 December 20107
Workpackages
OUNL 19 7 10 11 11 3 2 63
UU 3 5 5 2 18 7 3 1 44
UTU 7 6 18 6 3 1 41
WUW 10 3 4 7 4 3 1 32
UPMF 1 1 19 3 2 1 27
PUB-NCIT 1 3 19 17 8 3 1 52
AURUS
KTS
3 6 1 6 3 1 20
Work
packageWP1 WP2 WP3 WP4 WP5 WP6 WP7 WP8 WP9 TOTAL per
Beneficiary
9 December 20108
KTS
UNIMAN 5 12 6 14 3 1 41
IPP-BAS 3 6 11 15 9 3 1 48
BIT
MEDIA
15 3 5 6 1 30
OU 8 3 6 3 1 1 22
TOTAL 19 58 51 68 53 56 70 33 12 420
PUB Implementation Team� Prof.dr.ing. Stefan Trausan-Matu
� Prof.dr.ing. Valentin Cristea
� As.drd.ing. Traian Rebedea
� As.drd.ing. Vlad Posea
� As.drd.ing. Mihai Dascalu
� As.drd.ing. Costin Chiru
� As.drd.ing. Dan Mihaila
� Ing. Alexandru Gartner
� Ing. Erol Chioasca
Students which helped at the implementation:
� Dan Banica
� Mihai Nicolae
� Iulia Pasov
� Ionela Voinescu
� Iulia Moscalenco
� Oana Mihai
� Alexandru Georgescu
9 December 20109
PUB Publications on LTfLL� 29 papers, 12 in Proceedings ISI, 23 in
international databases,
� 2 NLPSL Workshops with Proceedings (Eds. S. Trausan-Matu, P. Dessus)
� 4 book chapters (including published at Springer � 4 book chapters (including published at Springer and Hershey)
9 December 201010
The Problems Solved by
PolyCAFe(a Polyphony-Based System for Collaboration (a Polyphony-Based System for Collaboration
Analysis and Feedback Generation)
9 December 201011
Chat Conversations with Multiple
Participants
� Multiple participants (≥3), conferencing style
� Particular features – multiple, parallel discussion
chains !!!
There is a need for� There is a need for
� Determining important utterances
� Contributions of the participants
� Degree of collaboration - inter-animation analysis
9 December 201012
Example: CSCL assignment� Students had to debate in chat sessions in
groups ranging from 3 to 8
� In the first part of the conversation, each student had to defend a technology by presenting its features and advantages and criticize the others features and advantages and criticize the others by invoking their flaws and drawbacks
� In the final part of the chat, they had to discuss on how they could integrate all these technologies in a single online collaboration platform
9 December 201013
CSCL assignment: Problems
�How to assist teachers in evaluating students’ work in chats?
�Offer assistance to students
�Abstraction tools�Abstraction tools
�Automatic feedback
9 December 201014
Experiments with Chat-based
CSCL
�K-12 students solving mathematics problems both individually and collaboratively in the VMT project at Drexel University, Philadelphia, USA
�Computer Science students at Bucharest “Politehnica” �Computer Science students at Bucharest “Politehnica” University, Romania at
�Human-Computer Interaction course in Romanian and French –role playing and debate�Natural Language Processing - role playing and debate�Algorithm Design – problem solving
9 December 201015
The VMT chat environment
9 December 201016
VMT Referencing facility
9 December 201017
Theoretical Basis of Theoretical Basis of
PolyCAFe
9 December 201018
Paradigms about KnowledgeCognitive Socio-cultural
Newell, Simon Vygotsky, Bakhtin
“Knowledge is in the head” “Knowledge is in the community”
Artificial Intelligence,
Natural Language Processing
Theory of Activity,
Collaborative systemsNatural Language Processing Collaborative systems
Ontologies Folksonomies
Semantic Web Social Web (Web2.0)
Intelligent Tutoring Systems Computer-Supported Collaborative Learning
9 December 201019
Computer Supported Collaborative
Learning
� A new paradigm in learning with computers (Koshmann, 1999):
� Knowledge is constructed socially (Vygotsky)� Knowledge is constructed socially (Vygotsky)� Induced by the spread of forums, chats, blogs,
wikis and folksonomies � learning in (on-line) virtual teams and/or communities
9 December 201020
Dialogism – Mikhail Bakhtin
• Basis for the CSCL paradigm (Koschman, 1999)
• “… Any true understanding is dialogic in nature” (Voloshinov-Bakhtin, 1973)nature” (Voloshinov-Bakhtin, 1973)
• Opposed to de Saussure ideas, which are the basis for Natural Language Processing
� Polyphony
� Inter-animation of voices
9 December 201021
The Polyphonic Model of CSCL(Trausan-Matu, Stahl and Zemel, 2005, http://mathforum.org/wikis/uploads/Stefan_Interanimation.doc)
� A polyphony of voices characterizes any linguistic phenomenon (Bakhtin) including CSCL chats
� Inter-animation (Bakhtin, Wegerif) may be detected in interactions and it may be used for analyzing collaboration and assessing learnersassessing learners
� Integrating NLP techniques with polyphony identification and Social Network Analysis may provide a way for analyzing the contributions of each participant and their collaboration.
� Inter-animation and polyphony appears also in non-verbal interactions
� Consider threads (voices which last) rather than analyzing pairs of utterances
9 December 201022
Polyphonic analysis(Trausan-Matu & Stahl, 2007, http://gerrystahl.net/vmtwiki/stefan.pdf)
9 December 201023
249 December 201024
Words, voices and threads� Different positions assigned to participants – different
voices
� Additional voices – frequent concepts – repeated
words become voices, stronger or weaker
� Voices continue and influence each other through
explicit or implicit links. explicit or implicit links.
� Voices correspond to chains or threads of utterances:� repeated words
� lexical chains
� co-references
� reasoning or argumentation
� rhetorical schemas
9 December 201025
Analysis Units in the Polyphonic
Model
� Words
� Utterances
� Pairs of utterances (links)
� Threads� Threads
� Voices
� Participants
9 December 201026
Units of Interaction
� Echoes of voices
� Polyphonic-contrapuntal weaving
� Inter-animation
Links between utterances and between words� Links between utterances and between words
� Links may be:
� implicit
� explicit
9 December 201027
Inter-animation Patterns(Trausan-Matu, Stahl & Sarmiento, 2007)
� Longitudinal� Adjacency pairs
� Repetitions
� Elaboration
� Cumulative talk (collaborative utterances)� Cumulative talk (collaborative utterances)
� Repair
� Transversal � Convergence
� Differential
� Dissonance
9 December 201028
Implementation Details of PolyCAFeImplementation Details of PolyCAFe
9 December 201029
Architecture of PolyCAFE
309 December 201030
9 December 201031
NLP pipe� spelling correction, stemmer, tokenizer, Named Entity
Recognizer, POS tagger and parser, and NP-chunker.
Stanford NLP software
(http://nlp.stanford.edu/software)
� Spellchecker : Jazzy
http://www.ibm.com/developerworks/java/library/j-http://www.ibm.com/developerworks/java/library/j-
jazzy/
� Alternative NLP pipes are under development,
� GATE (http://gate.ac.uk)
� LingPipe (http://aliasi.com/lingpipe/).
9 December 201032
Functional architecture
Surface Analysis
Readability & Page essay grading
Surface Analysis
Readability & Page essay grading
Morphological Analysis and POS Tagging
Morphological Analysis and POS Tagging
Semantic Evaluation
(LSA)
Semantic Evaluation
(LSA)
Social Network
Analysis
Grading Process
9 December 201033
Tagged LSA• Corpus of chats focused on collaborative
technologies
• POS Tagging
• Stemming
• Segmentation• Segmentation
– Participants
– Fixed Window
• Cosine similarity
Term-Doc Matrix + Tf – Idf
9 December 201034
Vector space visualization
Radial Model
Physical Model
9 December 201035
Social Network Analysis• Degree
• Centrality
– Closeness
– Graph
– Eigen Value– Eigen Value
• User Ranking
– Google Page Ranking
9 December 201036
Utterance evaluation
Social
Qualitative
• Degree
• Semantic similarity
• Predefined topics
• Overall discourse
Quantitative • NLP Pipe
• No of occurrences
)()())_log(1()( =mark(u) usocialuemphasisoccurencesnostemlengthwordsremaining
××
+×∑
)_,()_,()( keywordspredefineduSimdocumentwholeuSimuemphasis ×=
∏ +=
)(
))(log(1()(
equalitativandvequantitatiffactorssocialall
ufusocial
9 December 201037
Collaboration (1)
• Utterance graph
– Explicit links
– Implicit links
• Social cohesion
– Attenuation
– Trust
• Social cohesion
• Quantitative collaboration
explicit)(implicit/ links ofnumber total
trust(l)*n(l)attenuatio
=ioncollaborat vequantitatispeakersdifferent with
l links all∑
9 December 201038
Collaboration (2)
� Qualitative - Gain based collaboration =
ECHO
� Personal – individual knowledge building
� Links to previous utterances with same speaker
� Collaborative – collaborative knowledge � Collaborative – collaborative knowledge building
� Links to previous utterances with different speaker
∑speaker same have vandu
and utteranceearlier an is v v,andu between exists llink
trust(l))*n(l)attenuatio*v)(u,similarity*) gain(v)+((mark(v)=gain(u) personal
∑speakersdifferent have vandu and utteranceearlier an is v
v,andu between exists llink
trust(l))*n(l)attenuatio*v)(u,similarity*) gain(v)+((mark(v)=gain(u) ivecollaborat
9 December 201039
Collaboration (3)
9 December 201040
Services & Widgets� PolyCAFe is an online platform:
� Web services (Java and PHP-based)
� Web widgets using W3C Widgets1.0 standard
� The widgets can be integrated into any web platform that has a W3C widget container platform that has a W3C widget container (e.g. Wookie)
� There are services for maintenance tasks and analysis tasks (process discussion, search, etc.)
� The widgets have been deployed in Elgg (PLE)
9 December 201041
Presentation of Widgets
• Conversation Feedback (content + collaboration)
• Conversation Visualization (collaboration)
• Utterance Feedback (content + collaboration)
• Participant Feedback (content + collaboration)• Participant Feedback (content + collaboration)
• Search Conversation (content)
• Prepare chat/forum assignment
• Add conversations for assignment
9 December 201042
Conversation Feedback
9 December 201043
Conversation Visualization
9 December 201044
9 December 201045
9 December 201046
Search Conversation
9 December 201047
Validation and Verification of
PolyCAFePolyCAFe
9 December 201048
Validation Experiment 1.0
• Validation of PolyCAFe 1.0
– 9x students
– 5x tutors, 1x teacher
• Two chats with 4-5 students in a team
• All the students used PolyCAFe for the first time
• Only two tutors have previously used the system
• Students used the automatic feedback to get
insight about their discussion
• Tutors used PolyCAFe to offer manual feedback
9 December 201049
Validation Results – Tutors
� The tutors validated all the instruments with
suggestions to change part of them
� Quiz with 35 questions – all of them passed with an
average score between 3.50-5.00 / 5.00
� The system is relevant and useful for their � The system is relevant and useful for their
activity
� The time for providing final feedback to the
students is definitively reduced (between 30-
50%)
� The quality of the feedback is improved
9 December 201050
Validation Results – Tutors (2)
Category Average Perc. agree
Pedagogic effectiveness 4.11 83%
Efficiency 5.00 100%
Cognitive load 4.60 100%Cognitive load 4.60 100%
Usability 4.36 93%
Satisfaction 4.57 91%
Total 4.53 93%
1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree
9 December 201051
Validation Results – Students� The students have validated most of the
instruments� Quiz with 32 questions – 5 where not validated; all
others have scores over 3.66/5.00
� In the focus group, they reported that several misleads have been found using this widgetsmisleads have been found using this widgets
� These errors or misleads were reported to be only minor without influencing the overall feedback
� It has been suggested to try and fix them in order to gain the full trust of the users
9 December 201052
Validation Results – Students (2)
� For example, the students did not validate:
� “Overall, I believe that the support for my
learning PolyCAFe (Chat Analysis and
Feedback Service) provides is close enough
to the current support provided by humans. ”to the current support provided by humans. ”
� Average = 3.11
� Perc. agree = 33%
� However, we do not want to provide a
substitute for human evaluation!
9 December 201053
Validation Results – Students (3)
Category Average Perc. agree
Pedagogic effectiveness 3.94 77%
Efficiency 4.22 78%
Cognitive load 3.56 56%Cognitive load 3.56 56%
Usability 4.11 81%
Satisfaction 3.89 72%
Total 3.94 73%
1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree
9 December 201054
Validation Results - Widgets
Validation Statement Learners
Agreement Tutors
Agreement
PolyCAFe feedback is useful 100% 100%
PolyCAFe feedback is relevant 63% 80%
Conversation feedback is usefulConversation feedback is useful 78% 80%
Conversation visualisation isuseful
89% 100%
Utterance feedback is useful 83% 100%
Participant feedback is useful 78% 100%
Search conversation is useful 61% 100%
9 December 201055
Validation Experiment 2.0
� Validation of PolyCAFe 1.5
� 25x students in experimental group
� 10x students in control group
� 6x tutors in experimental group
� 2x 7 chats with 5 students in a team� 2x 7 chats with 5 students in a team
� Most of the students used PolyCAFe for the first time
� All the tutors have previously used the system
� Experiment is still under-way
� Preliminary results are very encouraging: very high correlation between ranking of participants by the system and by the participants themselves
9 December 201056
Verification� Accuracy of system ranking of participants
� Accuracy of grading utterances
� Accuracy of determining implicit links, collaboration areas, discussion threads
� Accuracy for determining speech acts� Accuracy for determining speech acts
� Accuracy for determining Model of Inquiry classes
9 December 201057
Example – Speech ActsSpeech act - label Precision Recall
Continuation 93% 92%
Statement 94% 93%
Greeting 100% 80%
Accept 92% 80%
Partial accept 71% 55%
Agreement 90% 51%
Understanding 96% 58%Understanding 96% 58%
Negative 97% 78%
Reject 73% 82%
Partial reject 35% 27%
Action directive 75% 90%
Info request 100% 71%
Thanks 100% 100%
Maybe 100% 69%
Conventional 66% 50%
Personal opinion 100% 36%
Sorry 66% 75%
9 December 201058
Conclusions and Future Work(PolyCAFe)
� The system is working well
� The system passed the validation with students and tutors
� The visualization widget proved to be the most usefuluseful
� The interface might be improved
9 December 201059
Transferability Issues• Domain
– The topic of the conversation should be easily
solved using discussions, no graphics or formulas
• Language
– Need for the components of the NLP pipe– Need for the components of the NLP pipe
– Corpus for training the LSA
– Maybe, a domain ontology
• Activity
– Collaborative activity
– Teams of 4-15 students (in the current design)
9 December 201060
Learning Support Based on Social
Networks Search and RecommendationNetworks Search and Recommendation
9 December 201061
Learning Scenario� Learner just uses social networking web sites
� He connects to friends and possible tutors
� Our application indexes the user’s friends and resources and their peers friends and resources
� Based on the data acquired we offer search and � Based on the data acquired we offer search and recommendation services
9 December 201062
Tag and Social Network-based Search
Community
=
Social networking application
Knowledge
=
Relevant content
Learner
=
Web 2.0 user
9 December 201063
Find Relevant Resources
Understand the learner’s request and lead him to a
resource that is both relevant and trusted as it is
recommended by a member of the community
9 December 201064
Find Relevant Peers
We search for the right person to offer feedback
and guidance
9 December 201065
Semantic Indexing of Learning Objects
• Crawled data
converted in semantic
formats •Vocabularies used: •Vocabularies used:
SIOC, Neuman’s tagging ontology,
SCOT, MOAT,
SKOS
•*using Amazon EC2
for crawling9 December 201066
Search and Recommendations
� Search
• Search algorithm based on PageRank
• Folkrank - a user is important if he annotates important
resources with relevant (well ranked) tags.
• Fokrank works on a folksonomy - hypergraph - F=(V,E)
∑∈
=)( )(
)()(
uBv vN
vRcuR
• Fokrank works on a folksonomy - hypergraph - F=(V,E)
where V=U ∪R ∪T and E⊂UxRxT
� Recommendation
• Uses clustering of resources, users and tags to
suggest relevant resources
9 December 201067
Architecture of the Application
9 December 201068
Using the Learning Services in a PLE
Search for
resourcesSearch for
users
username
Check the
learner’s
profile
Recommended
readings
Social networking platform
Relevant tags
and their
importance 9 December 201069
Conclusions and Future Work (Search and recommendations service)
� We use social data to provide feedback for learners
� We offer relevant resources to the user from his social network
� We help learners to find peers or tutors� We help learners to find peers or tutors
� The work was embedded in Moodle/WebCT and validated in Bucharest and Utrecht
� We’re centralizing the results
9 December 201070