Evaluation socio-technical environments integrated frameworks

MCSquared FP7 Project 610467, 1st Review Meeting, 06 November, 2014, Utrecht, Netherlands 1

Evaluation

socio-technical environments

integrated frameworks

Chronis Kynigos, CTI


•Social creativity in the

design of educational

resources for CMT

•the two ‘creativities’

•social creativity in design

•Creative mathematical

thinking

‘Evaluation’ in mc2 project

Focus Approach

•Design-based research paradigm

•Creativity as

•process

•product

•context

•Bottom-up / theory-driven

•Formative/ summative

•Integrative

•3 cycles of work

•4 (x3) empirical studies

•meta-analysis & synthesis


Socio-technical environments

consist of a community of individuals engaged

in a joint enterprise, whose creative

performance is mediated through

interactions with a particular ‘technical’

environment designed to amplify the

outcome of their collaborative efforts


Integrated Frameworks

• The medium integrates

• narrative/reading with

• expressivity/constructionism

• The evaluation integrates

• DG and BC to understand SC and

CMT affordances

• Diverse operationalizations of

DG/BC integrations


Communities of Interest

• heterogeneity, diverse expertise in

their members

• Bringing representations from

diverse CoP

• fueled by a culture of participation

• exemplified in social processes,

practices and mediations

• leading to novel/new, useful and

desirable outputs


CoI as SC enhancers• Heterogenous (members are representatives of CoPs)

• to solve a design problem of common concern (joint “interest”) that requires the combination of multiple knowledge systems

• a rich environment for interconnectedness, different perspective-taking, knowledge exchange and integration between diverse domains and more opportunities for creative thinking and learning

• CoI members may be participants in more than one CoPs

• One community can exhibit characteristics of both a CoI and a CoP

• can integrate aspects of both a CoI and a CoP its type may shift over time due to external factors


Collaborative design:

DG and BO

Documentational genesis

• Living documents

• Instrumentation andinstrumentalization

Digital artifacts as boundary

objects

• Boundary crossings

• Improvable objects


Integrations

CoI Diversity in operationalization of the integration between DG

and BC


• Narrative activity model (Vygotsky)

Exploration, inspiration, production, sharing

• Creative thinking spiral (Resnick)

Imagination Creation, Play, Sharing, Reflection, imagination

• Lateral, out of the box, playfulness, humor, imagination (Ackerman, DeCortis et al)

• Model of Collaborative Creativity (Aragon, Williams)

Focus, Frame, Create, Complete

• Product-process, general-specific, solving-posing problems interplay, ill structured open problems (Shriki, Jamir-Leikin)

• Creativity in techno-mathematical literacies and the importance of modeling and boundary crossings and artifacts as boundary objects (Hoyles-Noss)

Starting points for S-C and CMT


‘Evaluation’ in MC2 project: progress in Year 1

Theory grounding and integration

Research design and analysis

‘Integration’ tools (templates)

• lit review on s-c/ CMT

• ‘landscape’ of related concepts and approaches

• draft versions of D2.2 / D2.3

• operational definition

• research questions

• units of analysis

• levels of analysis

• methodological tools

• for sharing, reporting, reflecting, meta-analysing

• for contextualising evaluation

• for theory integration


Evaluating social creativity in Cycle 1:

theory grounding and integration

Social creativity in design

(Fischer, 1997, 2007)

Boundary crossing

approach

(Akkerman & Bakker, 2011)

Documentationalapproach

(Gueudet & Trouche, 2009)

The 3 theory strands

A landscape of related concepts


Operational definition of ‘social creativity’

a design activity: • structured, intentional (create edu resources for CMT)

• collective, participative

embedded and nurtured within a socio-technical environment• CoI (collective of professionals involved in the design of math edu

resources)

• C-book technology (authoring tool, widget factories, data-analytics engine, CoICode)

involving a set of processes and interactions: • socio-cognitive / group processes (within the CoI)

• and the CoI with the C-book technology

aiming at/ resulting in the production of ‘creative’ outputs:

• ideas born/ shaped in the interactions, and

• c-book units, which are:

novel/new

useful

elaborate

appropriate, w/r to the joint goal, and for the end-users


Evaluation studies of social creativity in

Cycle 1: research design and analysis

Common research

questions

• CRQ.1: How is s-c

stimulated/ enabled in

mc2 S-T env?

• CRQ.1: What kinds of s-c

emerge from the

interactions between CoI

and C-book tech?

Categories of data

collected

• Asynchronous discussion

transcripts from CoiCode

• Recorded conversations

from f-2-f / skype meetings

• CoiCode graphs, system

logs, user entries

• The c-book units


Evaluation studies of social creativity in

Cycle 1: focus of analysis

Focus of analysis

‘critical episodes’ of social creativity

Selected segments of an

activity with a single theme

as a focus, highlighting an

aspect of social creativity

a first lens to filter collected data

a mechanism to bring to the fore the

different approaches by the research teams

a contribution to our understanding of s-c


Evaluating social creativity in Cycle 1: analysis

of critical episodes

• the ‘idea’ concept

• stages in the creative

process

• interactions within S-T

environments

• the c-book unit’s

function

• prevalence of an idea if linked to a widget instance

• enablers of s-c:

• CoI synthesis (knowledge and

expertise complementarity)

• role of ‘boundary brokers’

• the c-book unit’s unified structure

• the c-book unit serving as a

‘boundary object’, encompassing

various considerations (math, ped,

tech, context)

main and recurrent themes identified

foci of critical episodes


Refining evaluation design for Cycles 2, 3:

units – levels – tools of analysis

•CMT

affordances

of the c-book

unit

•Modes of

interactions

with the C-

book tech

•The creative path of an idea

•The identityof a creative idea

the idea the idea

the c-book

the c-book


“The c-idea ID card”

The ID card of an idea

Idea “theme” What the idea refers to: e.g. a mathematical

idea, negotiation of the age of the students,

the nature of the narrative of the c-book unit, e.t.c.

Idea initiator

Number of idea commentators

Duration of discussion


“The creative idea indicators checklist”

Creativeideaindicators YES NO CommentsTheideaissupportedbyawidget. Ifyes,when(atwhatpointinthediscussion,forexample:thirdpostoftheinitiator)doesthewidgetappear?

Theideais“native”totheCoI(anemergingproductofCoIdiscussion,notpreviouslyusedinanotherresourcesystem).

Theideais“immigrant”totheCoI(hasbeenpreviouslyusedbytheinitiatorinanothercontext/comesfromanalienresourcesystem).

TheideaisintegratedintheCoIresourcesystem,leadingtosomereorganisation. (Ifno),whenistheideaabandoned? (Ifyes),numberofconsequentevolvingversionsofthesameidea. (Ifyes),howmanyCoImemberswereinvolvedintheelaborationoftheidea? (Ifyes),timeelapsedfromtheintroductionoftheinitialideatillthecommonapprovalofitslastversion Socio-cognitiveprocessesprocessesemployedbytheCoImembersinthedeploymentoftheideafrom

itsfirstwordingtoitsfinaldevelopment

OneormoreCoImembershaveexplicictlycharacterisedtheideaasnovel,unusualornew,atsomepointinthediscussion.

OneormoreCoImembershaveexplicitlycharacterisedtheideaasappropriate(conformingtotheusers’requirements),atsomepointinthediscussion.

Thediscussionoftheideaiscontinuous(discussionflowsinnon-interruptedthreads). Thediscussionoftheideaincludestheelementofquestioning/challenging/disagreement. Thediscussionoftheideaindicateselementsofoneormoreprocessesofboundarycrossing.


“The CMT affordances evaluation grid”The CMT affordances evaluation grid

Does the c-book unit engage the ‘reader’ in enough learning situations

involving…

YES NO JUSTIFY (when

necessary)• open problems/open questions ?

• problem-posing ?

• simulation ?

• the application of other (non-mathematical) subject domains ?

• the combination of more than one mathematical areas(or concepts) ?

• alternative uses of familiar digital tools ?

• real-life mathematics ?

• the combination of more than one available widgets ?

• multiple representations ?

• constructionist activities ?

Does the c-book unit trigger in the ‘reader’ a sense of …• pleasure/ amusement/ enthusiasm ?

• being motivated / challenged ?

• gamification/ playfulness ?

• beauty in mathematics ?

• personal engagement ?

Does the c-book unit give the ‘reader’ enough opportunities for…• collaboration/ communication/ Interaction with others ?

• discussion ?

• free-choice learning ?

• competition ?

• exploration ?

• creating a “story”?

• experimentation ?

Does the c-book unit allow / engage/ faciltate the ‘reader’ to…• formulate her own mathematical assumptions ?

• take initiatives ?

• to find multiple solutions to the same problem ?

• to use multiple paths/ strategies to get to the solution ?

• to apply various forms of reasoning ?

• to get to unexpected solutions ?

• to create new products/ original ideas ?

• to apply various math techniques ?

• to reflect on the solution ?


Evaluating CMT in the 3 cycles of CoI work

Cycle 1

•Identifying the CMT initial representations and criteria of the CoImembers/ designers

Cycle 2

•CoI members self-evaluate and cross-evaluate the CMT affordances of their c-book units

Cycle 3

•CMT affordances of c-book units are evaluated in the small case-studies with students

The CMT

affordances

evaluation grid

The revised CMT

affordances

evaluation grid


The ‘templates’: tools for supporting evaluation,

for meta-evaluation and theory integration

Purpose and intended use

• Description tools

• Support partners to

• design and report work

• externalise, exchange, and reflect on their approaches

• ‘Living documents’

• ‘Boundary objects’

• Analysis and integration tools

• Completed templates are used as raw material

• for meta-analysing and contextualising the evaluation studies (across the 4 CoIs/ research teams)

• for explicitation of research perspectives and synthesisof evaluation findings

The 7 ‘templates’: 5 (wp6), 1 (wp7) and 1 (wp5)


The ‘templates’: tools for supporting evaluation,

meta-evaluation and theory integration

CoIconstitution/

function

The CoI profile template

The CoImoderation

template

CoI activity report

The c-book unit description

template

The cross-case analyses template

Evaluation of the two

‘creativities’

The S-C evaluation

template

The CMT evaluation

template

Evaluation of the C-book affordances

The c-book analytics

requirements template


The “CoI profile” template:

• The CoI identity

• membership, diversity and

size of the CoI

• configuration of the CoI

• CoI 'interest' and

sustainability

• The CoI role and workplan

in Cycle 1 (or 2, 3 )

• intended role of the CoI

• aspired work plan of the

CoI

Structure and rationale:


Excerpts: membership & configuration of the CoIs

Communities of Practice Number of

representatives

1. Researchers in Education & Mathematics Education 5

2. School teachers & Teacher trainers 4

3. Technology developers and users in Education 3

4. ‘Experts’ on the dissemination of knowledge 2

5. Creative Industry 2

6. ‘Experts’ outside Maths Education 2

Communities of Practice Number of

representatives

1. Researchers in Education & Mathematics Education 6

2. Teacher educators 6

3. Game developers 1

4. Teachers 6

5. Mathematicians 1

6. Computer Scientists 2

The Community of Interest (CoI) is

composed, on the one hand, of several

communities of practice (CoPs) involved in

the teaching and learning of Mathematics

and, on the other hand, of external groups

outside Maths Education. This last CoP

gives rise to different contexts link to other

fields of knowledge, promoting an

interdisciplinary and creative interaction.

The final CoI represents various groups

involved and interested in themes involving

maths education, teaching, research,

design and creative industries. There is

some overlap in the various roles. Each

CoP will eventually be represented by at

least two, CoP active, unique people.

Spanish CoI

UK CoI


The “Social creativity evaluation” template

1. Operational definition for

Social Creativity

2. Common Research Questions

3. Specific Research Questions

4. Theoretical grounding of the

study

5. Research approach towards

Social Creativity

6. Context of each CoI-specific

study

7. Categories of data and data-

collection techniques

8. Data analysis techniques

Structure and rationale:


Excerpts from the completed templates - 1

Research Approach to Social Creativity

The UK CoI will use c-books as

‘improvable boundary objects’.

Therefore, one could first typify it as a

product approach. However, because

every actor involved in the evolution of

every c-book, from embryonic stage to

first version to later versions, records

his/her activities every ‘product’ is

accompanied by a ‘process’

describing exactly how the c-book

evolved. This product/ process

combination is documented by a rich

multimodal description.

To analyze social creativity we propose to

use the “componential theory of

creativity” from T.M. Amabile, 2012: “The

componential theory of creativity is a

comprehensive model of the social and

psychological components necessary for

an individual to produce creative work. […].

In this theory, four components are

necessary for any creative response: three

components within the individual -

domain-relevant skills, creativity-relevant

processes, and intrinsic task motivation -

and one component outside the individual -

the social environment in which the

individual is working.”UK research team

French research team


Excerpts from the completed templates - 2

S.R.Q1: How do (boundary crossing)

interactions between CoI members

with a different knowledge domain

(i.e. mathematics and

environmental education or

engineering education), affect the

CoI's performance in terms of social

creativity?

S.R.Q.1: What elements/features of

the c-book infrastructure are

critical to foster social creativity

among the CoI members?

Common Research Question 1: How is social creativity stimulated and/or

enabled within the mc2 socio-technical environment, as this is constituted by

each of the four CoIs interacting with the c-book technology with the aim to

design a series of c-book units?

Specific Research Questions

French

Research team

Greek

Research team