Carmel McNaught

R h i hi h d ti hReshaping higher education research

1

Carmel McNaught

Reshaping higher education research:p g gA few thoughts …

2

3

Role of a keynote speech

to stir things up … which

i ht tmight overturn ideas & beliefs

t l to explore some underlying assumptions &assumptions & concepts

http://londoncoder.files.wordpress.com/2007/12/iceberg.jpg 4

OutlineOutlineContext of my experience … and yoursHorizon report – key challenges Excellence Research Australia Polarity theory and inclusive models of research ‘Good’ & ‘bad’ research and reasons for ‘failure’ Ideas of learning design and T&L modelsMeta-analysis Evolving technologies New opportunities and summary of themespp y

5

Let’s begin with a little ‘social media’Let s begin with a little social media

“Conversation” is the “essential metaphor” foressential metaphor for university life. (Bennett, 2003)( , )

6

Essence and detail … You will see & hear different things in this talk depending

on your beliefs about scholarship & research, and your y p yexperience.

Core principles are transferrable.D t il ? M b b t Details? Maybe or maybe not …

Caveat: Examples will be in T&L design, rather than, for example, policy institutional changepolicy, institutional change, transnational matters, etc. … Perhaps during conversation time we can broaden the scope

http://tinyurl.com/2flmbwh 7

we can broaden the scope.

Shifts in geographical cultures

8Shifts in disciplinary cultures

2010 Horizon Report:C iti l h ll

The role of the academy – and the way we prepare students for their future lives is

Critical challenges

prepare students for their future lives – is changing.

New scholarly forms of authoring, publishing, and researching continue to emerge but appropriate metrics for evaluating them increasingly and far too often lag behind.

Digital media literacy continues its rise in importance as a key skill in every discipline and profession. http://wp.nmc.org/horizon2010/and profession.

Institutions increasingly focus more narrowly on key goals, as a result of shrinking budgets in the present economic climatein the present economic climate.

9Johnson, Levine, Smith & Stone (2010)

Excellence Research AustraliaExcellence Research Australia

Accountability is a reasonable objective

http://www.arc.gov.au/era/

Accountability is a reasonable objective … Emphasizing written outputs is good. Articulation

& reflection are assisted enormously by writing& reflection are assisted enormously by writing. For me, writing is an essential component of research.

However …

10http://www.skypoint.com/members/waltzmn/ChineseBook.jpg

Excellence Research AustraliaExcellence Research Australia

However “the devil is in the detail”

http://www.arc.gov.au/era/

However, the devil is in the detail– How exactly were rankings decided?– Is it all a bit old-fashioned? New formsIs it all a bit old fashioned? New forms

of publishing?– What impacts this will have on

scholarship?– How will new academics be inducted into the

academy?

http://tinyurl.com/2aw4fy4

academy? – What research is being established to examine trends

in research outputs?

11

Tensions in the research scenarioTensions in the research scenario The call for research productivity: Does this mean

quantity or quality?quantity or quality? The call for research quality: Does this imply journal

metrics or improving conceptual understanding? The call for research innovation: Does this mean seeking

new funding models or new strategies to explore understanding?understanding?

The call for research ethics: Does this imply focusing on curbing plagiarism or a desire to strengthen human values in research?

For each of these tensions (and others), should we use 'or' or 'and' or 'and/or'?or or and or and/or ?

12

Polarity theoryPolarity theory Polarities are sets of opposites which can’t

function well independently. Because the two sides of a polarity are

i t d d t it i t ibl t hinterdependent, it is not possible to choose one as a solution and neglect the other.

Th i f l it t i t t th The aim of polarity management is to get the best of both opposites while avoiding the limits of eachof each.

The solution resides within the tension between polarities.polarities.

13Johnston (1992)

Polarity theory

Research innovation -

Research quality - improving conceptual

understandingResearch innovation - new strategies to explore

understandingQuality in research productivity

AND rather th OR

Research ethics -strengthen human

Research ethics -curbing plagiarismthan ORvalues in research curbing plagiarism

Quantity in research productivity

Research innovation - seeking new funding

models

14

Research quality - journal metrics

‘Good’ & ‘bad’ researchTale of two theses – by an examiner

From two Australian universities Both in area of student reflection in technology-

enhanced environments Both ‘good’, non-trivial areas for researchOne action research; one a series of case

studies – both largely qualitative (Both ridiculously long – side plea from an

examiner!) (Both candidates native-English speakers)

15

One

One&

One

http://wishusuccess.com/images/bigstockphoto_Key_To_Success_509650%5B1%5D.jpg

16http://www.crestock.com/image/1563832-Back-To-The-Drawing-Board.aspx

What were the key differences?What were the key differences? Specific research questions Use of the literature Clarification of research methodology (action

research hanging around with the students!)

Clear presentation of research data (not just random student quotes)S ibl l i Sensible analysis

Implications of the findings

17

Why can this type of failure happen?Why can this type of failure happen? Four suggestions1. “Insistent individualism” (Bennett, 2003). My

career is what counts …2 L it t t h l hi (ALL f th2. Low commitment to scholarship (ALL of the

four Boyer (1990) scholarships)3 L k f i ht f h t d t t3. Lack of oversight of research at department,

faculty, university levelsANDAND4. Educational research is complex. What is good

is good research in a complex domain?is good research in a complex domain?18

Boyer scholarshipsBoyer scholarships

The scholarship of discovery – traditional p ydisciplined-based research

The scholarship of integration – connections across disciplines and contexts

The scholarship of application – professional and community-oriented research

The scholarship of teaching – where the i i l f h l l i i li d tprinciples of scholarly inquiry are applied to

planning and implementing teaching

19

Student learning needs

Aims/Aims/ desired learning

outcomes

Content/L i Feedback

Overall area of

researchContent/

fundamental concepts

Learning activities

Assessment

Feedback for

evaluation

Specific research

Actual learning

outcomes

questions

Disciplinary research

conventions

Research instruments & methods

Research data

Synergy between t hi &

Research analysis

teaching & research

Findings & discussion

20

Conversation in higher educationConversation in higher educationConversation implies active and open

t b t ll b f thengagement between all members of theuniversity – senior management, teachers and studentsstudents.

Rather than regulation, we may need more freedom to revitalize whatmore freedom to revitalize what Bennett described as the “virtue” of “hospitality” – a university community that cares for each other and for the values of that community.

21http://www.rdpl.org/images/penguin_community.jpg

Investigating (e)LearningInvestigating (e)Learning …

“… is interdisciplinary in that it seeks to combine … is interdisciplinary in that it seeks to combine and explore the interconnections between new and different approaches from different fields and specializations; it is multidisciplinary in that it simultaneously tries to respect the multiplicity of diff th t t hdifferences that can separate one research approach from another.” (Friesen, 2009, p. 12)

So, we are not talking about simple stuff!

22

A consideration of the term ‘l i d i ’‘learning designs’

Bringing technology Bringing technology and pedagogy together

Student learning needs

Aims/Aims/ desired learning

outcomes

Content/ fundamental

concepts

Learning activities

Feedback for

evaluation

Assessment

Actual learning

t

23

outcomesLockyer, Bennett, Agostinho & Harper (2009)

Learning designsLearning designs

Web functionality

INT

Learning objects/materials

EN

T Learning activities

TIONN

24

Decision questions

Student learning needs

Aims/ desired learning

outcomesDecision questions

Who are my students?

outcomes

Content/ fundamental

concepts

Learning activities

Feedback for

evaluation

Who are my students?Why am I teaching this … ?Which content ? In what form ? How much ?

Assessment

Actual learning

outcomes

Which content ? In what form ? How much ?Who finds it ?

Relationship between online and F2F ?Relationship between online and F2F ?Getting feedback to Ss on their learning ?Getting feedback to Ts on Ss’ learning ?g g Is this learning going to last ? Can I do this better ?

25

The e+ projectThe e+ project

Originally ‘ePLUS Web’ evaluating theOriginally ePLUS Web – evaluating the Potential for Learning: Use and Structure of the WebWeb

20 active & successful eTeachers– Courses: language learning, education, science, g g g, , ,

business studies, engineering and arts Examination of websites (range of LMSs) &

detailed interviews with teachers; survey of students. Mixed-methods research

26McNaught, Lam & Cheng (2009)

Using a two-way matrixg y

Management

*Rule-

b d*Incident-

b d*Strategy-

b d*Role-

b d*

Oliver et al. (2002); Agostinho et al. (2002)

* based* based* based* based*

Communication #

Assessment

Resources (simple)

Resources (enriched)

What do Why and how do you do it?

* Can be non-interactive or interactive# 0, 1 or 2 assigned to each cell

27

What do you do?

Why and how do you do it?

Summary of 20 course websitesy20

1716

20

1413

9

15

nsta

nces

45

4 4

9

45

10

umbe

r of i

1 12

0

5

m s s nt o Ts Qs ls nt y s s k ls

Nu

Asyn

. Foru

mQu

izzes

Past

pape

rsAs

signm

ent

Cours

e info

.No

tes/PP

TsFA

QsTo

ols

Exten

ded c

onten

tGl

ossa

ry Ca

ses

Game

s

ition o

f stud

ent w

orkLe

arning

skills

28

Exhib

itio

Web functions

Summary of 20 course websites50

40

50

Course info.

30ance

s

Extended content

Quizzes20

mbe

r of i

nsta

Glossary

Notes/PPTsPast papers

10

Num

Cases

Tools

FAQsAssignmentAsyn. Forum

Learning skillsExhibition of s tudent

work

Gam es

0Com m unication Assessm ent Resources (s im ple) Resources

29

Com m unication Assessm ent Resources (s im ple) Resources(enriched)

Web functions

Summary of 20 course websitesSummary of 20 course websites

50

40

ance

s

20

30

of in

sta

Interactive

Non-

10

20

Num

ber interactive

0Communication Assessment resources

( i l )Resources( i h d)

30

(simple) (enriched)Main functions

Summary of 20 course websitesSummary of 20 course websites50

30

40

nsta

nces

20

30

mbe

r of i

n Interactive

Non-interactive

0

10

Management Rule based Incident Strategy Role based

Num

Management Rule-based Incident-based

Strategy-based

Role-based

Learning focus

31

Student data from 20 cases infrom 20 cases in HK universities

~600 q’aires

Design of web features

32Outcomes

Implications for the meaning of digital literacy

Active eLearning

StudentStudent data from 20 cases20 cases

Kember, McNaught, Chong, Lam & Cheng (in press)

33

Why do this type of research?Why do this type of research? To set up a series

f tiof conversations … To learn what

teachers do andteachers do and why … and

To allow teachers to To allow teachers to see other design optionsp

Precursor to design-based research?

I t t bt t T t t !34

Important subtext: To get tenure!

Design-based research:Essential role of theoryEssential role of theory

P di ti h

Hypotheses based upon observations

and/or existing theories

Experiments designed to test

hypotheses

Theory refinement based on test

results

Application of theory by

practitioners

Predictive research

Specification of new hypotheses

Analysis of practical problems by researchers &

practitioners in

Development of solutions informed by existing design principles &

Iterative cycles of testing & refinement

of solutions in

Reflection to produce 'design principles' &

enhance solution i l t ti

Design-based research

pcollaboration

g p ptechnological innovations practice implementation

Refinement of problems, solutions, methods and design principles

35http://edutechwiki.unige.ch/en/Design-based_research

p g p p

Reeves (2006)

These findings fit with l t di T&L d llong-standing T&L models

E.g. Laurillard’s (1993, 2002) conversational model– between student(s) and teacher(s)– between the students– about the content

focused on concepts/– focused on concepts/ conceptions

– about tasks/ assessments– about artifacts produced– etc.

36http://dumais.us/newtown/blog/wp-content/uploads/2009/04/conversation.jpg

Diagnostic

Media-enriched explanations -

produced by teachers or students

Activities, e.g. discussions, quizzes, games, simulations,

debates, roleplays, etc.

DiscussionTeacher's conceptual knowledge

Student's conceptual knowledge

Diagnostic testing/ info. on learning prefs

A range of communication & collaboration

opportunities with other

Reflection on student

Reflection on

knowledge knowledge

Revision of learning

Revision of personal Peer reviews,

tests exams

opportunities with other students

Student's actions

work interactions

Teacher's

strategiesideastests, exams

Reflective spaces,Interaction in/ perception of

T&L environment conception of

learning design

Reflective spaces, e.g. blogs, ePortfolios

Feedback for evaluation

Assignments, presentations

37After Laurillard (1993, 2002)

Role of meta-analysisRole of meta analysis

Look for trends across isolated studies, each of which are multivariate.

More common in biomedical research

Good example of adapting a research approach

H l th i lHelps us answer the perennial “So, what …?”

http://www.hogrefe.fr/site/file/image/meta%20analysis.jpg

38

Role of meta-analysisRole of meta analysis

Large scale major study Large review teamLarge scale, major study. Large review teamE.g. Bernard et al. (2009) – 74 separate DE (no IT) vs DE (with IT) studies p ( ) ( )

“A strong association is found between strength and achievement for asynchronous DE courses compared to courses containing mediated synchronous or face tocourses containing mediated synchronous or face-to-face interaction. The results are interpreted in terms of increased cognitive engagement that is presumed to be promoted by strengthening ITs in DE courses.”

IT=Interaction Treatment

39

L ki b f ll l li k d Looking across a number of small-scale, linked studiesE g Lam McNaught & Cheng (2008)E.g. Lam, McNaught & Cheng (2008)– 70 small development & evaluation studies in HK

carried out by the same team. A ‘pragmatic’ approachcarried out by the same team. A pragmatic approach “… in Hong Kong the basic and non-interactive

eLearning strategies are often valued by students, while th i ti f i t ti t t i th ttheir perceptions of interactive strategies that are potentially more beneficial fluctuate. One possible explanation relates to the level of risk that teachers and pstudents are willing to take in venturing into more innovative teaching and learning strategies.”

40

Trends over time

Which ‘arm’ are you in?

http://farm4.static.flickr.com/3201/3122529077_805be34605.jpg

How do you relate to the whole? How do you engage with the whole?

41

Times are changing …

Web 3.0 ?“the smart read-write Mobile Web”

interconnected

user

published content

user generated

content

http://web2.socialcomputingmagazine.com/

2010 …???

Media since ~1990Semantic

Web 1.0 Web 2.0Web (3.0?)

A range of very different technologies:

Multimedia

g y gLMSs, mobile, virtual worlds, etc.

Social mediaEmail

431990 1995 2000 20102005

Technologies to watchHorizon report

One year or less: Mobile computingOne year or less: Mobile computing One year or less: Open content Two to three years: Electronic books (iPad!) Two to three years: Electronic books (iPad!) Two to three years: Simple augmented reality F t fi G t b d ti Four to five years: Gesture-based computing Four to five years: Visual data analysis

methodologymethodology

So lots more new tools & strategies coming44

So, lots more new tools & strategies coming …

Lots happening …C h ?Convergence or chaos?

New contexts New

opportunitiesOngoing

principles to guide us on this journeythis journey …

45http://www.parkenet.org/jp/challenges/convergence.jpg

Core principleshttp://tinyurl.com/2fk5xj7

Core principles

Conversation, rigour & community go togetherMixed-method research suits T&L researchMeta-analysis at key points is very usefulWriting is essential for clarification of ideasg ‘Strategic’ publishing for academic & personal

growth Appropriate networks really assist understanding

… and are fun Enjoy the journey!

46

Thank YouThank YouThank YouThank You

47

References Bennett, J. B. (2003). Academic life. Hospitality, ethics, and spirituality.

Bolton: Anker.

References

Bernard, R. M., Abrami, P. C., Borokhovski, E., Anne Wade, C., Tamim, R. M., Surkes, M. A., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research 79 1243 1289Educational Research, 79, 1243–1289.

Boyer, E. L. (1990). Scholarship reconsidered: Priorities of the professoriate. San Francisco: Jossey Bass.

Friesen N (2009) Re-thinking e-learning research: Foundations Friesen, N. (2009). Re thinking e learning research: Foundations, methods and practices. New York: Peter Lang.

Johnson, B. (1992). Polarity management: Identifying and managing unsolvable problems. Amherst Mass.: HRD Press.

Johnson, L., Levine, A., Smith, R., & Stone, S. (2010). The 2010 Horizon Report. Austin, Texas: The New Media Consortium.

Kember, D., McNaught, C., Chong, F. C. Y., Paul Lam, P. and Cheng, K. F. (in press). Understanding the ways in which design features of educational websites impact upon student learning outcomes in blended learning environments Computers & Educationlearning environments. Computers & Education.

Lam, P., McNaught, C., & Cheng, K-F. (2008). Pragmatic meta-analytic studies: Learning the lessons from naturalistic evaluations of multiple cases. Association of Learning Technologies Journal ALT-J, 16(2), 61–79.

Laurillard, D. (2002). Rethinking university teaching: a conversational framework for the effective use of learning technologies. (2nd ed.). London: RoutledgeFalmer.M N ht C L P & Ch K F (2009) U i t i t McNaught, C., Lam, P., & Cheng, K. F. (2009). Using expert reviews to enhance learning designs. In L. Lockyer, S. Bennett, S. Agostinho & B. Harper (Eds.). Handbook of research on learning design and learning objects: Issues, applications and technologies (pp. 243–262). Hersey, PA: objects ssues, app cat o s a d tec o og es (pp 3 6 ) e sey,Information Science Publishing.

Reeves, T. C. (2006). Design research from the technology perspective. In J. V. Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational design research (pp. 86–109). London: Routledge.

49