ONR/NETC Planning Meeting 18 July, 2003 UCLA/CRESST, Los Angeles, CA

Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

ONR/NETC Planning Meeting18 July, 2003

UCLA/CRESST, Los Angeles, CA

ONR Advanced Distributed Learning

Eva L. Baker

Intellectual and Social

Capital: Opportunities

2003 Regents of the University of California

UCLA CRESST

National Center for Research on Evaluation, Standards, and

Student Testing

2Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Intellectual Capital

• Fund of knowledge, skills, and cognitive propensities—analogous to ROI

• CRESST focus for ONR– Content understanding– Problem solving– Teamwork– Communication– Metacognition

3Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Intellectual Capital

ContentUnderstanding

ProblemSolving

Teamwork andCollaboration

MetacognitionCommunication

Learning

4Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Social Capital

• Team performance and skills• Networking and information

sharing• Trust• Efficacy (can do)• Adaptive communication• Environmental stress

management• Leadership roles• Effort

5Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Organizational Development and the Creation of Social

Capital

• Both intellectual and social capital needed

• Social capital is the sizzle• Social capital interacts with

individual differences• Individual differences in the past

has been conceived as general traits or traits that come into play on certain occasions

6Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Trust

EfficacyNetworks

EffortAdaptive

Communication

LearningOrganization

Leadership

Social Capital

Teamwork

7Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Measurement of Social Capital

• In institutions (e.g., schools), as a component of effectiveness

• In programs, as the context and outcome of quality instruction

• In organizations, to improve effectiveness

8Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

What’s New?

• Not trait-, state-, personality-focused• Componential and therefore

instruction-sensitive• Interactive with intellectual capital

formation• Measurement should be iterative and

focus on bottom-up cycles of reporting

• Organizations put self forward like awardees

9Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Program

• Design analytic framework for evaluating social capital, development, duration, predictive elements

• Contrast high-functioning with low-functioning systems to discriminate key predictors

• Create measures for self-assessment and improvement of individuals, teams, and organizations

10Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Components of Social Capital

• Individual behaviors (motivated, persistent, resilient, altruistic)

• Individual meta processes (intellectual and emotional awareness and control)

• Knowledge of how to make social systems work

11Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

• Measurement of Team Performance

– Knowledge maps for team cognition•Content knowledge

– Team process measured by pre-specified messages or online text analysis, e.g., latent semantic analysis E-Rater (ETS)

– Automated generated and scored After-Action Reviews

0

50

100

1st

Qtr

4th

Qtr

East

West

North

Opportunities

12Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Opportunities (cont.)

• Dynamic Testing of Team Performance– Vygotsky (ZPD), feedback (O’Neil),

help seeking (Webb)

• Cognitive Demands of Psychomotor Tasks (rifle marksmanship and SEALS)– Brief SEALS August 14, 2003, for

possible joint program

13Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Opportunities (cont.)

• Comparative Testing of Authoring Systems– Top Down

• Ontology, task analysis, experts

– Bottom Up• “Repurposing” simulations

– Dependent measures• Outcome of instruction is learning

• Outcome of assessment is reliable and valid decisions

• Online Analysis and Reporting

14Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Psychometrics of Simulation

• Psychometric Issues– Validity and accuracy– Design economies– Technical comparability

and equating– Estimation of difficulty– Generalization and

characteristics across topics, contexts, prior knowledge

– Sampling– Scaling – New functional approaches

to norming

• Online administration, diagnostic/ diagnostic/

scoring, reporting prescriptive prescriptive (AAR)

Individual Training Collective Training

15Baker ONR/NETC July 03 v.4 2003 Regents of the University of California

Opportunities (cont.)

Standardizing Quality

• Alignment (R&D)• Linguistics• Search capacity• Evaluation (program

assessment)• 5 Vector mixed models