Epilogue. Integration of Leadership, Education, Training, And Self-Development

8/12/2019 Epilogue. Integration of Leadership, Education, Training, And Self-Development

1/25


2/25

Epilogue 254

Asymmetric Warfare Group

Epilogue. Integration of Leadership, Education, Training, and Self-Development

Gary Riccio

The Wexford Group International

The purpose of this final chapter is to summarize the topics of discussion that, by the end of theAWGs scientific engagement in OBTE, were most urgent and vibrant with respect to

implications for social and programmatic decision-making about Army training, education, and

leader development.

Toward Values-Based Standards for Army Doctrinal Requirements

To date, the proponents and participants in OBTE have stressed that we should consider all the

outcomes of specific learning events that are relevant to preparation of Soldiers for Full Spectrum

Operations (FSO), not just the outcomes that are intended or explicit in a Program of Instruction

or Training Support Package. This has led us to the conclusion that OBTE provides anoverarching vision for training, education, and leader development that makes all these endeavors

relevant to FSO and to operational commanders in theater (see Chapters 3, 4, and 5; see Figure 1).The principles and practices of OBTE follow, arguably in a deductive fashion, from this vision of

relevance. While not especially surprising theoretically, at least in hindsight, this conclusion has

important practical implications. Most importantly, it suggests that OBTE is both a manifestation

and response to existing Army doctrine. The cascading implications are outlined below.

Figure 1. Triadic balance and equilibrium among intangible personal attributes

and the associated scaffolding for capabilities of the human dimension in Full

Spectrum Operations and competence in trans-extremis environments (e.g.,

transitions between strategically significant combat and noncombat operations).

See Chapter 3, section 3.2.3.


3/25

255 Riccio


The current notion of OBTE as vision for learning and development suggests that we can

approach the relationship between immediate learning objectives and long-term developmental

outcomes from the opposite direction, in a sort of reverse planning from what the Army expects

of its Soldiers coming out of programs of training and education. The intangibles such as

confidence, initiative, and accountability help bridge the gap between behavior that is

immediately observable and behavior that is meaningful and efficacious in FSO. But what are the

ultimate outcomes? Our current belief is that OBTE does not suggest a fundamentally new set ofconcepts to be added to doctrine. Given the vision and the purpose of preparing for FSO,

outcomes should be derived from doctrinally accepted concepts such as Army Values and

Warrior Ethos or at least they should be traced to them.

One of the strengths of OBTE is that its implementation can be verified. OBTE thus can help

translate existing doctrinal requirements such as Warrior Ethos and Army Values into practices

that actually are observable and around which programs of instruction can be designed and

developed. By showing that intangibles can be observed through their behavioral manifestations

(a common proposition in the physical, biological, and behavioral sciences) and that measures

can be developed for observable behavior, we also can claim that standards can be identified forsuch concepts. The logical conclusion of this line of argument is that a rigorous scientific

approach to development, verification, and validation of OBTE provides a way to identifystandards for value-based requirements in existing doctrine for which there currently are no

measures or standards.

A more axiomatic formulation of this line of argument is:

(1) The most fundamental aspects of doctrine are the requirements that are explicit or implicit inArmy Manuals, Regulations and associated documents. Concepts like Army Values and

Warrior Ethos are among the most common requirementsstated explicitly in Army doctrine.

(2) The Army is a standards-based organization but there currently are no doctrinally codifiedstandardsfor some of the most important requirements in Army doctrine (e.g., as Army

Values, Warrior Ethos).

(3) Verifiability and validityare two universally accepted criteria in capabilities development andquality assurance that all standards should address directly. Verifiability means that one can

observe something tangible that relates directly to the standard (e.g., one can verify that one

is doing what one says one is doing or should be doing). Validity refers to the relevance of

the standard to the requirement (e.g., does training to the standard help one meet the

requirement).

(4) The validity of all standards (e.g., current marksmanship qualification standards) is intangiblewith respect to some of the most important requirements in the Army (i.e., Army Values,

Warrior Ethos). The relationship is inductive not deductive. Nevertheless, not all standards

and inductive relationships are created equal. One should develop, refine, and utilizestandards for which confidence in validity is greatest (i.e., for which evidence about validity

has the most depth and breadth).

(5) To date, OBTE is the only approach to training, education, and leader development that hasgenerated measurespotential standardsfor some of the most important requirements in

the Army (e.g., Army Values, Warrior Ethos). While there is evidence for the validity of

these potential standards, the more important point is that OBTE has shown that such

standards can be developed in principle.


4/25

Epilogue 256


(6) Tasks, conditions, and standards are particular solutions to particular requirements. The mostdesirable solutions are ones that can evolve though refinement, upgrades, or other adaptations

to a changing context for the associated requirements over the life cycle of the solution.

Verification and validation fosters continued viability and relevanceof a solution by

providing feedback about what to sustain and what to improve.

(7) OBTE arguably is the only approach to training, education, and leader development that hasgenerated a methodology for integratedverification and validation.

The application of OBTE to doctrine embodies a broader conceptualization of standards, one that

connotes a quality of something that is attainable, not just standardization and evaluation with

respect to norms. The implications about standards and their employment reveal another way that

OBTE transcends particular instructional methods. Irrespective of the methods utilized in training

and education, there is a need for further development of standards for values-based requirements

in current Army doctrine. The validity of such standards cannot depend critically on any

particular methodology even though, in particular situations or under particular conditions, someinstructional methodologies may be more effective than others in achieving values-based

standards.

Nested Standards and Quality Assurance

The elaboration and refinement of values-based standards provides critical guidance to instructorsand their chain of command but it is not sufficient. More guidance is needed with respect to how

such standards can be achieved. Without clarity about the means to the ends, recognition of gaps

or shortfalls in achieving standards will not be actionable; there will not be a way to assure that

achievement of standards will be sustained let alone to identify ways to improve instruction with

respect to the achievement of standards. In this respect, the most important contribution of thescientific engagement in OBTE to date has been the development of formative measures for

instructors (e.g., Chapter 2, Appendix A) and the scientifically based approach to verification and

validation with which the measures are associated. The measures are the only way to verify thatOBTE is being practiced as it has been defined and thus to validate that it can be effective. They

provide a rigorous framework for the development of values-based standards that are achievable.

More generally, the most important realizations about OBTE are that there are requirements in

Army doctrine for which solutions are undefined or unverifiable, that measurable standards can

be developed for them, that values-based standards need not displace existing knowledge-based

or skill-based standards, that the achievability of values-based standards can be verified withformative measures for instructors, and that achievement of values-based standards need not

require additional resources for training and education. The logical implication of these

realizations is that multiple standards of different kinds should be addressed in every learning

event. Note that the implication is not that values-based standards are merely a higher level

quantitatively on measures associated with existing standards. They are standards of afundamentally different kind.

The formative measures developed for OBTE (Appendices A and B) provide a menu from which

instructors, and other stakeholders with an interest in quality assurance, can select a manageable

number of measures for planning, executing, and learning lessons from a particular learning

event. None of these measures precludes assessment with respect to knowledge-based or skill-

based standards, although they certainly can inform priorities and tradeoffs with respect to an

exhaustive list of tasks, conditions and standards that are not practicable (Appendix C). The


5/25

257 Riccio


evidence from this investigation supports these claims, in essence, that multiple measures of

different kind can be used in the same learning event and that values-based measures in particular

can inform decision-making in planning and executing instruction (e.g., Chapters 8, 9, 13). This

realization offers great promise for packaging training and education in more efficient ways (e.g.,

Perry & McEnery, 2009). OBTE, and specifically the associated formative measures, provide

clues about how to link tasks to achieve multiple goals at the same time. Thus, in addition to

showing that values-based standards dont impose an additional resource load, formativemeasures for OBTE may even show a path to achieving more with fewer resources.

It also is clear that there is much to be learned about the efficient and effective use of multiple

measures. Importantly, given the foundation provided by this investigation, increasing clarity on

such critical issues now is largely a matter of paying attention to the most important outcomes of

training and education whether intended or not, focusing collective priorities in a program of

instruction within a command intent that addresses the most important outcomes, and

documenting progress (Chapter 11; Appendix C). It is noteworthy that the value of these actions,

and the clarity they bring, is not limited to programs that adhere to the principles and practices of

OBTE. The principles and practices of OBTE will, however, bring greater value insofar as theywere developed pointedly to address values-based outcomes (Chapters 1-5).

Key constructs in OBTE are intangibles such as confidence, initiative and accountability. They

are assumed to have causal potency with respect to the developmental outcomes that motivated

OBTE. The trustworthiness of this assumption is revealed by the extensive dialectical reflection

and analysis we conducted during measure development, field-based observation, and thedevelopment of grounded theory for the practical wisdom of Soldiers as leaders and instructors

(e.g., Chapters 2-5). The intangibles emphasized by OBTE relate directly to values-based

doctrinal requirements such Army Values and Warrior Ethos (see e.g., Chapter 3; Appendix D).

The identification and promulgation of values-based standards will require additional attention to

outcomes that are influenced by these intangibles (cf., Bandura, 1997; see Figure 1). There is aneed to identify and refine the understanding of such outcomes as effects on students that can

transfer to different situations including performance in Full Spectrum Operations (cf., Leibrecht,

Wampler, & Pleban, 2009) if not to other aspects of a Soldiers life (see Chapter 4, section 4.3.2;Chapter 5, section 5.2).

We recommend that development of values-based standards consider the outcomes defined or

refined locally in collaborative decision-making among the instructor cadre within the local

commanders intent (Chapter 3, section 3.2.7; Chapter 14; Appendix C). Such outcomes and

associated measures currently are in the early stages of decentralized development across various

programs (Haskins, 2009; Perry & McEnery, 2009). Outcomes can be addressed systematicallyacross programs and within a time scale that is short relative to the refinement of doctrine.

Doctrine can assimilate this localized innovation and translate it into guidelines that stimulate and

focus innovation rather than discourage it. Within such guidelines, instructors and their chains of

command are in the best position to identify observable and achievable outcomes with the most

relevance to stakeholders for their programs of instruction. Collaborative identification ofoutcomes is a means for instructors to develop and direct internal sources of motivation for good

instruction. It is a means for the instructional organization to operate coherently with respect to

the commanders intent and to achieve a command climate within which unnecessary unintended

friction is readily identified and eliminated.


6/25

Epilogue 258


Figure 2. Programmatic view of the development and employment of values-based

standards as well as associated instructional principles, practices, and measures (i.e.,OBTE). Solid lines represent direct programmatic influences. Dotted lines connote

informal or indirect programmatic influences. Heavy solid lines connote direct

psychological causes and effects of instruction on students. Compare with Figure 8 in

Chapter 3.

To summarize key points about multiple standards:

Standards for values or ethos can be addressed in the same learning event as existingstandards.

Tasks, conditions, and standards can be designed to allow multiple standards to beaddressed in the same learning event, conditions permitting.

Addressing multiple standards in the same learning event suggests the potential toaccomplish more with the same resources or less.

Standards should reflect outcomes that are developed in decentralized innovation acrossprograms of instruction.


7/25

259 Riccio


Needs and Opportunities for Staff & Faculty Development

A Role for Science and Measurement

Several promising directions for instructor education in OBTE are outlined below. While the

recommendations are based on established practices, they are nuanced in the context of OBTE.

The nuances derive from the intent to be directive about process while not being directive aboutcontent (see e.g., Chapter 3, section 3.2.7; Chapter 14; Appendix C), to foster flexibility and

adaptability in instructional planning and execution, and to strike a balance between the science

of teaching and the art of teaching. The subtlety we seek was captured well by William James in

his talks to teachers over a century ago:

The teachers of this country, one must say, have its future in their hands. The

earnestness which they at present show in striving to enlighten and strengthen

themselves is an index of the nations probabilities of advance in all ideal

directions [p. 3] You make a great, a very great mistake, if you think that

psychology, being the science of the mind's laws, is something from which youcan deduce definite programs and schemes and methods of instruction for

immediate school-room use. Psychology is a science, and teaching is an art; andsciences never generate arts directly out of themselves. An intermediate inventive

mind must make that application, by using its originality [p. 7] A science only

lays down lines within which the rules of the art must fall, laws which the

follower of the art must not transgress; but what particular thing heshall positively do within those lines is left exclusively to his own genius [p. 8]...

many diverse methods of teaching may well agree with psychological laws [p.

9] To know psychology, therefore, is absolutely no guarantee that we shall be

good teachers. To advance that result we must have an additional endowment

altogether, a happy tact and ingenuity to tell us what definite things to say and dowhen that pupil is before us. That ingenuity in meeting and pursuing the pupil,

that tact for the concrete situation, though they are the alpha and omega of the

teachers art, are things to which psychology cannot help us in the least [p. 9]We know in advance, if we are psychologists, that certain methods will be

wrong, so our psychology saves us from mistakes. It makes us, moreover, moreclear as to what we are about. We gain confidence in respect to any method

which we are using as soon as we believe it has theory as well as practice at its

back. [p. 11] (James, 1899/1907).

A vast amount of pedagogically relevant research has been conducted in the century sinceJamess remarks but his advice is still relevant. Our advantage today is that psychology and other

academic disciplines are in a better position to help instructors become more keenly aware of

where certain methods will be wrong, to save us from mistakes, and to give us confidence in

respect to any method which we are using as soon as we believe it has theory as well as practice

at its back. We were mindful of Jamess advice from the beginning of the investigation, and itthoroughly infused our program of research. In general, through our interactions with the

progenitors and stakeholders of OBTE, we diligently sought to identify the tact and ingenuity to

tell us what definite things to say and do when that pupil is before us. Concurrently, we

identified relevant science and scholarship that lays down lines within which the rules of the art

must fall, laws which the follower of the art must not transgress (see e.g., Chapters 2 and 3).

Without something like the formative measures for instructors developed in this investigation, for

example, there is no principled basis for instructor education with respect to values-based


8/25

Epilogue 260


requirements in current Army doctrine. The student measures that were added toward the end of

the scientific engagement (Chapter 12; Appendix B) also can provide formative feedback to

instructors and to students about the efficacy of instructor behavior. They are measures that show

whether the instructor is having some effect on the student and what some of those immediate

effects are. However important, they are merely indicators (i.e., efficient causes) that outcomes

are likely to be influenced by instructors, especially if high scores are obtained on the instructor

measures (see Chapters 8 and 9). These can be dependent variables in a study of OBTE but theyshould not be confused with outcomes. With regard to outcomes emphasized by OBTE, they are

intervening variables. The grounded theory outlined in Chapters 3-5 provides a foundation for

further research into measures that provide a more complete picture of the impact instructors and

their programs of instruction can have on individual learning and development with respect to

values-based outcomes.

Toward Best Practices in Instructor Education

OBTE illuminates both needs and opportunities for development of staff and faculty associated

with Army training, education, and leader development. It is important to note that none of theseactionable implications are uniquely required by OBTE. They apply to any approach to training,

education, and leader development, in principle, and they apply to current practices in these areas.The need for increased resources often is associated with OBTE merely because practitioners

have a keener awareness about the best use of resources. They have greater motivation to achieve

more or at least they have a more coherently directed motivation. The insights of OBTE may

simply stimulate reconsideration about the distribution of resources. It may also reveal that anincrease in resources devoted to training and education, such as instructor education, can have a

disproportionately high increase in return on the investment. Again, this is not to be confused

with a requirement of OBTE for more resources.

Field-based training. The AWGs Combat Applications Training Course (CATC) is an effectiveway to familiarize instructors with OBTE (see Chapter 6). Such opportunities for experiential

learning are an important element of instructor education but CATC is not sufficient to ensure

that instructors will understand the general principles of the new approach. Instructors should befamiliarized with the application of OBTE to different skill sets or knowledge sets (see Chapter

11). We recommend use of formative measures (see e.g., Chapter 2) in such courses to assurequality and for collaborative reflection with participants in the course. The measures would help

student-instructors appreciate the ways in which the general principles and practices of OBTE are

manifested in particular instructional strategies and events. They would help see beyond the

particulars of the learning experience to reveal the deep structure of learning and instruction.

Seminars and workshops.The AWG has provided seminars on a regular basis to stakeholders in

OBTE (AWG, 2008a,b, 2009). These events have been very useful in helping instructors and

their chain of command come to understand the vision, purpose, principles, and intended

outcomes of OBTE even though the events are only several hours in duration. We prefer a

workshop format with two half-day periods on consecutive days given that discussion aboutOBTE generally simulates subsequent reflection with noteworthy personal implications. We

expect that workshops will be more successful when they are coupled with experiential learning

(e.g., CATC). This assumption is consistent with our informal observations during this

investigation.

We have recommended that workshops include multiple sessions in which the topic of discussion

is the experience of instructors (Sidman, Riccio, Semmens, et al., 2009). Toward that end, it

would be valuable to have a corpus of narrative vignettes that provide concrete examples of an


9/25

261 Riccio


instructors thinking and behavior. The vignettes could describe instructional events at a level of

detail that provides enough substance and enough left unstated to stimulate meaningful

collaborative reflection on a time scale commensurate with the time scale of the events described

in the vignettes. The formative measures for OBTE (e.g., Chapter 2) provide an excellent

framework both for generating such a corpus and for discussing the vignettes. In a sense,

instructional vignettes can provide a way for the student-instructor to get inside the head of a

more experienced instructor as conditions are observed, as in-stride decisions are made, and asconsequences unfold. We believe this would help demystify good instruction. It also would have

the added value of showing student-instructors how to use formative measures for quality

assurance and self-development.

Collaborative decision-making. We have stressed the importance of participation of instructional

cadre and their chain of command in making decisions about outcomes and associated measures

of interest in their own programs (Chapter 3; section 3.2.7; Chapters 7 and 14; Appendix C).

These collaborative discussions have the added value of helping individual instructors develop a

broader and deeper understanding of OBTE. It is another way of learning from peers and

contributing to that learning. It provides more direct connections between general theory andimmediate practice.

Multimedia products. An advantage of a field-based course is that a student-instructor can see

examples of good instruction. One of the limitations of this experiential learning is that it can be

difficult to differentiate the essential from the incidental, to appreciate the general principles and

practices in the particulars in which they are manifested. Exposure to a variety of instructors andapplications of OBTE can be a powerful and effective method to help students generalize and to

learn more deeply. Video records would be a simple, low-cost, and highly accessible method for

student-instructors to be exposed and re-exposed, on demand, to a variety of instantiations of

OBTE. Moreover, established methods in multimedia instruction could blend aspects of field-

based training and seminars. We have developed multimedia materials for instructor educationthat utilized formative measures for values-based requirements to guide viewers through video

segments of instructional events (Bruny, Riccio, Sidman, Darowski, & Diedrich, 2006; cf.,

Straus, Shanley, Burns, Waite, & Crowley, 2009).

There is added value to multimedia methods of instructor education that address a recognizedneed. Student-instructors and their leadership sometimes express skepticism that they are capable

of the kind or level of instruction demonstrated by master instructors or by individuals with

extraordinary credentials as Soldiers. While OBTE or any instruction in the Army can benefit

from extraordinary experience and expertise in Soldiering, we believe that OBTE does not require

such levels of mastery. Common instructors have the potential to improve their instruction withrespect to the principles and practices of OBTE (e.g., Chapters 8 and 9). Nevertheless, it would be

valuable for student-instructors to be exposed to instructors like me in situations like mine. This

need not be limited to instructors. Multimedia materials also can highlight the role of an effective

Brigade or Battalion commander, a tipping-point First Sergeant or Company commander, or other

influential leaders.

Distributed collaboration.Collaborative decision-making and peer-to-peer sharing of lessons

learned need not be limited to forums in which all participants are physically co-located (Chapter

11). There will be increasing opportunities for web-based social networking for the foreseeable

future. Such capabilities should be leveraged for formal or ad hoc instructor education. One of the

current needs in this context is finding outside experts who can help relative novices find the right

information in the right amount and in the right amount of time. More generally, there is a need to

help individuals share their respective expertise in the context of common or convergent interests.


10/25

Epilogue 262


Internet portals that facilitate such decentralized ad hoc task organization are a promising

potential solution to this need (Mikroyannidis, 2007; Riccio, Lerario, et al., 2006). Web-based

utilities that enable user generated content and its distribution are transforming use of the web

as well as the perception of what the web affords. This can lower barriers to entry into

communities of practice and thus extending the opportunities to engage in and benefit from

volunteerism (McKee, 2007; Putnam, 2001; Thoits, & Hewitt, 2001).

\Textbooks, pamphlets, and pocket guides.Textbooks are how novice instructors outside the

military teach with little or no background in teaching and even sometimes with little background

in the subject matter. One generally can use a text however one wants but at least it provides a

framework and reference for one's local and momentary initiative. If the textbooks dont help

instructors do this, they are not chosen or used. In any case, the authors get useful feedback about

the value of their contributions. This market-based solution does not necessarily require monetary

gain by the authors and may not even require monetary transactions. Other motives and

transactions can connect volunteers with people who value their special expertise. Recognition,

inclusion, and opportunities for impact can be powerful motivators (Maslow, 1943, 1968).

Content generated by volunteers would require review and perhaps editing by TRADOC but thisdemands fewer resources than generating content. With regard to content for instructor education,

reviewing and editing would be simplified significantly by the formative measures developed inthis investigation (see Chapters 2, 7, 12).

In the context of OBTE, textbooks need not and should not tell instructors what to do. They

should provide guidance about how to become a more effective instructor. We believe that anexcellent source of guidance would be a corpus of narrative vignettes that provide concrete

examples of an instructors thinking and behavior as suggested above in the context of workshops

for instructor education. In any case, formative measures for instructor behavior are a valuable

framework within which to seek, provide, and interpret guidance about instruction. Given that

such guidance is not a script for everything an instructor does in a course, it can be concise.Mnemonics also can be produced on pocket cards for instructors who have been familiarized with

the guidance represented in the cards. Pocket cards reflect the value of guidance that can be

accessed in situ. This is especially important for novices who may not have fully assimilatedlessons learned or retained knowledge of best practices or who may be overloaded during an

instructional event (Freeman, Jason, Aten, et al., 2008).

Portable electronic aids.Elaborate frameworks and context-dependent guidance can be

extremely valuable in situ. Hand-held electronic devices are becoming increasingly common aids

to performance in everyday tasks as well as on the job. As such, they are a natural extension to

pocket cards that are at once ubiquitous and excessive in the Institutional Army and theOperational Army. Electronic aids are interactive, thus, they allow the presentation of information

to be as simple or as complex as needed. Clever design of menus and the underlying structure

also can help a user find information without knowing exactly what they are looking for. More

generally, use of an electronic aid and the information it provides can be context dependent.

Moreover, read-write capabilities enable hand-held electronic devices also to replace the green

book that is a common tool for collecting notes and observations in the field, especially during

instructional events. The capability to record noteworthy events within a framework of formative

measures can facilitate both instruction and learning. It can greatly facilitate collaborative

reflection either in formal After-Action Reviews or in ad hoc groups after a noteworthy event (see

Chapter 3). Such devices also offer the added value of facilitating the capture and sharing of

lessons learned with peers and stakeholders (see e.g., Riccio, dEchert, Lerario, et al., 2006).


11/25

263 Riccio


Critical Considerations for Further Scientific Investigation

The Necessity of Long-Term Studies

In our view, all the debate about OBTE reduces to lack of agreement about the impact of training

and education, in general, irrespective of approach. By way of analogy, any competition that

would pit OBTE against some other approach would be impossible to initiate until all partiesagree on the rules of the game. The various parties in the debate about OBTE are not even close

to being on the same page about what outcomes matter. In fact, the use of the term outcomes

across the Army, and more broadly in education, has become so muddled it may be wise to

consider dropping the term altogether (see Chapter 3). In any case, in the terminology of

empirical science, the most important and difficult questions about OBTE involve the choice of

dependent variables. If we can achieve some agreement on what effects to measure, then we can

tackle the tough decisions about independent variables (i.e., interventions, treatments) or

comparisons (e.g., demographics and characteristics of samples, groups, situations).

OBTE emphasizes long-term developmental outcomes consistent with values-based requirementsin existing Army doctrine. Scientific inquiry can play an important role in identifying behavioral

manifestations (i.e., outcomes) of the values developed in Soldiers as well as antecedents thatreflect progress in the development of such values. Short-term investigations such the ones

reported in this manuscript are valuable but they are not sufficient for overarching programmatic

decisions about OBTE. The value of the empirical work summarized herein is in revealing

proximate effects of any approach to training and education and, more importantly, in verifyingthe implementation of any approach (Chapter 1). It cannot, by definition, reveal long-term effects

such as inculcation of values described in Army doctrine. Long-term studies are warranted if not

required if science is to help the Army meet its existing requirements. Moreover, theory

development is necessary to explore connections between short-term and long-term effects (see

e.g., Chapters 3-5). Without theory, it is difficult to take any empirical investigation seriously,and it certainly is impossible to draw any actionable conclusions based on the evidence. The

broader and deeper the theory, the more confidence one can be about interpretations of the

evidence, and the more responsible one can be in making recommendations based on theevidence. The seriousness of the tasks currently faced by Soldiers in FSO deserves theory that is

commensurate in depth and breadth.

OBTE claims that, when there is singular focus on a task and prescribed conditions associated

exclusively with a knowledge-based or skill-based standard, there is an unintended reliance on

short-term memorization and conditioning. This may result in a reasonable level of performance

in the learning event but our claim is that it also results in an unintended negative impact on long-term outcomes. In particular, it can be counterproductive with respect to attributes and values

necessary to perform in ambiguous situations or across the variations implicit in Full Spectrum

Operations (FSO).

OBTE claims that, when leadership at all levels seeks to create a command climate that stronglyencourages the development of values and attributes emphasized by OBTE, the instructors will

naturally begin to identify opportunities to achieve the knowledge-based or skill-based standard

established for performance while also developing a Soldier with respect to long-term outcomes.

Further, we argue that this will lead to the discovery of new and better ways to instruct with

respect to short-term objectives and long-term outcomes within current resource constraints; that

is, it will lead to a more efficient use of resources. Such adaptability and ingenuity is exactly what

is needed in FSO.


12/25

Epilogue 264


In essence, OBTE claims that the Army is already conducting outcomes based training and

education, in that there is always long-term outcomes from every event or encounter with a

student. A singular focus on knowledge-based or skill-based standards, however, can lead to

negative outcomes. These are empirical questions, and this is where science can help. These

issues suggest that further research would do well to include social psychological and even

econometric perspectives (see Chapters 4 and 5) more deeply than we were able to address in the

current investigation which initially drew mostly on industrial-organizational psychology and theexperimental psychology associated with human perception and performance, cognition and

learning, and human development (see Chapter 3). Further research should be re-conceptualized

at a higher level in terms of inter-temporal risk-benefit or cost-benefit relationships, for example:

Benefit: What is the impact of any program of training and education on outcomes thatmatter for mission effectiveness in Full Spectrum Operations?

Risk: Does OBTE have a positive or negative effect with respect to near-term learningobjectives in a particular program of instruction?

Risk-Benefit Analysis: Does impact of OBTE on outcomes justify additional risk, if any,to near-term learning objectives?

Cost: Does OBTE require additional costs for materiel or non-materiel resources orredistribution of such resources to maintain overall cost?

Cost-Benefit Analysis: Does impact of OBTE on outcome justify additional cost orredistribution of costs, if any,of its adoption?

Note that risk-benefit and cost-benefit analyses are relevant only if there is a tradeoff, that is, if

outcomes come at a risk relative to near-term learning objectives or at cost in additional resourcesused. With improved instructional design informed by experience with OBTE, we believe that

there will be no tradeoff. This seems to be the case at sites where instructor cadre and their

leadership have collaboratively re-designed their courses around OBTE (Haskins, 2009; Perry &McEnery, 2009; Appendix C).

False Dichotomy of Objective-Subjective

Measures and standards that relate to long-term developmental outcomes will look different from

measures typically used to assess performance with respect to knowledge-based and skill-based

standards. There is some concern about subjectivity in measures associated with values-basedstandards and, in particular, there are some assumptions about unreliability relative to objective

measures associated with more common standards. As a synonym for reliability and validity,

objectivity is a worthy pursuit and one that is convergent with rigorous subjectivity, especially

inter-subjectivity (Bridge, 2000; Dienes & Perner, 1999; Neisser & Fivush, 1994; Reed, 1996;

Searle (1997). The dichotomy is largely an illusion, however, for a variety of reasons.

Consider standards being applied in training today that are assumed to be objective. For any

standard, there are numerous ways to get to that standard that are not specified or precluded in the

associated regulations. Some of the unspecified ways of approaching a standard would be

interpreted as cheating, others are not so clear because of judgment on the spot when conditions

are not as assumed in the standard. In either case, the result depends critically on subjectivity. In

addition to this practical constraint on objectivity, there may be legitimate questions about the

putative validity of the standard with respect to the operational task to which it ostensibly is


13/25

265 Riccio


relevant. Potential violations of the assumption about validity generally are an even more serious

challenge to the common connotations of objectivity. The only way out of this conundrum is to

trust instructors and evaluators in using judgment (subjectivity), to establish a command climate

that motivates and fosters trustworthiness (shared values), to provide instructors and evaluators

with instruments and guidelines that are trustworthy (rigorous) without precluding judgment, and

to engage continually in meaningful collaborative reflection (inter-subjective validity).

When focused on nave objectivity (assessments that could be done by a novice), measures of

performance can pervert the purpose of training and education. Without the currency and

relevance (that novices are incapable of considering) mere verifiability and repeatability are

insufficient to assure development of the Soldier. Novice assessments tend to focus on specific

aspects of the nominal training and education process rather than on the various outcomes,

especially intangible outcomes, that the process influences. Novices can be expected to depend

upon these factors and to operate so as to maximize the arbitrarily observable aspects of their

instruction and minimize the role of their own expertise as a Soldier and instructor. On the other

hand, if instructors focus on intangible outcomes, their experience as a Soldier and instructor is a

critical foundation for reliable assessment of good training. Leaders are familiar with assessingintangible attributes. The systems for evaluating Officers and Non-Commissioned Officers

depend on the ability of experienced raters to make judgments about competencies, skills, valuesand other qualitative characteristics of an individual (AWG, 2009). This is no less valid for

assessing training and education.

The behavioral and social sciences have developed a plethora of methods that attempt to strike abalance between objectivity and subjectivity, that reflect both considerations rather than an

assumption that one has to choose between them, and thus that provide sophisticated and

trustworthy perspectives on verifiability and validity (Chapters 4, 5, and 11). There is no reason

for programmatic decision makers to ruminate over these problems of measurement or, worse, to

make suboptimal and even counterproductive decisions because of an assumption that theproblems are insolvable. Science provides plenty of options for moving forward, especially when

it is embedded in a broader scholarly context (Chapters 3-5; see also, Flyvbjerg, 2001; Godfrey-

Smith, 2003; Schrim & Caterino, 2006).

Clarity About What Is Evaluated

We have developed measures that reflect best practices of OBTE and, most importantly, are

verifiable in practice. Without this, there is no way of knowing whether, or the extent to which,

OBTE actually is implemented; and without such verifiability, investigating the results in a

program of instruction is meaningless. No scientific conclusions can be drawn nor can anyimprovements be suggested without verification of instructional practices. Documents describing

a program of instruction, a training support package, or any other kind of script for a learning

event do not satisfy this criterion. Measures are required so that scientists and evaluators can

verify what is occurring on the ground at the same time measures of effectiveness are being

applied and data on learning are being collected.

The focus of OBTE is on the instructor because instructors are the means through which the

organization has influence on the students. Measurement of instructor behavior is necessary for

this reason and, while it may be sufficient for scientific conclusions, it may not be sufficient to

come to conclusions that are actionable operationally. The reason is that there are organizational

factors that influence what an instructor can do (see Chapters 14 and 15; Appendix C). There is a

need to understand best practices within an instructional organization, including but not limited to

the command climate, with a level of theoretical grounding and practical detail commensurate


14/25

Epilogue 266


with the formative measures we have developed for instructor behavior. The thinking that

scientists would do to figure out what to measure within an organization is the very same thinking

that a programmatic stakeholder would have to do to reexamine and reprioritize activities and

objectives within a program. A provocative implication is that it would be useful to create a

forum in which scientists, leaders in programs of instruction, and active duty commanders or their

representatives could engage in collaborative reflection about the principles, practices, and

desired outcomes of training and education. The collaboration in such a forum could look verymuch like the COMPASS process used to develop formative measures for OBTE (see e.g.,

Chapter 2).

The return on the investment in science to support programmatic decision-making is greatest if

the scientific assessment yields information that either can improve the capability or refine the

requirements that motivated the capability. One of the most important contributions of the

scientific inquiry into OBTE to date is to argue for a reorientation toward the common

competencies and attributes of Soldiers that matter to active duty units involved in Full Spectrum

Operations, to strive for a fuller and actionable interpretation of requirements for training and

education, and to close any associated gap between the Institutional Army and Operational Army.Another important contribution is to show how best practices of instruction can become more

readily identifiable based on this reorientation. Together, this allows for systematic scientificinquiry and a broader scholarly dialog among scientists and nonscientists that can have the

following results:

Stimulate debate and facilitate decision making about instructional methodologies Identify and facilitate a coherent approach to training, education, and leader development Provide framework for curriculum-level design and development Provide framework for continuous and rigorous quality improvement

Next Steps

Further research should vigorously pursue development of (a) standards for the values-based

outcomes of interest to OBTE and Army doctrine; (b) measures that relate to various aspects andstages of individual progress with respect to values-based outcomes; (c) theory and empirical

methodologies that reveal inter-temporal relationships between immediate observable effects andlong-term outcomes; and (d) an approach to verification and validation that brings quality

assurance into a tight correspondence with staff and faculty development within a comprehensive

approach to program evaluation.

This broad program of research would not be possible without the grounded theory we developedover the course of this investigation (see e.g., Chapters 3, 4 and 5). These scientific underpinnings

are just a beginning for serious scholarship into OBTE and, more generally, into the critical

interrelationships among training, education, self-development, and leader development. At the

same time, this foundation is sufficient to initiate empirical research and to being to weave

together otherwise disparate sources of empirical evidence needed to support programmaticdecision-making in the institutional Army (cf., Flyvbjerg, 2001; Godfrey-Smith, 2003; Schrim &

Caterino, 2006; see also Chapter 11, section 11.2.3).

Figure 3 shows ways in which the scientific underpinnings for OBTE can guide the development

of values-based standards that can be verified and validated. This should be viewed as a road

map. The specific measures have yet to be identified and applied. It is important to note, however,

that these scientific underpinnings for OBTE are sources of measures that have been scientifically

validated and employed to come to a deeper understanding of situated behavior, situated


15/25

267 Riccio


experience, and situated meaning. That is, they preclude having to start from scratch and develop

new measures for all the important activities, decisions, and processes that are critical in OBTE

and, more generally, in the institutional Army. Our collaborative inquiry into the practical

wisdom as instructors and leaders has given us a good start. The integrative work should be

continued so that programmatic decision-makers can have the information they need to have the

greatest impact on preparation of Soldiers for FSO (see Chapters 11 and 15).

Figure 3. Sources of potential measures to support analysis, design, development,

implementation and evaluation of instruction that reflects a programmatic view of the

development and employment of values-based standards. Compare with Figure 2 in this

Epilogue and with Figure 8 in Chapter 3.


16/25

Epilogue 268


References

Asymmetric Warfare Group (2009, March). US Army Asymmetric Warfare Group workshop on

Outcomes-based Training and Education, Applied Physics Laboratory, Johns Hopkins

University, Laurel MD.

Bandura, A (1997). Self-efficacy: The exercise of control. New York, NY: Freeman and

Company.Bridge, G. (2000). Rationality, ethics, and space: on situated universalism and the self-interested

acknowledgement of difference.Environment and Planning D: Society and Space 2000, 18,

pages 519-535.

Bruny, T., Riccio, G., Sidman, J., Darowski, A., & Diedrich, F. (2006). Enhancing warrior ethos

in initial entry training. Proceedings of the 50th Annual Meeting of the Human Factors and

Ergonomics Society, San Francisco, CA.

Darwin, M. (2008a).Asymmetric Warfare Group combat applications training course (CATC)

senior leader discussion(Briefing, January 23, 2008). Fort Meade, MD: Asymmetric Warfare

Group.

Darwin, M. (2008b). Outcomes-based training and education: fostering adaptability in full

spectrum operations(Briefing, December 2008). Fort Meade, MD: Asymmetric Warfare

Group.Dienes, Z. & Perner, J. (1999) A theory of implicit and explicit knowledge.Behavioral and Brain

Sciences 22(5), 735-808

Freeman, J., Jason, J., Aten, T., Diedrich, F., Cooke, N., Winner, J., Rowe. L., & Riccio, G.

(2008). Shared Interpretation of Commander's Intent (SICI). Final Report to the ArmyResearch Institute for the Behavior and Social Sciences, contract number W74V8H-06-C-

0004.

Flyvbjerg, B. (2001).Making social science matter: Why social inquiry fails and how it can

succeed again. Cambridge, UK: Cambridge University.

Godfrey-Smith, P. (2003). Theory and reality.Chicago, IL: University of Chicago Press.Haskins, C. (2009, March).Development of outcomes based training. Presentation at the US

Army Asymmetric Warfare Group workshop on Outcomes-based Training and Education,

Applied Physics Laboratory, Johns Hopkins University, Laurel MD.James, W. (1907). Talks to teachers on psychology: and to students on some of life's ideals. New

York, NY: Henry Holt and Company. (Original work published 1899)Leibrecht, B., Wampler, R., & Pleban, R. (2009).Methodology for evaluating transfer of learning

from the U.S. Armys advanced leaders course. Research Product 2009-05. Washington, DC:

U.S. Army Research Institute for the Behavioral and Social Sciences.

Maslow, A. (1943). A theory of human motivation.Psychological Review, 50, 370-396.

Maslow, A. (1968). Toward a psychology of being. New York: Van Nostrand.McKee, J. (2007). The new breed: Understanding and equipping the 21st century volunteer.

Loveland, CO: Group Publishing.Mikroyannidis, A. (2007). Toward a social semantic web.

Computer, November 2007, 113-115. Los Alamitos, CA: IEEE Computer Society.

Neisser, U., & Fivush, R. (Ed.) (1994). The remembering self: construction and accuracy in the

self narrative. Cambridge, UK: Cambridge University Press.Perry, R. & McEnery, K. (2009). Army reconnaissance course: Defining the aim point for

reconnaissance leader training.Armor, July-August, 14-20.

Putnam, R. (2001).Bowling alone: The collapse and revival of American community. New York:

Simon & Schuster.

Reed, E. (1996). The necessity of experience.New Haven: Yale University.

Riccio, G., dEchert, B.C., Lerario, M., Pound, D., Bruny, T., & Diedrich, F. (2006).Enhancing

Joint Task Force Cognitive Leadership Skills. Report to the Army Research Institute for the


17/25


18/25

ix Riccio, Diedrich, & Cortes (Eds.)


TABLE OF CONTENTS

.

page

Prologue: A Programmatic View of the Inquiry into Outcomes-Based Training & Education....... 1Historicity of our Research on OBTE..........................................................................................1The Approach and Lessons Learned from the Research..............................................................3Documentation of the Research ...................................................................................................4

Section I. Development of Stakeholder Requirements for OBTE..............................................6 Chapter 1. Preparation for Full Spectrum Operations ......................................................................7

1.1 Requirements of Full Spectrum Operations...........................................................................81.2 Outcomes-Based Training and Education (OBTE)..............................................................10

1.2.1 Exemplar of OBTE: Combat Applications Training Course........................................111.2.2 OBTE as a Multifaceted Instructional System .............................................................12

1.3 An Appraisal of Instruction with Respect to OBTE ............................................................131.3.1 A Systems Engineering Framework for Integration and Development of OBTE........131.3.2 Preparation for Validation and Verification .................................................................14

1.4 References ............................................................................................................................17Chapter 2. Formative Measures for Instructors..............................................................................20

2.1 Development of Formative Measures ..................................................................................202.1.1 The COMPASS Methodology......................................................................................202.1.2 Development of Measures for OBTE ...........................................................................21

2.2 Description of Formative Measures .....................................................................................212.2.1 Results of the COMPASS Process................................................................................212.2.2 Elaboration on the Description of Measures.................................................................23

2.3 OBTE Performance Measures: Planning for Training.........................................................232.3.1 Define Outcomes ..........................................................................................................232.3.2 Create a Positive Learning Environment......................................................................252.3.3 Create the Parameters of Learning................................................................................27

2.4 OBTE Performance Indicators: Training Execution............................................................282.4.1 Communicate the Parameters of Learning....................................................................282.4.2 Training Emphasizes Broad Combat or Mission Success ............................................292.4.3 Customize Instruction When Possible Based on Constraints/Conditions ....................312.4.4 Facilitates Learning of Concepts ..................................................................................322.4.5 Creates a positive learning environment.......................................................................342.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation................................362.4.7 Uses scenarios to facilitate learning..............................................................................382.4.8 Instructors exhibit intangible attributes in own actions................................................402.4.9 Hotwashes and Mini-AAR............................................................................................42

2.5 Uses of the Measures ...........................................................................................................432.5.1 Formative Measures for Instructors..............................................................................442.5.2 Quality Assurance and Instructor Education ................................................................442.5.3 Continuous Improvement of Assessments....................................................................452.5.4 Program Evaluation and Organizational Change..........................................................46

2.6 References ............................................................................................................................46


19/25

Table of Contents x


Chapter 3. Principles and Practices of Outcomes Based Training & Education............................503.1 Multifaceted Inquiry.............................................................................................................50

3.1.1 Interaction with Progenitors of OBTE..........................................................................513.1.2 AWG Documents on OBTE .........................................................................................523.1.3 Collaborative Reflection on Participant Observation in CATC ...................................523.1.4 Interaction with Stakeholders .......................................................................................53

3.2 Essential Characteristics of OBTE.......................................................................................533.2.1 The Meaning of Developmental is a Critical Difference..............................................533.2.2 The Definition of Outcomes is a Critical Difference....................................................563.2.3 The Emphasis on Values and Causally Potent Intangibles is a Critical Difference .....583.2.4 The Meaning of Experience is a Critical Difference....................................................613.2.5 The Emphasis on Instructor-Student Interactions is a Critical Difference ...................623.2.6 The Emphasis on Learning to Learn is a Critical Difference .......................................633.2.7 The Emphasis on Collaborative Design and Development is a Critical Difference.....65

3.3 Toward a Grounded Theory for OBTE................................................................................663.3.1 Need for an Integrated Interdisciplinary Framework ...................................................663.3.2 Formative Measures of Instructor Behavior as Evolving Best Practices of OBTE......67

3.4 Emerging Best Practices in OBTE for a Community-Centered Environment.....................683.4.1 Leadership and Enculturation of Soldiers.....................................................................683.4.2 Robust and Adaptable Plan...........................................................................................703.4.3 Instructors as Role Models ...........................................................................................703.4.4 Collaborative Identification of Outcomes and Measures .............................................71

3.5 Emerging Best Practices in OBTE for a Knowledge-Centered Environment .....................713.5.1 Integrated Understanding of Basic Soldier Skills in Full Spectrum Operations ..........723.5.2 Task Relevance of Planned Instructional Events..........................................................723.5.3 Reveal Operational Relevance of Training...................................................................733.5.4 Incorporate Stress into Instructional Events .................................................................733.5.5 Identify General Lessons Learned and Extrapolate to New Situations ........................74

3.6 Emerging Best Practices in OBTE for an Assessment-Centered Environment...................743.6.1 Collaborative Reflection and Problem Solving ............................................................753.6.2 Communication.............................................................................................................753.6.3 Nature and Extent of Guidance.....................................................................................763.6.4 Establish a Pervasive Mindset of Collaborative Reflection..........................................76

3.7 Emerging Best Practices in OBTE for a Learner-Centered Environment ...........................773.7.1 Soldier Motivation and Development of Intangibles....................................................773.7.2 Plan for Development of the Individual .......................................................................783.7.3 Get Students to Take Ownership ..................................................................................783.7.4 Collaborative Reflection as a Means to Develop Self Efficacy....................................79

3.8 References ............................................................................................................................79Chapter 4. Grounded Theory for Values-Based Training & Education.........................................86

4.1 Exploration of Holistic and Functionalistic Underpinnings for OBTE ...............................864.1.1 Fundamental Units of Analysis.....................................................................................874.1.2 Nested Time Scales and Adaptability...........................................................................884.1.3 Adaptability and Ambiguity .........................................................................................904.1.4 Mechanistic Analogies and Predominant Experimental Paradigms .............................92

4.2 Three Pillars for the Scientific Foundation of OBTE ..........................................................93 4.2.1 Ecological Psychology..................................................................................................934.2.2 Self-Efficacy Theory.....................................................................................................974.2.3 Positive psychology......................................................................................................98

4.3 A More Integrated Scientific Infrastructure.......................................................................101


20/25

xi Riccio, Diedrich, & Cortes (Eds.)


4.3.1 Self Determination Theory .........................................................................................1014.3.2 Situated Learning Theory ...........................................................................................1034.3.3 Existential Psychology................................................................................................105

4.4 Building on the Scientific Infrastructure for OBTE...........................................................1094.4.1 Triadic Frameworks....................................................................................................1094.4.2 Further Development ..................................................................................................112

4.5 References ..........................................................................................................................112Chapter 5. Passion and Reason in Values-Based Learning & Development ...............................118

5.1 The Nested Self ..................................................................................................................1185.1.1 An Alternative to Individual versus Collective ..........................................................1185.1.2 Cognition and Reality .................................................................................................119

5.2 Conscious Experience and the Dynamics of Thinking ......................................................1225.3 Emotion, Information, and Engagement ............................................................................125

5.3.1 Ecological Perspective on Emotion ............................................................................1255.3.2 Emotion as Engagement .............................................................................................1265.3.3 Implications for Training and Education....................................................................129

5.4 Emotion, Decision-Making, and Inter-Temporal Choice...................................................1295.4.1 Toward a More Integrated Theory..............................................................................1295.4.2 Emotion and Decision-Making...................................................................................1305.4.3 Emotion and Nested Time Scales ...............................................................................1315.4.4 Neuroeconomics and Inter-Temporal Reasoning .......................................................1325.5.5 Inter-Temporal Reasoning and Adaptive Dynamical Systems...................................133

5.5 Beyond Science..................................................................................................................1345.5.1 Existentialism..............................................................................................................1345.5.2 The Soldier-Scholar as an Emergent Property of a Collective Pursuit.......................135

5.6 References ..........................................................................................................................137Section II. Verification and Validation of OBTE as a Service System ..................................142 Chapter 6. Initial Impressions of Participation in CATC .............................................................1436.1 Methods..............................................................................................................................143

6.1.1 Participants..................................................................................................................1436.1.2 Procedure ....................................................................................................................1436.1.3 Analyses......................................................................................................................144

6.2 Results ................................................................................................................................1446.3 Implications for Service System Development: Peer Review ...........................................1466.4 References ..........................................................................................................................147

Chapter 7. Local Development of Measures of Effectiveness .....................................................1497.1 What do Instructors Believe Soldiers Should Learn in Initial Entry Training?.................1497.2 Measure Development Process ..........................................................................................1507.3 What do OBTE-Trained DS Believe is Important to Assess in BRM/ARM? ................... 1517.4 Implications........................................................................................................................1567.5 Conclusions........................................................................................................................1587.6 References ..........................................................................................................................159


21/25

Table of Contents xii


Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training .....1608.1 Methods..............................................................................................................................160

8.1.1 Participants..................................................................................................................1608.1.2 Procedure ....................................................................................................................1608.1.3 Analyses......................................................................................................................161

8.2 Results ................................................................................................................................1638.2.1 Behavior of DS ...........................................................................................................1638.2.2 Behavior and Performance of Privates .......................................................................1658.2.3 Patterns of Communication ........................................................................................1688.2.4 Potential Influence of Instructor Behavior on Performance of Privates.....................170

8.3 Implications for Service System Development..................................................................1718.3.1 Verification of OBTE .................................................................................................1718.3.2 Validation of OBTE....................................................................................................172

8.4 References ..........................................................................................................................173Chapter 9. Impact on Rifle Marksmanship Training....................................................................174

9.1 Behavioral Data Collection During Basic Rifle Marksmanship ........................................1749.1.1 Method........................................................................................................................1749.1.2 Assessment..................................................................................................................1759.1.3 Results An Overview...............................................................................................1779.1.4 Evidence for Influence of OBTE................................................................................1789.1.5 Behavior of Drill Sergeants after Exposure to OBTE ................................................1809.1.6 Behavior of Privates....................................................................................................1829.1.7 Patterns of Communication ........................................................................................1869.1.8 Summary.....................................................................................................................186

9.2 Attitudes Toward an OBTE in Basic Training...................................................................1879.2.1 Method........................................................................................................................1879.2.2 Results.........................................................................................................................187

9.4 References ..........................................................................................................................191Chapter 10. Influence of CATC in an Operational Setting ..........................................................19210.1 Methods............................................................................................................................192

10.1.1 Participants................................................................................................................19210.1.2 Procedure ..................................................................................................................19210.1.3 Analyses....................................................................................................................193

10.2 Results ..............................................................................................................................19310.2.1 Downstream Impact on Marksmanship ....................................................................19310.2.2 Downstream Impact on Training in the Units ..........................................................19410.2.3 Downstream Impact on Self Efficacy.......................................................................195

10.3 Implications for Service System Development: Validation.............................................19610.4 References ........................................................................................................................197

Chapter 11. Implications for Service System Development.........................................................19811.1 Lessons Learned about Transfer of OBTE.......................................................................19811.2 Implications for Service System Development................................................................199

11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE..........19911.2.2 Further Development of OBTE as a Service System ...............................................199 11.2.3 Further Verification and Validation of OBTE..........................................................201

11.3 References ........................................................................................................................203


22/25

xiii Riccio, Diedrich, & Cortes (Eds.)


Section III. Further Development of OBTE as a Service System ..........................................206Chapter 12. Development of General Measures for Students ......................................................207

12.1 Intent ................................................................................................................................20712.2 Performance Measure Development Process...................................................................207

12.2.1 Phase One: Define Performance Indicators (PI).......................................................20712.2.2 Phase Two: Translate PI into performance measures...............................................20812.2.3 Phase Three: Measure refinement.............................................................................20812.2.4 Phase Four: Retranslation of Measures ....................................................................208

12.3 Product of Measure Development....................................................................................20912.3.1 Learner Perception of the Instructor and Course......................................................20912.3.2 Learner Engagement.................................................................................................21112.3.3 Student Relationship with Teacher ...........................................................................21212.3.4 Student Results .........................................................................................................21412.3.5 Self-Report Measures ...............................................................................................216

12.4 Conclusion........................................................................................................................21712.5 References ........................................................................................................................217

Chapter 13. Adapting OBTE in a Classroom Environment .........................................................21913.1 Intent ................................................................................................................................21913.2 Observing OBTE in the Classroom Environment............................................................219

13.2.1. Participants...............................................................................................................21913.2.2. Procedure .................................................................................................................22013.2.3. Measures ..................................................................................................................220

13.3 Utility of OBTE Measures in a Classroom Environment ................................................22013.3.1 Generality of Measures.............................................................................................22013.3.2. Implications for Improvement of Measures.............................................................22113.3.3 Implications for improvement of course design .......................................................222

13.4 Use of 360 Reviews for Collaborative Reflection..........................................................22313.4.1 The Role of a 360 Review in OBTE .......................................................................22313.4.2 Narrative of a Participant Observer ..........................................................................22513.5 Learning, cognitive load and motivation..........................................................................22813.5.1 The NASA Task Load Index as a subjective measure of student workload.............22813.5.2 Results.......................................................................................................................22913.5.3 Implications ..............................................................................................................230

13.6 Conclusions......................................................................................................................23013.7 References ........................................................................................................................231

Chapter 14. Organizational Climate and Creation of Durable Change ........................................23314.1 The Need ..........................................................................................................................23314.2 Initial Indications of Possible Resistance to Change .......................................................23414.3 Models and Considerations for Sustainable Change........................................................235

14.3.1 The Change Transition Period ..................................................................................23514.3.2 Organizational Culture..............................................................................................23714.3.3 Clarity of Mission and Shared Understanding..........................................................23714.3.4 Relevant Observations During the Current Investigation.........................................23814.3.5 Organizational Support and Incentives.....................................................................238

14.4 Conclusions......................................................................................................................23914.5 References ........................................................................................................................239


23/25

Table of Contents xiv


Chapter 15. Five ways OBTE can enable the Army Leader Development Strategy....................24215.1 Background ......................................................................................................................24215.2 An Emerging Consensus..................................................................................................244

15.2.1 What Part to Balance?...............................................................................................24415.2.2 Improving Training, by Design ................................................................................24515.2.3 Increased Use of dL and Dependence on Self-Development ...................................24615.2.4 Future Orientation, Unknown Requirements............................................................24715.2.5 The Quality Instructor Challenge .............................................................................24715.2.6 Purpose and Design are Key.....................................................................................24815.2.7 A Natural Advantage ................................................................................................24915.2.8 Task Specialization or Generalized Competency.....................................................249

15.3 Conclusion........................................................................................................................25115.4 References ........................................................................................................................252

Epilogue. Integration of Leader Development, Education, Training, and Self-Development.....254Toward Values-Based Standards for Army Doctrinal Requirements ......................................254Nested Standards and Quality Assurance.................................................................................256Needs and Opportunities for Staff & Faculty Development.................................................... 259

A Role for Science and Measurement .................................................................................259Toward Best Practices in Instructor Education....................................................................260Critical Considerations for Further Scientific Investigation ....................................................263

The Necessity of Long-Term Studies ..................................................................................263False Dichotomy of Objective-Subjective...........................................................................264Clarity About What Is Evaluated.........................................................................................265Next Steps ............................................................................................................................266

References ................................................................................................................................268Section IV. Appendices...............................................................................................................270 Appendix A. OBTE Principles & Practices: Instructor Measures................................................271

A.1 Genesis of Formative Measures for Instructors ................................................................271A.2 Principles of Outcomes-Based Training & Education......................................................272A.3 Guide to Using Measures of Instructor Behavior..............................................................276A.4 Complete Menu of Instructor Measures............................................................................279

Appendix B. OBTE Principles & Practices: Student Measures...................................................318B.1 Guide to Using Measures of Student Behavior ................................................................. 318B.2 Complete Menu of Student Measures ...............................................................................319

Appendix C: A Commanders View of Outcomes-Based Training and Education..................... 340Summary ..................................................................................................................................340

Definition.............................................................................................................................340Description...........................................................................................................................340 Elements of OBTE. ..................................................................................................................341Developing the Outcomes....................................................................................................341Developing the Training Plan ..............................................................................................341Conducting Training............................................................................................................342How Training is Assessed....................................................................................................344

Conclusion................................................................................................................................344

8/12/2019 Epilogue. Integration