18 1541-1672/09/$25.00 © 2009 IEEE iEEE iNTElliGENT SYSTEMSPublished by the IEEE Computer Society

H I S T O R I E S & F U T U R E SH U M A N - C E N T E R E D C O M P U T I N GEditors: robert r. hoffman, patrick J. hayes, and Kenneth M. Ford, Institute for Human and Machine Cognition, rhoffman@ihmc.us

Sherlock tutor for electronics troubleshooting, for example, condensed four years of on-the-job train-ing to approximately 25 hours, compressing the du-ration of the experience-feedback-learning cycle:2

Sherlock presented a concentration of useful cases in a brief period of time. The real world mostly pro vides opportunities to do the routine. Exper tise involving the nonroutine is harder to get from ev-eryday work experience because the right sit uations occur rarely and often are handled by established experts when they do occur, not by students.3

But accelerated learning should refer to more than the hastening of basic profi ciency. It reaches across the profi ciency scale to the question of how to accel-erate the achievement of expertise, and whether that is even possible. Paralleling this question are practi-cal issues, including the military’s need to conduct training at a rapid pace, and the issues of workforce and loss of expertise. Many organizations such as the US Department of Defense, NASA, and the electric utilities are at risk because of the imminent retirement of domain practitioners who handle the most diffi cult and mission-critical challenges.4

To accelerate profi ciency, we must facilitate the acquisition of extensive, highly organized knowl-edge. We must also accelerate the acquisition of ex-pert-level reasoning skills and strategies.5 But that’s just the beginning of the challenge.

The ChallengeExperts are repositories of vast historical informa-tion that enables them to exercise effective techni-cal leadership in ambiguous or complex situations, often by communicating subtle features that other people won’t see until those features are pointed out. A classic estimate states that the development of very high-level skills in any complex domain takes at least 10 years.6 But extraordinary experts who conduct mission-critical activities in industry settings have proven their value and earned extra-ordinary respect through the course of 25 to 35 years of experience.

It’s clear that mere time in grade doesn’t enable just anyone to adequately fi ll such functions; rou-tine practice isn’t suffi cient for the development of expertise. There needs to be

a constant stretching of the skill, defi ned by in-•creasing challenges (tough or rare cases), high levels of intrinsic motivation to work hard, •on hard problems,practice that provides meaningful feedback, and•practice with an expert mentor’s support and •encouragement.

Edwin Thorndike, one of the founders of educa-tional psychology, called this “practice with zeal.”7 Appropriate to his milieu, Thorndike focused on classroom learning of simple tasks. More recently in the fi eld of expertise studies, Anders Ericsson has referred to “deliberate practice” to achieve ex-pertise, in such domains as music and chess.8 But the notion also holds for domains such as weather forecasting, engineering, and military command.9

At its most general level, the core idea is that mere

We wish to pose accelerated learning as a

challenge for intelligent systems technol-

ogy. Research on intelligent tutoring systems has

proved that accelerated learning is possible.1 The

Robert R. Hoffman and Paul J. Feltovich, Institute for Human and Machine CognitionStephen M. Fiore, University of Central FloridaGary Klein, Klein Associates Division of ARADavid Ziebell, Electric Power Research Institute

Accelerated Learning (?)

March/april 2009 www.computer.org/intelligent 19

repetition of what’s already known is not enough—what’s necessary is practice that systematically engages the learner in increasingly challenging ways and provides a meaningful con-text for feedback, remediation, and growth.

Let’s accept for the moment the finding from studies of experts that 10 years is a lower limit and that a lengthy period of deliberate practice is necessary. In that case, the best that organizations could hope for is to “preburn” a few years during the novice-to-apprentice and apprentice-to-journeyman spans of training. Our vision, in contrast, is that we might accelerate the achievement of profi-ciency across the journeyman-to- expert span.

BackgroundWe can leverage a few decades of re-search on “cognition in the wild”—that is, studies of complex problem solving by experts in a variety of domains, including firefighting, nurs-ing, weather forecasting, emergency response, and military command.10−13 Experts possess rich, detailed, highly organized knowledge. They are distin-guished by their ability to

develop rich mental models of cases •or situations to support sensemak-ing and anticipatory thinking,create new procedures on an ad hoc •basis,cope with novel and tough cases,• 14 andperform under stress and high levels •of mental workload.

Expertise takes a long time to achieve and is bounded in its range.8 Thus, traditional learning methods that focus on cursory exposure and short-term results might be insufficient to accelerate the achievement of profi-ciency. It’s often assumed that the way to help learners cope with complexity is to simplify the situation and then

incrementally introduce increasingly complex elements. However, initial learning that’s more difficult and that even demonstrates inferior immediate results can lead to greater flexibility and transfer.15 Perhaps most impor-tantly, research on high-end learn-ing (for example, in medical educa-tion) has shown that when learners are initially exposed to simplifications of complex topics, serious misunder-standings can become entrenched and then interfere with achieving richer,

accurate understandings.16

Leverage Points for AccelerationBackground research also provides pointers to ways of achieving ac-celeration.

Tapping the Right TheoriesTwo related proposals about high-end learning, based upon findings from expertise research, can inform accel-eration: cognitive flexibility and cog-nitive transformation.

Research on how people deal with difficult problems led to the idea of cognitive flexibility.17,18 Problems that involve dynamics, interacting pro-cesses, nonlinear causation, simulta-neous events, and other complexities are particularly difficult for learn-ers. When confronted with gaps and

inaccuracies in their knowledge or reasoning regarding these types of complexity, learners invoke what are called knowledge shields—ill-formed counterargu ments that let them pre-serve their simplistic understanding in the face of contradictory evidence.16 To achieve expertise, a learner must be flexible—that is, willing and able to recognize and overcome the knowl-edge shields.

Closely related is the idea of cog-nitive transformation: the hypoth-esis that to achieve expertise, people must unlearn.19 That is, they must work through experiences that force them to lose faith in their entrenched, simplistic mental models so that they can move to deeper levels of understanding.

Studying How Mentors Do ItMentoring has not been studied at the higher levels of expertise and ex-traordinary expertise. Some research-ers have investigated what makes for a good coach in sports20 and what makes for a good teacher.21,22 But little is known about how to find good mentors or what makes for a good mentor in the context of the modern sociotechnical workplace. Also, little is known specifically about mentoring from the journeyman level to expert.

In Sources of Power, Gary Klein describes the techniques that skilled mentors use and the kinds of skills they need to develop for evaluat-ing mentees and setting the right climate for learning.23 The good mentor knows how to create appro-priate learning content and guide those who are less experienced. The expert mentor can rapidly form a rich mental model of the learner’s knowledge and skill. From this, the expert mentor can predict when and why the learner will form a simplis-tic or inaccurate understanding. The mentor anticipates the kinds of cases that will lead the learner to err and the kinds of practice experiences that

Time in grade doesn’t enable just anyone to adequately fill such functions; routine practice isn’t sufficient for the development of expertise.

20 www.computer.org/intelligent iEEE iNTElliGENT SYSTEMS

will push the learner to greater un-derstanding of complexity.14,24

There is an outstanding practical need for methods to rapidly iden-tify good mentors and then apply the right cognitive task analysis methods to reveal their reasoning strategies, especially as they deal with challeng-ing mentoring situations. Many expe-rienced workers are intrinsically moti-vated to pass along their “tough case” knowledge, but they’re often left with out organizational support, not knowing how to proceed. They often don’t even know that their “tough case” knowledge is critical and highly constructive to apprentices.

Giving Corrective FeedbackStudies of expertise reveal great vari-ability between domains (and special-izations within domains) in the extent to which workers receive any feed-back, let alone timely, high-quality feedback.25 In some cases, the inher-ent nature of the domain (for example, long-term weather forecasting and intelligence analysis for policy pro-jection) makes it impossible for the practitioner to receive timely feedback. This might be one reason why it can take a decade or more for an individ-ual to achieve expertise.

Experts learn more from their mis-takes than from what they get right. It’s been said that apprentices make the same mistake twice, journey-men make the same mistake once, and experts work until they never make mistakes. Domain specialists are intrinsically motivated and often seek out corrective feedback that al-lows them to perceive their errors.26 When everything works the way it’s supposed to, practitioners are less likely to receive feedback about what didn’t work or what might have been done better. Experts seek out correc-tive feedback that points out targets for improvement. Cognitive flexibility and cognitive transformation suggest that feedback should help learners

transcend the inclination to invoke a knowledge shield (that is, rationalize away a reductive misunderstanding) and help them unlearn notions that incorrectly simplify their understand-ing of the domain.

Tough-Case Time CompressionPeople learn what they practice. So, to achieve adaptive expertise, they should practice in many ways, across many kinds of cases (including ones that involve the same basic principles

but in different kinds of contexts), us-ing many kinds of conceptual tools, points of view, mental organizational structures, and investigation and prac-tice strategies.17 The modes and means of training should engage real work practice—the job’s challenges, con-texts, and duties—to the greatest ex-tent possible.

Training to accelerate proficiency will have to include an appreciation of the importance of dynamic or causal reasoning, anticipatory thinking, problem detection, and other macro-cognitive functions.5 In order for practitioners to better handle complex-ity, especially unexpected complex-ity, training must involve helping them acquire a more powerful toolbox and the knowledge of how to build entirely new tools when tough cases arise.

Tough cases, more or less by defini-

tion, are rare. This might be another reason why it takes so long to achieve expertise. A library of tough cases could be utilized to accelerate exper-tise development … and here we come to the role of intelligent systems.18,27 We propose that intelligent technolo-gies, including virtual reality and other case-oriented modeling tools, could enable time compression. We could eliminate the time spans be-tween tough cases, and also shrink the chunks of time inside tough cases, when not much is happening.

The initial step would be to gener-ate a case library. This would be a job for cognitive systems engineers. Although laborious and time con-suming, this would be a “two-fer.” That is, the process of capturing and preserving expert knowledge would merge with the process of generating training materials. Methods of knowl-edge elicitation and cognitive task analysis would be used to generate cases. For each case, the expert’s nar-rative would weave together the hid-den chain of causally related events leading to a particular problem. Each case will have to form around a rich narrative, a timeline, a roster of in-formation requirements, a roster of decision requirements, and other re-sources spanning the range of media (photos, video, and so on). Each case would be scaled for the level of profi-ciency, relative to individuals’ levels of ability, both for routine practice and for skill stretching. In collaboration with computer scientists, the cases would be formed into instructional materials and exercises, which could range from desktop exercises to full simulated scenario experiences driven by intelligent systems technologies.

We can also use tough, rare cases in unlearning to reveal knowledge shields and help trainees overcome dimen-sions of difficulty for given cases.17 Rare cases would be juxtaposed with routine or typical cases in such a way that the learner could experience the

Apprentices make the same mistake twice, journeymen make the same mistake once, and experts work until they never make mistakes.

March/april 2009 www.computer.org/intelligent 21

differences that produce difficulties. Another application of the case

corpus involves the constantly chang-ing sociotechnical workplace.28 New technologies, and the work methods that they shape, typically must be inte-grated with legacy work methods and technologies. Conceptually, we can view a set of tough-case scenarios as opportunities for practice at adapta-tion, not merely to handle familiar and routine cases but also to “stretch” the new technologies and work methods through application in tough cases.

Simulations allow for tremendous flexibility in manipulations of fidel-ity—differing kinds and amounts of contextual richness. With regard to the application of intelligent systems, the acceleration simulations would be set up such that the simulations would sometimes lead and sometimes mis-lead the learner in seeking solutions. The idea is that the simulation, like an expert mentor, purposely misguides a learner into a common solution, but the reality is that the error is due to a subtle issue. In addition, an appropri-ately crafted scenario would man-age temporal sequencing. This would juxtapose real-world time with nar-rative time, so that the learner could experience the event through the ex-pert’s perspective (real-world time) and through the more omniscient, third-person perspective of compressed nar-rative time—that is, presenting events to highlight certain occurrences. These variations would emphasize particular elements of the scenario and promote both perceptual learning and the ac-quisition of expert strategies.

The modern sociotechnical work-place and the world in which it’s em-bedded present significant challenges that aren’t easily solved, in part be-cause they force us to take traditional notions of training and transfer to new levels. The people who work in

sociotechnical systems must be trained to be adaptive, so that they can cope with the ever-changing world and an ever-changing workplace. People must be trained to be resilient, so that they can cope with complexity when unex-pected events stretch resources and ca-pabilities. And people must be trained faster. Intelligent systems technology, and intelligent use of technology, will certainly play a critical and perhaps necessary role in this.

References

1. K.D. Forbus and P.J. Feltovich, eds., Smart Machines in Education, AAAI Press, 2001.

2. A.M. Lesgold, “Sherlock: A Coached Practice Environment for an Electron-ics Troubleshooting Job,” Computer Assisted Instruction and Intelligent Tutoring Systems: Shared Issues and Complementary Approaches, J. Larkin and R. Chabay, eds., Lawrence Erlbaum Associates, 1992, pp. 201−238.

3. A.M. Lesgold, “On the Future of Cogni-tive Task Analysis,” Cognitive Task Analysis, J.M. Schraagen, S.F. Chip-man, and V.L. Shalin, eds., Lawrence Erlbaum Associates, 2001, pp. 451−466.

4. R.R. Hoffman and L.F. Hanes, “The Boiled Frog Problem,” IEEE Intelligent Systems, July/Aug. 2003, pp. 68−71.

5. G. Klein et al., “Macrocognition,” IEEE Intelligent Systems, May/June 2003, pp. 81−85.

6. J.R. Hayes, “Three Problems in Teach-ing General Skills,” Thinking and Learning Skills: Research and Open Questions, vol. 2, S.F. Chipman, J.W. Segal, and R. Glaser, eds., Law-

rence Erl baum Associates, 1985, pp. 391−406.

7. E.L. Thorndike, Education Psychology, Routledge, 1913.

8. K.A. Ericsson et al., The Cambridge Handbook on Expertise and Expert Per-formance, Cambridge Univ. Press, 2006.

9. R.R. Hoffman, ed., Expertise Out of Context: Proc. 6th Int’l Conf. Natural-istic Decision Making, Taylor & Fran-cis, 2007.

10. E. Hutchins, Cognition in the Wild, MIT Press, 1995.

11. G. Klein and C. Zsambok, eds., Natu-ralistic Decision Making, Lawrence Erl-baum Associates, 1995.

12. J. Lave, Cognition in Practice: Mind, Mathematics, and Culture in Everyday Life, Cambridge Univ. Press, 1988.

13. S. Scribner, “Studying Working Intel-ligence,” Everyday Cognition: Its De-velopment in Social Context, B. Rogoff and S. Lave, eds., Harvard Univ. Press, 1984, pp. 9−40.

14. R.R. Hoffman, “How Can Expertise Be Defined? Implications of Research from Cognitive Psychology,” Exploring Ex-pertise, R. Williams, W. Faulkner, and J. Fleck, eds., Macmillan, 1998, pp. 81−100.

15. R.A. Schmidt and R.A. Bjork, “New Conceptualizations of Practice: Com-mon Principles in Three Paradigms Suggest New Concepts for Training,” Psychological Science, vol. 3, 1992, pp. 207−217.

16. P.J. Feltovich, R.L. Coulson, and R.J. Spiro, “Learners’ (Mis)Understanding of Important and Difficult Concepts: A Challenge to Smart Machines in Educa-tion,” Smart Machines in Education, K.D. Forbus and P.J. Feltovich, eds., AAAI/MIT Press, 2001, pp. 349−375.

17. P.J. Feltovich, R.J. Spiro, and R.L. Coulson, “Issues of Expert Flexibility in Contexts Characterized by Complex-ity and Change,” Expertise in Context, P.J. Feltovich, K.M. Ford, and R.R. Hoffman, eds., AAAI/MIT Press, 1997, pp. 125−146.

18. R.J. Spiro et al., “Cognitive Flexibility Theory: Advanced Knowledge Acquisi-tion in Ill-Structured Domains,” Proc. 10th Ann. Meeting Cognitive Science Soc., Lawrence Erlbaum Associates, 1988, pp. 375−383.

19. G. Klein and H.C. Baxter, “Cognitive Transformation Theory: Contrasting Cognitive and Behavioral Learning,” The PSI Handbook of Virtual Envi-ronments for Training and Education: Developments for the Military and Be-yond, Vol. I: Learning, Requirements, and Metrics, Praeger Security Int’l, 2009, pp. 50–65.

People must be trained to be resilient, so that they can cope with complexity when unexpected events stretch resources and capabilities.

22 www.computer.org/intelligent iEEE iNTElliGENT SYSTEMS

20. S.M. Fiore and E. Salas, “Cognition, Competition, and Coordination: Un-derstanding Expertise in Sports and Its Relevance to Learning and Performance in the Military,” Military Psychology, vol. 20, 2008, pp. S1−S9.

21. A.J. Minstrell and E.H. Van Zee, In-quiring into Inquiry Learning and Teaching in Science, Am. Assoc. for the Advancement of Science, 2000.

22. R.W. Proctor and K.-P.L. Vu, Stimulus-Response Compatibility Principles: Data, Theory, and Application, CRC Press, 2006.

23. G. Klein, Sources of Power: How Peo-ple Make Decisions, MIT Press, 1998.

24. G. Klein, “Coaching Others to Develop Strong Intuitions,” Intuition at Work, Doubleday, 2002, pp. 208−226.

25. E. Salas, D.R. Nichols, and J.E. Driskell, “Testing Three Team Training Strategies in Intact Teams: A Meta-Analysis,” Small Group Research, vol. 38, no. 4, 2007, pp. 471−488.

26. S. Sonnentag, “Excellent Performance: The Role of Communication and Coop-eration Processes,” Applied Psychol-ogy: An Int’l Rev., vol. 49, 2000, pp. 483−497.

27. R.J. Spiro et al., “Cognitive Flexibil-ity Theory: Hypermedia for Complex Learning, Adaptive Knowledge Appli-cation, and Experience Acceleration,” Educational Technology, Sept./Oct. 2003, pp. 5−10.

28. R.R. Hoffman and W.C. Elm, “HCC Implications for the Procurement Pro-cess,” IEEE Intelligent Systems, Jan./Feb. 2006, pp. 74−81.

Robert R. Hoffman is a senior research sci-entist at the Institute for Human and Ma-chine Cognition. Contact him at rhoffman@ihmc.us.

Paul J. Feltovich is a research scientist at the Institute for Human and Machine Cogni-tion. Contact him at pfeltovich@ihmc.us.

Stephen M. Fiore is director of the Cogni-tive Science Laboratory at the University of West Florida. Contact him at sfiore@ist.ucf.edu.

Gary Klein is senior scientist at the Klein Associates Division of ARA. Contact him at gary@decisionmaking.com.

David Ziebell is manager of human perfor-mance technology at the Electric Power Re-search Institute. Contact him at dziebell@epri.com.

CallPublish Your Paper in IEEE Intelligent Systems

IEEE Intelligent Systems seeks papers on all aspects

of artifi cial intelligence, focusing on the

development of the latest research into

practical, fi elded applications.

For guidelines,

see www.

computer.org/

mc/intelligent/

author.htm.

The #1 AI Magazine www.computer.org/intelligent

Be on the Cutting Edge of Artifi cial Intelligence!

IEEE

Articlesfor