International Journal of Forecasting 8 (1992) 251-267 North-Holland
Bridging the gap between theory and practice in forecasting
American Graduate School of International Management, Glendale, AZ 85306, USA
General Motors Corporation, Detroit, MI 48202, USA
Materials Management Systems, Inc., P.O. Box 239, Thetford Center, VT 05075-0239, USA
American Graduate School of International Management, Glendale, AZ 85306, USA
Abstract: The successful implementation of forecasting in many organizations is hampered by gaps in communication and understanding between forecast preparers and forecast users. Different individuals in organizations may have varying political agendas which impact the forecasting process. Advances in forecasting can be limited because of the gap between forecasting theorists and practitioners. This paper discusses these issues. In particular, it reports on a series of roundtable discussions on the gap between theory and practice which took place at the 1991 International Symposium on Forecasting. The paper includes suggestions on ways to bridge the gap.
Keywords: Forecast practice, Implementing forecasting, Forecasting process.
Correspondence to: E. Mahmoud, American Graduate School of International Management, Glendale, AZ 85306. USA
We wish to express our thanks to all those who participated in the roundtables and shared their experiences and views. We are also particularly grateful to roundtable participants and others who offered suggestions which enabled us to improve an earlier draft of the paper: Wally Albers, Scott Armstrong, Stuart Bretschneider, Chris Chatfield, Bob Clemen, Estela Bee Dagum, Xiang Ying Du, Robert Fildes, Benito Flores, Wassily Leontief, Russ Maddox, Lars-Erik Oiler, Keith Ord, Robert Raeside, Debra Schramm, Janet Sniezek, Ralph Snyder, Herman Stekler, Arthur Taylor, Tom Yokum, Peg Young, Lilian Wu, Shawa Zhang and an anonymous reviewer.
Some observers believe that there is a gap between forecasting theory and practice, for ex- ample, see DeRoecks (1991) editorial in the International Journal of Forecasting. Several years ago, a study by Wheelwright and Clarke (1976) revealed just such a discrepancy between users and preparers of forecasts. This was re- flected in the technical emphasis of the preparers and the managerial emphasis of the users. Wheelwright and Clarke make the interesting finding that, although technical competence was
~~tt~ibuted to the preparers by the users, users lacked confidence in the ability of the preparers to choose the best techniques or to provide cost-effective forecasts. Further support can bc found in De Gooijcr (1990) who argued that if we acknowledge the fact that the discipline of forecasting derives its reason for existence from its practical relevance. then we should try to close the gap between theory and practice.
This paper reports on a series of roundtable discussions at the 1991 International Symposium on Forecasting (ISF) which addressed the prob- lem of the gap between forecasting theory and practice. Roundtable participants also suggested ways of bridging the gaps that exist between academic forecasters and practitioners and be- tween forecast preparers and forecast users. The longer term goal of this paper is to encourage others to come forward with their ideas and experiences so that the International Institute of Forecasters (IIF) can better serve its members and society in general.
The paper is organized as follows. First is a brief review of the experience of the IIF with respect to the building of relationships between academics and practitioners. Section 3 reports on the roundtahle discussions. Section 4 suggests some solutions and ways to bridge the gap. Section 5 presents some concluding remarks.
Before publication, an attempt was made to have this paper reviewed by all participants in the roundtable discussions. IIF members were also encouraged to contribute their own com- mentary. At the end of the paper are a series of short commentaries in which other authors share their views and experiences regarding the gap between theory and practice.
2. IIF experience
Many recognize the need to improve relation- ships between practitioners who forecast for business and governmental purposes and aca- demics who develop and research forecasting methods. The IIF, from its inception in 1981, has been active in fostering exchanges and building links between forecasters in academia and prac- ticing forecasters in government and the corpo- rate world. Practitioner members of the IIF are becoming more numerous and now represent
more than 40% of IIF membership. The Institute places much emphasis on inviting practitioners to participate in its symposia and publish in its journals. Practitioner members have always served on the editorial boards of the IIFs journals.
In a survey, Armstrong (1988) asked prac- titioner members of the IIF if they actually used the research published in the ZF?~ernario~#i Jour- nal of Forecasting (IJF). Thirty-six per cent re- ported applications of research in the journal and about half of this number reported success- ful applications. While these results appear im- pressive, there is much yet to be done. Gardner and Makridakis (1988) believed that there was a substantial amount of confusion experienced by practitioners interested in using research litera- ture to select and apply forecasting methods. Before 1980, research focused on the theoretical deveIopment of quantitative methodologies in forecasting. Research since 1980 has emphasized empirical studies which challenge the theoretical expectations of methods developed earlier. In the literature, advocates of the various methods still disagree about the most appropriate method for any situation and about how particular meth- ods should be applied and evaluated.
According to Dagum (1989), if forecasting is to become recognized as a profession, there must be unity between theorists and practitioners. The challenge before the Institute in the 1990s is to establish this unity [Gardner (1991)]. In 1985, the IIF introduced the Consultants Clearing House, to help match forecast users to forecast providers. This has had limited success, how- ever, despite considerable publicity in the IIF newsletter and at conferences. At the Interna- tional Symposium on Forecasting (ISF) in 1991, the IIF presented the first practitioners award, as another step towards joining theory and practice.
At the tenth ISF in Delphi, Greece, in 1990, a small group of participants, upon the suggestion of DeRoeck, addressed the need to bridge the gap between academics and practitioners in a more formal and structured way. DeRoeck and Mahmoud agreed to take the initiative in de- veloping the approach. As a result, at the 1991 ISF, three workshop sessions dealt with Bridg- ing the Gap between Theory and Practice in Forecasting. These roundtables reffected the
E. Mahmoud et al. I Bridging the gap between theory and practice 253
theme of the entire symposium: Practice and Potential. All roundtable participants are listed in Appendix A.
3. The three gaps between theory and practice
Discussions at the three roundtables centered on the question of whether a gap between theory and practice exists, and, if so, what exactly is its nature and what can be done about it? Discus- sion was purposely unstructured, yet seemed to fall naturally into three areas: understanding of forecasting, data requirements, and organiza- tional politics. There also appeared to be three, and not just two, groups of people at variance with each other: academics, practitioners and forecast users. These discrepancies arise because of the differences in behavior and perceptions between the members of these groups. The gaps are listed in Exhibit 1 and are elaborated upon below.
3.1. The understanding gap
According to Fildes (1990): The various sur- veys of (market forecasting) practice all have one thing in common: the conclusion that subjective forecasting techniques based on expert, usually executive, opinion are more widely used than any of the quantitative approaches to forecasting recommended in the management science litera- ture. As Ralph Snyder (ISF 91) explained, man- agers often possess ambivalent attitudes towards management science in general and forecasting in particular. When confronted with manage- ment science, as a time and resource intensive process for problem-solving and as an iterative
Understanding gap Managerial
Data sharing gap Industry need for. confidentiality
Political gap Hidden agenda of. management
. . Technical detail of forecasts
Researchers needs for data to build and test useful models
Analysts viewpoint is to forecast honestly
scheme of successive model refinement, the inevitable failures experienced often provide the convenient excuse required for jettisoning such an approach. Unless a solution can be bought off the shelf and provide instantaneous success, the chances of engaging managers in a scientific approach are, whether we like it or not, rather remote.
Wally Albers (ISF 91) explained the under- standing gap in terms of workplace culture. There is, he observed, an accepted division of labor between the academics who develop new knowledge and the practitioners who apply it. The academics live in a different culture, receiv- ing different rewards. Academics receive recog- nition and advancement for research that might have limited applicability in a business environ- ment. Academics traditionally have had to be less concerned with the immediate practical orientation of their work. A forecast may be technically brilliant, but understood by few (Russ Maddox, ISF 91). There has to be a translation of the forecast by the forecaster so that it can be used by others. Forecasters need imagination to perform this task successfully. In todays business world, learning how to deliver research products may be more important than the production of the products (Shawa Zhang, ISF 91). In the words of Martin Morman (ISF 91): If you have to sell a forecast, then someone doesnt understand it. Thats a communication problem.
The way in which forecast performance mea- sures are used is an example of the understand- ing gap. Mean Absolute Deviation, for example, is important to a statistician but the measure that should be used, perhaps, is the extent of the financial impact upon business performance (Richard DAngelo, ISF 91). In another exam- ple, Leitch and Tanner (1991) investigated profit measures of interest rate forecasts. They argued that the main reason that firms pay for profes- sional forecasts, which do not seem to work much better than do naive forecasts, is because the former enhance profits and seem to be linked to turning point information. In contrast, most academic studies of forecast accuracy argue for the importance of using conventional statistical error measures in the face of some limited evi- dence as to their inadequacy. From a forecast user perspective, it may be better to communi-
254 E. Mahrnoud et 01. I Bridging the gup between theory und practiw
cate the forecast results in terms of customer service rates, profits, etc. (Richard DAngelo, ISF 91). Some researchers, such as Price and Sharp (1986) and Mahmoud and Pegels (1990), have illustrated the use of a dollar-based forecast accuracy measure, but more work is needed here.
3.2. The data sharing gap
Wassily Leontief (ISF 91) emphasized the im- portance of disseminating information among re- searchers and making data available to them. Managers often consider data to be sensitive and confidential. Yet, the sharing of data with aca- demics may be a good way for organizations to learn more about forecasting theory and about how to analyze data more successfully using appropriate techniques.
The data sharing gap reflects more than a mere unwillingness to provide data because of reasons of confidentiality. There may also be an attitude on the part of some academics that data have to be in a particular form before they will use them, Views on data quality and the accep- tability of data for model-building differ between academics and practitioners. Academics prefer to have clean data; practitioners must work with what managers give them or with what is available from various internal or external sources. Contaminated data are a fact of life for the practitioner; the art of purifying data to get at targeted elements is often critical to the de- velopment of a valid and reliable forecast (RUSS Maddox, ISF 91). If the purpose of the forecast is to predict the future behavior of the market, available data may need to be adjusted for non- market-related events. In addition, adjustments may need to be made to account for prior (non- recurring) market-related events which do not have a continuing impact. The resultant series provides a sounder basis for forecasting. See Rice and Mahmoud (1984) for a discussion of issues concerning data quality for forecasting.
3.3. The political gap
The political gap exists when some of the
participants in an organizations forecasting pro- cess have an agenda in support of decision- making that is often hidden from other particip- ants, and which may be in conflict with statistical accuracy. This can result in biased forecasts.
Depending on their role in an organization, people may have different perceptions of the rewards of over- or under-forecasting. The fol- lowing examples are typical, although not uni- versal. l Sales people want to underestimate the fore-
cast so that they will be sure of surpassing it and gaining bonuses.
l Production prefers to overestimate the forecast to ensure that there will be no backlogs or delays in the production schedule.
l Top management seeks a forecast that is ori- ented towards satisfying board members as to corporate profitability.
l Marketing likes to overestimate the forecast in order to gain a larger budget for marketing expenditures. Because of the inherent conflict of interest
between different managerial functions, forecasts are negotiated. Forecasting is more and more being viewed as a process where people start with a figure and elicit discussion on why it should be changed (Keith Ord, ISF 91). In busi- ness, the making of adjustments to statistically determined forecasting is becoming more accept- able; as is anything that appears to increase forecasting accuracy (Russ Maddox, ISF 91). Note, however, that adjustments may be under- taken as part of the negotiation process, despite the fact that the accuracy of the forecast may be adversely affected.
If forecasters are of low status in the organiza- tion, this means that their forecasts can readily be altered. Forecasters are perceived as provid- ing input to decision-makers; forecasters do not make operational decisions (Russ Maddox, ISF 91). They are, there...