Studies in Managerial Problem Formulation Systems

Decision Support Systems 9 (1993) 413-423North-Holland

413

Studies in managerial problemformulation systems

James F. Courtney and David B. ParadiceTexas A&M University, College Station, TX, USA

A series of projects to develop decision support systemsfor managerial problem formulation is described. Problem-solving theory from cognitive psychology is integrated withproblem structuring techniques (cognitive mapping and struc-tural modeling) to provide the theoretical foundation for thework. The first project involved the development and testingof graphics software to support problem formulation. Subse-quent studies examined the use of this software in groupdecision-making, and extended the system to include a prob-lem diagnosis module, statistical routines to test relationshipsbefore they were stored in the systems knowledge base, anadvisory module, and a discovery module which searches thedatabase for relationships as yet untested by users. Currentwork includes the addition of a dialectical module and exten-sion of the system to support an organizational perspective onthe management of causal knowledge.

Keywords: Problem formulation, Problem solving, Graphicalinterface, Decision support system, Expert system,Knowledge-based system, Automated discovery.

James F. Courtney is Tenneco Professor of Business Adminis-tration in the Business Analysis and Research Department atTexas A&M University. He received his Ph.D. in BusinessAdministration (Management Science) from the University ofAustin in 1974. His academic experience includes facultypositions at the Georgia Institute of Technology, Texas TechUniversity and the State University of New York at Buffalo.Other experience includes positions as Database Analyst atMRI Systems Corporation and Visiting Research Scientist atthe NASA Johnson Space Center. His papers have appearedin several journals, including Management Science, Communi-cations of the ACM, IEEE Transactions on Systems, Man andCybernetics, MIS Quarterly, Decision Sciences, Decision Sup-port Systems, the Journal of Management Information Systems,Database, Interfaces, the Journal of Applied Systems Analysis,and the Journal of Experiential Learning and Simulation. He isthe co-developer of the Systems Laboratory for InformationManagement (Business Publications, 1981), a software pack-age to support research and education in decision supportsystems, co-author of Database Systems for Management (sec-ond edition, Irwin Publishing Company, 1992) and DecisionSupport Models and Expert Systems (Macmillan Publishing,1992). His present research interests are knowledge-baseddecision support systems, intelligent organizational systemsand management information systems.Correspondence to: J.F. Courtney, Business Analysis and Re-search Department, College of Business Administration, TexasA&M University, College Station, TX 77843-4217, USA, Tel:(409) 845-9541, Bitnet: Courtney@Tamcba.

1. Introduction

This paper summarizes a series of studies in-volving the development and testing of decisionsupport systems (DSS) to assist in managerialproblem formulation. Problem formulation refersto the process of deciding what variables areincluded in a problem domain and how thosevariables fit together and interact [2]. Problemformulation has been recognized as a critical as-pect of the modeling process [5, 17,22,25]. If thedecision-maker formulates the problem incor-rectly, then a "Type III error" [35] will occur;that is, the wrong problem will be solved. Sincean optimal answer to the wrong problem maylikely be worse than a good answer to the realproblem, it makes sense to develop systems tohelp get the problem right. Although problemformulation studies have been reported in theliterature of psychological organizational behav-ior, when this stream of research was initiatedDSS researchers had not yet begun to developsystems to support the problem formulation pro-cess (as documented in [9]).

With this in mind, a series of projects wasinitiated in the early 1980's to determine howsystems might be developed to support the prob-

David B. Paradice is Assistant Professor of MIS in the De-partment of Business Analysis and Research at Texas A&MUniversity. He holds a B.S. in Information and ComputerScience and an M.S. in Industrial Management from GeorgiaTech and a Ph.D. in Business Administration (MIS) fromTexas Tech. Dr. Paradice's primary research interests areinvestigating means for applying artificial intelligence tech-niques in managerial domains and studying the impact oftechnology on society. His research includes investigatingmethods of dynamic model construction in managerial advi-sory systems and utilizing qualitative models of organizationalsystems, as well as investigating the influence of computer-based information systems on ethical decision-making pro-cesses. Aspects of his research have appeared in the Journalof MIS, Annals of Operations Research, IEEE Transactions onSystems, Man and Cybernetics, Decision Sciences, Informationand Management, Journal of Business Ethics, Information Re-sources Management Journal, and Journal of Education inBusiness.

0167-9236/93/S06.00 © 1993 - Elsevier Science Publishers B.V. All rights reserved

414 J.F. Courtney, D.B. Paradice / Managerial problem formulation systems

lem formulation process. The purpose of thispaper is to summarize the results of those pro-jects and provide directions for future research inproblem formulation systems. The research de-scribed here is unique even considering the morerecent efforts by DSS researchers. This researchstream has emphasized computer-based modeldevelopment and manipulation, while otherDSS-related research has emphasized computer-based automation of problem-solving heuristics.For example, Niwa [27] developed a system basedon a knowledge base of experimental knowledge.This system contained rules designed to exploit ahuman user's ability to associate knowledge whichwas not related to other knowledge at the time ofacquisition. The primary tool provided by thecomputer was an "association guidance" mecha-nism (i.e., heuristic) for relating pieces of knowl-edge. Cats-Baril and Huber [9] studied the effectof delivering heuristics for problem structuring todecision makers via computer. They found thatcomputer-based delivery of these heuristics hadno affect on any dependent variable they studied.Recently, Paradice [28] studied the use of ques-tion/ answer process theory as a basis for guidingthe development of cognitive maps for particu-larly wicked problem situations. His research sug-gests that systems are needed to assist problemsolvers in identifying desired goal states, but sucha system has not yet been developed and testedthat is based on question/ answer process theory.

The next section of the paper describes thetheoretical foundations for this stream of re-search. After that, the experimental context ispresented and the series of projects is described.As the research proceeded, it became clear thatmanagerial problem formulation required a muchbroader view than had been taken at first. Theimplications of this finding are discussed in thesection on future research.

2. Problem formulation research in cognitive psy-chology and systems engineering

The first research question to be faced was todetermine what unique features would be re-quired of a DSS to assist in problem formulation.Second, if such a system could be built, howcould one be sure that it does indeed help a userformulate problems correctly? Answers to the

first question were found in the literature oncognitive psychology and systems engineering.

Research in cognitive psychology relevant toproblem formulation is based on various theoriesof human problem solving (see, for example, [16]or [26]). For the most part, these findings arebased on rather simple problems, such as syllo-gisms and memorization of short word lists. Theprocess can be described as consisting of twogeneral phases: problem formulation (also calledpredecision [14,15]) and problem solving [38].

Problem formulation contains three phases[38]. Problem identification occurs when the prob-lem solver perceives the need for a decision to bemade. This phase is typically marked by concernor worry and the belief that a problem exists.Problem definition concerns the cognitive concep-tualization of the problem situation. During defi-nition, the problem solver determines the rele-vant properties of the problem situation. Problemstructuring examines the relevant components ofthe problem situation in order to determine astrategy for addressing the problem. This phase ischaracterized by instrumental reasoning, fromwhich a strategy for addressing the problememerges.

Similarly, problem solving contains two phases[38]. Diagnosis occurs when the problem solveridentifies a problem's causes. Consequently,causal reasoning characterizes this part of theprocess. Alternative generation occurs next andconcerns the design of alternative courses of ac-tion.

Although convenient for discussion, this modelof problem solving suggests a linear progressionthat may not always appear in problem solvingprocesses. Problems, depending on the problemstructure, the experience of the problem solver,and the impact of information gained at variousphases, may require backtracking from laterphases to earlier ones as the problem structuredevelops. Also, solutions to some problems giverise to other problems, implying a cyclical struc-ture. However, the model serves the purpose ofproviding a means for examining an inherentlyunstructured process.

Experiments in problem formulation and prob-lem solving have revealed several important char-acteristics of the process. The set of alternativeacts that might solve a problem has been claimedto represent the inherent structure of a problem.

J.F. Courtney, D.B. Paradice / Managerial problem formulation systems 415

The ability to generate these alternative acts hasbeen found to be positively influenced by diver-gent thinking [14]. Subjects who exhibited diver-gent thinking abilities generated alternative actsets that were closer to the optimal set for a givenproblem. Additionally, the use of divergentheuristics increases the number of alternativesgenerated [1]. However, the use of either conver-gent or divergent heuristics, as opposed to alimited heuristic strategy, resulted in high qualityalternative generation. This work seems to indi-cate that structuring heuristics in general aresupportive of the problem solving process.

As stated in [15, page 24], "The elements ofdecision problem structures - acts, states of theworld, and outcomes - are inherent in decisiontasks, and hence should be considered by a moti-vated decision maker who is faced with an impor-tant problem." However, decision makers are notnecessarily adept at generating these structureswhen they confront problems. Thus, problemsolvers act with limited knowledge of problemstructure, and in the worst case, produce a solu-tion which is ineffective. If the most importantpart of predecision behavior from the point ofview of decision analysis is developing problemstructure [1,15,24,25], then aids to structuringproblems should provide great benefits. Organi-zation theorists believe that these problem struc-tures are not only used to solve current problems,but also dictate how future information is filteredand interpreted [39]. Thus, determining the cor-rect problem structure in a given situation haslong range implications.

Studies have also shown that diagrams andimages were useful in the problem understandingphase of the process [16,31]. The right hemi-sphere of the brain, which is known to be associ-ated with the manipulation of mental images, wasfound to be more active during the problem for-mulation phase of problem solving. Thus, it hasbeen theorized that images and diagrams areeffective ways of representing the understandingof a problem's structure [16]. Paivio [31] evenproposed a "dual coding theory" founded on thebelief that images are encoded differently in thebrain than verbal and procedural knowledge.

In rather loosely related work, systems engi-neers developed "structural modeling" [5,17,22]and "cognitive mapping" [5], to represent thestructure of a problem. These techniques, which

are based on the mathematical theory of graphs,were developed independently of the research incognitive science described above, and have littleor no theoretical basis in cognition.

Cognitive maps are simply box and arrow dia-grams in which boxes are used to represent vari-ables and arrows represent relationships. Cogni-tive maps can also be represented in an "ad-jacency matrix," formed by creating a row andcolumn for each variable in the problem, thenplacing a 1 in each cell where there is an arrowfrom the variable in row i to the variable incolumn j. Zeroes are entered in other cells of thematrix. Signs may also be entered.

If weights on relationships are entered, thenthis is usually referred to as a structural model,rather than a cognitive map. Various matrix mod-eling techniques are available to analyze struc-tural models.

Since work in cognition has indicated imageswere useful in problem structuring and structuralmodeling uses graphs to represent problem struc-ture, it seems to follow that structural modelingwould be a useful element in a decision supportsystem to support problem formulation. The firstresearch task was to integrate the work in thesetwo fields and relate it to problem formulation ina DSS context. Fortunately, work in the twofields was quite compatible, even though doneindependently. Despite the fact that structuralmodeling uses both graphic and matrix forms, nosoftware could be found to support the graphicalaspect of the cognitive mapping process. Thus,the next step undertaken involved the develop-ment of graphic software to support the develop-ment of structural models of managerial prob-lems, the domain of decision support systems.

3. Pracht's study

Pracht [32,33,34] developed this software,known as GISMO, a Graphical, Interactive,Structural Modeling Option. This system permitsthe interactive development of structural modelsin either graphic or tabular form. It is integratedwith previously existing software for performingvarious kinds of mathematical analysis, includingpulse analysis, matrix powering, etc. The questionremained whether the system actually did help informulating problems correctly.


To address this question, a rather complexmanagerial decision-making environment wasneeded. However, in order to know whether userswere formulating problems correctly, it was nec-essary to know the actual problem structure. Sincethe actual structure of real managerial problemsis not known with certainty, validation requiredanother approach.

Validation against a simulated managerial en-vironment was chosen for several reasons. Thereal relationship between variables in the simu-lated environment could be known by examiningthe source code of the simulation model. Thus,users could develop models of the simulated envi-ronment, and these could be compared to thereal model (that is, the source code) to determinetheir accuracy. Furthermore, the initial tests in alaboratory environment could be conducted wherebetter internal controls would ensure that thefindings were derived from system use, and werenot confounded by other factors.

Since it was not possible to obtain practicingmanagers neither in sufficient numbers nor for anappropriate length of time (several weeks) foradequate statistical power, senior business majorswere used as subjects in this study. The use ofstudent subjects continues to be debated in theliterature. Some studies indicate the students re-spond similarly to managers, while others showthem to behave differently [34]. However, thesubject pool is a possible limitation of the experi-ment.

The Business Management Laboratory was thesimulation selected. It is a rather complex simula-tion, requiring up to 50 decisions per round. Italso provides a database and query languageknown as SLIM, which had been used by De-Sanctis [12] and Kasper [20,21] in previous DSSstudies. The SLIM database contains informationon over 120 highly interrelated variables. Theinteraction between these variables is sufficientlycomplex that "finding" the underlying structureof the simulation is highly unlikely. In fact, theunderlying structure is exceedingly complex. Thus,although the structure is programmed, it wouldnot be generally described by subjects as "wellstructured" in the sense that a linear program-ming model or some other type of model (or setof such models) could be developed to reflect thestructure. Consequently, subjects must createmodels of various aspects of the simulation in an

attempt to cope with the complexity of the deci-sion problem at hand. An informal survey con-ducted in class showed that many students whohad made decisions in this simulated environ-ment for 8 weeks still felt that many of thedecisions were ill-structured.

Based on the research in cognition mentionedpreviously, it was hypothesized that subjects withgood spatial ability would be most capable ofusing GISMO. Since Witkin's Group EmbeddedFigures Test had been used in several previousDSS studies and is related to spatial ability, it wasused as the measurement instrument. A 2 X 2factorial design was developed, based on a con-trol group that did not use GISMO versus anexperimental group that did, and field indepen-dence (high spatial ability) versus field depen-dence. Instruments were developed to test theability of subjects to formulate the structure ofthe BML simulation. These instruments includedstructural models (drawn by hand by the controlgroup), the ability to list factors important in twoselected decision problems, and signed adjacencymatrices filled in by the subjects.

Data analysis determined that field indepen-dents using GISMO more closely formulated thecorrect problem structure than any other group.Field independents without the software per-formed the worst of all groups. They had thecognitive ability to formulate the problem, butfailed to use that ability in the absence of a toollike GISMO. Field dependents performed aboutthe same with or without the tool. Their cognitiveskills are such that they are ineffective in usingthe tool to formulate the problem structure.

This result may indicate that the methodologyunderlying automated tools is a critical factor ineffective use of the tool. Where the methodologywas compatible with the user's cognitive predis-position, the tool provided great support. Whenthere was little relationship between the tool'smethodological basis and the user's cognitive pre-disposition, the tool's impact was minimally effec-tive.

4. Loy's small group experiment

Given the promising results of the Pracht study,Loy [23] decided to test GISMO in a small groupdecision-making context. The main objective of


the experiment was to test whether small groupsusing GISMO would better understand the prob-lem domain and perform better in terms of netincome. However, since much of the research onsmall groups deals with techniques for structuringgroup interaction, the experiment was also de-signed to analyze the effect of GISMO with astructured decision process. The Nominal GroupTechnique (NGT) was chosen as the structuredprocess to test because it is the most widelystudied technique in the literature. Since mostsmall groups making management decisions usean unstructured format, the NGT format wascompared to "interacting groups" which weregiven no predefined rules regarding their interac-tion and were allowed to interact as they saw fit.This resulted in a 2 X 2 design with the factorsGISMO use versus non-use, and NGT versusinteracting groups. Again, BML was used as theexperimental environment, and the instrumentsdeveloped by Pracht were used to measure theaccuracy of the problem formulation, and netincome was used to measure decision quality.

One would hope that better problem domainunderstanding is associated with better decisions,thus, a correlation analysis was conducted to ana-lyze the relationship between these variables. Theresults bear out this belief in that the correlationcoefficient is between the two is 0.7 (p > r =0.002).

Multivariate and univariate analyses of vari-ance were conducted to test the hypotheses thatthe overall performance of groups as measuredby decision quality and problem understandingwas influenced by GISMO use, discussion format(NGT versus interacting group), and the interac-tion of these two independent variables. GISMOuse was a significant variable in these analyses,the statistical significance being supported at analpha level of 10 percent. Thus there is evidenceto support the assertion that GISMO use helpssmall groups make better decisions.

There were no statistically significant differ-ences between NGT and interacting groups foreither dependent variable. Thus in this study,NGT was not found to be superior to interactinggroups in terms of either decision quality or prob-lem domain understanding.

Since Pracht's study showed that spatial abilityhas an effect on understanding, GEFT scoreswere used to develop a measure of group spatial

ability and were included in the analysis. TheGEFT variable was significant at an alpha of 5percent. Thus there is evidence to support theassertion that GISMO use helps small groups tobetter understand a problem domain, especiallywhen group members have high spatial ability asmeasured by the GEFT. The finding regardingspatial ability is consistent with Pracht's resultsregarding individuals.

Informal observations by the researchers anddebriefing of subjects seemed to indicate that thediscussions of the GISMO groups were muchmore focused and productive, communication wasmore effective, and the decision-making processwent much more smoothly. Non-GISMO groups,even when using NGT, seemed to waste a greatdeal of time in circular discussions, and ended uprushing to make the decision at the end of themeeting (which was limited to 45 minutes for allgroups). This may help to explain why GISMOgroups outperformed their non-GISMO counter-parts on both dependent variables, but these areopinions for which there is no statistical evidence.

NGT groups, especially those using GISMO,seemed to feel that the process was actually ahindrance and that they could have done betterwithout it. The subjects just didn't seem to likeNGT. Clearly it did not help in the decisionprocess, at least in this environment. Again theseare observations, not statistical facts, but theyseem to help in understanding the result thatNGT groups failed to outperform IG groups.

The results from Pracht's and Ley's studiesmay indicate that GISMO use provided severalefficiencies that benefitted decision makers. Thevisual nature of the models supplemented thecognitive processes of individual decision makers,allowing them to bring cognitive capabilities tobear on the problem that otherwise may not havebeen utilized. Also, the graphical models pro-vided a means to "store" a problem formulation,thus requiring less "redundant" mental effort toreformulate the strategic situation each time indi-viduals or groups had to process results from thesimulation. Taken together, these aspects ofGISMO use result in a methodology that requiresa lower "thinking cost" (i.e., less "units of think-ing" to arrive at a decision) than would haveotherwise been required [37]. On the other hand,the NGT methodology, with its regulated meansof interaction, may require a higher thinking cost


than perceived necessary, and thus may be per-ceived as a cumbersome decision-making ap-proach.

5. Ata's diagnostic system

Whereas Loy made no refinements to theGISMO software, but rather tested it in a groupcontext, Ata [3,4,11] extended the system itself toinclude problem diagnosis. As noted earlier,problem diagnosis refers to searching for or hy-pothesizing about the causes of a problem. In apioneering study based on protocol analysis,Bouwman [8] found that financial analysts usedrelationships that he modeled as "causation trees"to diagnose financial statements. Causation treesare hierarchical diagrams similar to structuralmodels. The root node of a causation tree is the"problem" for which potential causes are sought.For example, in financial analysis the problemmight be an unexpectedly high ratio of debt toequity. Branches of the tree represent paths fromthe problem to variables representing base causes.For example, a high debt/equity ratio the levelbelow the debt/equity node might simply beseparate nodes for debt and equity. The debtnode might have nodes under it for long-termand short-term debt. Short-term debt might becomprised of loans and accounts payable. Thepath from accounts payable to short-term debt todebt to debt/equity ratio is a possible explana-tory path for a high debt/equity ratio (if ac-counts payable are unusually high).

Ata integrated Pracht's and Bouwman's workby using structural modeling techniques to repre-sent causation trees. User models of the BMLproblem domain, including estimates of coeffi-cients between pairs of variables, were inputs toAta's system. Then, given some unexpected varia-tion of a goal variable such as profit or salesrevenue, the system used the variables and coeffi-cients entered earlier by users to construct linearmodels of the problem area. Current data wasdynamically extracted from the database and fedto models in an attempt to diagnose or explainthe variation. If the model results were not signif-icantly different from actual results, then theuser's model was accurate and could be used asthe basis for a diagnosis. If not, then the modelitself was deficient and in need of improvement.

To test the system, models developed byPracht's subjects were input and diagnoses ofproblem areas were attempted. This approachwas very sensitive to the accuracy of the model,and the users' models were not very good. Eventhough GISMO helped user's formulate moreaccurate models, they were still not accurateenough for effective, semi-automated support ofproblem diagnosis. This deficiency led the way tothe next project, which was designed to produce asystem that could formulate more accurate repre-sentations of the BML problem domain.

6. Paradice's knowledge-based system

The Ata study found that many user-devel-oped models were flawed. These errors werethought to be due to different cognitive biasesidentified in various studies (see [36] for a review).Paradice [29,30] developed SmartSLIM, a systemthat controlled these biases through the use oflinear and higher order statistical models whichused data from the SLIM database to test users'assertions about relationships in the BML do-main at the time they were input. A dialogue wasconducted with the user to determine whetherthe asserted relationship would actually be re-tained, and if so, to gather additional informationabout the relationship.

Early testing of the system indicated simplecategorization of relationships into causal andnoncausal classes would be inadequate for prob-lem diagnosis. (Validation of the system is dis-cussed below.) Put simply, the system generatedmodels that tended to include all relationships inthe knowledge base when asked to diagnose par-ticular problems, due to the extent in which allvariables in a complex managerial domain inter-act to some degree. These models were not veryuseful in structuring a particular problem context.Consequently, a revised taxonomy of managerialdomain relationships, based primarily upon ananalysis of relationships between data items infinancial proforma statements, was developed andused [29]. In addition to representing cause andeffect relationships between data items, the re-vised taxonomy explicitly considered the existenceof data items that (1) act as upper and lowerbounds on other data items, (2) redefine otherdata items, (3) combine with other data items to


compose a data item, and (4) are correlated withother data items. The revised taxonomy sup-ported model formulations that were much moreparsimonious during the validation stage de-scribed below.

Since it was possible that an asserted relation-ship could be valid, but not statistically sup-ported, the user could insert relationships evenwhen statistical validity was not upheld. User-re-jected relationships were stored in a "rejectionbase" for testing later by the system as additionaldata was added to the database.

This system also incorporated features of ex-pert systems to represent model knowledge andprovide a rather unusual advisory facility. Given aquestion about how to increase or decrease atarget variable, the advisory module was capableof extracting relationships from the knowledgebase (possibly input by different users), construct-ing a model of the target variable, and offeringadvice on what other variables could be manipu-lated to achieve the desired change in the target.The system used a breadth-first search to traversea hierarchy of relationships in offering this ad-vice. Thus, variables that most directly impactedthe target variable were brought to the attentionof the user first.

SmartSLIM was validated in a rather uniqueway. First, it was "trained" in the BML domainby inputting relationships in models developed byvarious users. The advisory module, along withother system features, was then used to answerthe questions Pracht posed to his human subjects.One set of questions dealt with the level of confi-dence the respondent had in various decisions.The system's confidence was based on variousstatistical measures including correlations andbeta coefficients. It was confident in most mar-keting decisions, but not confident in the areas ofplant and production, and unable to addressquestions in finance and administration. Theseresults are due to deficiencies in the models usedto train the system, which had few model compo-nents in production and financial areas but hadwell developed marketing concepts. In otherwords, the subjects that developed the modelscreated models that had a strong marketing ori-entation. Still, the confidence levels illustrate howthe system itself could use information in its ownknowledge base to determine where it (and itsusers) needed additional knowledge.

Another test required the system to determinewhether a relationship between two variables waspositive, negative or zero. Eighteen (18) pairs ofvariables were included in the test. Pracht's sub-jects had identified an average of 12.9 of theserelationships correctly. The system had enoughinformation to answer only 12 of these cases, butgot all 12 of those correct. Perhaps more impor-tantly, it was correct in one case that was missedby 29 of 32 subjects and another missed by 22subjects. Thus SmartSLIM could have affordedsubjects the opportunity to improve their models,at least in the case of two relationships. In thatsense the system improved on the problem diag-nosis concept developed by Ata. It also demon-strated that computer-based knowledge supportof the managerial domain will require complexknowledge modeling capabilities.

7. Billman's discovery modules

The architecture of SmartSLIM was based onBlum's RX project [7], which included a "dis-covery module" intended to search the databasefor relationships unknown to (or at least untested)by its users. The objective of this module was toimprove representations of the problem domainwithout user direction. The rejection base wasdesigned to support the discovery process, butonly a crude implementation existed in Smart-SLIM.

Billman [6] undertook the development of thediscovery module. Whereas RX used statisticaltests to search for relationships, Billman basedher system on three different models for hypothe-sizing causal relationships. For brevity, only themodule based on Einhorn and Hogarth's [13]model is described here. For a discussion of theother two approaches see Billman [6].

Einhorn and Hogarth develop a model of theway people evaluate the potential strength of acausal relationship between two variables. The"gross strength" measure between a potentialcause x and effect y is defined by the equation:

s(x, y) = T*B*L*[w*C + (l-w)S],

where C is covariation or the frequency withwhich x and y occur and do not occur together,T is 1 if x precedes y in time, and is 0 otherwise,B is 1 if the change in x is large enough to be


perceived and is 0 otherwise, L is a measure ofthe length of the causal chain between x and y,computed by multiplying the covariation of eachlink in the chain, w is a weight reflecting theattitude of the individual, and S is the "similarity"between the two variables. To illustrate similarityin a management context, two marketing vari-ables might be perceived as more similar (moreclosely related) than a marketing variable and aproduction variable.

Note that if T is zero (the effect precedes thecause) the entire equation reduces to zero. Thesame is true for B - if the potential cause x is notrecognized as such by the observer, then theequation goes to zero. Also, since covariation isbetween 0 and 1, the value of s(x, y) decreasesas the length of the causal chain from x to yincreases.

One additional factor affects the perceivedstrength of the causal relationship, the number ofother variables which may have caused the effectto occur. To account for this, the gross strengthmeasure is reduced by a weighted sum of thegross strengths of all other possible causes. Theweights reflect the perceived likelihood that thecorresponding variable is the actual cause.

The Einhorn and Hogarth model is directedprimarily at non-quantitative variables so that xand y either occur or do not. On the other hand,cognitive mapping and structural modeling dealwith concept variables, which, in principle at least,are measurable. In developing an approach toautomated discovery based on the Einhorn andHogarth model, Billman extended the model tothe case in which x and y can increase, decrease,or remain constant. Thus, her approach does notrequire variables that can be measured precisely,but can handle "soft data," such as morale, stress,satisfaction, and so forth.

To describe how the approach was imple-mented, let x' be the change in x from oneperiod to the next. A threshold value represent-ing B was selected (5% in the baseline case) andvariables in a SLIM database were transformedusing the mapping:

( 1 ifx'>Bx"= -1 if x' <B

\ 0 otherwise,

where the terms in braces correspond to an in-crease, decrease, and no change in x, respec-

tively. Flags were set to indicate the time order ofvariables in the database, so that T was known.In the initial implementation, causal chain lengthwas ignored, and only gross strength measureswere calculated.

The system was tested in two ways: (1) bycomparing its ability to find relationships whichactually exist in the BML code, and (2) by com-paring its results to a composite model built byintegrating four of the best cognitive maps devel-oped by subjects who had participated in 12rounds of the BML simulation. Cost of goodssold was used as the test variable. There are 39variables in BML that have an impact on cost ofgoods sold.

A problem arises in interpreting the grossstrength measure. At what level of gross strengthare we to say that a relationship has been discov-ered? Clearly if s(x, y) approaches 1, we wouldsay a relationship has been found, and if it ap-proaches 0, the system believes no relationshipexists. In the test case, if a gross strength ofgreater than 0.15 is taken as an indication that arelationship exists, then 30 of the 39 actual rela-tionships would be "found" by the system and 6spurious relationships would be hypothesized.The composite model based on experienced hu-man subjects contained only 23 of the 39 relation-ships, but no spurious variables.

The system finds spurious relationships be-cause it looks at the data blindly without regardto any underlying theoretical or logical rationaleas a basis for a relationship. User interventioncould be used to overcome this problem, suggest-ing a symbiotic human-machine relationship (asin [27]) in which the machine could present rela-tionships and the human could validate them.This approach would exploit the talents of eachparticipant. The computer would tirelessly exam-ine combinations of variables, testing various sta-tistical relationships or possibly drawing upondata item descriptions to direct its search throughthe database. The size of most complex corporatedatabases requires automated intervention likethis. The human would then examine the rela-tionships detected by the computer, drawing uponhuman judgment and experience to validate orreject relationships proposed by the computer.Thus, the computer's processing power is used toidentify candidate relationships, and the human'sevaluative power is used to endorse them.


8. Future research

Pracht's study indicated the methodologicalbasis of an automated tool may impact the effec-tiveness of the tool's use. Studies need to beconducted that examine this supposition. CASEtechnology provides a particularly attractive envi-ronment for this study, as there are currentlyseveral methodologies that have been imple-mented and CASE use is becoming increasinglywidespread.

Both Ata's and Paradice's work highlight thepotential danger in using knowledge bases thatare either flawed or incomplete. Knowledge basesare generally considered to be fragile and brittle:when they fail, they generally do not fail grace-fully. Research into improving ways for a systemto determine the level of it's own expertise isneeded Some means for developing and usingsystem meta-knowledge, knowledge about thesystem's knowledge, must be pursued. Such re-search could greatly reduce the fragile and brittlenature of knowledge base use.

Paradice's "rejection base" represents one ofthe first attempts to make use of "discarded"knowledge. His assumption in retaining thisknowledge 'was that a relationship hypothesizedby a user has an inherent worth, even if thecurrent data does not support it. 1 He used therejection base to drive a crude version of thediscovery module prior to Billman's work. Aninteresting avenue of research would be findingadditional means for exploiting this "knowledge,"including heuristics for determining when re-jected knowledge should be reevaluated.

Work is continuing on ways to improve theaccuracy of cognitive maps produced by subjectsoperating in the BML domain. In an intriguingstudy, Hodges [18] is developing software basedon dialectical theory. This software will take cog-nitive maps developed by users, form a diametri-cally opposed counter-map, and conduct a debateintended to lead the user to a fuller understand-ing of the problem domain and a more accuraterepresentation thereof.

1 Many organizations retain files of project proposals or ideasthat have been rejected at earlier points in the organization'shistory. These files become reservoirs of ideas that can bereexamined at later dates.

Another project under consideration would in-volve an attempt to develop software that wouldconstruct cognitive maps directly from documentsdescribing problems the organization has ana-lyzed in the past. When faced with importantproblems, organizations usually form a committeeor task force to study the situation and make arecommendation as to how the problem shouldbe solved. Reports produced by such teams ofteninclude a diagnosis of the problem's causes. Asystem that could scan these reports looking forphrases such as "A caused B" or "A was causedby B" might be able to construct at least a roughmap of the environment. Rules for manual cod-ing of such documents [5] could provide the basisfor an expert system's approach to this problem.While the resulting map might not be perfect,human coders would at least have an initial mapto begin with and could then take over and makethe necessary corrections and additions. Whilethe development of an automated mapper isclearly a nontrivial problem, the existence ofcomputerized grammar and style checkers, the-sauri and other natural language processors sug-gest that such a system might be feasible.

One of the observations made during thecourse of these projects was that managerialproblem formulation is not an isolated processthat must be undertaken each time a problem ordecision situation is faced. Rather, organizationaldomains, while dynamic, consist of a variable setand corresponding relationships that should becaptured, made explicit and shared with appro-priate people in the firm. This is made obvious bythe use of a simulated environment in the pro-jects described above. An accurate cognitive mapof the BML environment, if passed from onesubject group to the next, could be extremelyhelpful in the successors' decisionmaking activi-ties.

Likewise, in the "real world," mental maps areoften passed from one cadre of managers toanother. Decision-making processes could be im-proved by making those maps explicit, and storingthem in a form where they can be manipulated,improved, used and shared. Exploring this hy-pothesis is the ultimate objective of our researchnow. It will hopefully lead to the development ofsystems for the support of "intelligent organiza-tions," or those that use artificial intelligencetechniques at an organizational level rather than


simply on narrow domain-specific problems re-lated to only a small aspect of the organizationalmanagement process.

9. Summary

Problem formulation is an important part ofthe managerial decision-making process, but onethat has only recently received much attention inthe DSS literature. The series of projects de-scribed in this paper began with the objective ofdeveloping systems to support the problem for-mulation process. Pracht integrated concepts fromcognitive psychology and structural modeling anddeveloped graphics software to assist in problemstructuring. An experiment testing the software'sability to assist in problem formulation led to theconclusion that the software was helpful to sub-jects with good spatial ability.

Since many management decisions are made ina group situation, Loy tested the software in asmall group context. He found limited support forthe contention that the software would be usefulin helping groups arrive at more accurate prob-lem formulations. Use of the Nominal GroupTechnique in conjunction with GISMO did nothave any additional effect, and according to sub-ject comments may have even hindered the pro-cess.

Ata extended the system to managerial prob-lem diagnosis, and in the process discovered thathis system was very sensitive to errors in modelformulations. Even though GISMO was helpfulin problem formulation, models developed byrather experienced subjects were still fraught withbias and were not accurate enough for an effec-tive diagnosis.

Paradice incorporated statistical analysis anddialogues to mitigate the errors due to user bias.Validated relationships were stored in a knowl-edge base and an advisory system was developedto counsel users on how a target variable could bemanipulated. This system was capable of answer-ing the questions posed to Pracht's subjects at arather high level of performance.

Billman extended the discovery module initi-ated by Paradice. Her work adapted that of Ein-horn and Hogarth to a cognitive mapping envi-ronment. Her discovery module was able to lo-cate relationships unknown to experienced sub-

jects in the problem domain, but also discoveredspurious relationships.

In summary, several conclusions can be drawnfrom these studies. First, tools can be developedto support problem formulation in a managerialdomain, although one tool may not be effective insupporting all managers. Second, tools can alsobe developed to support managerial domainproblem formulation by groups. However, theinfluences of group dynamics on the problemformulation process may render tools effective insome group settings ineffective in other groupsettings. Third, problem formulation systems canbe used as a basis for supporting later stages ofthe problem solving process, such as problemdiagnosis. When they are used in this manner,however, a critical concern must be the validityand scope of the knowledge base supporting thesystem. Fourth, creative aspects of problem solv-ing, such as discovery of relationships betweenvariables, can also be supported by computer-based systems. As the use of these system extendsinto the most abstract phases of the problemsolving process, the interaction between the userand the system becomes more critical. Fifth, andfinally, these studies have identified many inter-esting avenues for further research. These studieswill require increasingly sophisticated systems tobe developed by DSS researchers.

References

[1] R.A. Abualsarah, B. Carlin and R.R. McDaniel, Jr.,Problem Structuring Heuristics in Strategic DecisionMaking, Organizational Behavior and Human DecisionProcesses, 45 (1990) 159-174.

[2] R.L. Ackoff, Toward a System of Systems Concepts,Management Science 17, No. 4 (Jan. 1971) 661-671.

[3] N.H. Ata Mohammed, A Knowledge-Based DecisionSupport System for Managerial Problem Recognition andDiagnosis, Unpublished Ph.D. dissertation, (Texas TechUniversity, Information Systems & Quantitative SciencesDepartment, Lubbock, TX, 1985).

[4] N.H. Ata Mohammed, J.F. Courtney and D.B. Paradice,A Prototype DSS for Structuring and Diagnosing Man-agerial Problems, IEEE Transactions on Systems, Man,and Cybernetics 18, No. 6 (Nov./Dec. 1988) 899-907.

[5] R.M. Axelrod, The Structure of Decision (Princeton Uni-versity Press, Princeton, NJ, 1976).

[6] B. Billman, Automated Discovery of Causal Relation-ships in Managerial Problem Domains, UnpublishedPh.D. Dissertation (Texas A&M University, Departmentof Business Analysis & Research, College Station, TX,1988).


[7] R.L. Blum, Discovery, Confirmation and Incorporationof Causal Relationships from a Large, Time-Oriented,Clinical Data Base: The RX Project, Computer andBiomedical Research 15 (1982) 164-187.

[8] MJ. Bouwman, Human Diagnostic Reasoning by Com-puter: An Illustration from Financial Analysis, Manage-ment Science 29 (1983) 653-672.

[9] W.L. Cats-Baril and G.P. Huber, Decision Support Sys-tems for Ill-Structured Problems: An Empirical Study,Decision Sciences 18, No. 3 (1987) 350-372.

[10] J.F. Courtney, G.R. DeSanctis and G.M. Kasper, Conti-nuity in MIS/DSS Research: the Case for a CommonGaming Simulator, Decision Sciences 14, No. 3 (July1983) 419-439.

[11] J.F. Courtney, D.B. Paradice and N.H. Ata Mohammed,A Knowledge-Based DSS for Managerial Problem Diag-nosis, Decision Sciences 18, No. 3 (Summer 1987) 373-399.

[12] G.R. DeSanctis, An Examination of an Expectancy The-ory Model of DSS Use, Unpublished Ph.D. Dissertation(Texas Tech University, Management Department, Lub-bock, TX, Jan. 1982).

[13] H.J. Einhorn and R.M. Hogarth, Prediction, Diagnosisand Causal Thinking in Forecasting, Journal of Forecast-ing 1, No. 1 (1982) 23-36.

[14] P.D. Englemann and C.F. Gettys, Divergent Thinking inAct Generation, Acta Psychologica 60 (1985) 39-56.

[15] C.F. Gettys, R.M. Pliske, C. Manning and J.T. Casey, AnEvaluation of Human Act Generation Performance, Or-ganizational Behavior and Human Decision Processes 39(1987) 23-51.

[16] J.G. Greeno, The Structure of Memory and the Processof Solving Problems, in: R.L. Solso, Ed., ContemporaryIssues in Cognitive Psychology: The Loyola Symposium(Wiley, New York, 1973).

[17] F. Harary, R.Z. Norman and D. Cartwright, StructuralModels: An Introduction to the Theory of DirectedGraphs (Wiley, New York, 1965).

[18] W.S. Hodges, DIALECTRON: A Dialectic Engine forStrategic Planning, Ph.D. Dissertation (Texas A&M Uni-versity, Department of Business Analysis & Research,College Station, TX, forthcoming).

[19] R.L. Jensen, The Business Management Laboratory Par-ticipant's Manual, (Irwin, Boston, MA, 1977).

[20] G.M. Kasper, The Effect of User-Developed DSS Appli-cations on Forecasting Decision-Making Performance,the Journal of Management Information Systems 2, No. 2(1985) 26-39.

[21] G.M. Kasper and R.P. Cerveny, A Laboratory Study ofUser Characteristics and Decision-making Performancein End-User Computing, Information and Management9, No. 2 (1985) 87-96.

[22] G.G. Lendaris, Structural Modeling - A Tutorial Guide,IEEE Transactions on Systems, Man and Cybernetics(1980) 807-840.

[23] S.L. Loy, An Experimental Investigation of a GraphicalProblem-Structuring Aid and the Nominal Group Tech-

nique for Group Decision Support Systems, UnpublishedPh.D. Dissertation (Texas Tech University, InformationSystems & Quantitative Sciences Department, Lubbock,TX, 1986).

[24] H. Mintzberg, D. Raisinghani and A. Theoret, TheStructure of 'Unstructured' Decisions, AdministrativeScience Quarterly 21, (1976) 246-275.

[25] I.I. Mitroff and T.R. Featheringham, On Systemic Prob-lem Solving and the Error of the Third Kind, BehavioralScience 19, No. 6 (1974) 383-393.

[26] A. Newell and H.A. Simon, Human Problem Solving(Prentice-Hall, Englewood Cliffs, NJ, 1972).

[27] K. Niwa, A Knowledge-Based Human-Computer Coop-erative System for Ill-Structured Management Domains,IEEE Transactions on Systems, Man and Cybernetics 16,No. 3 (May/June 1986) 335-342.

[28] D.B. Paradice. A Question Theoretic Analysis of Prob-lem Formulation: Implications for Computer-Based Sup-port, in: Questions and Information Systems (ErlbaumPublishers, Hillsdale, NJ, 1990.

[29] D.B. Paradice and J.F. Courtney, Controlling Bias inUser Assertions in Expert Decision Support Systems forProblem Formulation, Journal of Management Informa-tion Systems 3, No. 1 (Summer 1986) 52-64.

[30] D.B. Paradice and J.F. Courtney, Causal and Non-causalRelationships and Dynamic Model Construction in aManagerial Advisory System, Journal of ManagementInformation Systems 3. No. 4 (Summer 1987) 39-53.

[31] A. Paivio, Mental Representations: A Dual Coding Ap-proach (Oxford University Press, New York, 1986).

[32] W.E. Pracht. An Experimental Investigation of a Graphi-cal, Interactive Problem Structuring Aid for DecisionSupport Systems. Unpublished Ph.D. Dissertation (TexasTech University, Lubbock, TX, 1984).

[33] W.E. Pracht. GISMO: A Visual Problem Structuring andKnowledge Organization Tool, IEEE Transactions onSystems, Man, and Cybernetics 16 (1986) 265-270.

[34] W.E. Pracht and J.F. Courtney, The Effects of an Inter-active, Graphics-Based DSS to Support Problem Struc-turing, Decision Sciences 19, No. 3 (Summer 1988) 598-621.

[35] H. Raiffa, Decision Analysis (Addison-Wesley, Reading,MA, 1968).

[36] A.P. Sage, Behavioral and Organizational Considerationsin the Design of Information Systems and Processes forPlanning and Decision Support, IEEE Transactions onSystems, Man and Cybernetics 11 (1981) 640-678.

[37] S.M. Shugan, The Cost of Thinking, Journal of Con-sumer Research 7 (1980) 99-111.

[38] G.F. Smith, Defining Managerial Problems: A Frame-work for Prescriptive Theorizing, Management Science35, No. 8 (1989) 963-981.

[39] J.B. Thomas and R.R. McDaniel, Interpreting StrategicIssues: Effects of Strategy and the Information-ProcessingStructure of Top Management Teams, Academy of Man-agement Journal 33, No. 2 (1990) 286-306.

Documents

Studies in Managerial Problem Formulation Systems