Semantic Data Fusion Through Visually-enabled Analytical Reasoning

Guoray Cai College of Information Sciences and Technology

Pennsylvania State University University Park, PA 16802 USA

cai@ist.psu.edu

Jake Graham College of Information Sciences and Technology

Pennsylvania State University University Park, PA 16802 USA

jgraham@ist.psu.edu

Abstract— Investigating terrorist activity patterns and predicting threats involve collecting and analyzing data from both hard sensors and humans as part of analysts' reasoning process (evidence building, hypothesis creation and testing and decision making). Although automated data fusion methods have been proposed in previous studies, they tend to operate on low-level linguistic features of events and fail to connect to high-level conceptual categories that analysts need to make judgment. This paper argues for extending data fusion models and architecture with an explicit component of visual analytics that integrates human and machine analytical capability through interactive visual analysis. We motivate this argument by the need for human-driven analytical reasoning in counter-intelligence investigation domain. Our extended data fusion architecture follows the sensemaking theory of Pirolli and Card, which provides a framework for understanding specific details on how investigative analysis weave computation, visualization and human reasoning to support coherent analytics. The feasibility of this data fusion architecture is demonstrated through an Analysts’ Workbench that allows analysts to construct intelligence reports through discovering, assessing, and associating evidences.

Keywords—visual analytics; sensemaking; investigative intelligence

I. INTRODUCTION Traditional data fusion systems focus on processing of

physical sensor data to achieve an understanding of an observed environment. The rapid dissemination of mobile phones and social media allows human analysts to act on both as (hard) sensor-generated data and human-generated textual data (or “soft” data). In the same time, methods for fusing hard and soft data are increasing driven by the practical goals of extracting information, knowledge, and insight from data about complex situations. Such fusion processes stress both computational algorithms for event processing and pattern detections, as well as supporting human reasoning when making sense of the data.

In homeland security domains, analysts face the challenge of fusing overwhelming amounts of disparate, conflicting, incomplete and dynamic information that requires human judgment and expertise to make the best posisble evaluation of emerging threats or other disasters. The significant advances in our capacity of hard and soft data collection must be matched

by improvement in analyzing such data. Current information fusion technologies do not address the needs for analytical reasoning and insight discovery with messy and ever-changing data. In a Multidisciplinary University Research Initiative (MURI) project at Penn State University, we are integrating visual analytic tools into an extended hard-soft information fusion architecture in order to achieve “human-in-the-loop” situation assessment. This paper present a conceptual framework that we used in developing information fusion architecture driven by the needs for situation understanding and tactic insights.

II. AN ANALYTICAL PERSPECTIVE ON INFORMATION FUSION

Data fusion is a complex process with a wide range of issues that must be addressed. Fusion of data from multiple sensors can be understood in three levels of processing: the data level, the feature level, and the decision level [1], with information created at higher level of abstraction. More general data fusion models are mostly influenced by the JDL fusion model [2-4]. JDL is a functional model of a fusion process with several levels (from level 0 to level 5): sub-object, object, situation, impact, and refinement. Later models [5] elaborated the 4th level (situation level) of the JDL model with Endsley’s situation awareness model [6] which characterizes situation awareness in three levels of mental representation: perception, comprehension, and projection. It explains how human perceive world elements in their meaning, and project their status in the near future states [7].

Our research program addresses the need for fusion of hard sensor data and soft (human-generated text) data in support of situation awareness and intelligence gathering in counter-insurgence operations [8] [9]. In this domain, data can come from multiple sources. Hard data coming from physical sensors deployed in the field to track and identify targets (people, vehicles, etc.) and soft data from human sources such as soldiers. This information is collected in response to priority information requirements (PIRs) issued from commanders. These PIRs determine the priority for intelligence focus and support that the commander needs to understand the adversary and operational environment [10]. Each PIR is associated with a set of indicators that field commanders must consider in planning actions. These include in-part, organization, size and composition of group, motivation and goals, religious and

This work is sponsored by US Army Research Office (ARO) under a MURI program (Number W911NF-09-1-0392) “Unified Research on Network-based Hard/Soft Information Fusion.”

ethnic affiliation, international and national physical, financial), identities of group leadand idealist, sources of supply and support andand operations, among others. The analyst neethe PIRs and fulfill them based on the current o

Intelligence analysis in COIN domainchallenges to the existing methods of data fusmodels have been focusing on automated extraand identifications of entities, attributes, arelevant to situations, but none of these mature enough to generate reliable outcome.data fusion research [2], developing mathemaof perception and recognition of basic objeremain difficult, due to the problem ocomplexity [11] of pattern recognition and leaThis problem can be partially overcome bdomain-specific semantic descriptions in the and ontologies. In practice, structures of thontologies have to be fixed and they cavariations of semantics with new situations.

On the side of soft data fusion and prochallenges exist. Soft data fusion architecturebased on natural language processing technsyntactic parsing, semantic mapping anresolutions, towards knowledge representationpropositional graph or association graph [9, knowledge, semantic and pragmatic leveprocessing technologies remain premature afrom the way human communicate throughexisting mechanisms for natural language un14, 15] are mainly limited to relations of wordand phrases, but they fail to capture the rich objects in the surrounding world, not tcomplexity of human cognition and thinking process linguistic artifacts and fixed rulescognition operates on concepts attached emotion, intuition, and imagination [17].

In addition to the difficulties in processinsoft data, methods for fusion of hard and softeven more problematic, due to lack of coknowledge models between the two categoriesFor example, Gross et al [13] prescribes to a fruns two processing pipelines for hard and sofan “attributed soft data propositional graph (“attributed hard data propositional graph (AHHard-soft data fusion is achieved by mergAHDG into a cumulative data association measured similarities between entities and resulting systems are brittle, since the semused for creating the language-semantic mapsituation-specific and evolving in the dominsurgence.

In order to address the “missing link” betwand analytical reasoing, there have been muconceptualize the data fusion, situation awarenmaking in a human-cetered framwork [18]. Enawareness model [6] inspired much researchsituation understanding from data fusion proce

support (moral, ders, opportunists d preferred tactics eds to understand observed data.

n imposed new sion. Data fusion action of features and associations technologies are . In hard sensor

atical descriptions ects and entities

of combinatorial arning algorithms. by incorporating forms of models

hese models and annot deal with

ocessing, similar es are commonly niques involving nd co-reference n in the forms of 12-14]. To our

els of language and far removed

h language. The nderstanding [12, ds to other words semantics of the to mention the [16]. Computers s, while human to experiences,

ng hard data and t data seem to be ommonly shared s of data sources. fusion model that ft data, producing (ASDG)” and an

HDG)” separately. ging ASDG and graph, based on attributes. The

mantic knowledge pping tends to be main of counter-

ween data fusion multiple efforts to ness, and sensing-ndsley’s situation h on construting

ess. The outcome

of such system is often in the form“what-where-who-when” knowledgas a critical element of understandtime and their relationships. In situation knowledge have been reprthat support the discovery of associaDecisions to establish association gating procedure and a measure of an indicator of association likelihoowhich data association determinelements (entities, events and relaSimilarity scores are calculated by tfeature similarities, taking into acscores (see Figure 1).

Fig. 1. Computing graph associations representation

However, the use of data fusiogeneration of situation knowledge used in making intelligence judgunknown reliability. Intelligence anprior knowledge of situation underthe broader analytical contexts to desituation are the most relevant. Succan inform data fusion process. If natural to ask if human situation (epart of data fusion process.

Perlovsky [20] believes that higterms of understanding ‘who’ and cannot be developed separately frcommunication. The reason is thacapable of high-level fusion, cognitsupported by research in the last twelinguistics, mathematics of intelligcognitive science, neuro-physiologysignificantly advanced understandinmind involved in learning and usingperception and cognition [16, 21-30

For data fusion methods to matcanalytical strategies and focuses, wedata fusion to include not only cofeatures, attributes and entities, butconcepts and their relationships. Fhuman-centric information fusion [1conceptual-emotional understandingactions [20], we present here architecture based on our understand

m of situation description of e. Data fusion is advanced

ding things, people, places, previous work [9, 13, 19],

resented as attributed graph ations through link analysis. are commonly done by a similarity / dissimilarity as

od. Gating is the process by nes eligibility for graph ationships) for association.. the weighted aggregation of ccount the missing feature

from attributed data graph

on methods for automated is unlikely to be directly

gment, due to it low and nalysts normally have some r investigation, and they se etermine what aspects of the ch knowledge and expertise f this assumption holds, it is even incomplete) should be

gher levels of cognition in ‘why’ from hard-soft data

rom human cognition and at only the human mind is tion, and language. This is enty years in computational

gence and neural networks, y and psychology that have ng of the mechanisms of the g language, mechanisms of ].

ch such new development in e must re-conceptualize the omputational processing of t also human processing of Following the argument for 18] and a elevated focus on g of world observations and

a semantic data fusion ding of analytical reasoning

in intelligence analysis. This is followed by a visual analytic method for supporting human analytical reasoning.

Using the counter-insurgence investigation of IED attacks as motivating scenarios, Graham and Rimland [10] argued that data fusion community should pay attention to the recent trend that intelligence analysis practices have migrated from their traditional focus on objects (the ‘what’ such as IED devices) towards higher level insights that target better understanding of the people (the ‘who) involved in terrorist activities and the reasons (the why) of their involvement [31] (see figure 2).

Fig. 2. Evolution of intelligence analysis (after [31]).

In this paper, we take the perspective of intelligence analysis to assess the progress in data fusion. We argue that data fusion systems must be designed to collaborate with human analysts.

III. THE SCIENCE OF ANALYTICAL REASONING Intelligence analysis is both an art and science [32, 33].

Analytical reasoning refers to the process that intelligence analysts perceive, understand and reason about complex and dynamic data in order to collect evidences for reaching conclusions or judgment. The science of analytical reasoning focuses on analytical discourse and reasoning processes that makes the collected information relevant, focused and effective. To make judgments about an issue, analyses are often done on smaller questions relating to a larger issue. In addition, analysts must often reach their judgments under significant time pressure and based on limited and conflicting information. Their judgment necessarily reflects their best understanding of a situation, complete with assumptions, supporting evidences, and uncertainties. They seek and process a set of information, ideally from multiple sources; assert and test key assumptions; and build knowledge structures using a chain of reasoning to incorporate new information into knowledge they already have [34]. When forming their judgment, analysts often form multiple competing hypotheses and evaluate these alternative explanations in light of evidences and assumptions.

Analysis is generally not a linear process. During an analytic session, the analyst engages in multiple, iterative dialogues with the information available. This process is known as analytical discourse [32, 35]. Information is consulted and extracted to (1) refine and elaborate the issues

and questions, (2) gather data for identifying and evaluating evidence, and (3) using evidences and assumptions to evolve knowledge and insights.

We conceptualize analytical discourse with data using the sense-making loop of Pirolli and Card [36] (See Figure 3). Boxes in the diagram represent data and arrows represent processes. An analyst filters messages and actively searches for information and collects it in an information store (called a shoebox). Relevant information nuggets from this store are assembled into evidence files and organized by evidence schemas. Schemas correspond to the internalized mental representations of the analysts. The sense-making loop characterizes how users obtain insights through analytical reasoning with data that directly support situation assessment, planning, and decision-making. The analytical process involves diverse tasks such as

• understanding historical and current situations, as well as the trends and events leading to current conditions;

• identifying possible alternative future scenarios and the signs that one or another of these scenarios is coming to pass;

• monitoring current events to identify both expected and unexpected events;

• determining indicators of the intent of an action or an individual;

Fig. 3. The sense-making loop (after Pirolli & Card [36])

Analytical work itself is inherently difficult, due to the cognitive difficulty of critical thinking and sensemaking. This difficulty is exacerbated by heavy workload, time pressure, high stakes, and high uncertainty. Analysts must deal with data that are dynamic, incomplete, often deceptive, and evolving. Problem solving with such data involves breaking data into elements, examining the patterns to reveal evidence, accumulating and relating evidences, and assembling and harmonizing many different insights from different observations.

The field of visual analytics has been developed as partial solutions to the challenge of supporting human analytical reasoning. It seeks to marry techniques from information visualization with techniques from computational

transformation and analysis of data. Information visualization amplifies human cognition by expanding human working memory, speeding up search of relevant patterns, enhancing the recognition of important relationships and trends, and monitoring changes. All these tasks benefit from human visual perception capacity. Visual representations in visual analytics tools are not static pictures, but are ‘windows’ for analysts to interact with the underlying data, which enables human thought process to translate data into information and knowledge. Visualization becomes the medium of a semi-automated analytical process, where humans and machines cooperate using their respective distinct capabilities for the most effective results. Interaction is central to both perception and cognition. From the point of view of distributed cognition [37], interactive visual representations is a way to allow cognitive processes to operate on a human-machine joint cognitive architecture[38].

Fig. 4 Data fusion driven by analytical reasoning

The science of analytical reasoning (above) provides the reasoning framework upon which data fusion technologies can be designed. Figure 4 shows our conceptual model of data fusion driven by the analytical reasoning process of intelligence analysis. This model highlights the centrality of human analytical reasoning in the proces of fusing data for making judgment.

IV. INTRODUCING COINSIGHT – AN ANALYST’S VISUAL WORKBENCH

Based on our vision of the analysis-driven data fusion process, we have designed and implemented a testbed where human-analytical reasoning and data fusion capabilities are woven together using visual analytic methods. The resulting tool is called COINsight, targeting intelligence analysis in counter-insurgence operations. Our initial effort focused on investigative analysis of IED attacks based on a synthetic COIN dataset (SYNCOIN) [10]. SYNCOIN data includes 595 messages (“soft data”) and synthetic complimentary simulated physical sensor data (“hard data”). The scenarios cover a four-month period between January 1st, 2010 and May 10th, 2010; centered in Baghdad, Iraq. The central theme throughout the dataset involves Improvised Explosive Device (IED) operations and associated networks.

There are several stories and sub-stories that have been interlaced throughout the message set. These stories are about people, their plans, and the timeline towards fulfilling their intended IED related activities. The data set involves six threads of parallel activity including; a bio-weapons thread, a Bath’est resurgence thread, an Iranian Special Group thread, a sectarian conflict thread, a Sunni criminal thread (SUN) and a

Rashid IED Cell thread. In this paper we are concentrating on the Sunni criminal thread. This thread contains 114 messages consisting of both soft and hard reports. These SYNCOIN messages are front line reports from soldier’s and information gathered by soldiers from informants in response to Priority Information Requests (PIRs).

To motivate design choices, we often cite a concrete scenario. In this scenario, an analyst has previously been assigned to monitoring the Sunni Criminal Group. On March 4, 2010, the analyst received this message:

This targeted jailbreak of Dhanun Ahmad (a Sunni Criminal Group member) has elevated the priority of understanding his connections to the criminal group and identification of how the well-coordinated breakout could take place under tight security.

A. Processing of SYNCOIN messages COINsight operates on human concepts level. Therefore, SYNCOIN messages must first be analyzed to extract entities, attributes and relationships. This was done in semi-automated fashion, where text analysis algorithms are used to identify candidates of entity references and feed to human for verification and fixation of errors. After entities and their attributes are identified, we trained human coders to create association graphs capturing relationships among entities.

Fig. 5 SYNCOIN message processing.

For messages that contain geographical information (place descriptions, GPS coordinates, or location phrases), we assign geographic footprints (in the form of point, line, or polygon) to the messages according to the geocoded location and extent. The recognized entities and their relationships are encoded into a relational schema. Figure 6 is the entity-relationship diagram of our relational data. This relation model representation is specific to the SYNCOIN data. From this data representation, we can generate a variety of visual representations of SYNCION data.

On March 4, 2010, “SIGACT: Complex attack on INP HQ in Karkh using VBIED and grenades. Dhanun Ahmad escaped with 3 other detainees. Detainment area destroyed. Several prisoners killed.”

Fig. 6 The entity-relationship diagram of the SYNCOIN data.

B. Design Considerations COINsight is designed to support the “information foraging

loop” and the “sense-making loop” of the analytical reasoning process. The “information foraging loop” is supported by a visual analytic dashboard that consists of a coordinated multiview environment. The “sense-making loop” is supported by an interactive notebook (see the window of the lower right corner in Figure 7. Priority information requests (PIRs) come into the notbook panel where the analyst reason on the investigative needs and develops strategies for collecting evidences from data. The analyst will turn to the visual dashboard area whenever it is time to explore the data. Findings from the visual dashboard can be recorded into the notebook messages where the analysts make judgment and claims.

Fig. 7 The design of COINsight

Our design of visual analytic dashboard considers several models of multiple view systems [39, 40]. It supports three different dimensions visual data exploration: selection of views, presentation of views, and interaction among views [41]. Coordinated multiviews have been widely used for visual analytic support to exploratory data analysis. Here we use a

specific design of multiple views to support exploration of events, messages, criminal groups, IED devices and their relationships through space, time, and domain knowledge schemas. Among many choices of visualization forms [42, 43], we have chosen four basic representations: tables, timeline, map, and network. Table views are used to visualize and interact with messages and events. Map view is used to discover associations through spatial relationships. Timelines allow effective query for filter and zero into specific time range. Network view is constructed from the relational data model of the SYNCOIN data and it allows exploration of associations among entities and objects. By making live linking across all views, different visualizations are coordinated to ease the analyst effort to find corresponding items depicted in different views.

C. Implementation We took an iterative approach to incrementally design and

refine the behavior of COINsight. It is implemented as an HTML 5 application driven by D3 and JSON JavaScript libraries. Data are centrally managed in a relational database store, and requests for data are handled through a gateway server program to ensure coordination. This web-based architecture works equally well for single user or a distributed intelligence team working together remotely.

One important design decision is to give the analysts the maximum flexibility in controlling their reasoning and interactions with data. For example, Figure 8 shows that the system allows any number of windows to be opened, resized, re-positioned, and closed, so that they can organize their workspace. Coordination across views are also controllable by the analysts so that the analysts can use multiple views to provide “overview + detail,” “filter/subset,” and “comparison” analysis.

Fig. 8 Flexible layout and evidence collector functions

The true power of this system originates from the joint use of the notebook and dashboard functions, where the assessment of competing hypotheses in the notebook creates need for gathering evidences about situations from the dashboard. This situations become information requests for data fusion

subsystems to decide what aspects of the sitfurther characterized and analyzed.

D. Scenario-based Evaluation and RefinemenWe have applied the data fusion framework atool in our research of hard-soft data fusionimprove the coordination of analytical reasonIn the same time, we are planning to usscenarios developed in our project to systemabenefit of this design methods and the utility otool. Towards this goal, we have identifiscenarios where intelligence analysis inprocess of hypothesis-driven investigaunderstanding, and data fusion. As an exshows a scenario that the investigation of futuattack revealed suspects of a number of crimindividuals who might be the organizers potential attacks. However, neither hard senmessages gathered by our data fusion systemresolving the uncertainties in who is related twhen. On the other hand, human analystknowledge about how criminal groups transactions and their likely motives of effectively eliminates certain possibilities usiof coherence in reasoning. We are impscenarios for both demonstrating the utility owell as refining the design of COINsight tool.

Fig. 9 Sample scenario for evaluation experiment

tuation are to be

nt and the COINsigt n environment to ning data fusion. se the analytical atically study the of the COINsight ed a number of

nvolves iterative ation, situation

xample, Figure 9 ure thread in IED minal groups and

or executers of nsor data nor soft m is adequate for to what events at ts bring in their

work in IED attacks, which

ng the principles plementing such of our method, as

V. DISCUSSIONS AN

We have argued that the theorydiscourse with data provides an beprocess. Our goal is to create conceptual processing and computhrough interactive visual represenCOINsight system was motivated byfeasibility of such idea. However, wto understanding the sense-making pcorrect choice on the interaction andis to be learned about the procreasoning before the benefit canFuture work is planned to use theanalytical discourse of experts whenintelligence requests.

ACKNOWLEDG

We gratefully acknowledge thabeen supported in part by a MResearch Initiative (MURI) grant0392) for “Unified Research on Information Fusion”, issued by the (ARO) under the program managem

REFERENC

[1] B. V. Dasarathy, "Sensor fusion potarchitectures and illustrative applicativol. 85, pp. 24-38, 1997.

[2] M. E. Liggins, D. L. Hall, and J. Llinafusion: theory and practice. Boca Rato

[3] D. L. Hall and J. Llinas, Handbook Raton, FL: CRC Press, 2001.

[4] D. L. Hall, "An introduction to multisethe IEEE, vol. 85, pp. 6-23, 1997.

[5] D. A. Lambert, "Grand challenges o213-220.

[6] M. R. Endsley, "Toward a Theory of Systems," Human Factors: The JourErgonomics Society, vol. 37, pp. 32-32

[7] M. R. Endsley, "Design and evalenhancement," presented at the ProceeErgonomics Society Annual Meeting, 1

[8] "Unified Research on Network-baseded. Multidisciplinary University Re(Number W911NF-09-1-0392) by t(ARO): awarded to University at University, and Tennessee State Univer

[9] K. Date, G. A. Gross, S. Khopkar, Rassociation and graph analytical procesdata," in Information Fusion (FUSConference on, 2013, pp. 404-411.

[10] J. L. Graham, D. L. Hall, and J. Rimldataset for qualitative evaluation of h14th International Conference on Infor

[11] L. I. Perlovsky, "Conundrum of coTransactions on Pattern Analysis and M666-670, 1998.

[12] S. C. Shapiro and D. R. Schlegel, "Nasoft information fusion," presented at on Information Fusion, Istanbul, Turke

[13] G. A. Gross, R. Nagi, K. Sambhoos, DG. Tauer, "Towards hard+soft data fuimplementation for the joint fusion

ND CONCLUSION y of analytical reasoing and etter framing of data fusion

synergy between human uter-based data processing ntation. Initial design of y that vision, and tested the we have come to more need process in order to make the d coordination design much cess of human analytical

n be expirically observed. e tool to gather data about n they engage in a vareity of

GMENT at this research activity has

Multidisciplinary University t (Number W911NF-09-1-

Network-based Hard/Soft US Army Research Office

ment of Dr. John Lavery.

ES tential exploitation -- Innovative ions," Proceedings of the IEEE,

as, Handbook of multisensor data n, FL: CRC Press, 2009. of multisensor data fusion. Boca

ensor data fusion," Proceedings of

of information fusion," 2003, pp.

Situation Awareness in Dynamic rnal of the Human Factors and , 1995. luation for situation awareness edings of the Human Factors and 1988. d Hard/Soft Information Fusion," esearch Initiative (MURI) grant the US Army Research Office

Buffalo (SUNY), Penn State rsity, 2009-2014.

R. Nagi, and K. Sambhoos, "Data ssing of hard and soft intelligence

SION), 2013 16th International

land, "A COIN-inspired synthetic hard and soft fusion systems," in rmation Fusion, 2011, pp. 1-8. ombinatorial complexity," IEEE Machine Intelligence, vol. 20, pp.

atural language understanding for the 16th International Conference

ey, 2013. D. R. Schlegel, S. C. Shapiro, and usion: Processing architecture and

and analysis of hard and soft

intelligence data," in 15th International Conference on Information Fusion (Fusion 2012), 2012, pp. 955-962.

[14] M. Prentice, M. Kandefer, and S. C. Shapiro, "Tractor: A framework for soft information fusion," presented at the 13th Conference on Information Fusion (FUSION), Edinburgh, UK, 2010.

[15] C. Mellish and J. Z. Pan, "Natural language directed inference from ontologies," Artificial Intelligence, vol. In Press, Corrected Proof, 2008.

[16] W. Yeh and L. W. Barsalou, "The situated nature of concepts," The American journal of psychology, vol. 119, pp. 349-384, 2006.

[17] L. W. Barsalou, W. Kyle Simmons, A. K. Barbey, and C. D. Wilson, "Grounding conceptual knowledge in modality-specific systems," Trends in Cognitive Sciences, vol. 7, pp. 84-91, 2003/2 2003.

[18] D. L. Hall and J. M. Jordan, Human-centered information fusion. Boston: Artech House, 2010.

[19] G. Gross, R. Nagi, and K. Sambhoos, "A fuzzy graph matching approach in intelligence analysis and maintenance of continuous situational awareness," Inf. Fusion, vol. 18, pp. 43-61, 2014.

[20] L. I. Perlovsky, "Cognitive high level information fusion," Information Sciences, vol. 177, pp. 2099-2118, 2007.

[21] R. Jackendoff, Languages of the Mind. Cambridge, MA: MIT Press, 1992.

[22] L. W. Barsalou, W. Yeh, B. J. Luka, K. L. Olseth, K. S. Mix, and L. Wu, "Concepts and meaning," in Chicago Linguistics Society 29: Papers from the parasession on conceptual representations, K. Beals, G. Cooke, D. Kathman, K. E. McCullough, S. Kita, and D. Testen, Eds., ed: University of Chicago: Chicago Linguistics Society, 1993, pp. 23-61.

[23] S. Harnad, "Symbol grounding and the origin of language," in Computationalism: New Directions, M. Scheutz, Ed., ed: MIT Press, 2002, pp. 143-158.

[24] L. Barsalou, "The human conceptual system," in Proceedings of the international conference on formal ontology in information systems, 2001, pp. 186-186.

[25] L. W. Barsalou, "Simulation, situated conceptualization, and prediction," Philosophical transactions of the Royal Society of London. Series B, Biological sciences, vol. 364, pp. 1281-1289, 2009.

[26] L. W. Barsalou, C. Breazeal, and L. B. Smith, "Cognition as coordinated non-cognition," Cognitive Processing, vol. 8, pp. 79-91, 2007.

[27] R. L. Goldstone and L. W. Barsalou, "Reuniting perception and conception," Cognition, vol. 65, pp. 231-262, 1998.

[28] L. Perlovsky, "Language and cognition," Neural Networks, vol. 22, pp. 247-257, 2009.

[29] D. Reisberg, Cognition: exploring the science of the mind. New York: W. W. Norton, 2010.

[30] L.-l. Wu and L. W. Barsalou, "Perceptual simulation in conceptual combination: Evidence from property generation," Acta Psychologica, vol. 132, pp. 173-189, 2009.

[31] S. Downey, "Intelligence, Surveillance and Reconnaissance Collection Management in the Brigade Combat Team during COIN: Three Assumptions and Ten “A-Ha!” Moments on the Path to Battlefield Awareness," Small Wars Journal, 2008.

[32] J. J. Thomas and K. A. Cook, "The Science of Analytical Reasoning," in Illuminating the Path: The Research and Development Agenda for Visual Analytics, J. J. Thomas and K. A. Cook, Eds., ed: National Visualization and Analytics Center, 2005.

[33] R. J. Heuer, Psychology of Intelligence Analysis. Washington, D.C.: US Government Printing Office, 1999.

[34] L. Bradel, J. Z. Self, A. Endert, M. S. Hossain, C. North, and N. Ramakrishnan, "How analysts cognitively "connect the dots"," presented at the 2013 IEEE International Conference on Intelligence and Security Informatics (ISI), 2013.

[35] G. Cai, "Formalizing Analytical Discourse in Visual Analytics," in VAST 2007- IEEE Symposium on Visual Analytics Science and Technology 2007, pp. 217 - 218.

[36] P. Pirolli and S. Card, "The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis," presented at the Proceedings of 2005 International Conference on Intelligence Analysis, 2005.

[37] J. Hollan, E. Hutchins, and D. Kirsh, "Distributed cognition: toward a new foundation for human-computer interaction research," ACM Transactions on Computer-Human Interaction, vol. 7, pp. 174 - 196, 2000.

[38] Z. Liu, N. J. Nersessian, and J. T. Stasko, "Distributed Cognition as a Theoretical Framework for Information Visualization," IEEE Transactions on Visualization and Computer Graphics, vol. 14, pp. 1173-1180, 2008.

[39] C. North and B. Shneiderman, "Snap-together visualization: can users construct and operate coordinated visualizations?," International Journal of Human-Computer Studies, vol. 53, pp. 715-715, 2000.

[40] J. C. Roberts, "State of the Art: Coordinated & Multiple Views in Exploratory Visualization," in Coordinated and Multiple Views in Exploratory Visualization, 2007. CMV '07. Fifth International Conference on, 2007, pp. 61-71.

[41] M. Wang Baldonado, A. Woodruff, and A. Kuchinsky, "Guidelines for using multiple views in information visualization," 2000, pp. 110-119.

[42] M. Bostock, V. Ogievetsky, and J. Heer, "Data-Driven Documents," IEEE Transactions on Visualization and Computer Graphics, vol. 17, pp. 2301-2309, 2011.

[43] W. Pike, J. Bruce, B. Baddeley, D. Best, L. Franklin, R. May, D. Rice, R. Riensche, and K. Younkin, "The Scalable Reasoning System: Lightweight visualization for distributed analytics," Information Visualization, vol. 8, pp. 71-84, 2009.