Open Source Intelligence (OSINT) - OSS.Net, Inc. … · Web viewOpen Source Intelligence and Joint or Coalition Operations OSINT is a vital component of NATO’s future vision. Through

Draft Chapter for The Handbook of Intelligence Studies

Open Source Intelligence (OSINT)

Robert David Steele

Executive Summary..................................................................................2Definition & Scope................................................................................2Open Source Intelligence and Joint or Coalition Operations................2Private Sector Information Offerings....................................................3The Open Source Intelligence Cycle.....................................................3OSINT and the Emerging Future Intelligence Architecture of NATO.3

Introduction to Open Source Intelligence.................................................4Definitions.............................................................................................6OSINT in Context.................................................................................7OSINT and Information Operations......................................................9OSINT and National Security.............................................................10OSINT and the Larger Customer Base for Intelligence......................12OSINT and the Levels of Analysis......................................................13OSINT and Coalitions.........................................................................13OSINT and Saving the World.............................................................14OSINT as a Transformative Catalyst for Reform...............................15

Open Sources of Information..................................................................16Open Source Software and Software for Exploitation............................18Open Source Services..............................................................................20The Open Source Intelligence Cycle.......................................................20Applied Open Source Intelligence..........................................................28

Open Source Intelligence Tradecraft...................................................29Mission Relevance of Open Source Intelligence................................42Mission Area Applications..................................................................43

Conclusion...............................................................................................44Money Matters....................................................................................44The Value of Sharing..........................................................................46

References...............................................................................................47Abbreviations..........................................................................................47

Version 2.4 dated 30 March 2006 Posted for Public Comment

1


Executive Summary1

Definition & Scope

Open Source Intelligence, or OSINT, is unclassified information that has been deliberately discovered, discriminated, distilled and disseminated to a select audience in order to address a specific question. It provides a very robust foundation for other intelligence disciplines. When applied in a systematic fashion, OSINT products can reduce the demands on classified intelligence collection resources by limiting requests for information only to those questions that cannot be answered by open sources.

Open information sources are not the exclusive domain of intelligence staffs. Intelligence should never seek to limit access to open sources. Rather, intelligence should facilitate the use of open sources by all staff elements that require access to relevant, reliable information. Intelligence staffs should concentrate on the application of proven intelligence processes to the exploitation of open sources to improve its all-source intelligence products. Familiarity with available open sources will place intelligence staffs in the position of guiding and advising other staff elements in their own exploitation of open sources.

Open Source Intelligence and Joint or Coalition Operations

OSINT is a vital component of NATO’s future vision. Through its concentration upon unclassified open sources of information, OSINT provides the means with which to develop valid and reliable intelligence products that can be shared with non-NATO elements of international operations. Experience in the Balkans, and the increasing importance of the Partnership for Peace and Mediterranean Dialogue members in security dialogue, illustrates the need to develop information sources that enable broader engagement with these vital partners. 1 The executive summary is a precise replication from the NATO Open Source Intelligence Handbook (November 2001), which remains the standard in the field. Drafted by the author, with important refinements from LCdr Andrew Chester, RN Canada, and under the leadership of Capt David Swain, RN, United Kingdom, this volume was approved by General William Kernan, USA, then Supreme Allied Commander Atlantic. The NATO documents and other essential references on OSINT, including the original OSINT Executive Overview, are easily accessible by going to http://www.oss.net/BASIC. This chapter is of necessity a very summative rendition of the 30,000 pages of accumulated knowledge in the Archives at http://www.oss.net.


2

http://www.oss.net/

http://www.oss.net/BASIC


Private Sector Information Offerings

The Internet is now the default C4I architecture for virtually the entire world. The principle exceptions are most militaries and intelligence organizations. The Internet facilitates commerce, provides entertainment and supports ever increasing amounts of human interaction. To exclude the information flow carried by the Internet is to exclude the greatest emerging data source available. While the Internet is a source of much knowledge, all information gleaned from it must be assessed for its source, bias and reliability.

As a source of reliable information, the Internet must be approached with great caution. As a means with which to gain access to quality commercial sources of validated information, the Internet is unbeatable.

A vision of open source exploitation must not be limited exclusively to electronic sources. Traditional print, hardcopy images and other analog sources continue to provide a wealth of data of continuing relevance to NATO intelligence.

The Open Source Intelligence Cycle

As the range of NATO information needs varies depending upon mission requirements, it is virtually impossible to maintain a viable collection of open source materials that address all information needs instantly. The focus should be on the collection of sources, not information. With knowledge of relevant and reliable sources of open source information, an intelligence staff can quickly devote collection energy and analytical expertise to develop tailored OSINT products to the mission need.

OSINT and the Emerging Future Intelligence Architecture of NATO

OSINT is an essential building block for all intelligence disciplines. Open sources have always played a role in classified intelligence production. In the NATO context, a robust OSINT capability greatly increases the range of information sources available to intelligence staffs to address intelligence needs.

Nations are capable of tasking classified intelligence sources to address intelligence gaps. Lacking organic intelligence collection assets, NATO intelligence staffs are unable to task classified collection. Rather than immediately directed a Request For Information (RFI) to a national intelligence


3


centre, a robust OSINT capability enables intelligence staffs to address many intelligence needs with internal resources.

While unable to replace classified intelligence production, OSINT is able to compliment an all-source intelligence production process with essential support including tip-offs, context, validation and cover for information sanitation.

Author’s Note: The most important new concepts to emerge since the release of the NATO documents are those of the Seven Tribes, Collective Intelligence, and the World Brain. The seven tribes, each of which has unique access and perspectives, are those of government, military, law enforcement, business, academic, ground truth (media and non-governmental organizations), and civil (citizens, labor unions, and religions). Collective Intelligence and the World Brain are discussed below.

Introduction to Open Source Intelligence

OSINT is not a substitute for satellites, spies, or existing organic military and civilian intelligence capabilities. It is, however, a foundation—a very strong foundation—for planning and executing coalition operations across the spectrum from humanitarian assistance to total war. OSINT provides strategic historical and cultural insights; it provides operationally helpful information about infrastructure and current conditions; and it provides tactically vital commercial geospatial information that is not available from national capabilities. In coalition operations, OSINT is both the foundation for civil-military cooperation, and the framework for classified bilateral intelligence-sharing.

OSINT is distinct from academic, business, or journalistic research in that it represents the application of the proven process of national intelligence to the diversity of sources, with the intent of producing tailored intelligence for the commander. OSINT is also unique, within a coalition operations context, in that it simultaneously provides a multi-lateral foundation for establishing a common view of the shared Area of Operations (AOR), while also providing a context within which a wide variety of bi-lateral classified intelligence sharing arrangements can be exploited. Figure 1 illustrates these relationships.


4


Figure 1: Relationship Between Open and Classified Information Operations

Since 2001, the Swedish government has advanced a concept for Multinational, Multiagency, Multidisciplinary, Multidomain Information Sharing (M4IS), and the author has put forward the need for regional Multinational Information Operations Centers (MIOC). At the same time, in the private sector, organizations such as the Co-Intelligence Institute has brought forward robust concepts for Collective Intelligence, and books have been written about Smart Mobs and Wisdom of the Crowds. It is clear from these developments that OSINT is taking on a life of its own outside the government, in keeping with the author’s original depiction of the seven tribes of intelligence, illustrated below.2

2 The Swedish concept was advanced at the third Peacekeeping Intelligence Conference sponsored by the Folke Bernadotte Academy & Swedish National Defence College under the direct leadership of the Supreme Commander, 4-6 December 2004. The Co-Intelligence Institute was founded by Tom Atlee, author of The Tao of Democracy: Using CO-INTELLIGENCE to create a world that works for all (The Writer’s Collective, 2003). Howard Rheingold, former editor of The Whole Earth Review, is the author of Smart Mobs: The Next Social Revolution (Perseus, 2002) as well as seminal books on Tools for Thinking, Virtual Reality, and Virtual Communities. James Surowiecki is the author of The Wisdom of the Crowds: Why the Many Are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations (Doubleday, 2004). Key works on the emerging World Brain include those of H. G. Well, World Brain (Admantime, 1994 from 1938); Pierre Levy, COLLECTIVE INTELLIGENCE: Mankind’s Emerging World in Cyberspace (Plenum Trade, 1997); Willis Harman, Global Mind Change: The Promise of the 21st Century (Noetic


5


Figure 2: Information Continuum and the Seven Tribes

OSINT is less about specific sources such as are listed in the column on the right of Figure 1 (previous page), and more about “knowing who knows,”3

Definitions

There are four distinct categories of open information and intelligence.

Open Source Data (OSD). Data is the raw print, broadcast, oral debriefing or other form of information from a primary source. It can be a photograph, a tape recording, a commercial satellite image, or a personal letter from an individual.

Sciences, 1998); and Howard Bloom, GLOBAL BRAIN: The Evolution of Mass Mind From the Big Bang to the 21st Century (John Wiley, 2000).3 This term was developed by Dr. Stevan Dedijer, a Swede who was born and died in Croatia (former Yugoslavia), widely recognized as the father of modern business intelligence. He led fifteen Swedes to the first Open Source Intelligence Conference in 1992, where he made a passionate plea for government attention to this vital independent discipline.


6


Open Source Information (OSIF). OSIF is comprised of data that can be put together, generally by an editorial process that provides some filtering and validation as well as presentation management. OSIF is generic information that is usually widely disseminated. Newspapers, books, broadcast, and general daily reports are part of the OSIF world.

Open Source Intelligence (OSINT). OSINT is information that has been deliberately discovered, discriminated, distilled, and disseminated to a select audience, generally the commander and their immediate staff, in order to address a specific question. OSINT, in other words, applies the proven process of intelligence to the broad diversity of open sources of information, and creates intelligence.

Validated OSINT (OSINT-V). OSINT-V4 is OSINT to which a very high degree of certainty can be attributed. It can be produced by an all-source intelligence professional, with access to classified intelligence sources, whether working for a nation or for a coalition staff. It can also come from an assured open source to which no question can be raised concerning its validity (images of an aircraft arriving at an airport that are broadcast over the media).

OSINT in Context

In this summary chapter we will touch lightly on context of OSINT, while distinguishing between OSINT as it supports government Intelligence & Information Operations (I2O) where secret sources and methods play a paramount role, and OSINT as the sole legal means of decision support for non-governmental organizations.

While OSINT is not “new” in that Nations and organizations have always understood the value of legal travelers, direct observation, structured reading, and legal purchases of information services, what is new about OSINT is the confluence of three distinct trends: first, the proliferation of the Internet as a tool for disseminating and sharing overt information in all languages; second, the consequent and related “information explosion” in which published useful knowledge is growing exponentially; and third, the collapse of formerly denied areas accompanied by the explosion of non-traditional threats in the form of

4 Dr. Joseph Markowitz, the first and only Director of the Community Open Source Program Office (COSPO) before it was destroyed by the Community Management Staff (CMS), devised this important distinction between OSINT such as can be done by private sector practitioners, and OSINT as validated by government analysts with full access to classified sources & methods.


7


failed states and transnational non-state threats to public security and prosperity.

Below, and provided together to conserve space, are four snapshots of how OSINT relates to the larger secret intelligence, to the specific secret disciplines, to the wisdom of the crowds, and to the decision support process of any commander or Chief Executive Officer (CEO).

Figure 3: OSINT in Larger Contexts

Four simple points can be made:

1. Governments are wasting billions on secrets and still not taking OSINT seriously.

2. Change the rules of the game and every catch by a private citizen becomes an “out.”


8


3. OSINT is both an essential foundation for secret disciplines, and its own discipline.

4. OSINT provides a basis for institutionalizing sharing of information with everyone.

OSINT and Information Operations

Information Operations (IO) is comprised of Information Peacekeeping (IP) and Information Warfare (IW).

At the strategic level, IO is broadly related to influencing and messaging all parties (hostile, neutral, and friendly) for national advantage. IO must integrate OSINT (understanding their reality as well as our own), Joint Information Operations Centers or Commands (JIOC) as well as multinational and national variants (MIOC, NIOC) which comprise the tool-sets as well as the mind-sets; and Strategic Communication (the message).

At the operational and tactical levels, this translates into assuring one’s own ability to see, hear, know, understand, decide, and act on “all information, all languages, all the time,” while denying or distorting adversarial information capabilities to support the campaign.

This is an extraordinarily complex undertaking that has not been intellectually defined. The concepts, doctrines, tools, and mind-sets are a long way from being robust. What this means in practice is that nations and organizations must be able to devise unified campaign plans that fully integrate on an interagency or inter-departmental basis, the activities of public diplomacy and public affairs or relations, strategic communication and influence (as well as strategic acquisition and force structure management), perception management, psychological operations (PSYOP), the propaganda and agent of influence aspects of covert operations (among governments), denial and deception, space control, network attack and defense, electronic warfare, information and communications and electronic security operations, information assurance operations, counter-intelligence and counter-deception operations, and so on.5

5 The author is indebted to Admiral Bill Studeman, USN (Ret.), former Deputy Director of Central Intelligence (DDCI), former Director of the National Security Agency (NSA), and former Director of Naval Intelligence (DNI), who in his post-retirement years has become a master of IO and all that this implies. In the age of information, IO is the manifestation of “total war” and the need—not yet realized—to harness every source of national power to further national advantage.


9


Rarely emphasized except by the author, all of these demand that we be understand reality, and not allow the Republic to be driven into bankruptcy by ideological fantasies and consequent policy-level misjudgments.

OSINT and National Security

It is a common mis-perception that most “intelligence” is classified and must come from secret sources and methods that are very expensive and relatively risky. In fact, as the “temple” slide in Figure 3 suggests, the “cult of secrecy” has put us in a very disadvantageous position, where in the United States of America (USA) roughly $70 billion a year is spent on collecting the 5% of the information that is secret and can or must be stolen, and virtually nothing is spent on the 95% of the information in all languages that is relevant to all but the most secretive threats.

The importance of this observation can be emphasized by listing the top threats to global security as documented in the Report of the High-level Panel on Threats, Challenges and Change, A more secure world: Our shared responsibility6

Economic and social threats including 95%o poverty, 99%o infectious disease and 95%o environmental degradation 90%

Inter-State conflict 75% Internal conflict, including 90%

o civil war, 80%o genocide and 95%o other large-scale atrocities 95%

Nuclear, radiological, chemical, and biological weapons 75%

Terrorism 80% Transnational organized crime 80%

Figure 4: OSINT Relevance to Global Security Threats

6 (United Nations, 2004), The endeavor benefited from the participation of The Honorable LtGen Dr. Brent Scowcroft, USAF (Ret), former national security advisor to President George Bush. Terrorism is either fifth on this list or seventh if the first is counted as three. The report, 262 pages in length, can be seen at http://www.un.org/secureworld/report2.pdf.


10

http://www.un.org/secureworld/report2.pdf


The average utility and relevant of OSINT to these global threats is—on the basis of my informed estimate—82.5%, which comes very close to the generic “80-20” rule. We must conclude that any nation that persists in spending 99.9 percent of its intelligence funds on collecting secrets,7 and less than one half of one percent of its intelligence funds on OSINT, is quite literally, clinically insane (or insanely corrupt) at the highest levels.

Naturally there are those who will quibble about whether the budgets of the National Aeronautics and Space Agency (NASA) or the Environmental Protection Agency (EPA) or the Department of Justice (DoJ) should be “counted.” What matters here is that intelligence is nothing more or less than decision-support for the President and the top members of the Cabinet, as well as Congress in its oversight role. Most of the U.S. Government budget, by way of example, is spent on weapons, manpower, and administration. Research & development (R&D) is focused on investigation, design, and the creation of capabilities, not on decision-support. Intelligence is decision-support.

It merits comment that those business enterprises and religions that choose to emphasize industrial espionage or the covert subversion of governments, are making the same fundamental error of confusing “secret sources & methods” with “intelligence.” Intelligence is information that has been collected, processed, analyzed, and presented in order to support a decision that increases security or profit, or reduces risk or cost. Nowhere is it written that “intelligence” must be secret or that intelligence is improved by a reliance on secret sources and method. Indeed, it has been demonstrated on more than one occasion, with Viet-Nam and Iraq as the extant examples,8 that not only is secret intelligence easy to ignore and manipulate, but a reliance on secret

7 While not the focus of this chapter, it merits comment that according to the Commission on the National Imagery and Mapping Agency, as published in December 1999, most of the intelligence money is spent on esoteric collection systems, and almost none at all is spent on actually making sense out of the collected information.8 For Viet-Nam, the single best reference cooking the books and spinning the truth is George Allen’s NONE SO BLIND: A Personal Account of the Intelligence Failure in Vietnam (Ivan R. Dee, 2001). On the topic of Peak Oil, 9-11, and Iraq, there are numerous books, of which three stand out: James Bamford, A PRETEXT FOR WAR: 9/11, Iraq and the Abuse of America’s Intelligence Agencies (Doubleday, 2004); James Risen, STATE OF WAR: The Secret History of the CIA and the Bush Administration (Free Press, 2006); and Michael C. Ruppert, CROSSING THE RUBICON: The Decline of American Empire at the End of the Age of Oil (New Society, 2004). Many other books address various aspects of how 9/11 represented both a break-down of secret intelligence and a celebration of ideological fantasy unchecked by responsible oversight.


11


intelligence can lead to a “shutting out” of overt common sense and open sources of information.9

Consider this, Daniel Ellsberg lecturing Henry Kissinger:10

The danger is, you’ll become like a moron. You’ll become incapable of learning from most people in the world, no matter how much experience they have in their particular areas that may be much greater than yours” [because of your blind faith in the value of your narrow and often incorrect secret information].

OSINT—intelligence that is publicly disseminated—is the single best antidote to the pathologies of secret executive power.

OSINT and the Larger Customer Base for Intelligence

Most citizens, and most legislators, assume that national intelligence or corporate intelligence is in the service of every part of the government, or every part of the corporation. This is not actually the case. In the USA, specifically, the focus continues to be on “secrets for the President,” and on a few “hard targets” considered to be of the gravest possible concern—China, Cuba, Iran, North Korea. Within corporations, the emphasis is on serving the Chief Executive Officer (CEO). Consider the following questions as both a litmus test for intelligence managers, and as a broad definition of the possibilities for OSINT. To be explicit: every single customer ignored by the mandarins of secrecy or the sycophants to the CEO, is a customer for OSINT.

Do you believe that secrets are the ultimate form of knowledge, or do you believe that all sources including open sources should be brought to bear on decision-support?Do you believe that intelligence should focus only on the gravest of threats, what some call the “hard targets,” or do you believe there is

9 During the eighteen years of my campaign to secure added funding for and emphasis on OSINT, open source information has been derisively referred to as “Open Sores” by nominally intelligent but foolishly unprofessional managers and some analysts at the Central Intelligence Agency (CIA). Even the so-called open source professionals in the Foreign Broadcast Information Service (FBIS) have refused to be serious about anything other than mainstream broadcast media until allies of OSINT finally got an Open Source Agency into the 9-11 Commission Report (page 413). The mind-sets within CIA and its runt orphan FBIS have not yet matured on this topic. 10 Daniel Ellsberg, SECRETS: A Memoir of Vietnam and the Pentagon Papers (Viking, 2002). The three pages on the pathological effects of falling prey to the cult of secrecy, on pages 237-239, should be forced rote memorization for all who receive clearances.


12


merit to “global coverage,” seeking to monitor and understand all threats at some minimal mandatory level of detail?

Is intelligence something that should be done only for the leadership, or should intelligence support—decision-support—be provided to agency heads, department heads, and even the individuals in the field, the front line that interacts with the real world?

Is federal or corporate level intelligence only for the members of the federal government or the corporate headquarters, or should it support state and local jurisdictions, or subsidiaries?

OSINT and the Levels of Analysis

It is in the above context that we can conclude this overview by starting without equivocation that OSINT must be provided to all levels of any enterprise. This about empowering every individual, every segment of the enterprise, with decision-support.

Figure 5: OSINT and the Four Levels of Analysis

OSINT and Coalitions

Although the concepts and doctrine that I have been developing for eighteen years recognize the seven tribes of intelligence as distinct historical, cultural, intellectual, and direct access entities, it is the military and the concept of the military coalition that really serves as the spinal cord and nervous system for


13


“harnessing the distributed intelligence of the Whole Earth.” Within the USA, as within most countries, the military is consistently the most professional, the most disciplined, the most structured, and the most reliable organization. It is also the only one that treats Command and Control, Communications, Computing, and Intelligence (C4I) as a distinct discipline with its own Military Occupational Specialty (MOS) for each aspect.

It is a fact that the USA is simply not capable of fielding sufficient citizens with sufficient language and foreign area qualifications. Given the rapid rate at which available information doubles (a rate that will accelerate as hand-held devices become the instrument of choice, and are used to register photographs, videos, voice recordings, and text inputs from tens of millions operating in all languages, all the time), there is only one possible solution for mastering “all information, all languages, all the time.” We must provide our coalition partners, are particularly our military coalition partners, with the means to digitize, translate, and analyze (using both automated tools and their own unique human expertise) all information of mutual interest, and we must provide a global Information Arbitrage™ capability that enables all coalition partners, each responsible for harnessing and nurturing their respective seven tribes, to participate in what I call the Open Source Information System – External (OSIS-X).

Bi-lateral intelligence sharing may still predominate in the secret world, but in the open source world, it is M4IS—multilateral sharing—that will define the common approach.

OSINT and Saving the World

C. K. Prahalad has taught us that our government and business focus to date, on the one billion richest people on the planet, who represent a one trillion a year marketplace, is short-sighted. His brilliant book, The Fortune at the Bottom of the Pyramid, makes the important point that the five billion poorest people on the planet, because of their numbers and despite their low wages (an average of $1,000 a year, with half that number earning as little as $1 a day), actually represent a four trillion a year marketplace—in short, a marketplace four times larger than the one that is active today.

It was not until I absorbed the wisdom of C. K. Prahalad that I understand that OSINT can help the poor cut costs, reduce disease, improve health, and increase revenue. It is now possible to show religions, labor unions, and civil societies how to leverage the Internet and low-cost hand-held devices (instead


14


of the more expensive laptops or personal computers) to apply OSINT from the “bottom up,” and consequently to double or triple revenue at the bottom of the pyramid. The creation of sustainable indigenous wealth is without question the single fastest way to save the world from itself.

OSINT as a Transformative Catalyst for Reform

America has been adrift for some time. The “me” generation spawned the disengaged generation, and we suffer now from the twin curses of an uneducated public that is also inattentive to its civic responsibility. This affects the rest of the world. It prevents us from keeping our politicians and corporate leaders honest, and its spawns terrible mis-adventures undertaken on the basis of ideological fantasies, without due policy process, or any semblance of a coherent affordable sustainable grand strategy.

Below are illustrated the possibilities of reform that could begin with Electoral Reform.

Figure 6: OSINT as a Transformative Catalyst for Reform

Electoral Reform, which could be inspired by multiple compounding failures of any Administration across the board, could lead to Governance Reform. A Coalition Government could demand that Intelligence Reform be substantive


15


and comprehensive. This would have the happy outcome of imposing National Security Reform, which would not only reduce America’s risk around the world, but would reduce the cost of the heavy metal military, and free up resources for waging peace. From peace, will follow prosperity. The low-cost, high-return value proposition from OSINT cannot be exaggerated.

Alvin and Heidi Toffler have focused in the manner in which information is a substitute for violence, for capitol, for labor, for time and space. Others followed, including Thomas Stewart in The Wealth of Knowledge and Barry Carter in Infinite Wealth. 11 This is real.

Open Sources of Information12

Traditional Media Sources. Many policymakers, commanders, and corporate executives consider themselves to be their own open source collector and analyst, and they limit themselves to traditional print, broadcast, and (today) online sources. They do so unwitting of the fact that 90% of what most journalists know never gets published (including some of the more important information, such as who is prone to take bribes), and without applying the methods of OSINT that help track source bias, contextual history, and related non-media source inputs.

Commercial Online Premium Sources. Commercial aggregators of media and journal information have provided a vital service in the past, notably by integrating multiple sources into databases that could be searched and that benefited from editorial review. However, the premium sources have focused mostly on current information in English for the business audience, and are not suitable for studying emerging threats through all sources in all languages.

Other Forms of Commercial Online Information. Prior to 9-11 there were a number of niche sources online, such as industry or topic specific newsletters. Since 9-11 a number of additional sources have sprung up, including sources that specialize in monitoring foreign language blogs. These are of mixed value,

11 PowerShift (Bantam, 1990). Stewart (Currency, 2001). Carter (Butterworth Heineman, 1999).12 The next four sections consist of alternative updated observations that are distinct from those to be found in the NATO Open Source Intelligence Handbook. There is no substitute for down-loading and studying that reference as well as the NATO Open Source Intelligence Reader and the NATO guide to Intelligence Exploitation of the Internet. These and other key references are freely available at http://www.oss.net/BASIC.


16


and at present no one service has emerged with which to access and evaluate all relevant online sources.

Gray Literature. Gray literature is that information that is both legally and ethically available, but only from specialized channels or through direct local access. Gray literature includes working papers, pre-prints, technical reports, dissertations and lower school papers, as well as yearbooks, organizational telephone directories, and trip reports or other notes available without any violation of accepted access standards..

Overt Human Experts and Observers. The ultimate open source is a human expert or human observer with direct experience. In many places of the world, Africa, for example, it is not possible to obtain published information on specific locations or conditions. The human expert is often the most efficient as well as the most inexpensive means of creating new open source intelligence that is responsive to a specific need at a specific time. The identification and interviewing of those with direct on-the-ground experience is a valuable means of ascertaining “ground truth.” It merits comment that official communications from Embassies and corporate offices, and most media reporting, tend to rely on second and third-hand reports, and are often replete with biases and misconceptions. More often than not, a direct human observer will be more reliable.

Commercial Imagery and Geospatial Information. Commercial imagery and commercially-collected, processed, or presented geospatial information (e.g. precise locations for street corners and specific buildings) is both essential to establish a good understanding of terrain obstacles and conditions in 90% of the world where the U.S. has not created 1:50,000 combat charts with contour lines.13 It is also very helpful in meeting a wide variety of needs including agricultural watering plan, telecommunications route planning, foreign industrial and residential site development, and so on.

The Internet. The Internet is growing from the 400 million users in 2000, to an estimated 3.5 billion users by the year 2015.14 The Internet, despite its robust growth since the early 1990’s, is still in its infancy. The Internet has been over-

13 Fortunately, for most of the Third World countries engaged in “wars of national liberation,” the Russian military has produced excellent 1:50,000 combat charts, and these are available from commercial providers with special access, such as East View Cartographic.14 According to Dr. Vint Cerf, one of the founders of the Internet, and today employed by Google.


17


sold.15 It is, however, maturing, and by one estimate, now offers over 250,000 useful databases or collections of information in the “deep web,” a portion of the web that is distinct from “the Internet” and is also not covered by Google or other common search engines.

Figure 7: Internet, World Wide Web, and Information16

Open Source Software and Software for Exploitation

As a general statement, Open Source Software is one of the five “opens” that will converge to create the World Brain. The others are, apart from OSINT, Open (Electromagnetic) Spectrum, Open Access Copyright, and Open Hypertext Document System (OHS).

15 One study, by the Community Open Source Program Office (COSPO), found that in 1994 there were about 450 useful sites, and that 99% of the Internet was pornography, opinion, or advertising.16 Peter Lyman % Hal Varian, “How much information 2003?” (UC Berkeley, 27 October 2003, http://www.sims.berkeley.edu/research/projects/how-much-info-2003. “These” notes added.


18

http://www.sims.berkeley.edu/research/projects/how-much-info-2003


Standards. The following standards are emergent as enablers of M4IS while still compliant with copyright or other individual caveats desired by the originator or owner of the information.

RDF Resource Description Framework OWL Web Ontology Language SOAP Simple Object Access Protocol OHS Open Hypertextdocument System17

XML Geo eXtended Markup Language Geospatial

Functions. There is no one offering that meets the need for a fully integrated analyst toolkit. This is partly because of the lack of agreement on standards in the past, and partly because of the lack of coherence in government and corporate contracting, where the emphasis has been on hardware and proprietary software instead of generic functionality and ease of data integration. The good news is that newly available offerings such as CISCO’s Application Oriented Network (AON) are eliminating middleware, at the same time that Google’s innovative approach to commodity storage has eliminated configuration management and back-up costs, while also reducing the cost for efficient global distributed storage and fast retrieval to one third of the industry standard. The table below lists the functions established by the Office of Scientific and Weapons Research at the Central Intelligence Agency (CIA) in 1986.18

Revision Tracking, RT Review Desktop Publishing Graphics/Multimedia ProductionCollaborative Work Notetaking & Organizing Ideas Structured ArgumentInteractive Search & Retrieval Graphic Map-Based

VisualizationModeling and Simulation

Clustering & Linking of Data Statistical Analysis for Anomalies

Detection of Changing Trends

Detection of Alert Situations Easy Digitization of Hard Copy Automated Language TranslationProcessing of Images, Signals Automated Data Extraction Data Standardization/Conversion

Figure 8: Fundamental Functions for Online Analysis

17 This is the only standard that may not be readily apparent when this chapter is published. Invited by Doug Englebart, inventor of the mouse and hypertext, this standard enables linkage of related content to take place at the paragraph level, which also allows copyright compliance to be executed at the paragraph level, for pennies instead of dollars.18 Diane Webb, under the leadership of Dr. Gordon Oehler, developed “CATALYST: A Concept for an Integrated Computing Environment for Analysis” (CIA/DI SW 89-10052, October 1989). To not have this now, close to 18 years after precise requirements definition, tells us clearly of the sustained pathos of U.S. Intelligence Community “leadership.” Follow link at www.oss.net/HISTORY.


19

http://www.oss.net/HISTORY


Open Source Services

Collection Services. Collection services include online collection (searchers that specialize in Internet, deep web and premium commercial online source exploitation, inclusive of foreign language competency); off-line gray literature or document acquisition; telephone surveys and electoral or other forms of polling; private investigations and human intervention services (“boots on the ground”); and aerial surveillance or reconnaissance services.

Processing Services. Processing services include data conversion from hard-copy or analog to digital, indexing and abstracting of hard-copy or soft-copy textual data or images, interpretation and annotation of imagery or signals, database construction and stuffing, and complex modeling & simulation projects with the best ones including geospatial and time-based visualizations. When integrated with well-planned open source collection and the right analytical expertise, complex processing services can yield substantial dividends by compressing large amounts of data into manageable tailored products that address specific intelligence requirements.

Analysis & Production Services. A wide variety of commercial and academic organizations offer diverse analysis and production services. As a general rule, the best value is found through the hiring of single individual experts with no overhead, rather than through broad contracts with organizations that then add a substantial fee for their considerable overhead expenses. The very best value results when niche collection, niche processing, and niche analysis services can be "mixed and matched" to obtain precisely the desired results. The very worst value comes when an organization is hired because of a convenient contract, they do not have a niche expert, and choose to dedicate an analyst that does not bring sufficient other organizations with similar intelligence experience or skill to the task.

The Open Source Intelligence Cycle

Overview. Regardless of whether one is trying to obtain data, information, or intelligence, the proven process of intelligence, known as the “intelligence cycle,” can dramatically enhance one’s return on investment in relation to time and money in, and outcomes achieved. An easy way to remember the essential elements of the process is by remembering “the four D’s” of Discovery (Know Who Knows); Discrimination (Know What’s What); Distillation (Know What’s Hot); and Dissemination (Know Who’s Who).


20


Requirements Definition. The first step always begins with defining the requirement.

What do you need to know? Why do you need to know this?When do you need an answer? How long do you want the answer to be?Who else has an interest in this? Do you have a preliminary bias?How much can I spend? What level of security do you want on this?

Practical Triage. Most organizations do not know what they already know, nor do they know what other allied or approachable organizations know. Indeed, despite huge sums spent by governments or corporations, the information and intelligence reaching the President or Chief Executive Officer is invariably biased, filtered, late, or incomplete.19

The OSINT cycle cannot make up for pathological mind-sets (including ideological fantasy and political corruption) or poor management.

Over time the following strategy improves responsiveness and lowers cost, but it is one that requires a concerted effort to “map” internal, allied, and commercial open source options.

FIND the information, for free, from internal resources. Example: in considering a new location for a distribution plant, consider asking the truck drivers their opinion, and charge them with direct observation and elicitation from existing local contacts.

GET the information, for free, from allied sources open to sharing. Example of such organizations include Embassies (not just one’s own), Chambers of Commerce, Non-Governmental Organizations, academics expert on the country or topic in question, and labor unions or religious organizations with direct experience.

BUY the information, at low-cost, from appropriate providers of niche information services. This is generally much cheaper if one serves as one’s own “general contractor.”

TASK either distributed managers, or a specific private sector organization, with a major effort to acquire exactly the right information, process it, and present decision support intelligence tailored to the need.

19 This point is made very ably by Ben Gilad in Business Blindspots (Infonortics UK, 1996).


21


Collection. Armed with clear-cut requirements, it is possible to achieve cost and time effectiveness in collection, as well as to satisfy any operational security requirements. Collection can be scheduled, event-driven, or analyst-driven. It can focus only on internally-available resources, or contract for external actions and assistance. It must at a minimum factor in time and cost allowed, operational security concerns, copyright compliance, foreign language parameters required to achieve a competent answer, and external networking needed if internal resources are insufficient.

Elementary and Inadequate: Google or any other search engine Journeyman Good Enough: SILOBREAKER (sources plus tools) Advanced: SILOBREAKER plus specialized sources Master of the Craft: Above plus qualified human inputs

Processing and Exploitation. The adage, “the network is the computer” is gradually becoming true, and the best offerings now integrate both direct access to sources, and some form of advanced processing capability including, in addition to the ability to cut and paste information into a workbook, varied forms of visualization include map-based visualization and network linkages that can be weighted by frequency or mention or weighted factors. The bottom line is that there are three “scales” at which the OSINT cycle takes place:

Small (Individual) . At the small scale, which is where most analysts are today, OSINT is not treated as a separate discipline and the individual is expected to do their own requirements definition, collection management, mostly human processing (“cut and paste”), and personal presentation, all without proper desktop analytic tools.

Medium (Team) . At the medium scale, teams of individuals may self-organize to share the burden of collection, translation, gisting, and database stuffing. Generally this kind of effort will have one primary language, and be limited by the physics of the 24-hour day and the number of team members that can be harnessed, generally part-time.

Large (Man-Machine Coalition) . Large-scale processing is not, contrary to most views, simply a matter of digitizing vast amounts of information (generally text in English—very few coalitions are capable of digitizing all mediums in all languages and then making sense of what they have). Instead, it requires an elegant combination of tactical collection where expert humans can selectively use cell phones to digitize key pages or enter key information into templates; or send documents back to a processing center for digitization; of man-machine translation combining machine learning, online


22


dictionaries, and a distributed network of human translators and subject matter experts; and advanced statistical analysis that detects anomalies, patterns, and—when combined with rapid response modeling and simulation (RRMS)—can fulfill some predictive analysis functions.

Security. Security, within reason, is important. When security becomes an excuse for hiring the unskilled and not allowing them to talk to informed individuals from all corners of the earth, it has become an albatross. Within the OSINT world, the emerging poster child for pathological security is the current obsession with “anonymous” access to the Internet. This violates both the integrity of the open source environment where “what you look at leads to other stuff” has real value; and consequently anonymity obstructs the functionality using information of known interest to connect humans to one another. The bottom line is clear: if security by obscurity is essential, contract the work out or do it from what are quaintly called “unresolved numerical addresses.” However, if you take pride in your expertise and you want to be part of the World Brain, then forget about anonymity, and focus instead on being a recognizably intelligent and reliable participant in the global network.

Analysis and Production. It is a common misconception, particularly among those who are accustomed to being spoon-fed secrets, that open sources of information are somehow less accurate, credible, authoritative, current, objective, or relevant than secret sources. All sources require the analyst to evaluate these factors.20 The reality is that today, those who collect clandestine human intelligence or secret signals or covert images, cannot be relied upon to authenticate their sources. Case officers will “buy in” to fabricators, and in some cases clandestine managers will deliberately send to the White House reports from known double-agents simply to avoid admitting they have lost all access in Moscow. Signals codebooks will be stolen under circumstances that make it certain the country being robbed knows the next morning, and consequently all subsequent captured signals should be considered suspect. Imagery satellites have known trajectories, such that India could easily hide its preparations for a nuclear test, as the U.S. Intelligence Community blithely ignored published open source reports in both India and British Columbia with respect to the forthcoming explosion.

20 Twenty-two sources of misperception and bias are listed in Appendix C of the NATO Open Source Intelligence Handbook, as drawn from Lisa Kirzan, Intelligence Essentials for Everyone (Joint Military Intelligence College, June 1999). Follow link at www.oss.net/BASIC.


23



OSINT has one advantage over the other sources, its exposure to millions of pairs of eyeballs. OSINT also offers analytic frames of reference that have stood the test of time.21

Another misconception relates to production. Too many people misconstrue reports and page counts as “production” when in fact production includes reports, link tables, distance learning, and professional networking.

Reports should have analytical summaries, should specify the date and time information collection ended, the period covered by the report, and the sources upon which the report is based. A major difference between classified reports and OSINT reports is that it is easier to embed hot links within OSINT reports, so that the reader can if desired go directly to the full text of the raw source, and ideally directly to the applicable paragraph.

Link Tables are a professional means of cumulatively tracking, weighting, and describing online sources. Using a table feature or a spreadsheet, columns can be created for rank, key word, link, and source name and description. Over time link tables can be shared, integrated, updated, and used as a distance learning device.

Distance Learning in the online age is much different from the “scrapbooks” of the past. It is now a simply matter for any analyst to create their own relational database of people, places, events, organizations, and so on.

Professional Networking rounds out the “production” performance of any top-notch analyst. Information really is a substitute for money, and the best analysts are skilled at sharing. More about this below.

Dissemination and Evaluation. The OSINT cycle concludes (and begins anew) with dissemination and evaluation. The best analysts will obtain peer review of their work before sending it forward, and they will share their work to the fullest extent possible, both informally, and formally through official publications including academic or industry journals. There is no greater accomplishment for an analyst than to be recognized for having added, however minutely, to the cumulative knowledge of mankind.

Time Management. Discipline, in the form of time management, is an essential attribute for anyone who aspires to be effective as an analyst. Below is a

21 See the analytic references at http://www.oss.net/BASIC.


24



recommended timetable for a standard OSINT evolution that relies on direct access to internal and online sources.

15 Minutes Requirements Definition. Ensure understanding including context.30 Minutes Internet Collection. Use search tools, identify top items & review.15 Minutes Internet Table. Create table for future use, sharing, and reader reference.60 Minutes Commercial Collection. Use fee sources, purchase top 20 items.60 Minutes Analysis. Understand, evaluate, and structure collected information.60 Minutes Production. Create summary, table of contents, slides, concise text.4 Hours Total time to create standard OSINT report using direct access sources.

Figure 9: OSINT Time Management

Putting It All Together. The illustration below is applicable to all three levels of endeavor.

Figure 10: World Brain Operational Planning Group Virtual Private Network

The common mistake that most vendors of OSINT make is to confuse the weekly report or database stuffing with “answering the mail.” In fact, the weekly review is the foundation for a more complex process that requires each of eight distinct iterative and interactive capabilities to be present at all times.


25


OSINT is a continuous process of collection, processing, analysis, sharing, feedback, and expansion. Unlike secrets (for a spy, a secret shared is a secret lost), OSINT is enhanced, strengthened, validated, and monetarized by sharing.

Weekly Review . This can be as inexpensive as an intern using SILOBREAKER. The key is to create a structured foundation for reviewing the week past and the week ahead. One week is about the right amount of time for easy separation of wheat from chaff. This serves as an archival basis for detecting trends and supporting distance learning; as a current events framework for identifying materials to be entered into the virtual library, as a basis for learned discussion among experts.

Expert Forum . Most “forums” die because they permit indiscriminate participation. The fastest way to kill any forum is to allow 80% of its members to be inexperienced askers of the question, “Hi, I am new and want to know….” Expert Forums should be limited to experts who are either on retainer or agree to participate in return for access to the OPG VPN; others may be allowed to “lurk” but should not be allowed to query or comment.

Distance Learning is as essential for experts as it is for those new to the topic. Over time a biographic, events, and places registry can be created, as well as structured notes on organizations, tribes, cultures, just about any topic that requires recurring review. Every “expert” will inevitably be transferred, retire, die, or get another offer. This function is essential to achieving “organizational intelligence.”

Virtual Library . A virtual library uses an Internet table, illustrated below, to make the deep web and private collections accessible. The Internet table can be automated to permit relevance and preference rankings. Ideally the virtual library can utilize all of the standards that will enable semantic web and synthetic information applications, as well as paragraph-level citation linkages. At its best, the virtual library will include copies of all raw information in all languages, with the translation into the home language visible side by side, or if no translation is visible, ready access to online dictionaries, and a “save” button for online dictionary translations as they are done.

Shared Calendar . Trip and event reports are a lost art-form that must be revived. Every analyst would benefit from a single consolidated calendar that lists relevant conferences and meetings and trade shows taking place all over the world. In combination with the Shared Rolodex below, it should be possible for global distributed groups to help one another by ensuring that at least one local person attends and


26


reports on each event on the calendar. Of special interest should be the list of speakers, the list of exhibitors if any, and to the extent that it can be done legally and ethically, any list of participants that may be available. Content can be evaluated on the basis of posted presentation titles, abstracts, direct evaluation, and copies of the presentations where made available.

Shared Rolodex . “Knowing who knows” is a big part of what makes an expert the expert they are. However, most personal “Rolodex” directories will tend to have just a name and a number. They do not have why that person is important, a record of interactions, or any other relevant information. This needs to change, and it needs to adopt new forms of assigning “karma points” to individuals from more than one peer.

Shared Budget . As we all move forward with M4IS, the amount of money that will become available for redirection will be nothing less than astonishing. There is absolutely no reason why fifteen different collection endeavors must take place on a single tsunami, or why different parts of any government should pay three different contractors for the same information on Chinese telecommunications. A “virtual” shared budget quickly identifies redundancy and waste. It allows for the barter of shared information for redirected collection.

Shared 24/7 Plot . We are rapidly approaching a time when all information in all languages all the time will be locationally-aware, and thus subject to being “plotted” on a 24/7 map. This will also enable the rapid visualization of changes in data over time against any given geospatial framework, and it also allows alternative depictions of the same data for the same space, contingent on the preferences of the individual analyst. A shared “plot” allows for faster more nuanced discussions of what can be known and what may not be known, and serves as a shared space for making decisions about collection gaps, collection strategies, processing strategies, analytic judgments, and presentation possibilities.

At root, OSINT is about smart people creating smart organizations by sharing the burden of conceptualizing requirements, collecting all information in all languages all the time, doing multi-cultural inter-agency analysis, and then producing credible reliable intelligence that is actionable—it is useful and it leads to constructive outcomes. OSINT crosses all boundaries, and in so doing, brings us all close together and helps us both understand, and address common problems at every level of community and governance.


27


Applied Open Source Intelligence

“Intelligence must be able to tell us, down to individual personalities and neighborhoods, ‘who,’ ‘where,’ and ‘how much’ of ‘what’ is needed, and whether what has been applied has been effective. If it doesn’t know, it must have assets able to obtain and report the information within six hours of demand.”

General Al Gray, Commandant, U. S. Marine Corps

When Dr. Stephen Cambone, the Undersecretary of Defense for Intelligence, said in January 2004 that he needed universal coverage, 24/7, in all languages all the time, he was the first person at the highest levels of the U.S. Government to formally adopt what General Al Gray recommended in 1988.22 Sadly, despite various Commissions including the 9-11 Commission, as of this date the U.S. Government is still not serious about open source intelligence.23

There is a simple reason for any leader to apply OSINT. It offers the best possible return on investment (ROI) for whatever resources—be they man-hours or dollars or Command interest—that can be earmarked for this emerging discipline. OSINT is the best possible way for any mission area specialist or professional to enhance their knowledge and increase their influence.

22 General Al Gray, “Global Intelligence Challenges of the 1990’s,” American Intelligence Journal (Winter 1988-1989). At www.oss.net, Google for title.23 The appointment of an Assistant Deputy Director of National Intelligence for Open Source (ADDNI/OS) on 5 December 2006 was a step in the right direction, but this individual has no program authority, no money, and no staff. Meanwhile, the Open Source Center at the Central Intelligence Agency, a cosmetic re-definition of the Foreign Broadcast Information Service (FBIS), has absolutely no likelihood of being relevant to anyone outside CIA in the next ten years. Fortunately, the Open Source Information System (OSIS) is a national-level system belonging to the Senate-confirmed Chief Information Officer for the Director of National Intelligence, an Air Force Major General who understands that the key to open source exploitation is sharing rather than secrecy, standards rather than security. Applied OSINT will flourish outside the secret intelligence world—to the extent that OSINT is ably developed by the U.S. Intelligence Community, it will be through OSIS embracing the 90 nations forming the Coalition, rather than through the OSC/FBIS. We continue to lack a national Open Source Center under the auspices of the Department of State, a sister agency to the Broadcasting Board of Governors.


28

http://www.oss.net/


This portion of the chapter is a summary of the more detailed information contained in the draft Special Operations Forces OSINT handbook.24

Open Source Intelligence Tradecraft

The Expeditionary Factors Analysis Model

In 1988, the U.S. Marine Corps was given its own intelligence center, in part because it was not able to obtain adequate intelligence support from any national, defense, or service intelligence center, for the distinct area of expeditionary operations, a hybrid form of operation that required a unique combination of blue water, brown water, coastal zone, and deep inland intelligence. The model that I introduce here25 was created when our Marine Corps warfighters asked us to find a way to generate an automatic “paragraph 2” to any operational order. We worked with them to define both what they needed to know, and how they defined “degrees of difficulty,” an aspect of intelligence support that had never been clarified before that time.

There are five key points to “take away” from this section:

First, no mission area factor can be evaluated unless there is a clear understanding with the operator being supported as to what constitutes a high, medium, or low level of threat or degree of difficulty.

Second, no mission area factor can be considered in isolation. The military, the geographic, and the civil factors all need to be taken into account together, as a whole.

Third, the threat changes depending on the level of analysis. A tank, for example, may be a high threat technically, low tactically because of poor training and parts availability, medium operationally because of the number scattered around,, and low strategically because they cannot be sustained in battle over two weeks.

Fourth, we cannot limit our “order of battle” calculations to conventional uniformed opponents. Terrorists, criminals, and corrupt officials are all part of the “threat.” Smugglers of drugs, small arms, and people are an

24 As with other key references, follow the link from www.oss.net/BASIC. 25 Both the model and the 1991 strategic generalizations are available by following the links from www.oss.net/BASIC.


29




especially important threat in terms of unexpected detection of SOF infiltration and covert observation activities.

Fifth, open sources of information are very helpful in arriving at “strategic generalizations” against when SOF can plan for generic capabilities and expectations. It is helpful to know, for example, that 1:50,000 combat charts for the Third World are generally not available from the National Geospatial Agency, but they are available from the Russian military through a commercial vendor based in the US. It is helpful to know that most of the time you are going to be under canopy, with a line of sight distance of less than 1000 meters, in hot humid temperatures, and so on.

Now let’s look at the high-level depiction of this model.

Figure 11: Integrated Intelligence Analysis Model

Beneath this model, for each mission area, we went on to define “degrees of difficulty.” I have to stress that this advanced model of analysis has not yet been adopted by the U.S. Intelligence Community, which still lives by the “list theory” of threat reporting—long lists with very little in the way of integrated analysis of training, logistics, terrain compatibility, or related factors. Of


30


course this is only when they realize a threat exists, and their fancy satellites can actually find the things on the ground. Our ability to support you when you go in harm’s way in what are quaintly called “lower tier” countries—this is code for “not important enough to warrant classified coverage”—is severely limited. OSINT—and direct covert human observation—may be your primary INT, not just a supporting INT.

What I have found is that a common understanding of “degrees of difficulty” is helpful in evaluating the threat intuitively and holistically, rather than in a cookbook or “list” fashion. It also allows you to create a “country profile” that in one snapshot tells anyone in the SOF family precisely where that country fits in terms of the threat, the environment, and the logistics challenge. It also allows the leader to focus in right away on the areas where command attention is needed, for example, the lack of maps, the need for hard language interpreters, the urgency of having man-portable air defense, active camouflage, and so on.

Below is just one example, for aviation, of how “degrees of difficulty” can be used to distinguish among special aviation threat environments.

Difficulty Description Representative Countries1 No substantive air OOB or air defense Grenada, Surinam, Liberia, Papua NG2 DC-3/props, early anti-air guns Haiti, Lebanon, Uganda, Sri Lanka3 Day jets, hand-held SAMS Venezuela, Oman, Sudan, Bangladesh4 Early radar, early Smart, EW radar Nicaragua, Ethiopia, Thailand5 3rd Gen. radar & SAM, stand-off PGM Cuba, Libya, Angola, Viet-Nam6 NATO/Bloc level night all-weather Egypt, Iran, Syria, N. Korea, China

Figure 12: Degree of Difficulty Example (Aviation)

Available systems must always be evaluated in relation to the specific mission and specific location. Many tanks around the capital city might represent a high threat there, but if there is no cross-country mobility, the bridges in your AO are limited to ten tons, and there is zero likelihood of the tanks being moved into your immediate vicinity, then this is an irrelevant threat. It is not enough to ask “how many of what do they have?” You also have to know where they are, whether they are trained up, gassed up, loaded up, mobile, and capable of being a factor in your specific operation.

Available systems also have to be evaluated based on the level of analysis that you need to apply to your mission. Below is one example, that of the Libyan T-72 tank in the late 1990’s, at which time the standard U.S. Intelligence


31


Community evaluation was that this represented a “high” threat to any US force entering the country.

Level of Analysis Description Threat LevelStrategic Insufficient supplies to keep active over 2 weeks. LOW

Operational Significant numbers scattered around the country. MEDIUMTactical Badly trained troops, stored in open, cannibalized. LOW

Technical Latest available Soviet technology and armor plate. HIGH

Figure 13: Threat Changes Depending on Level of Analysis

Don’t ever let anyone tell you something is a “high” threat. Ask them where, exactly, and at what level of analysis. More often than not, it will reduce the threat. Sometimes it will cause you to take the threat more seriously.

Within this model, and the information you obtain from conventional classified sources, it is extremely important to be very sensitive to the distinction among four distinct types of forces: government forces; para-military forces (some government, some not); tribal forces; and criminal forces. Most of our all-source intelligence is relatively strong on the first and weak on the remaining three. The latter three forces are very important to you primarily because they are more likely to be both invisible to our technical collection systems, and also most likely to detect your infiltration and oppose you or report you when you least expect to be noticed.

Tribal and criminal gang studies are especially interesting in the context of learning about open sources of information because it turns out that classified sources are next to useless in creating orders of battle for sub-state or non-state actors. This is where you have to really learn how to go after academic, law enforcement, commercial, and non-governmental organization (NGO) sources.

Below is an illustration of one high-level look at the spheres of influence of specific tribes relevant to the US efforts to construct the road from Kabul to Kandahar, with a view of the specific Afghan tribes in the inset box.


32


Figure 14: Tribal Areas of Influence

Finally, we come to the concept of strategic generalizations.

I like to tell every generation of warfighters that they are not only responsible for fighting the force they are given, but they are also responsible for putting in the requirements to create the force that will be fought by the next generation.

It helps to know that your world is one quarter jungle, one quarter mountains, one quarter urban, and one quarter desert. You must be trained, equipped, and organized for all four.

It helps to know that cross-country mobility is next to nothing in 80% of your world; that bridge loading constrains most heavy offensive weapons systems from being a factor in the Third World; that line of sight distance is generally under 1,000 meters; that three-quarters of the time the remote satellites cannot see below the canopy; that rain is common and your unmanned aerial vehicles really do have to be all-weather capable; that the Navy cannot get your there half the time; that just about everywhere is C-130 capable and we do not ever want to give up that platform; that your standard aviation temperature is not


33


what the Navy and Air Force build to (warm and not humid), but rather hot and humid—this has severe implications for range, load, and loiter.

In short, OSINT will help you understand the real world better, and help explain to your acquisition program managers why they need to fight for small war and unconventional capabilities. Use this model to improve the value of OSINT to what you do.26

The Revolutionary Analysis Model

The day of being able to simply do “Order of Battle” analysis is over. This is especially true for those responsible for Civil Affairs (CA) and Psychological Operations (PSYOP), but it is also true for every mission area specialist, for every Team, for every Commander.

In addition to the Expeditionary Factors Analysis Model introduced earlier, it is necessary for the all-source intelligence analyst as well as the CA/PSYOP specialist to utilize a broader framework for thinking about sources of revolutionary change across every dimension of a society.

This framework is created by combining societal dimensions with personality aspects to create a matrix for structured evaluation and reflection. This matrix has the added advantage of helping to frame potential Civil Affairs lines of inquiry, and potential Information Operations or Strategic Communication themes in support of the larger campaign plan.

The societal dimensions include:27

Political-Legal. Who governs whom, how, to what end.

Socio-Economic. Allocation of goods and services in process of managing the nation.

Ideo-Cultural. Spiritual means of co-coordinating the population, managing inter-relatedness.

26 From www.oss.net/BASIC, a number of specific generalizations about military, civil/cultural, and geographic conditions for which we must structure the future force, are outlined in MCIA documents.27 These are all as developed for an unpublished graduate thesis, “Theory, Risk Assessment, and Internal War: A Framework for the Observation of Revolutionary Potential,” (Lehigh University, 1 April 1976).


34



Techno-Demographic. Technology and human resources as instruments of the state or tribe.

Natural-Geographic. Water, energy, mineral, agricultural factors, geographic access.

The personality attributes, a big part of the “why” of revolution, include:28

Perception. Ability to see discrepancies between what is and what might be.

Identity. Ability to recognize one’s own limitations and inherent external limitations.

Competence. Result of solid perception and solid identity.

Investment. Authentic and intense dedication to a common good or larger purpose.

Risk taking. Ability to risk one’s self and one’s beliefs in open confrontation.

Extroversion. Enabler for investment and suspension, opening way to interact with others.

Transcendence. Result of all of the above, as the immediate sub-set group is energized.

Synergy. Elevation of the larger group as sub-sets interact and reinforce one another.

Complexity. Reconciliation of contradictions and larger integration of all elements.

Don’t get bogged down with nuances. Just absorb this as a frame of reference. However, do note that each of these conditions has been operationized, i.e. specific sources, thresholds, and criteria exist for defining the degree to which each revolutionary condition is present and how it changes over time.This is what the shorthand matrix of indicators looks like:

28 These are drawn from Charles Hampden Turner, Radical Man: The Process of Psycho-Social Development (Anchor, 1971).


35


Figure 15: Framework for Analyzing & Predicting RevolutionI want to make one final point here, and that is this: governments are not good at monitoring all of these factors within their own country, much less other


36


countries and sub-state sectors. You will not succeed at OSINT on revolutionary factors unless you harness the distributed knowledge of foreign nationals and private sector parties. OSINT requires networking at the collection level, at the processing level, and at the analysis level. Ideally, OSINT also leads to multinational policy understanding, not just unilateral appreciations.

Analytic Tradecraft

Mr. Jack Davis is the de facto dean of the U.S. national intelligence community’s analytic cadre. In 1995 he published a series of notes on what is now known as “analytic tradecraft”, and in 1997 these were published by the Central Intelligence Agency and have become the standard reference on the subject. Below we offer a summary of these notes, recommending that the work of Mr. Davis be read in its entirely.

Analytic Tradecraft #1: Trade-offs analysis in client context Greatest value of analysis is strategic--helping evaluate trade-offs

in policy and acquisition and operational costs Analysis must be done in client context but with external views

carefully represented

Analytic Tradecraft #2: Nine evaluation criteria for credibility Relevance to policy Analytic sophistication Unique information Timely and concise Facts/sources are stated Conclusions are offered Assumptions are stated Options are outlined

Analytic Tradecraft #3: Assumptions must be specified• Separate “box” for assumptions gains consumer confidence• List variables and relationships--some are factual, others not• Obtain second views• Question assumptions

Analytic Tradecraft #4: Alternative outcomes must be specified• What are the alternative outcomes?• If they occur, what signposts or events might be triggers?• In choosing one outcome over others, what judgments were

made?


37


Analytic Tradecraft #5: Facts, opinions, sources• Distinguish between direct & indirect sources• Be precise about sourcing of information• Understand policy sensitivities• Respect complexity• Account for deception• Label judgments

Analytic Tradecraft #6: Harnessing internal and external expertise

• Must rise above the competition--and use external experts• Demonstrate research, scholarship, database access, ground truth,

teamwork, experience• Customize intelligence for specific individuals

Analytic Tradecraft #7: The art of distillation and summarization

• 30-second lead buys you 3 minutes for summary which might get them to read it all

• Focus on what’s new• Make it easy to read• A few points, clearly• Role play the audience

Analytic Tradecraft #8: Deception, counterintelligence, timeliness

• Account for denial & disinformation• Gaps in collection?• Odd contradictions?• Odd confirmations?• Use textbox to show reader that steps were taken to consider &

defeat deception

All analysts, however junior, are managers: they are managers of customer relations; of money for open sources; of external experts; and of classified collection. The purpose of analysis is to help key individuals make intelligent decisions. 29

Social Networking & Expert Networks

There are two separate concepts that will help you connect to humans who know stuff. The first, originally popularized in the 1960’s and 1970’s as the 29 The compendium, the “New Rules” chapter from The New Craft of Intelligence, and a lecture on “Analysis: Making Magic” are all accessible via www.oss.net/BASIC. .


38



“small world phenomenon” or “six degrees of separation,”30 refers to the very high likelihood that you already know someone who knows someone….out to a sixth person who knows exactly what you need to know. And if you do not know someone directly, it is very likely you can find someone resident in the U.S. whom you can reach, that does know someone who knows someone…you get the idea.

A very famous example of this comes from President Bill Clinton and his need to know, the night that the U.S. bombed the Iraqi intelligence headquarters, whether our strike had been successful. Our (then) $35 billion a year intelligence community could not tell him. We had no one on the ground and the satellites were not within range of the target for a thermal reading. President Clinton, without telling anyone, called CNN, CNN called its Jordan bureau, whose cameraman had a cousin who lived near the intelligence headquarters, who confirmed the strike." Four calls.31

Operationally, this can be useful as a means of helping you think about how to obtain both in-depth historical and cultural understanding for a specific mission area, and specific tactical information that might not be available through current online or offline sources—what is called “local knowledge” that has not been published and is only known to those in the immediate area. An example of this local knowledge might be which water wells are free of bodies and safe to drink from, and which have recently had bodies dumped in them and are therefore dangerous.

If you think back to both the expeditionary factors and the revolutionary analysis models, you can easily conceptualize a number of categories of individuals whom you might get in touch with through either direct personal contacts or indirect web and other directory identification.

For example, and skipping over the obvious availability of the U.S. defense attaché and foreign military assistance teams as well as any ground walkers that are or have been in country recently:32

30 Stanley Milgram, “The Small World Problem,” Psychology Today, 1967. Google “small world problem.” 31 As told by Dick Clarke, Against All Enemies (Free Press, 2004), page 84.32 It is a common misperception among US military personnel that they are not allowed to call strangers to ask for information. Maj Mark Coffin, USA, then Officer in Charge of the OSINT Cell within the Special Operations Command Joint Intelligence Center (SOCJIC) personally visited the office of the Assistant Secretary of Defense responsible for intelligence oversight, and determined that calls can be made provided that there is no misrepresentation by the uniformed person, and they identify


39


Military Vendors. Who is selling military hardware, software, vehicles, and munitions into the target area?

Airlines. What American airlines are flying into the area in question and how do you get hold of the pilots and air crew?

Ships. What US-based companies are running merchant shipping in and out of the ports, and what kind of access can you get to their officers and crew?

Political. Who is the U.S. Government or U.S. labor union person that has been in-country most recently to help the local democratic party get organized?

Economic. Who is running the US Chamber of Commerce in that country? What US companies have a US manager in place?

Media. What journalists are writing extensively on that country, either from in-country or as regular visitors?

Academic. Where are the US, UK, and Australian centers of excellence on this specific country? Who can you talk to there? Don’t overlook the archeologists, anthropologists, and medical researchers—a lot of field work is done by these groups.

Religion. What evangelical or other religious missions have gone to the country from the US recently?

Citizens. Who is running tours into the area? Don’t overlook the fanatical climbers and cavers who are willing to bribe their way into a dangerous area, as in Indonesia.

The above is not intended to suggest that you spend a lot of time “shot-gunning” for information, only that you be very aware of two realities: first, there are no clandestine assets immediately available to help you understand the specifics of your target area; and second, there are a significant number of people going in and out of that area who could be tapped into where the situation warrants, using cover support plans and commercial cut-outs.

This concept overlaps with the second concept, that of “citation analysis” or “the web of knowledge.” Mr. Eugene Garfield, founder of the Social Science Citation Index and the Science Citation Index, is the originator of this

themselves. SOF also has recourse to commercial intelligence support specialists, who can meet Operations Security (OPSEC) needs by not identifying the client interest, and converting any SOF OSINT requirement into a graduate paper by a foreign student.


40


extraordinary method for both identifying networks of experts, and discovering the very latest knowledge in any field.

Mr. Garfield’s genius was in realizing that the best and most relevant work tends to get cited by others, and that by tracking citations, he could do three things:

1) Based on numbers alone, determine the influence of the article or book.

2) Based on tracking specific relationships between articles and/or authors and/or their parent organization, identify networks or the “web” of knowledge and how it was growing, changing, or responding to external influence.

3) By providing a link from past publications known to be “on target,” by following citations of that work, rapidly identify the latest work, within a month of its publication, which might not have been discoverable any other way. This is especially true for work in adjacent fields, for example, discovering useful recent anthropological studies highly relevant to understanding Muslim politics in southern Thailand.

For any given topic or country, for $1,000, we routinely identify the top 100 experts on anything, based on citation analysis. In turn, the published experts know who the unpublished experts are (e.g. within governments and non-governmental organizations).

Our model, using real-world experts with localized knowledge, is vastly superior to the beltway bandit model, which seeks to profit from overhead at the client’s expense, using “butts in seats” and the Library of Congress in lieu of doing the hard work of actually “knowing who knows.”

Mission Relevance of Open Source Intelligence

This section must of necessity be abbreviated. Twenty-five pages replete with search examples using only Google, are available in the free SOF OSINT Handbook.

Strategic Historical & Cultural Understanding addresses the critical importance to any SOF mission of going back in time to understand the history of the region, the history of foreign powers as well as the US in the region, and


41


the history of Anti-Americanism is the region. If there is one thing SOF cannot afford when going in-country, it is to be delusional about just where we stand as we go about trying to win hearts & minds or capture single hostile individuals in a context where we do not realize the odds are stacked against us. The greatest threat to SOF missions is not armed forces but rather hostile observers. Understanding history and culture is fundamental.

Operational Understanding for Campaign Planning connects open sources of information to the theater level of warfare, and helps develop an understanding of open sources in relation to the current situation. Regional power sources, status discrepancies among tribal groups, change agents that are present or emergent, internal security and stability issues (water, food, energy, health, crime, for example) are all essential to understanding the weak links in a current social structure that SOF can either leverage for operational advantage, or must be aware of to avoid operational failure.

Tactical Sub-State Understanding for Unit Effectiveness gets to the heart of the matter for SOF units that will be working in-country. This chapter focuses on tribal orders of battle down to the village and elder level, on key leaders and value-based biographies, on understanding the local media and how groups and individual communicate with one another, and finally, on content analysis—understanding their PSYOP themes.

Technical Understanding for Policy, Acquisition, & Operations begins with an introduction to the NATO Open Source Intelligence Handbook, which is the technical reference and companion to this volume, and then provides a very brief overview that relates open sources of information to policy, acquisition, and operations in general. One of the great things about OSINT is that it can be used to study domestic US policy debates as well as allied debates. Understanding the players, both friendly and third party, and understanding how the players are perceived locally, is at the heart of any successful CA or PSYOP endeavor. OSINT can also enhance SOF acquisition, and help SOF decide what to leave on the pier and what to take along on the mission.

Mission Area Applications

Thirty-three pages with additional detail are in the SOF OSINT Handbook.

Civil Affairs can use OSINT in relation to human intelligence (understanding the demographics, the socio-economic environment, displaced persons, and crime, among other topics); to technical intelligence about the local command & control, communications, computing, and intelligence environment, the


42


infrastructures of transportation, power, and finance; to welfare intelligence (water, food, medical); cultural intelligence about protected or restricted targets, and liaison intelligence.

Psychological Operations can use OSINT in relation to strategic, operational, and tactical campaign plans; revisits the mapping of themes in play, especially anti-U.S. themes; the original collection and testing of themes for possible U.S. play, and the litmus test for successful PSYOP: does the message produce actionable intelligence from indigenous volunteers?

Target Analysis discusses how OSINT might fulfill team needs in the absence of classified intelligence support, to create a detailed description and vulnerability assessment, evaluate the natural environment and the human environment, and carry out route planning.

Terrain Analysis uses OSINT to establish key factors relevant to special aviation and covert ground movement, in part by leveraging commercial and Russian military combat charts, commercial imagery, and alternatives for terrain reconnaissance including unmanned aerial vehicles and indigenous scouts.

Weather Analysis uses OSINT as a means of rapidly getting to the basics of temperature, visibility and timing of sun and moon, wind, and inclement weather.

In addition to the SOF OSINT Handbook, see the Quick Links Guide for the Military Analysis, included in the one-page list of links at www.oss.net/BASIC.


43



Conclusion

Money Matters

Funding Trade-Offs. As our larger world comes to grips with the end of cheap oil, the end of free clean water, the rise of pandemic disease, the twin deficits and rampant militarism of the USA under the Bush-Cheney Administration, the bottom line is clear: we have no slack left, we are at a tipping point, every mistake could be fatal. It is no longer adequate to muddle through, drawn down on savings, or “make the best of it.” Any manager, any person, that does not invest the time and as needed the money to make informed decisions using OSINT, is derelict in their duty to their employers and themselves. OSINT is now an established discipline required for “due diligence.” Perhaps more importantly, information is a substitute for time, money, labor, and space. Practicing OSINT is a way of printing your own money! Practicing OSINT is also a means of restoring power to the people, allowing them to better hold accountable their policymakers and corporate executives all too inclined to manipulate or ignore secrets, or claim special knowledge that does not exist, as a means of justifying actions and expenditures that are not in the public interest.

Contracting Mistakes. As a general comment, we have found that the biggest failure among both government and private sector clients is that of almost total ignorance with respect to the diversity and quality of open source services, and most especially of those offered by foreigners in their own localized environments. Even those organizations that have the wit to contract for a variety of open source service support, generally do not have a single focal point nor do they attempt to monitor best prices and best practices. The worst possible mistake is to attempt to procure OSINT from a major defense corporation that specializes in massive expensive projects to deliver technology that often does not work and “butts in seats,” rather than niche expertise or direct access to all information in all languages all the time. It is also a mistake to contract for the delivery of OSINT without making provision for a working requirements process that will save time and money by getting the questions right in the first place, or to contract for the delivery of OSINT in hard-copy, without making provision for its delivery in a form that will allow its easy dissemination throughout the sponsoring organization’s network. A more nuanced contracting mistake is to avoid seeing that information, once purchased, has a tangible value that can be used to barter for more information. Copyright issues not-with-standing, a coherent program for sharing information with varied members of the seven tribes in one’s own home country, and with


44


counterpart organizations from other countries, will generally produce a ten to one return on investment—ten new useful pieces of information for each single piece of information that is shared broadly.

Metrics for Measuring Return on Investment. There are four valuation metrics that can be applied in evaluating the role of OSINT in any organization’s Information Operations (IO).

Cost of Secrecy. Transaction costs are higher. Classification reduces competition from nee and foreign providers of better information. Functional costs come from non-interoperability and operational disconnects. Clients tend not to access all that is offered because of the obstacles imposed by handling secret information (e.g. reading on a trip).

Relative Value. Is the OSINT “good enough” now? Does it provide, in context, “good enough” understanding to move forward? For the decision at hand, it is “good enough” to allow the decision to be made? Can the information be shared and thus engage other stake-holders?

Return on Sharing. Does this information, shared openly, attract other information that is equally useful? Does this information, shared openly, reach others who have a “need to know” and consequently include them and engage them in an expanded network for mutual benefit?

Commercial Strategy. Dr. Joseph Markowitz, the only truly competent manager of open source information endeavors within the U.S. Intelligence Community, published a commercial strategy prior to resigning from government service. It has yet to be implemented.

Budget and Manning Recommendations. Detailed proposed budgets are online33 for a national Open Source Agency (OSA), a theater Multinational Information Operations Center (MIOC) and network, and a subordinate commercial imagery and geospatial procurement plan. It remains then, to simply illustrate a standard OSINT “cell” such as could be added to any corporate or government library, with the observation that OSINT should be accomplished in three tiers:

If it can be done online in less than 15 minutes, the analyst should do it.

33 See the links under Policy & Investment at www.oss.net/BASIC.


45



If it will take 15-60 minutes, or require specialized knowledge, the OSINT Cell should receive the task.

If it will take more than 60 minute or require very specialized knowledge or direct access, it should be out-sourced to exactly the right source or service, by the OSINT Cell, which should be expert at best prices and best practices for all sources in all languages all the time.

Figure 16: Standard OSINT Cell

The Value of Sharing

We have, as J.F. Rischard puts it so well in HIGH NOON: 20 Global Problems, 20 Years to Solve Them,34 reached the point of no return. OSINT is relevant to individual security and prosperity; to organizational and national security and prosperity; and to global security and prosperity. He writes about sharing our planet, sharing our humanity, and sharing our rule book. Tom Atlee, founder of the Co-Intelligence Institute and author of The Tao of

34 (Basic, 2002). The author is Vice President for Europe of the World Bank.


46


Democracy: Using CO-INTELLIGENCE to create a world that works for all35

adds another group to these three groups: sharing our wisdom. Sharing our wisdom. That is what distinguishes OSINT from the secret collection disciplines, and that is what distinguishes the role of OSINT in the world of analysis: it can be shared without restriction. OSINT is democracy. OSINT is moral capitalism. OSINT will make our lives better and offer hope to future generations. E Veritate Potens.36

ReferencesVisit http://www.oss.net and see especially http://www.oss.net/BASIC. See also the books by Robert Steele:

ON INTELLIGENCE: Spies and Secrecy in an Open World (Foreword by Senator David Boren, D-KS), first published in 2000.

THE NEW CRAFT OF INTELLIGENCE: Personal, Public, & Political—Citizen’s Action Handbook for Fighting Terrorism, Genocide, Disease, Toxic Bombs, & Corruption (Foreword by Senator Pat Roberts, R-KS), 2002

PEACEKEEPING INTELLIGENCE: Emerging Concepts for the Future (contributing editor, Foreword by Dame Pauline Neville Jones), 2004

INFORMATION OPERATIONS: All Information, All Languages, All the Time—The New Semantics of War & Peace, Wealth & Democracy (Foreword by Congressman Rob Simmons, R-CT-02), 2006

AbbreviationsAbbreviations are included in the Glossary to this Handbook.

35 (The Writer’s Collective, 2003). Observation made in a personal communication (electronic mail) of 18 March 2006. The Co-Intelligence Institute merits more attention and support.36 This is the motto for OSS.Net, Inc., and before that for the Marine Corps Intelligence Command which the author helped create. It means “from truth, power” or literally, “one is made powerful by the truth.” Thus does OSINT contribute to the power of every individual regardless of their race, nationality, religion, or station in life.


47


http://www.oss.net/

Documents

Open Source Intelligence (OSINT) - OSS.Net, Inc. … · Web viewOpen Source Intelligence and Joint or Coalition Operations OSINT is a vital component of NATO’s future vision. Through