Air Force Material Command

USAF T&E Roadmap:

A Key Driver in Test Capability Development and Modernization

Steven W. Leas(1) United States Air Force, Air Force Flight Test Center, Modernization Planning Manager

AFFTC/XPR, Edwards AFB, CA 93524

Rachel Waddell(2) United States Air Force, Air Armament Center, Chief, Strategic Initiatives 46th TW/XPX Eglin AFB, FL

In an effort to meet the challenge of balancing limited investment funds between operations of today and the requirements of tomorrow; the AF DT&E community is developing a test and evaluation way ahead or roadmap. The goal is to lay out a “roadmap” which will characterize the health of current T&E capabilities as well as highlight future capability shortfalls. This tool in consideration with a stringent set of processes will inject discipline into the investment decision making process. In addition it will allow for stronger traceability back to warfighter’s needs.

The AF T&E Roadmap will serve many functions. As a single repository for capability requirements and associated shortfalls, it will provide resource planners explicit insight into capability gaps and the timeframe in which the gap needs to be filled to meet customer requirements. It would allow planners from various locations to collaborate on common solutions much like the Reliance Panels seek to do across service lines. Additionally, test capability owners would have insight into Enterprise resources and test capabilities, leveraging existing capabilities at other test locations when possible. Maturation of the AF T&E Roadmap and associated processes will provide the means and information to effectively support prioritization of investment proposals through the cascading requirements process; providing traceability back to high-level AF requirements. The prioritization process will focus on project value to the stakeholders, examining projects from both a technical and business perspective. Lastly, as a living document the roadmap will directly support regular re-examination of planned investment projects, ensuring their on-going alignment with evolving requirements. The AF must choose wisely to ensure projects yield maximum return on investment while balancing today’s mission against the needs of the future. This will be accomplished through the implementation of a rigorous set of processes and tools such as the T&E Roadmap. This combination of processes and tools will ensure effective management of the T&E test infrastructure and capabilities.

_

1 Senior Analyst; AFFTC/XPR, Edwards AFB, CA 93524, AIAA Member 3 yrs. 1Senior Analyst, AEDC/XPR, Arnold Engineering and Development Center, TN 39389, AIAA Member 3 yrs.

1 1

U.S. Air Force T&E Days 5 - 7 February 2008, Los Angeles, California

AIAA 2008-1686

This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.

Nomenclature 46th TW =46th Test Wing (Eglin AFB, FL) AEDC =Arnold Engineering Development Center (Tullahoma, TN) AFFTC =Air Force Flight Test Center (Edwards AFB, CA)

AFMC =Air Force Material Command Wright-Patterson AFB, OH) CTEIP =Central Test and Evaluation Investment Program I&M =Improvement and Modernization MRTFB =Major Range and Test Facility Base PE =Program Element T&E =Test and Evaluation TIPP =Test and Evaluation Investment Planning and Programming TRMC =Test and Evaluation Resource Management Center NOTE: The processes described herein are the views of the authors and are not the official position of the Air Force, the Air Force Material Command or the Air Force Flight Test Center. The authors reside within Air Force Material Command (AFMC). Consequently, much of the information contained within this document predominately reflects the concepts and processes of AFMC organizations and the AFMC enterprise.

I. Introduction he cornerstone of all enduring industries lies in the strength of leadership’s ability to balance today’s needs against tomorrow’s vision. This also holds true for the test and evaluation (T&E) of military weapons systems. The Department of Defense (DoD) attempts to meet future test capability infrastructure requirements through a

combination of joint venture and service specific programs. These programs are referred to as improvement and modernization (I&M) or “investments”. Just as industry has been required to adapt a strategic approach and long term strategies to compete in a global market; the DoD must make tough decisions between executing the mission of today and preparing for the future. Limited funding across the DoD is driving a need for greater precision and more effective resource management, including investments. In an effort to meet these challenges, the AF DT&E community is developing a test and evaluation way ahead or roadmap. The goal is to lay out a “roadmap” which will characterize the health of current T&E capabilities as well as identify future requirements. This tool in consideration with a stringent set of processes will inject discipline into the investment decision making process. In addition it will allow for stronger traceability back to warfighters’ needs. This paper provides insight into both the USAF and DoD T&E I&M programs, and highlights the importance of the USAF T&E roadmap in identifying and prioritizing I&M solutions to T&E capability shortfalls.

T

II. T&E Resource Management The majority of the Air Force’s T&E infrastructure resides within AFMC and is spread across the three primary

test centers, the Air Force Flight Test Center, 46th TW at Air Armament Center, and the Arnold Engineering Development Center; with a fourth tenant activity, the 46th Test Group, located at Holloman AFB. The annual I&M budget across AFMC averages one hundred five million dollars per year (Figure 1). This budget addresses not only improvement and modernization of existing capabilities, but is also intended to fund development of new capabilities. Projected T&E infrastructure requirements across AFMC are expected to run into the billions of dollars within the next several years. But this should come as no surprise as the Operational Air Force faces the same issues with fleet recapitalization. This clearly highlights the need for smart decision-making when it comes to allocation of I&M funds, as well as effective use of existing resources both within and across the Services.

This paper focuses on AFMC's T&E resourcing processes, although similar resource programs exist across DoD. The T&E resources are broken down into two primary categories within AFMC, T&E Operations and T&E Investments. Although the focus of this paper is on investment accounts, operational accounts are briefly covered. Operational funds or operations and maintenance (O&M) fall into three accounts: T&E Support, Restoration and Modernization, and Facility Sustainment. T&E Support (Program Element 65807) funds civilian pay and all non-pay requirements associated with the daily operations of an organization. This includes contractor labor, supplies, utilities; aircraft operations, test support aircraft costs; facility operations such as ground test tunnels, cells, chambers, track, and other testing facilities; preventative maintenance of test equipment, photolabs, x-ray facilities, and data processing. Restoration and Modernization (Program Element 65976) funds repair, replacement, alterations

2 2

and modernization of facilities, while Facility Sustainment (Program Element 65978) funds real property maintenance. The AFMC T&E O&M accounts account for 85% to 90% of all AFMC T&E dollars (see Figure 1).

$-

$200

$400

$600

$800

$1,000

$1,200

FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13

Thou

sand

s

AFMC T&E

AFMC T&E I&M

Figure 1: AFMC T&E Resource Funding Profile

III. T&E Investment Programs There are multiple investment programs within the AF and DoD as shown in Figure 2. Each funding program

targets specific capability thrusts, some narrower than others. In virtually all cases, due to the number of resource requirements (ie. gaps in capabilities), competition for available funds is intense.

The majority of I&M projects shown in Figure 2 are associated with one of two major programs, the AF T&E Investment Planning and Programming (TIPP) process or the OSD Central T&E Investment Program (CTEIP). Within AFMC, resources are planned and programmed using the TIPP process, which is centrally managed by AFMC/A3F with oversight from HQ AF/TER. The TIPP process facilitates the identification of capability “Needs” and associated “Solutions” against one of two funding sources, Major T&E Investments and Threat Systems Development. Proposed Needs and Solutions packages are ultimately prioritized, with select projects recommended for funding within available projected budget. The TIPP process is executed on a two-year cycle. Solicitation for new Needs and Solutions proposals occurs in February of each odd calendar year.

Figure 2: T&E Investment Programs

3 3

Major T&E Investments (PE 759) funds T&E investment projects in all T&E capability areas, excluding Electronic Warfare (EW). It supports airframe, propulsion and avionics: armament and munitions; space; and command, control, communications, computers and intelligence test areas. The program is intended to help test organizations keep pace with emerging weapon system technologies through the creation of new test infrastructure, while also modernizing existing T&E capabilities. However, the task of satisfying both requirements with limited funds drives the need to shrewdly balance between competing priorities.

Threat Simulator Development (PE 256) funds projects which develop representative high fidelity threat environments via digital modeling and simulation, hardware in the loop, multi-spectral simulation and integration of live, virtual and constructive capabilities. Electronic warfare infrastructure is located throughout the T&E Enterprise. The program also provides funding to support the AF Foreign Material Program, which supports the acquisition and exploitation of foreign material. Threat Simulator Development faces the same challenge of balancing new capabilities while modernizing existing capabilities.

The Central T&E Investment Program (CTEIP) (PE 940) is the DoD managed T&E investment program. Oversight is provided by the Test Resource Management Center (TRMC) and HQ AF/TER leads all AF multi-service programs. The principle focus of CTEIP is the development of T&E capabilities which promote range commonality and interoperability across the Services. CTEIP is further broken down into two project categories: Joint Improvement and Modernization (JIM) and Resource Enhancement Program (REP). JIM projects address critically needed investments in major functional areas of T&E and comprise the bulk of CTEIP funds. JIM projects must address joint/multi-service needs. Projects include automated data collection and advanced data transmission, embedded instrumentation, advanced sensors, modeling and simulation and advanced time-space position instrumentation. REP projects are tailored to resolve near-term shortfalls when the timeframe between identification of a test need and critical test dates does not allow sufficient time in the budget cycle to program for the required capability.

In addition to CTEIP, the Science and Technology for T&E (T&E/S&T) program (PE 941) funds projects which exploit new technologies and expedite their transition from the laboratory to T&E capabilities. The program is managed by TRMC and leverages ongoing efforts in DoD, academia and industry. T&E/S&T are intended to serve as risk reduction for future CTEIP projects. Other CTEIP and OSD –supported investment programs include Target Management Initiative, which applies new technology to improve target threat representations in support of T&E and training, and Threat Simulator Investments, which anticipate threat simulator shortfalls by applying new technologies and innovations to improve threat representations.

Looking across the multiple I&M programs described above one may assume the T&E infrastructure is and should continue to be healthy; however, rapidly advancing technologies, escalating costs, and limited funding have led to a continuous disconnect between needs and funded solutions. In addition long programmatic lead time (up to 8yrs) and the lack of common priorities severely limit programmatic flexibility in responding to emerging requirements.

IV. Effective Investment Program Management and the USAF T&E Roadmap Much like a roadway map offers multiple options for reaching the desired destination; the T&E Roadmap will

provide useful, effective information which will allow us to make smart decisions on the most prudent route to take. Figure 3 shows how the relationships between independent processes will link to form a cohesive resource or PPBE cycle. The left side of the diagram reflects the strategic/ roadmapping portion of the cycle, while the resourcing and execution focused activities are shown inside the dotted oval. Roadmap output or needs are prioritized and fed through the Analysis of Alternatives process which forms the bridge between the planning and execution activities. From this process comes a set of prioritized solutions. From that point the solutions are sourced and fed into the capabilities portfolio.

4 4

Figure 3: T&E Resource Management Process One of the key attributes of the mature roadmap will be the creation of a single repository which will document

all T&E validated capability requirements, characterize existing capabilities, and highlight capability resource shortfalls. Any capability requirements which cannot be met due to resource shortfalls (“gaps”) will be highlighted. Additionally, a single repository will enable greater insight into existing resources across the enterprise. This will provide capability owners insight into available resources/test capabilities, allowing them to consider the option of leveraging an existing capability at another test center to meet a local resource shortfall. A single repository will also allow resource planners from various centers to collaborate on solutions to common requirements, much like Reliance Panels seek to do across service lines. In the example shown in Figure 4 (next page), possible funding solutions have been earmarked against requirements, based upon inputs from resource planners. Bottom line, resource information, including resource requirements, existing capabilities, and resource shortfalls ("Needs") represent the principle inputs into the investment planning arena. Clear insight into explicit T&E resource information would facilitate more effective resource planning and budgeting.

5 5

Figure 4: Example of Capability Functional Analysis

The above capability based functional breakdown Figure 4) provides an example of a product which would

provide capability owners the ability to characterize and track requirements within their given capability area. Such a functional capability analysis would be important to resource planners as it could provide explicit insight into capability gaps and the timeframe in which the gap needs to be filled to meet customer requirements. Without such products significant time can (and has been) lost in ferreting out needs.

While identification of capability gaps is a principle input to the investment planning arena, the greatest challenge in the face of budget constraints lies in prioritization of investment proposals associated with defined gaps.

Currently, investment planning is largely accomplished via a set of federated processes unique to each test center. Identification, prioritization and coordination of TIPP and CTEIP Needs and Solutions occur independently at each center. Center-specific proposals are first introduced to command during the bi-yearly Spring Investment Planning meeting. Proposals are then reviewed by independent multi-service Reliance Panels which verify requirements and validate solutions, ensuring that no unwarranted duplication of capabilities exists across services. Once validated by Reliance, a TIPP Summit is convened by HQ AFMC, with representation from the test centers and HQ AF/TER. Proposals are briefed, discussed and ranked within their specific program element (Major T&E Investments or Threat Simulator Development). The prioritization process and criteria have been fluid during the last several cycles, each round employing a modified set of criteria to determine relative ranking of proposals. Examples of particular criteria include: competitiveness, effectiveness, jointness, bang for the buck, executability, risk, and emphasis on modeling and simulation. While each test center’s internal investment prioritization process may use slightly different criteria or varying terminology, they all basically address these common themes:

6 6

1. What is the value (or impact) to the stakeholders?

a. Warfighter b. Program Office c. Test Center (MRTFB)

2. How well does the solution address the capability requirement (gap or disconnect)? 3. Is the solution executable?

These three considerations should be examined from both a “technical” and a “business” perspective, as fiscal

implications often drive resource decisions. This is particularly true in considerations one and three. For example, the value of enabling a weapon system to meet mission requirements through effective T&E is a primary prioritization driver from a “technical” perspective. However, the cost of developing or modernizing the required T&E capability is certainly an important opposing “business” consideration. As a second example, improvement of a test capability which supports a moderate or non-critical warfighter objective may not have the appearance of a worthy investment, however, if that capability significantly shortens testing, reducing schedule and cost risk, it may be deemed a worthy investment. Similar thinking can be applied to criteria three. Development of a capability which supports an important warfighting requirement, which in turn supports a key AF CONOPS objective, may merit a high priority, but if the investment solution is very expensive and contains significant technology risks, its overall relative priority may be affected. Suffice it to say, prioritization of T&E investments for funding consideration can present a daunting task.

Development of a roadmap prioritization process is currently underway (refer to AIAA Prioritization Paper by Brown and Babilon). This process is intended to support prioritization of investments. The prioritization process (Figure 5) parallels the Roadmap’s cascading requirements traceability model, examines high-level AF requirements, and their associated priorities, as a basis for establishing prioritized T&E capability requirements and then subsequently defines gaps between existing capabilities and required capabilities.

PrioritizeT & E

CapabilitiesPrioritize

Disconnects(Gaps)

PrioritizeSolutions

PrioritizeDrivers

(Guid, Acq) S&T, Tech)

PrioritizeT & E

CapabilitiesPrioritize

Disconnects(Gaps)

PrioritizeSolutions

PrioritizeDrivers

(Guid, Acq) S&T, Tech)

Figure 5: Roadmap Cascading Prioritization

These two outputs, capability priority and assessment of capability gap, are will heavily influencing the priority

of investment “Needs and Solutions” proposals, principally from a technical perspective. They provide meaningful information on the importance of the overall T&E capability to the warfighter and the program office. When combined with additional criteria addressing technical considerations at the test center level and business considerations at the program office, T&E Enterprise, and test center levels, they establish a sound set of criteria for use in prioritization of investment proposals. These “criteria”, which embrace the three investment prioritization

7 7

themes listed earlier, are more simply represented in Figure 6 below. The criteria depicted, in conjunction with the prioritization approach outlined to the right of Figure 6, originated from the AFFTC, and were recently used at the FY 07 TIPP Summit to partition new proposals into bands as placeholders for FY 14/15 I&M funds. Interestingly, as a result of this initial prioritization process, three proposals targeting development of Network-Centric Warfare test capabilities (1 from AAC and 2 from AFFTC), along with a wind tunnel modernization project from AEDC, emerged as the highest priority proposals. These rankings were consistent with warfighter themes reflected in high-level AF strategic documentation.

PrioritizeT & E

CapabilitiesPrioritize

Disconnects(Gaps)

PrioritizeSolutions

PrioritizeDrivers

(Guid, Acq) S&T, Tech)

Prioritization Approach

- Assess common set of criteria - Utilize High, Medium, Low or Full, Partial, Minor/None as ranking criteria where applicable - Use collected info as basis of sub-jectivediscussion- Document rationale behind final ranking of each proposal

Capability Priority

Capability Status

Disconnect Consequence

Solution Impact &

Executability

CapabilityPriority

T1, T2, T3Capability

AssessmentR, Y, G

What is impact of

disconnect on

capability? F, P, N(Include

affect on R, Y, G)

Disconnect consequence

to: -T&E Center-- SPO-Warfighter-H, M, L

How well does solution address

disconnect?F, P, M

Is the solution executable?

H, M, L

PrioritizeT & E

CapabilitiesPrioritize

Disconnects(Gaps)

PrioritizeSolutions

PrioritizeDrivers

(Guid, Acq) S&T, Tech)

PrioritizeT & E

CapabilitiesPrioritize

Disconnects(Gaps)

PrioritizeSolutions

PrioritizeDrivers

(Guid, Acq) S&T, Tech)

Prioritization Approach

- Assess common set of criteria - Utilize High, Medium, Low or Full, Partial, Minor/None as ranking criteria where applicable - Use collected info as basis of sub-jectivediscussion- Document rationale behind final ranking of each proposal

Capability Priority

Capability Status

Disconnect Consequence

Solution Impact &

Executability

CapabilityPriority

T1, T2, T3Capability

AssessmentR, Y, G

CapabilityPriority

T1, T2, T3Capability

AssessmentR, Y, G

What is impact of

disconnect on

capability? F, P, N(Include

affect on R, Y, G)

Disconnect consequence

to: -T&E Center-- SPO-Warfighter-H, M, L

What is impact of

disconnect on

capability? F, P, N(Include

affect on R, Y, G)

Disconnect consequence

to: -T&E Center-- SPO-Warfighter-H, M, L

How well does solution address

disconnect?F, P, M

Is the solution executable?

H, M, L

How well does solution address

disconnect?F, P, M

Is the solution executable?

H, M, L Figure 6: Investments Prioritization Process

Although the FY07/08 TIPP cycle was intended to target programming of funds for the FY10-15 Fiscal Year

Defense Plan (FYDP), funds for FY 10-13 had previously been programmed during the FY 05/06 TIPP cycle. This highlights an additional concern that the roadmap may effectively address. As the roadmap is a living document, it can provide an on-going accurate status of capability requirements. Today the programming of investment funds occurs several years before the start date of the actual project. In the interim, capability requirements, technology, and costs continue to evolve. To reduce programmatic risk all programs should be re-evaluated at each TIPP Summit. This will ensure that all funded projects are addressing current requirements and that the project planned years earlier is still supporting a viable requirement. This process should be developed in coordination with the associated roadmap processes.

V. Summary and Conclusions The AF T&E Roadmap will serve many functions. For starters, as a single repository for capability requirements

and associated shortfalls, it will provide resource planners explicit insight into capability gaps and the timeframe in which the gap needs to be filled to meet customer requirements. This will allow them to focus on development of solutions to documented shortfalls and avoid wasted time identifying requirements. Secondly, it would allow planners from various locations to collaborate on common solutions much like the Reliance Panels seek to do across service lines. As an added benefit capability owners would have insight into Enterprise resources and test capabilities, leveraging existing capabilities at other test locations when possible.

The maturation of the roadmap and associated processes will provide the means and information to effectively support prioritization of investment proposals through the cascading requirements process; providing traceability back to high-level AF requirements. The prioritization process will focus on project value to the stakeholders, examining projects from both a technical and business perspective. Lastly, as a living document the roadmap will directly support regular re-examination of planned investment projects, ensuring their on-going alignment with evolving requirements.

The AF must choose wisely to ensure projects yield maximum return on investment while balancing today’s mission against the needs of the future. This will be accomplished through the implementation of a rigorous set of processes and tools such as the T&E Roadmap. This combination of processes and tools will ensure effective management of the T&E test infrastructure and capabilities.

8 8

APPENDIX A

Within the AF T&E resource planning community, some key vernacular exists which are useful in understanding T&E resource planning.

Test Resources encompass all components necessary to conduct T&E of any kind. This includes, but is not

limited to: test ranges and associated facilities, targets, data management, instrumentation, communication, range safety, air, space and ground support equipment, support aircraft and flying hours, research and development; threat systems development, modeling and simulation, distributed assets (live, virtual and

“Investment(s)” is the expression commonly used to describe a formal I&M acquisition project. The purpose of

these projects or “investments” is to develop, or improve and modernize resources which comprise a “T&E capability”. Investment projects predominately focus on the infrastructure (tools and equipment) aspects of a test capability.

An “Investment Program” is a formal I&M activity executed by a T&E support organization via formally

defined processes, which facilities solicitation, selection, and execution of investment projects targeted at specific aspects of T&E (for example: T&E of targets or threat systems). The two major investment programs within the AF and DoD, respectively, are the T&E Investment Planning and Programming (TIPP) program, managed by AFMC/A3F, and the Central T&E Investment Program (CTEIP), managed by the DoD Test Resource Management Center (TRMC).

Major Range and Test Facility Base or MRTFB references the DoD collection of test and evaluation facilities.

All three AFMC primary test locations are considered part of the MRTFB. Air Force Instruction 99-109, MRTFB Test and Evaluation Resource Planning (17 May 2006) provides a definition of MRTFB resources and identifies the various types of resource programs, their purpose and respective planning processes, and organizational roles and responsibilities.

9 9