U.S. ARMY EVALUATION CENTER

OTA Panel on Cybersecurity: Survivability Evaluation Directorate (SVED)March 30, 2017

Approved for public release: distribution is unlimited.

Agenda

• OTA Perspective on Cybersecurity

• Cybersecurity T&E Success Stories

• Army OTA challenges

2 Approved for public release: distribution is unlimited.

OTA Perspective on Cybersecurity

• Recruiting and Retaining skilled workforce

• Technically Diverse Area and Ever changing ―Requires investment in personnel training

• Improving outcomes with earlier life cycle testing and evaluations

3 Approved for public release: distribution is unlimited.

Cybersecurity T&E Success Stories• Defenders are Getting Better at all echelons (BDE, DIV,

RCC)

• Defenders’ tools are improving and making threat movement across the network more challenging

• TTPs are developing on protecting “key cyber terrain”

• Defense in Depth works both doctrinally and architecturally

• Programs are able to mitigate concerns quickly

4 Approved for public release: distribution is unlimited.

Army OTA Challenges

• Defender Manning and Training is always an issue for T&E as it is across the Army

• Replicating network defense in depth by echelon and evolving Defender toolkit

• PLT -> CO -> BN -> BCT -> DIV -> CORPS/ACERT

• Recurring Cyber Vulnerabilities: ― Exposed or poorly managed credentials― Systems not configured to identified standards― Systems not patched for known vulnerabilities― System/network services and trust relationships that provide avenues

for cyber compromise

5 Approved for public release: distribution is unlimited.

6

SVED “Top 5” Future PrioritiesLooking Out Looking In

• System Evaluations 25 systems over next two years

• NDAA 1647 25 total Army systems

• Strengthen Stakeholder Partnerships• Cross-Service Coordination

• OTA TEM on Cybersecurity Hosting March 2017

• Cross-Service Cyber T&E Summit 28 Feb – 2 Mar

• Growing the Workforce

• Investing in the Workforce Training & Certifications

• CEMA Test Management

• ATEC Cyber Initiatives National Cyber Range

Approved for public release: distribution is unlimited.

Air Force Operational Test and Evaluation Center

Operations Directorate (A3)

OTA Panel on Cybersecurity

Prepared by Col Matthew Magness, DirectorFor FSK ITEA Cybersecurity Workshop30 March 2017

Approved for public release: distribution is unlimited.

Agenda

• OTA Perspective on Cybersecurity

• Cybersecurity T&E success stories

• Air Force OTA challenges

8Approved for public release: distribution is unlimited.

OTA Perspective on Cybersecurity

Competing for talent is a highly competitive career field

Flexibility in thought and execution

Define Cyber Requirements early

Drive cybersecurity testing to the left…way left

9 Approved for public release: distribution is unlimited.

Cybersecurity T&E Success StoriesDocumenting Cyber test requirements into the TEMPs

Crosswalking LLs with other OTAs

Expanding AF Adversarial Assessment Capacity

Expanding Cybersecurity within tests…flexibility to assess as own COI (Critical Operational Issue)

Integrated cyber/EW/RF testing between the OTAs, JEPAC, AFRL

10 Approved for public release: distribution is unlimited.

Challenges

Manpower and Training is always an issue― Attracting the right folks to want to come to test in a competitive market

Investing in a single collaborative environment to bring the community to standardize process, tools, info, lexicon

Out striping our CVPA and AA capacity

Ensuring the test community is resourced to build Integrated cyber/EW/RF testing between the OTAs, JEPAC, AFRL

Putting dollars in the right space for the best return― Cyber ranges, FFRDCs, UARCs, RTs, BTs, ?

11 Approved for public release: distribution is unlimited.

Unclassified

Marine Corps Operational Test and Evaluation Activity

OTA Panel on Cybersecurity30 March 2017

Our job in Cyber Security is to ensure the technological advantage our fighting men and women have in the face of adversity, remains intact. We protect the hard-won intellectual property that protects those that place themselves in harm's way. It is our responsibility to defend the edge developed for our warfighters from those that would take it. We are the Cyber Guards protecting the information that furnishes a degree of security to our forces. We are here because they are there.

Unclassified

Agenda

• OTA Perspective on Cybersecurity• Cybersecurity T&E Success Stories• Marine Corps OTA Challenges

Unclassified

OTA Perspective on Cybersecurity

• Requirements have not caught up with defining a level of protection needed against a specified threat level or time to remain operationally effective while in a cyber contested environment.

• Accurate and representative cyber threats against systems under test are difficult to determine (i.e. capability + intent, appropriate levels).

• Limitation in Red Teams to support cybersecurity test priorities.

Unclassified

Cybersecurity T&E Success Stories

• Program Managers are becoming aware of cybersecurity and are beginning to incorporate best practices and appropriate testing early in lifecycle development.

Unclassified

Challenges

• Integration of Cybersecurity testing with the rest of the operational test program.– Cyber testing is confined to its own event similar to live fire testing.– Cyber mission effects are often not allowed due to operational concerns or

unpredictable affects on the rest of the test.• Marine Corps defenders are usually not co-located with systems under

test (SUT).– Tier 2 is provided by the MCCOG and most SUTs are at the tactical level

(Tier 3).– Tier 3 is typically the users and maintainers where cyber defender training is

usually not provided.

QUESTIONS

19

20

ICE

CBP FEMATSA CIS

USCG USSS

DHS OTAs

Organic OTA Outsourced OTA

• No central OTA, component driven

OTA Perspective on Cybersecurity

• Varies by maturity of component• Many programs defer responsibility to CIOs• Pros & Cons with the outsourced OTAs• ST&E (compliance) is enough to address cyber

21

DHS Cybersecurity T&E Success Stories

• Cyber T&E campaign still in its infancy• Components adding organic capability• Starting to see Cyber KPP/COI in ORDs

22

DHS Cyber T&E OTA Challenges

• Resources & Workforce• Embedded IT • ST&E (compliance) is enough to address cyber

• “I do penetration testing”• Focused on the technical aspects vs mission• Definitions, example – threat vs vulnerability• What do you want me to do? How much is

enough?23