Programming Network Policies by Examples: Platform, Abstraction and User Studies

Boon Thau Loo University of Pennsylvania NetPL workshop @ SIGCOMM 2017

Joint work with Yifei Yuan, Dong Lin, Siri Anil, Harsh Verma, Anirudh Chelluri, Rajeev Alur, Boon Thau Loo

Programming Abstractions in SDN

n  Declarative Networking [CACM’09]

n  Frenetic [ICFP’11] n  NetCore [POPL’12] n  Pyretic [NSDI’13] n  NetKAT [POPL’14] n  …

2

•  C/C++ (NOX) •  Python (POX) •  Java (Floodlight) •  …

General-purpose Programming Languages

Domain Specific Languages (DSL)

Missing: SDN programming for non-programmers

NSF I-Corps Program

n  Interviewed 101 network operators/architects, software engineers and router vendors q  Need for programming skills among network

operators q  Frequent network configuration changes that

should be automated q  (Lack of) programming expertise an impediment

towards SDN usage.

3

Need for Programming Capabilities

60.81% 21.62%

17.57%

Needs for programming skills among network operators

Strong need Good to have

4

Outline

n  NetEgg overview n  Controlled user study

NetEgg Overview

Example behaviors

Controller

Action Instructions / Rule installation

Network events

Synthesizer

Interpreter

Policy

Bad behaviors

conflicts

6

NetEgg Overview

Example behaviors

Controller

Action Instructions / Rule installation

Network events

Synthesizer

Interpreter

Policy

Bad behaviors

conflicts

7

A First Example

Learning switch: 1. Learn the mapping between hosts and ports 2. Forward packets according to the learnt

mapping

3A2

1 B

flood <p1, hA, hB    >

fwd(p1) <p2, hC, hA    >

<p3, hB, hC    > fwd(p2)

8

C

<port, src, dst    >

A Stateful Policy Model

n  Intuition: states + cases (policies) n  Learning switch:

q  States: Mapping between hosts and ports q  Case 1. Flood packets if the destination is

unknown; store the port for the source q  Case 2. Forward packets otherwise; store port for

source in packets

9

10

Match Test Action Update

* ST(p.dst).state=0 flood ST(p.src):= (1,port)

* ST(p.dst).state=1 fwd(ST(p.dst).value) ST(p.src):= (1,port)

MAC State Value A 1 1 MAC State Value A 1 1 C 1 3

Controller

3A2

1 C

B

State Table Policy Table

Execution Model

Why This Model?

n  Simple: q  Fast execution q  Deployment on dataplane (FAST [HotSDN’14]，

openState SIGCOMM CCR’14) n  Expressive:

q  Dynamic policies (Kinetic [NSDI’2015])

11

Synthesizer

Synthesizer

.

.

.

Match Test Action Update * ST(dstmac).s

tate=0 flood ST(srcmac

):=(1,port)

* ST(dstmac).state=1

fwd(ST(dstmac).value)

ST(srcmac):=(1,port)

12

Synthesis Algorithm

n  Goal: consistent, minimal

13

Synthesis Algorithm

n  Goal: consistent, minimal n  Greedy search

14

Match Test Action Update

* ST(dst).state=0

Match Test Action Update

* ST(dst).state=0

* ST(dst).state=1

…

Synthesis Algorithm

n  Goal: consistent, minimal n  Greedy search

n  Backtrack with pruning

15

Match Test Action Update

* ST(dst).state=0

Match Test Action Update

* ST(dst).state=0

* ST(dst).state=1

…

Runtime Rule Installation

n  Idea: Keep rules not updating state tables on the switch

16

MAC State Value A 1 2 B 1 3

Match Test Action Update * ST(dstmac)

.state=0 flood ST(srcmac):

=(1,port)

* ST(dstmac) .state=1

fwd(ST(dstmac).value)

ST(srcmac):=(1,port)

Match Action

inport=3, srcmac=B, dstmac=A fwd(2)

… …

<inport=3, srcmac=B, dstmac=A>

Evaluation

n  Is scenario-based programming feasible? q  Expressiveness q  Intuitiveness q  Conciseness q  Efficiency

n  Is the performance of synthesized implementations comparable to hand-crafted implementation? q  Controller response latency q  End-to-end performance

17

Breadth and Synthesis Effort/Time

18

Comparison in Code Size

NetEgg Pyretic POX Mac learner 3 17 29 Stateful firewall

3 21 58

TCP firewall 9 24 68

19

Response Time

0

0.5

1

1.5

2

2.5

mac learner

stateful firewall

tcp firewall

Res

pons

e tim

e (m

s)

netegg pox

20

End-to-end Performance

n  Topology: fattree, 20 switches, 16 hosts n  Policy: learning switch n  Setup:

q  1 host as HTTP server q  other hosts send HTTP requests to the server q  benchmark connection time (i.e. time between a

request is issued and it is finished.)

21

End-to-end Performance

200

400

800

1600

3200

0 10 20

HTT

P co

nnec

tion

time

(ms)

Connection rate (req/s)

netegg pox

22

NetEgg Toolkit

23

Outline

n  NetEgg overview n  Controlled user study

Controlled User Studies (2016/17)

n  Hypothesis: q  “Programmers are more productive in programming language X

compared programming language Y for solving problems type Z”

n  Keep all factors as constant as possible: q  Academic/industry backgrounds, …. q  Solve the same problem Z once q  In a controlled setting under our watch q  Software environment is heavily instrumented

n  Vary only one factor: programming language X vs Y

Controlled NetEgg user study (IRB approval was needed)

The Case for User Studies in PL Design

n  Goal: identify early on the mental model of target audience and programming abstractions:

n  Complementary Approaches q  Release as open-source and measure adoption q  Projects in large classes (>100 students) : Declarative

networking (SIGCOMM’11 Education workshop), Kinetic (NSDI’15)

26

User Study

n  Q1: Can NetEgg reduce the error rate of programming SDN policies?

n  Q2: Can NetEgg improve user’s productivity?

n  Q3: How the users interact with NetEgg?

27

User Study Design

n  Object: NetEgg VS. POX n  Users: masters/PhD students from the

Engineering School of Penn q  With Networking/Distributed System background q  No SDN programming background

n  15 users in NetEgg, 9 users in POX q  All users in the POX group are required to be

proficient in Python programming q  No requirement for NetEgg group

n  More than 21 users in early pilot test 28

Programming Assignment: Firewall++

29

Requirment: 1.  Allow all non-IP. 2.  Block all UDP traffic from the Internet. 3.  Allow all UDP traffic from the CS department. 4.  Allow all TCP traffic from the CS department. 5.  Block all SSH traffic from the Internet. 6.  TCP traffic from any host A in the Internet at TCP port P to any

host B in the CS at TCP port Q is blocked, unless B sends TCP traffic using TCP port Q to A at TCP port P before.

Q1: Error Rate

30

67% 33%

POX

Correct Incorrect

100%

NetEgg

Correct Incorrect

3 users 6 users

15 users

Q2: Programming Time

31

NetEgg POX

Avg. 73 mins

Avg. 146 mins

NetEgg achieves 50% reduction in programming time compared with POX.

Q3: User Interaction Pattern

n  Monitored the number of (unique) scenarios at each call to the synthesizer

n  Identified three interaction patterns: q  Smooth interaction q  Back-and-forth interaction q  Stuck interaction

32

Smooth Interaction Pattern

33

# of Scenarios

Time (minute)

Gradually adding scenarios to describe new behaviors until NetEgg converges

Back-and-forth Interaction Pattern

34

# of Scenarios

Time (minute)

Delete a scenario

Stuck Interaction Pattern

35

# of Scenarios

Time (minute)

Keep modifying the scenario

Statistics across Patterns

9 2

4

# Users

Smooth Interaction

Stuck

Back-and-forth

36

0

20

40

60

80

100

120

Programming Time

Smooth Back-and-forth Stuck

User Feedback

“the tool has a small learning curve and is quite intuitive . . . it is probably easier to use than a programming language.”

37

“I am very clear about what I am doing when I use the interface, and know exactly how the packets are handled under what situations.”

“(I) may get confused about how to define variables.”

Some Takeaways on User Study

n  Effective way to evaluate programming abstractions from the user point of view q  Participants found the tool intuitive to use q  Interesting observations: variables confusion and

tendency to want to tweak compiled policy tables n  Finite pool of participants à need to get the

design of the study correct to maximize participants’ time

n  Missing element: longitudinal analysis

38

Summary n  Scenario-based programming for SDN policies:

q  Expressive to program a range of policies by examples q  Target non-programmers who manage networks q  Comparable performance to hand-crafted implementations q  User study shows evidence of improved productivity

n  Other similar opportunities in networking? q  Measurements, data-plane processing, security attack

patterns ….

39

Thank You!

n  netdb.cis.upenn.edu/netegg

NSF Expeditions in Computer Augmented Program Engineering excape.cis.upenn.edu

q  Designer expresses “what”, using multiple input formats q  Synthesizer discovers new artifacts via integration q  Synthesizer solves computationally demanding problems using

advanced analysis tools q  Interactive iterative design q  Integrated formal verification

41

Software-Defined Networking (SDN)

42

Controller

App App

Dst

NextHop

A 2

… … Match Action

Src=A drop

… …

Openflow

e.g. POX, NOX, Floodlight, etc.

APIs distributed protocols

Control plane

Data plane

NetEgg Scenarios: Stateful firewall

43