Engineering ReportMark Kosters

Staffing

• Tim Christensen

QA Manager

– Passed away August 5,

2014

– Worked for ARIN for 14

years

• DBA

• System Architect

2

Staffing

• Operations– 6 operations engineers + 2 managers (One vacancy in

operations)

• Development– 8 programmers + manager

• New Software Integration head taken from engineering• New hire – filled vacancy created by transfer to SI

• Software Integration formerly known as Quality Assurance– Leadership Change– 5 engineers, 1 contractor + manager

• Project Management – 1

• CTO– 1

3

Accomplishments since ARIN 33

• DNS (and DNSSEC) now have near-real time updates– TTL’s added for NS’s and DS’s

– Hardening of signing infrastructure

• DNSSEC enabled for ARIN’s forward zones (and reverse)

• Shared tickets

• Display agreements associated with organizations

• User interface improvements for payment processing

• Transfers – 8.3 Released

– 8.2 and 8.4 underway

• Movement away from ARIN HQ to Colo for back office production underway

• Movement from EMC to NetApp underway

4

Accomplishments Cont…• Fault Tolerance Improvements

– More efficient system backups

– Moving Production Systems from ARIN HQ to Colo

– Moving backend services to physical hardware when

merited

• Corporate Help Desk and IT Support

• ARIN Member Meeting Support

• Care and Feeding of Servers & Network

• OT&E

5

OT&E• Operational Test & Evaluation

– Place to test code

– Place to test process

– All services now under ote.arin.net

• Replicated Core services

– Reg-RWS (provisioning API)

– Whois-RWS (directory API)

– Web Interface

– RPKI suite (up/down and hosted)

• Participation

– 30 new requests since February

– 152 networks registered to access OT&E

6

YTD Efforts Cont…

• IETF Participation

– SIDR (RPKI), WEIRDS (RDAP)

• ICANN Participation

– SSAC

– RSSAC

– Technical Advisory Group

7

Operational Challenges

• UPS incident

• DOS attacks

• ISP availability

8

UPS Incident• ARIN HQ is in a cheap location – however suffers from

– Lack of power diversity

– Lack of connectivity options

• ARIN hosts the provisioning systems at HQ

– Mail, web, and reg-rws

– Long running project to move gear out to colocation site

• UPS outage

– Complicated and long-running fix

– Resulted in “almost” new UPS

• Lessons learned

– We did a pretty good job recovering gear that had not

suffered power cycles for a long time

– Renewed emphasis on moving to colo for production services

9

Other Operational Concerns

• Periodic DOS attacks

– Hitting our provisioning network

– Need to implement DOS mitigation with

upstreams

• ISP Availability

– Multiple connectivity outages with ARIN HQ

• Unannounced maintenance

• Fiber availability

– Issues with west coast PFS site

10

Operational Highlights

• 100% uptime on our public facing sites

over the past 6 years

• Services include

– Whois

– Whois-RWS

– DNS

– Mailing lists

– FTP

• Will do the same with RPKI11

ARIN Online Usage

• 81,984 accounts activated since

inception through Q3 of 2014

12

2008

2009

2010

2011

2012

2013

2014*

Number of Accounts Activated

5000 10000 15000 20000

* Through Q3 of 2014

Active Usage of ARIN Online

13

0

10000

20000

30000

40000

0 1 2 - 5 6 - 10 11 - 15 >16

Logins

# o

f U

sers

Times logged in

• Logins from inception through Q3 of 2014

Reg-RWS Transactions

14

ARIN

29

ARIN

30

ARIN

31

ARIN

32

ARIN

33

ARIN

34

Template 408383 595858 846943 1066037 1311403 1498204

REST 40374 320197 841105 3524124 4296734 4715231

0

500000

1000000

1500000

2000000

2500000

3000000

3500000

4000000

4500000

5000000

Template

REST

Reports Via REST

ARIN 33 ARIN 34

Associations 176 9,445

Reassignments 25,219 69,320

WhoWas 253,135 382,619

15

• Requests since inception

DNSSEC

ARIN 34

DNSSEC Secured Zones 648

Number of Orgs with DNSSEC 94

Total Number of Delegations 552,329

16

RPKI Usage

ARIN XXX ARIN XXXI ARIN XXXII ARIN33 ARIN34

RPAs Signed 27 72 130 162 208

Certified

Orgs47 68 108 153

ROAs 19 60 106 162 239

Covered

Resources30 82 147 258 332

Web

Delegated

(REMOVED)

0 0 0 0

Up/Down

Delegated0 0 0

17

Whois Queries Per Second

18

0

500

1000

1500

2000

2500

3000

3500

4000

20

01

-07

20

01

-11

20

02

-03

20

02

-07

20

02

-11

20

03

-03

20

03

-07

20

03

-11

20

04

-03

20

04

-07

20

04

-11

20

05

-03

20

05

-07

20

05

-11

20

06

-03

20

06

-07

20

06

-11

20

07

-03

20

07

-07

20

07

-11

20

08

-03

20

08

-07

20

08

-11

20

09

-03

20

09

-07

20

09

-11

20

10

-03

20

10

-07

20

10

-11

20

11

-03

20

11

-07

20

11

-11

20

12

-03

20

12

-07

20

12

-11

20

13

-03

20

13

-07

20

13

-11

20

14

-03

20

14

-07

RESTful

Port 43

Whois via IPv6

19

Percentage of traffic over IPv6

0.00%

1.00%

2.00%

3.00%

4.00%

5.00%

6.00%

7.00%

20

09

-01

20

09

-03

20

09

-05

20

09

-07

20

09

-09

20

09

-11

20

10

-01

20

10

-03

20

10

-05

20

10

-07

20

10

-09

20

10

-11

20

11

-01

20

11

-03

20

11

-05

20

11

-07

20

11

-09

20

11

-11

20

12

-01

20

12

-03

20

12

-05

20

12

-07

20

12

-09

20

12

-11

20

13

-01

20

13

-03

20

13

-05

20

13

-07

20

13

-09

20

13

-11

20

14

-01

20

14

-03

20

14

-05

20

14

-07

V6 Percentage

V6 Percentage

IRR Maintainers

2011 2012 2013 2014

Maintainers 1726 1850 1951 2102

0

500

1000

1500

2000

2500

20

IRR Route / Route6

2011 2012 2013 2014

Route 18636 19969 21204 23535

Route6 242 527 698 1072

1

10

100

1000

10000

100000

Route

Route6

21

IRR InetNum / Inet6Num

2011 2012 2013 2014

InetNum 419 481 531 621

Inet6Num 13 25 38 51

1

10

100

1000

InetNum

Inet6Num

22

Systems at the forefront

• RPKI– Up/Down available – no takers so far

– Removed web delegated

– Upgraded the HSM’s to IBM 4765

• RDAP (IETF WEIRDS)– “Soon” will be an RFC

– Public testbedhttp://rdappilot.arin.net/restfulwhois/rdap

– Work is underway to make it production

– ARIN has open source software at http://projects.arin.net

– Other RIRs are also deploying RDAP

23

Systems (cont..)

• We are a small engineering shop

– Lots of demands

– Attempting to provide exceptional service

• Creating API’s to core services

– Allows YOU to create tools

– Allows YOU to follow your timeline

• projects.arin.net (ACSP completed years ago)

– If you find your tool is cool

– Way to allow others to come find and use it

– We had one taker so far since ARIN 33 with a new tool (ArinWhois.NET)

24

What we have accomplished

since ARIN 33

• Finished up more ACSPs

• DNSSEC on forward zones (arin.net/arin.com)

• Moved RPKI to a new HSM

• Making DNS changes near real-time

• Started automation on transfers

• Moving core production from ARIN HQ to colo

• Moving SAN from EMC to NetApp

25

What we are working on through

2015 Q2 (provisional)

• Moving the RDAP pilot into production *

• Further automation on transfers *

• Complete core production from ARIN HQ to colo *

• Complete migration of SAN from EMC to NetApp *

• Add Links to Whois Query Responses (ACSP 2011.29)

• Change Whois Output for Certain /8 Records (ACSP 2013.4)

• Start “SWIP Easy” – a web-based tool to send in reassignment information

• Deploy Two-Factor Authentication (ACSP 2013.8)

* Part of 2014 Board-approved Operating Plan

26

27

Comments?