Virtualising 5G Network Capabilities
Gerry Foster: 5G Systems Architect
Serdar Vural: SoftFIRE Project Manager
Agenda
Thursday, 17 November 2016 2
1. 5GIC in context and Testbed
2. The 5G Flat Distributed Cloud (FDC) architecture
3. Virtualising the FDC over the 5GIC testbed
4. Results and Conclusions
5GIC and partners, In-Confidence
1. 5GIC In Context
Introduction to 5GIC, Testbed and Trial activity
Thursday, 17 November 2016 3
What is the 5GIC?
Thursday, 17 November 2016 4
Based at the University of Surrey (UoS) in the Institute of Communications Systems (ICS)
World’s largest academic/industry research partnership & test facility for the development of future 5G Communications.
£58m from industry partners below: Operators, Vendors & Systems providers)
£12m investment from the Higher Education Funding Council.
£5m from Local Enterprise Partnerships to support:
• 5GIC test facilities development
• Step-out 5GIC facilities to SMEs within the region
UK based and with significant international connections to China, Korea & Japan
LEP Board link between 5GIC and China Britain Business Council.
160 Researchers, PhDs and support staff
Outdoor Campus Base Station Deployment
Ultra-Dense Outdoor Network
ISD : 10m to 100m
Size: 1km x 0.5 km
#Sites: 44
Thursday, 17 November 2016 5
Indoor 5GIC Building Second Floor Base Station Deployment
Thursday, 17 November 2016 6
Notes:i) Not to Scale
Lampsite_04 Wi-Fi AP_04
Femto_05
Lampsite_05
Wi-Fi AP_05
Lampsite_06Wi-Fi AP_06
N
W
E
S
02)
Indoor Network
ISD : 5-10m
Size: ~50mx 50m
Deployed 2 Floors
Per Floor
3 x Wi-Fi Access Points
3 x FDD LTE-A Femtos
3 x TDD LTE-A micro RRH
5GIC and partners, In-Confidence
2. The Flat Distributed Cloud
The 5GIC view on 5G architecture evolution
Thursday, 17 November 2016 7
Architecture
LTE & UMTS today
Ue(01)
X2
Control Plane
Data Plane
Control & Data
Management
S6a
S1-MME
S5
SGi
S11S3Gn
Iub
IuPS
S3
Gi
Ue(02)
Key
S4S10
Gx
IP address/APN/Usr – complexIPv4-NAT – slow
RAN has integrated UP and CP at Radio Unit/RAT level!
2x GTP Tunnels2x GTP Tunnels in breakout path= notable latency
IuPS CP and UP are combined and distinct only by logical trunks and GTP!
SGSN has integrated CP and UP functions at same node IP address/APN/Usr – complex
IPv4-NAT – slow
RAN has integrated UP and CP at Radio Unit/RAT level
2x GTP Tunnels2x GTP Tunnels in breakout path= notable latency
S1 is CP and UP separated - Good evolution from 3G
Architecture
LTE Rel-14 evolutions (DC) + CUPS + MEC
UE(01)
UPc(S)
External PDN
S1-MME
S1-U
S5-C
SGi
S10S11
S1-U
S5-U
RAN now operates CP + UP anchor to MeNB& additional SeNB UP xN, adding to MeNBCan bond capacity across multiple eNBsUsing “Dual Connectivity”
NAS UP Control now fully separated with CUPSBetter scalability
MEC Svr
MEC Server applications are enabling Content and App ctrl at the edge of the network lower latency
UPc(P)
New UP control or UPc interfaces required (TBD)
NAS UP Control now fully separated with CUPS Better scalability
UE(01)
Friday, 04 December 2015 9
Control PlaneData Plane
Control & DataDC = Dual ConnectivityMeNB = DC, Master eNBSeNB = DC, Secondary eNB
Key
Architecture
5GIC Flat Distributed Cloud (FDC) in context with NGR & LTE
5gD(01)
External PDN
5gD(02)
NG6
5g Radio
5g Radio
5gRadio
UPNCPN UPN UPN
NAS CP integrated into common Control Plane Node (CP + UPc)- Expedited integrated signalling- Context Awareness EnabledFull CP and UP separationFaster access/ Better QoE
NAS UP functions now integrated in UPN- collapses one GTP tunnel in UP pathLower LatencyNG1,
NG2
NG6
UserProfile:Type = IoTMobility = FastThroughput = LowConnect = Fast
UserProfile:Type = Mobile BroadbandMobility = SlowThroughput = HighConnect = Moderate
Fully Separated CP and UP by Radio unit now possible.Can bond capacity across multiple RU and RATs, network scalability
UPN’s still remains under control for LI/Charging/ Policy Ctrl via CPN
CPN connects to Legacy MME and HSS
Friday, 04 December 2015 10
Context Enablement to make network context aware = Flexibility
Full adoption of NFV and SDN to provide OMC/ SON based Network Slicing & R&A Slices for large degree of reconfigurability for optimisation and disaster support
NG1
NG3 NG3 NG3
Control PlaneData Plane
Control & DataNG = Next Generation
Key
Architecture
Full FDC Architecture
5gD(01)
NG1,2
NG4
Cluster(01) Cluster(02)
OMC(Nwk)Orchestration Control
iMgt
NG6External PDN
Radio Radio
5gD(02)
Radio
CPN + UPN for fast moving/ wide and/or Wide area Users integrated into Cluster Controller Connected to Macro Radio System
UPN for MBB/ high capacity slower moving devices integrated into Cluster Members connected to Small Cells
Orchestration Ctrl added to OSS. - Enables ACO
CPN enabled with Meta-Data Protocol to provide context awareness to control plane for anchoring etc.
Supports Cloud-RAN, Distributed –RAN and Hybrid RAN (Split MAC)
11Friday, 04 December 2015
Control Plane
Data Plane
Control & Data
Management
Key
NG3 NG3 NG3
NG6
NG1
iMM
iCC
H/W Cluster(N) (datacentre)
H/W Cluster(01) (datacentre)
Operator Technology Deployment
Architecture
5gD(01)
iMM
Cluster(01) Cluster(02)
OMC(Nwk)Orchestration Control
iMgt
iCC
External PDN
Radio Radio
5gD(02)
Radio
FDC operator wide scope
12Friday, 04 December 2015
Each Data Centre (DC) supports 1 to N x Clusters
Each network supports multiple data centres (Regional and Local)
NG1,2
NG4
NG3
NG6
NG1
NG3NG3
5GIC and partners, In-Confidence
3. Virtualising The Flat Distributed Cloud
… over the 5GIC testbed
Thursday, 17 November 2016 13
5GIC Testbed Segmentation
Thursday, 17 November 2016 14
Segment(01):
Outdoor Fixed Segment
Segment(02):
Indoor Fixed Segment
LTE-A TDD Small-Cells LTE-A FDD Femtos (3)
Segment(03):
Indoor and Outdoor SoftFIRE Segment
Wi-Fi APs, LTE-A TDD Lamp-sites, LTE-A TDD Small-Cells
Core(01)
Vodafone Test Core
Newbury, UK
Segment(04):
Indoor Soft Test Segment
LTE-A FDD Femtos (2)
Vodafone Breakout
Server
Core(02)
Fixed Quortus Core
Web Breakout Server
Core(03)
Soft Quortus Core
UoS
Intranet
Servers
Vodafone
Intranet
Servers
CORE
RAN
SoftFIRE Breakout Server
Fokus Control
Test & Development
Admin Server
Test & Development
Compute Server + Test Web Svr
PLMNid = 235 91
(VF Test)
Label = 5GIC
PLMNid = 235 92
(unused)
Label = 5GII
PLMNid = 235 91
(unused)
Label = 5GIC
PLMNid = 235 92
(unused)
Label = 5GII
VNF/ OTSBlade Core
Production VNFSegment
BespokeVendor Core
ReferenceSegment
SW/ OTS Blade Core
SW Segment
VNF/ OTSBlade Core
Bench VNFSegment
Production VNF Segment - RAN Summary
Core Network
EPC is Virtualised, Evolved EPC from Quortus and 5G FDC components CM and CC from 5GICLocal Intranet services include Web, Content, App and MEC servers, implemented as bare metal instancesExternal Internet connection is through JANET university network 10Gbit/s from UoS to network and easily provides 300Mbit/s per user towards the internet for capacity testing Wi-Fi and 100-200Mbit/s for LTE-A testing
Radio Access Network(s)
Thursday, 17 November 2016 15
System Deployment
Site Type # off Sites
# offCells
AccessType
Band BW(MHz)
Mode Handover Mobile CA UL Mbps
Typ(Max)
DL Mbps
Typ(Max)
Notes
Outdoor2xSector
14 28 LTE-A B38 20 TDD Yes CAT6 Yes 10(18) 100(220)
OutdoorOmni
5 5 LTE-A B38 20 TDD Yes CAT6 Yes 10(18) 100(220)
IndoorLampsites
6 6 LTE-A B41 20 TDD Yes CAT6 No 7(18) 75(100) 3xLS/per floor,@ ground & 2nd, 1st has none
Indoor AP 6 6 Wi-Fi 2.4GHz 30 N/A none N/A N/A 400 400 3xAP/per floor,@ ground & 2nd, 1st has none
Wi-Fi(AP) Lampsite 2xSector Omni
Production VNF Segment - RAN Architecture
Thursday, 17 November 2016 16
1 6 1 21
24 161
Indoor
LTE-A TDD Lampsites
Outdoor
LTE-A TDD Omni’s
and 2xSector Sites
Sx: RANCoreBBU(SF-01) 10.5.22.51 (S1-MME)
10.5.22.52 (S1-U)
10.5.22.53 (X2)
10.5.22.54 (OMC)
BBU(SF-15)10.5.22.107 (S1-MME)
10.5.22.108 (S1-U)
10.5.22.109 (X2)
10.5.22.110 (OMC)
BBU(SF-16) 10.5.22.111 (S1-MME)
10.5.22.112 (S1-U)
10.5.22.113 ( X2)
10.5.22.114 (OMC)
Key:
SF: SoftFIRE
pRRH: pico Remote Radio Head
rHub: radio Hub
Wi-Fi_0110.5.3.110 (AP)
10.5.3.111 (Ctrl)
Wi-Fi_0610.5.3.120 (AP)
10.5.3.121(Ctrl)
Wi-Fi_Ctrl10.5.3.201 (Ctrl)
Sx:Wi-Fi Agg
Legacy 5GIC
Network Testbed
Sx:Nwk_02
Production VNF Segment - Resources/ OS and Platforms
Thursday, 17 November 2016 17
“uos-comp-svr”
(NFV Compute Server)
(Dell R920)
“uos-admin-svr”
(Admin Server)
(Dell R430)
“uos-res-svr”
(Resource Server)
(Dell R430)
90 Core4 x 500Gbyte90 core8 x Ethernet 1Gbit/s(25 VMs planned)
16 Core2 x 500Gbyte16 core2 x Ethernet 1Gbit/s
16 Core2 x 1Tbyte16 core2 x Ethernet 1Gbit/s
14.04, kernel 4.2V7.0V7.0
(Liberty)
(Beryllium)
(Liberty)
Sx: SoftCtrl10.5.10.13
VLAN’s: 10, 20, 21
Production VNF Segment - Test Peripherals
Thursday, 17 November 2016 18
Svr(Web B/O) External IP 131.227.90.4
Internal IP 10.5.3.10
Svr(SoftFIRE Web B/O)External IP 131.227.90.5
GW: 131.227.90.1
Internal IP 10.5.21.10(F/W, Web)(Wi-Fi RADIUS/DHCP – VLAN 21)
Svr(RAN Ctrl)10.5.1.200(Femto-DHCP/ - VLAN 1)
Sx:Nwk_0110.5.10.6VLAN’s: 10,1,2,3,4,5
Sx:Nwk_0310.5.10.5VLAN’s: 10,1,2,3,4,5
Sx:Wi-Fi Agg10.5.10.3VLAN’s: 10,1,2,3
Svr (Content)10.5.3.22(FTP)
Svr (Intranet HTTP:/ pages)10.5.3.21
(Web)
Sx:Nwk_0210.5.10.1VLAN’s: 10,1,2,3,4,5,20,
21, 23
Svr(App)10.5.3.222
Sx: RANCore
10.5.10.14VLAN’s: 10, 23
BBUs LTE
RRHs
WiFi
APsSx: ICS
Dual link (Trunk) eth
Single link eth
Key
Svr(SmokePing) 10.5.21.27 (except EIT-SF)
Production VNF Segment - Network Architecture
Thursday, 17 November 2016 19
“uos-comp-svr”
(NFV Compute Server)
(Dell R920) (10.5.21.24)
VI-Ha
OS-Ma(CN)
Ve-Vnfm
Vi-Vnfm
Virtualised Infrastructure Mgr(Mgr/Hypervisor VIM)
Nf-Vi
Or-Vnfm
MANO
Or-Vi
Vn-Nf
“uos-admin-svr”
(Admin Server)
(Dell R430) (10.5.21.20)
“uos-res-svr”
(Resource Server)
(Dell R430)
(10.5.21.21)
vNF library
SoftFIRE Web B/O Svr
(Dell R430) (10.5.3.10)
CN EM Adaptation layer
Generic Virtualised NF Mgr (VNFM)
2
VirtualisationLayer
Instantiated VirtualResources
NFV MANOOrchestrator
“uos-lic-svr” (Licence Server)
Svr(Dell R430) (10.5.21.26)
Remote Open Baton
Server Host
Interfaces all over
VPN(UoS-Fokus)
Account: admin
Projects: admin(UoS) project(OB)
UE addresses (10.5.20.x)
Sx: RAN_CORE
2
VNFD, NSD Deployed
Eth:0
Sx: SoftCtrl
Production VNF Segment - VNFD/ NSD plan
Thursday, 17 November 2016 20
VM(01), VNFD(EPC_CPN) Common
OpenStack Account: admin
openStack Project: admin (UoS)
10.5.21.30(10.5.21.30) Account: admin(admin)
VM(02), NSD(EPC_UPN_CC) UoS
OpenStack Account: admin
openStack Project: admin (UoS)
10.5.21.31
VM(03), NSD(EPC_UPN_CC) Experimenter#01
OpenStack Account: admin
openStack Project: project (OpenBaton)
10.5.21.32
VM(04), NSD(EPC_UPN_CC) Experimenter#02
OpenStack Account: admin
openStack Project: project (OpenBaton)
10.5.21.33
VM(05), NSD(EPC_UPN_CC) Experimenter#03
OpenStack Account: admin
openStack Project: project (OpenBaton)
10.5.21.34
VM(06), NSD(EPC_UPN_CM) UoS
OpenStack Account: admin
openStack Project: project (UoS)
10.5.21.35
VM(07), NSD(EPC_UPN_CM) Experimenter#01
OpenStack Account: admin
openStack Project: project (OpenBaton)
10.5.21.36
VM(09), NSD(EPC_UPN_CM) Experimenter #03
OpenStack Account: admin
openStack Project: project (OpenBaton)
10.5.21.38
KEY
Virtualisation Terms
NSD: Network Service Descriptor
VNFD: Virtual Network Function Descriptor
VM: Virtual Machine
3GPP Packet Core terms:
EPC: Evolved Packet Core
MME: Mobility Management Entity
HSS: Home Subscriber Server
PGWc: Packet Gateway control
SGWc: Serving Gateway control
5G Evolved Core terms:
UPN: User Plane Node
CPN: Control Plane Node
CC: Cluster Controller
CM: Cluster Member
UoS: University of Surrey
CC
slic
es (
UP
N)
CM
slic
es (
UP
N)
CP
N S
lice
5GIC and partners, In-Confidence
4. Results and Conclusions
… the 5GIC Virtualisation Experience
Thursday, 17 November 2016 21
Virtualisation Experience
5G FDC features (Virtualised and Orchestrated)- used a componentised version of OTS Rel-12 EPC code, from
- added custom Rel-14 modifications for migration to CUPS architecture
- added FDC overlay components CC and CM to demonstrate 5G features: - Traffic Redirection,
- Moveable Breakout,
- Soft Mobility,
- Association
No user plane throughput impact noted
Additional virtualisation delay > LTE-A, ~5ms - Due to mapping of internal to external addresses ‘virtualisation’
- Unavoidable with current OpenStack due to excessive NAT’ing required when using OpenStack
- Mitigated to ~5ms using S/W-F/W subversion of OS OVS to control mappings
MANO Deployment time ~ 5-10 minutes per VNF or NSD (<3Gbyte images)
Results
Thursday, 17 November 2016 22
Virtualisation Experience
Strengths
- Proven rapid deployment of VNFD/NFD (5-10min)
- Proven multiple networks can easily share same RAN
- Proven rapid re-arrangement of sharing configuration
Thursday, 17 November 2016 23
Weaknesses
- Integrated OVS in OpenStack is very slow
- Subversion of OpenStack is complex
- Only one OVS per Compute server (irrespective of # of VMs) can’t control performance between VMs on same machine
- OpenStack is not a robust codebase for Telecoms at present
- OpenDaylight programming approach is cumbersome (5 tables to program per flow is excessive (really only need a match and route 2 entry table (dest/pk and O/P port)
Opportunities
- Massive potential market for VNF platforms and SDN platforms (Orchestrators and Controllers)
- We need standardisation of the control procedures and interfaces from ETSI across platforms at the Controller and MANO levels
- Need a much better NFV platform controller to complement MANO based platforms,
- Better model than FOSS or Bespoke, maybe scalable support form of FOSS with service contracts from FOSS to fully paid up maintenance and roadmap support
- External OVS to NFV platforms for scalability/ P/Pre-emption
- Firmware OVS with formal I/F to NFV platforms to reduce latency in UP
Threats
- If the complexity, support and latency of NFV and SDN and MANO FOSS platforms are not significantly improved quickly then Telecoms operators will either build /commission proprietary platforms or sidestep this technology.
3GPP, 5GRoll-outs
2020
Early Commercial 5G trials(3GPP Stds based)
2019
International SpectrumAgreements (WRC2019)
5G ETE Int. mmWave
Spectrum Agreed (WRC2019)
Early mm-Wave Cellular added
Multi-beam/cell
5GIC Test-bed: Roadmap Goals
5G Technology Showcase
N x Gbit early 5G Systems
2018
5G ETE multi-user 2-4Gbps 10Gbps/Cell 128 branch Nwk.
MIMO Concurrent Multi-RAT
- RF/ Wi-Fi/ mm-Wave 5G Applications
(concurrent/cell) 5G Integrated RAN +
FDC
2017
5G ETE PDCP layer Radio
System demo (16 sites)
4Gbps/Usr & 7Gbps/Cell
Slicing of Flat Distributed Cloud (NFV/SDN Networking)
Context Aware network
User Profile driven Wireless Briefcase
2016
5G RAN MAC layer Radio System
demo (4 sites) Single User 2Gbps &
5Gbit/cell 1ms actuation Multi-Stream 4k Video (4) Tactile Internet Demo5G CN over LTE-A Moveable Breakout IoT Concentration mHTTP Virtualisation Orchestration4G LTE-A (retained as underlay)
2015
5G RAN Bench Component
Demos SCMA, RCA, F-OFDMA, FBMC, FD, evolved MAC, Int. WRC2015 RF spectrum agreed
Meta-Data Protocol (CN)4G-LTE-A LTE-A (R13+) TDD-O/D, FDD+TDD+ Wi-Fi-
I/D 2 x TDD modes(SA1,2) Ultra-Dense HetNet Demo:
Macro & Small Cell Coverage Cat 7 testing, CCA(2) Softcore testing Intranet/ Internet Web &
Video
Rel-14/Rel-15 Subset of urgent Commercial needs
Rel16: IMT2020 submission addressing all UC
Thursday, 17 November 2016 24
5GIC Testbed Roadmap – Network Detail
Thursday, 17 November 2016 25
5G CN over LTE-A/5G
Jan2016 Context Aware FDC Feb2016 Moveable Breakout
Traffic Redirection Feb2016 IoT Concentration Feb2016 mHTTP
Access Agent (DASH) Jun2016 FDC Virtualisation Nov2016 FDC Orchestration
Dec2016 Basingstoke Step-out
2016
5G ETE
Feb2017 Virtualised MECJun2017 Network Slicing of FDCJun2017 Brighton Step-outDec2017 NGP Protocol Demos
2017 Context Aware ETE network
TBC User Profile driven ETE networkWireless Briefcase
2017
With and Without Subversion of OVS for OpenStack
Thursday, 17 November 2016 27
Without subversion of OVS
OpenStack Controller
uos-admin-svr
OVS
SWNAT
Sx: to RAN
OpenStack Compute
uos-comp-svr
OVS
Sx: SoftCtrl
Sx: to Core
Int NFV
NATVM(02),
UPN(CC) Slice(A)
OpenStack Controller
uos-admin-svr
OVS
OpenStack Compute
uos-comp-svr
OVS
Int NFV
NATVM(02),
UPN(CC) Slice(A)
Key
VxLANPath of UP
uos-res-svrOpenFlow
1.3 Ctrl
With subversion of OVS (routing config via OpenStack CP )
- Software NAT from External IP to OpenStack Internal = Slow
- Using OpenStack Control path for system UP = non-CUPS Virtualisation
- UP path is SW NAT‘ed under control of Neutron I/F from ODL
(CUPS separation)
SWNAT
SWNAT
Sx: to RAN
Sx: SoftCtrl
Sx: to Core
With full Subversion of OVS for OpenStack
Thursday, 17 November 2016 28
OpenStack Controller
uos-admin-svr
OVS
OpenStack Compute
uos-comp-svr
OVS
Int NFV
NATVM(02),
UPN(CC) Slice(A)
Key:
VxLANPath of UP
uos-res-svrOpenFlow
1.3 Ctrl
With full subversion of OVS
- Hardware NAT from External to Internal = Fast
- UP path is truly UP = CUPS virtualisation
(Configure UP NAT using CP ODL)
HWNAT
HWNAT
Sx: to RAN
Sx: SoftCtrl
Sx: to Core