Toward Networks with Toward Networks with Cognitive PacketsCognitive Packets

Erol Gelenbe Ricardo Lent Erol Gelenbe Ricardo Lent

Esin Seref Zhiguang XuEsin Seref Zhiguang XuSchool of Electrical Engineering & School of Electrical Engineering &

Computer ScienceComputer Science

University of Central FloridaUniversity of Central Florida

Orlando, FL 32816Orlando, FL 32816

OverviewOverview Introduction to CPNIntroduction to CPN Cognitive Packet NetworksCognitive Packet Networks Packet Routing Using Neural NetworksPacket Routing Using Neural Networks

– Random Neural Networks & LearningRandom Neural Networks & Learning– Random Neural Networks with Reinforcement Random Neural Networks with Reinforcement

LearningLearning Simulation (LFRNN, RNN RL, & Bang-Bang)Simulation (LFRNN, RNN RL, & Bang-Bang) Cognitive Adaptive RoutingCognitive Adaptive Routing Analytical Model Predictions using G-NetworksAnalytical Model Predictions using G-Networks CPN Test-BedsCPN Test-Beds Future WorkFuture Work

Problem DefinitionProblem Definition

Packet networks have capabilities for flow Packet networks have capabilities for flow control, error control and routing control, error control and routing

Cognitive Packet NetworksCognitive Packet Networks (CPN)(CPN)

concentrate intelligent capabilities for concentrate intelligent capabilities for routing in the packets rather than in the routing in the packets rather than in the nodes and protocols nodes and protocols

For the time being, we assume that error For the time being, we assume that error and flow control in CPN are handled using and flow control in CPN are handled using current techniquescurrent techniques

Introduction to CPNIntroduction to CPN

A packet switching network where packets A packet switching network where packets route themselves route themselves

Packets are assigned goals before entering the Packets are assigned goals before entering the networknetwork

Packets learn to achieve their goalsPackets learn to achieve their goals Learning is performed by sharing information Learning is performed by sharing information

between packetsbetween packets Packets sharing same goals can be grouped Packets sharing same goals can be grouped

into into classesclasses Packets do not rely on nodes for routingPackets do not rely on nodes for routing

OSI LayersOSI Layers TCP/IP is a layered protocol stackTCP/IP is a layered protocol stack ApplicationApplication handles particular handles particular

applications, applications, PresentationPresentation handles handles compression and encryption of data, compression and encryption of data, SessionSession controls establishment, controls establishment, management, termination of sessionsmanagement, termination of sessions

TransportTransport provides flow of data provides flow of data NetworkNetwork handles the transmission of handles the transmission of

packets in the networkpackets in the network Data-linkData-link is responsible for the is responsible for the

interaction of the device driver in the interaction of the device driver in the operating system and the network operating system and the network card in the machinecard in the machine

PhysicalPhysical defines electrical and defines electrical and mechanical specificationsmechanical specifications

A pplica tion

T ransport

N etwork

D ata L ink

P hysica l

P resenta tion

S ession

Transport Layer (TCP)Transport Layer (TCP)

Connection-oriented:Connection-oriented: client and server client and server applications establish connection before data applications establish connection before data exchangeexchange

Reliable:Reliable: sends checksum on data and header, sends checksum on data and header, takes care of the order of arriving packets, takes care of the order of arriving packets, takes care of duplicate packets, sends takes care of duplicate packets, sends acknowledgments, maintains timeracknowledgments, maintains timer

Flow-control:Flow-control: receiving side allows the sending receiving side allows the sending side to send as much as it can bufferside to send as much as it can buffer

Network Layer (IP)Network Layer (IP)

Unreliable:Unreliable: best effort service, no guarantees best effort service, no guarantees to get to the destinationto get to the destination

Connectionless:Connectionless: no state information about no state information about successive datagrams, each handled successive datagrams, each handled independently independently

Routing:Routing: done on a hop-by-hop basis according done on a hop-by-hop basis according to the routing tables, required information is to the routing tables, required information is kept in the IP header (Routing table kept in the IP header (Routing table information comes from the routing protocols) information comes from the routing protocols)

Current Routing Current Routing ProtocolsProtocols

Distance-vector ProtocolsDistance-vector Protocols– Routing tables updated according to a vector of Routing tables updated according to a vector of

distances (hop counts)distances (hop counts)– Regular and triggered updatesRegular and triggered updates– Whole network topology is distributedWhole network topology is distributed– No information about the network topologyNo information about the network topology

Drawbacks:Drawbacks:– Regular updates consume bandwidthRegular updates consume bandwidth– Takes too long to stabilize after failureTakes too long to stabilize after failure– Using only hop count omits other variablesUsing only hop count omits other variables

Current Routing Current Routing ProtocolsProtocols

Link-state ProtocolsLink-state Protocols– Each router tests the status of its link to its neighbors Each router tests the status of its link to its neighbors

and sends this information to its other neighborsand sends this information to its other neighbors– Converges faster than distance-vector protocols Converges faster than distance-vector protocols – Each router can be assigned a cost based on Each router can be assigned a cost based on

throughput,reliability, round-trip time, etcthroughput,reliability, round-trip time, etc– Load balancing possibleLoad balancing possible– Multicasting reduces load on systems not Multicasting reduces load on systems not

participatingparticipatingDrawbacks:Drawbacks:– No policy based routingNo policy based routing– Needs more memory and CPU powerNeeds more memory and CPU power

CPN and CPCPN and CP

CPNCPN: Packet switching networks in which : Packet switching networks in which intelligent capabilities for intelligent capabilities for routing are concentrated in packets routing are concentrated in packets rather than in the nodes and protocolsrather than in the nodes and protocols

CPCP: Packets in CPNs which have the route : Packets in CPNs which have the route themselvesthemselves

Structure of CPsStructure of CPs

IdentifierF ie ld

DATACognitive

M apExecutable

Code

•Identifier Field: The unique identifier of a packet

•Data Field: Payload

•Cognitive Map: S-D information, the packet’s view of the state of the network, etc

•Executable Code: Code used to update the CM and the decision algorithm

Storage area for CPs: Storage area for CPs: Input and Output Input and Output BuffersBuffers

Mailboxes are used to Mailboxes are used to exchange data exchange data between CPs between CPs

The Node executes the The Node executes the code for each CP in the code for each CP in the input bufferinput buffer

Nodes in a CPNNodes in a CPN

Update of CM by Update of CM by NodeNode

Packet arrives at the input buffer of a nodePacket arrives at the input buffer of a node The code of each CP in the input buffer is The code of each CP in the input buffer is

executedexecuted– The CP retrieves relevant information from the The CP retrieves relevant information from the

mailboxmailbox– The packet’s CM is updatedThe packet’s CM is updated– Some information is moved from the CP to the MBSome information is moved from the CP to the MB– The CP is moved to an output bufferThe CP is moved to an output buffer

M ailbox(M B)

CodeUpdatedCognitive

M ap

CognitiveM ap(CM )

Cognitive Packet Cognitive Packet ClassesClasses

S

D

S

D

NODE(X,Y)

NODE(X,Y+1)

NODE(X+1,Y)

S D

S D

Class(S,D,L)

A Class is a set of packets having the same QoS requirements, sets of internalstates, control rules, input-outputsignals

Random Neural Random Neural NetworksNetworks

A spiked RNN with a mathematical structure A spiked RNN with a mathematical structure similar to a queuing network similar to a queuing network (Gelenbe in Neural (Gelenbe in Neural Computation 93, IEEE Trans. on NN 99 etc.)Computation 93, IEEE Trans. on NN 99 etc.)

Probability that the neuron is excited:Probability that the neuron is excited:

wherewhere

is the rate at which neuron i sends is the rate at which neuron i sends excitation spikes to neuron j when i is excitedexcitation spikes to neuron j when i is excited

is the rate at which neuron i sends is the rate at which neuron i sends inhibition spikes to neuron j when i is excitedinhibition spikes to neuron j when i is excited

ii

ii

rq

jijiji wq and

jijiji wq

ijw

ijw

Learning Feed-Forward Learning Feed-Forward RNNRNN

Input to each neuron Input to each neuron is represented as a is represented as a pair of excitatory pair of excitatory and inhibitory signalsand inhibitory signals

Each connection is Each connection is represented as a pair represented as a pair of weights of weights

Output of the system Output of the system is the q of the output is the q of the output layer neuronslayer neurons

ii

LFF RNN EquationsLFF RNN Equations Normalized training set Normalized training set

Network parameters are computed from the Network parameters are computed from the training settraining set

Parameters adjusted to minimize the error Parameters adjusted to minimize the error functionfunction

Weight updatesWeight updates

0,)(2

1

1

2

i

n

iikiik ayqaE

)},({ jiwW kk )},({ jiwW kk

n

ikiikikikk vuwqyqavuwvuw

11 )],(/)[(),(),(

,*9.0 1

l

ll D

DT

1 ll DD1

*9.0

l

ll D

DT otherwise

LFF RNN EquationsLFF RNN Equations

WhereWhereu

u

qvuWvuwQvuwQ

qvuWvuwQvuwQ

),(),(/),(/

),(),(/),(/

)}(/]),(),({[

),...,( 1

jqjiwjiwW

qqQ

j

n

0

)(/1

)(/1

i

i

i

0

)(/

)(/1

)(/)1(

iq

i

iq

i

i

i

if u=i,v!=iif u!=i,v=iotherwise

if u=i,v=iif u=i,v!=iif u!=i,v=iotherwise

RNN with Reinforcement RNN with Reinforcement LearningLearning

Fixed input valuesFixed input values Each connection is Each connection is

represented as a represented as a pair of negative pair of negative and positive and positive weightsweights

Output of the Output of the system is the system is the largest q of all the largest q of all the neuronsneurons

i

j

RNN RL AlgorithmRNN RL Algorithm

Decision thresholdDecision threshold

Compute Compute

If If

11 ,)1(

GRRaaTT lll

jkn

Rkiwkiw

Rjiwjiw

l

l

,2

),(),(

),(),(ll RT 1

l

l

Rjiwjiw

jkn

Rkiwkiw

),(),(

,2

),(),(otherwise

1lT

RNN RL AlgorithmRNN RL Algorithm

Re-normalize all weights by calculatingRe-normalize all weights by calculating

andand

n

i miwmiwr1

* )],(),([

*

*

),(),(

),(),(

i

i

i

i

r

rjiwjiw

r

rjiwjiw

SimulationSimulation 100 nodes100 nodes Link speeds normalized 1Link speeds normalized 1 Arrivals are PoissonArrivals are Poisson Buffer sizes are unlimitedBuffer sizes are unlimited Packet loss simulated Packet loss simulated

probabilistically, probabilistically, intentionally high in intentionally high in some areassome areas

GoalGoal is to minimize a is to minimize a weighted combination of weighted combination of delay W and loss Ldelay W and loss L

LWG

Bang-Bang to Minimize Bang-Bang to Minimize Average DelayAverage Delay

Bang-Bang to Minimize Bang-Bang to Minimize Avg Delay + LossAvg Delay + Loss

RNN Reinforcement Learning RNN Reinforcement Learning to Minimize Avg Delay + Lossto Minimize Avg Delay + Loss

LFF RNN Control to LFF RNN Control to Minimize Avg Delay + LossMinimize Avg Delay + Loss

Bang-Bang & Shortest Path Bang-Bang & Shortest Path in the Network Without Lossin the Network Without Loss

Cognitive Adaptive Routing Cognitive Adaptive Routing (CAR)(CAR)

There are three types of packets: There are three types of packets: – The packets sent into the network by the source with The packets sent into the network by the source with

small rate small rate which travel intelligently to their which travel intelligently to their destination (SMART or Cognitive Packets) –destination (SMART or Cognitive Packets) –

– A SMART packet which reaches its Destination A SMART packet which reaches its Destination generates an ACK back to the source; ACKs travel generates an ACK back to the source; ACKs travel back along the same (reverse) route as the packetback along the same (reverse) route as the packet

Reception of an ACK at the Source may trigger Reception of an ACK at the Source may trigger the transmission of a DUMB packet along the the transmission of a DUMB packet along the route which the older packet had takenroute which the older packet had taken

ACKs may not be necessary for each packetACKs may not be necessary for each packet

The ACK coming back from Destination The ACK coming back from Destination D deposits Delay Information D deposits Delay Information W(n,O,D) when it passes through Node W(n,O,D) when it passes through Node n, entering it from Node On, entering it from Node O

W(n,O,D) is used to Update the CAR W(n,O,D) is used to Update the CAR Learning Algorithm (eg Reinforcement Learning Algorithm (eg Reinforcement Learning) either by the Smart Packet, Learning) either by the Smart Packet, or by the Node itselfor by the Node itself

Cognitive Adaptive Routing Cognitive Adaptive Routing (CAR)(CAR)

Analyzing CAR with G-Analyzing CAR with G-NetworksNetworks

G-Networks are Queuing Networks with Three G-Networks are Queuing Networks with Three Types of Customers: Types of Customers: – Normal CustomersNormal Customers– Negative Customers: They Destroy Normal CustomersNegative Customers: They Destroy Normal Customers– Triggers: They Move Normal Customers to Another QueueTriggers: They Move Normal Customers to Another Queue

Multiple Class Models AllowedMultiple Class Models Allowed G-Networks Have Product Form … They Can be G-Networks Have Product Form … They Can be

Solved Analytically and then NumericallySolved Analytically and then Numerically Work of Gelenbe (89, 91, 93, 98) and Many OthersWork of Gelenbe (89, 91, 93, 98) and Many Others Key Papers Mainly in Journal of Applied ProbabilityKey Papers Mainly in Journal of Applied Probability G-Networks Can Model “Worst-Case CAR” G-Networks Can Model “Worst-Case CAR”

and “Best-Case CAR”and “Best-Case CAR”

G-Network EquationsG-Network Equations Non-Linear Equations for the Queue UtilizationsNon-Linear Equations for the Queue Utilizations

- queue S is Source, queue D is Destination- queue S is Source, queue D is Destination

- is a route from S to D with queue i and queue j as - is a route from S to D with queue i and queue j as components and j is the predecessor of i components and j is the predecessor of i

- ACKs will trigger packets from queue 0, which constantly feeds - ACKs will trigger packets from queue 0, which constantly feeds

queue S with small rate queue S with small rate 00<S,D><S,D>

D,RS,

iD,RS,

jD,RS,

jD,RS,i

D,RS,

DD,RS,

DD,RS,

0DS,

0DS,0

DS,

SD,RS,

D,RS,0

DS,0

DS,0

DS,D

D,RS,D

D,RS,SD,RS,

μ

μqq

)μ(qμq

μ

Pμqqμqq

R

Analytical SolutionAnalytical Solution

Calculate the average No of packets in each Calculate the average No of packets in each queue:queue:

Calculate the average transit delay for a Calculate the average transit delay for a packetpacket

S D

0DS,

i

i

0DS,

0DS,0

DS,

S R D

iD,RS,

i

S R D

iD,RS,

iD,RS,i

D,RS,

NNN

q1

qN

NN

q1

qN

S D

0DS,Δ

NW

Cognitive Adaptive Routing Cognitive Adaptive Routing PerformancePerformance

All Packets Are Smart

but Not Always Successful

Fewer PacketsAre Smart:

Dumb Packets Follow Successful

Smart Packets

Cognitive Adaptive Routing Cognitive Adaptive Routing PerformancePerformance

CAR Performance as a Function of CAR Performance as a Function of Smart Packet Injection RateSmart Packet Injection Rate

Test-beds for CPNTest-beds for CPNGoalsGoals Performance evaluation and capability enhancementsPerformance evaluation and capability enhancements Demonstration of applications Demonstration of applications

RequirementsRequirements Explicit separation and clear borderline between the Explicit separation and clear borderline between the

different protocol layersdifferent protocol layers Standardized interfaces both between the application Standardized interfaces both between the application

layer and the CPN layer, and between the CPN layer layer and the CPN layer, and between the CPN layer and the data transport networkand the data transport network

Compatibility with a wide range of computing Compatibility with a wide range of computing platformsplatforms

Interchangeability of the data transport networking Interchangeability of the data transport networking layerlayer

A CPN-VN (Virtual A CPN-VN (Virtual Network)Network)

Standard Linux on Intel-PCsStandard Linux on Intel-PCs New CPN Code Instead of IPNew CPN Code Instead of IP

CPN Test-BedCPN Test-BedDifferent Selected Logical Different Selected Logical

ConnectivitiesConnectivities Different Selected Different Selected Physical Physical

ConnectivitiesConnectivities

CPN-VP Test-BedCPN-VP Test-BedCurrent Test-Bed ConnectivityCurrent Test-Bed Connectivity

Special Purpose Special Purpose CPN-Router Test-BedCPN-Router Test-Bed

Special-Purpose CPN-Special-Purpose CPN-RouterRouter

PowerPCCPUCore

MPC860sar

EthernetMACSCC1

EthernetTransceiver

ATMSARSCC4

ATMTransceiver

ATMTransceiver

ATMTransceiver

RAMMemory

32K

MemoryController

FlashROM16K

10Mbps155Mbps

155Mbps

155Mbps

Smart & Dumb PacketsSmart & Dumb Packets

SMART or DUMB PacketIndicator

Routing Data + Code

Payload

CPN CPN HeaderHeader

Cognitive Mapand Executable Code

Identifier

Packet ContentsPacket Contents

CPN CPN HeaderHeader

Mail Boxes in CPN-Routers Mail Boxes in CPN-Routers

The Mailbox (MB) The Mailbox (MB) is a LRU stack where is a LRU stack where information is stored: delaysweights for information is stored: delaysweights for different directions associated with certain different directions associated with certain <S,D>, ... <S,D>, ...

Packets (esp. ACKs) deposit information in the Packets (esp. ACKs) deposit information in the MB MB

CPN-Router SoftwareCPN-Router Software

State Diagram of CPN-State Diagram of CPN-RouterRouter

PDU Assembled

Update Route

Complete RouteAvailable

Consult MBUpdate CM

Incomplete RouteAvailable

No RouteAvailable

PDUReady

Send

Complete / Incomplete Route

PDU Arrived (either from higher layer or from peer entity)

Proceed at Random

No Route Found

Work AccomplishedWork Accomplished

CPN: packet network architecture in which routing is carried under CPN: packet network architecture in which routing is carried under Packet Control --Packet Control --

Cognitive Adaptive Routing: Smart or Cognitive Packets Control the Cognitive Adaptive Routing: Smart or Cognitive Packets Control the Routing of the Dumb PacketsRouting of the Dumb Packets

Cognitive Packet Routing is Designed Using Random Neural Network Cognitive Packet Routing is Designed Using Random Neural Network based Reinforcement Learning, Feed-Forward Learning, Bang-Bang based Reinforcement Learning, Feed-Forward Learning, Bang-Bang ControlControl

Analytical Modeling and Simulation Results are PresentedAnalytical Modeling and Simulation Results are Presented

Test-Bed architectures are being implementedTest-Bed architectures are being implemented

Work to be Done: IWork to be Done: IRun and Measure Significant ApplicationsRun and Measure Significant Applications(eg File Transfer, Video-Conferencing, Voice, (eg File Transfer, Video-Conferencing, Voice, Web Browsing …)Web Browsing …)

Evaluate Network Adaptation under Time-Varying WorkloadsEvaluate Network Adaptation under Time-Varying Workloads

Deploy a “Large” Test-BedDeploy a “Large” Test-Bed

Examine Implications with respect to TCPExamine Implications with respect to TCP

Consider CAR as a Joint Routing & Flow Control SchemeConsider CAR as a Joint Routing & Flow Control Scheme

Work to be Done: IIWork to be Done: II

Can CAR be used to enhance IP ?Can CAR be used to enhance IP ?

Can CPN run in conjunction with IP ?Can CPN run in conjunction with IP ?

Can Generalized CAR be a way of Can Generalized CAR be a way of selecting paths in selecting paths in BGP ?BGP ?