• Home

  • Documents

  • Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering)...
62
Strategies of packet buffering inside Routers Rafal Jan Szarecki #JNCIE136 Solu@on Architect, Juniper Networks

Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

  • Upload
    ngophuc

  • View
    232

  • Download
    3

Embed Size (px)

Citation preview

Page 1: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Strategies  of  packet  buffering  inside  Routers  

   

Rafal  Jan  Szarecki  #JNCIE136  Solu@on  Architect,  Juniper  Networks  

Page 2: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Why  should  I  care  ?  

•  Under  load,  Your  router’s  discard  behavior  according  to  queuing  strategy  selected  by  vendor.  – Could  be  quite  unintui@ve  !  – BePer  to  know,  how  to  live/deal  with  this  ar@ficial  “intelligence”.  And  turn  it  for  your  benefit.  

– Do  not  troubleshoot  if  there  is  nothing  unexpected/misbehaving.  

Page 3: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

How  it  manifest  •  Something  is  going  on  

–  SLA  monitoring  system  rise  alarm  

–  Customer  calls  and  complains  

•  Which  node  in  network  cause  it?  –  You  may  need  go  node-­‐by  node.  

–  Other  expert/anali@c  systems  my  help.  

–  Out  of  scope  •  When  guilty  node  is  nail  

down  …  

Traffic  loss  

Too  high  jiPer/latency  

Page 4: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

 Unintui@ve  behavior  

Defect  (bug)  or  expected  behavior  ?  

“wirespeed”  router  

1  Gbps   10GE

 po

rt  

100G

E  po

rt  

<1Gbps  

Other  traffic  

Other  traffic  

Port  u'liza'on  ~1%    &  traffic  loss  !!!  

“wirespeed”  router  

FLOW  A  po

rt   FLOW  A’  

+    FLOW  B’  

Port  u'liza'on  100%,  port  buffer  size  is  Xms  

But:  •  Unfair  Latency:    

Flow  B  <<  Flow  A  •  Latency  is  >  Xms  FLOW  B  

Other  traffic  

Other  traffic  

Page 5: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Queuing  !=  QoS  •  Queuing  goal  –  avoid  traffic/

packet  drops  during  temporal  conges@on  

•  QoS  goal  -­‐  provide  differen@al  treatment  and  separa@on  among  traffic  of  different  classes.    

–  Avoid  traffic/packet  drops  during  temporal  conges@on  in  some  classes  at  expenses  of  losses  in  other.  

–  Re-­‐order  packets  in  a  way  to  deliver  data  of  some  classes  as  fast  as  possible.  

This  talk  is  not  on  QoS.    We  will  look  at  best-­‐effort  only  

Queueingvs.

QoS

tx-rate reduction, class blind - red and blue

impacted proportional

tx-rate reduction, class aware - blue impacted more

Page 6: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Router  is  the  embedded  network      

Router  anatomy  

PFE  –  a  CPU,  NPU  or  ASIC  that  process  packet  

Ingress  PFE  

Ingress  PFE  

Egress  PFE  

Egress  PFE  

MUX     DEMUX    SWITCH  FABRIC    

Page 7: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

1 x

Tx

N x

Rx

N x Rx rate <= 1 x Tx rate

The  MUX  

•  Mul@ple  low-­‐speed  In  (Rx)  port  and    

•  Single  high-­‐speed  Out  (Tx)  interfaces  

•  Many  to  One  •  No  conges@on  risk  –  no  need  for  buffer  

Page 8: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

N x

Tx

1 x

Rx

Rx rate == N x Tx rate

The  de-­‐mux  

•  Simple  model  •  High-­‐speed  In  (Rx)  port  and    

•  Mul@ple  lower-­‐speed  Out  (Tx)  interfaces  

•  One  to  many  

•  1  Rx  to  1  Tx  @  same  @me  è  Conges@on.  

Page 9: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Queuing  architectures  for  de-­‐mux  

•  Simple  Output  Queuing  –  OQ  

•  Usually  implemented  in  single  (shared)  memory  

Pn RX

IF0 Tx

IF1 Tx

IFn Tx

Page 10: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

The  (asynchronous)  switch  fabric  •  N  x  In  (Rx)  and  N  x  Out  (Tx)  

port  of  switch  of  same  speed  

•  Any  to  Any  •  Each  ingress  port  is  

independent  –  Traffic/datagram  may  appear  at  any  @me  

–  Not  aware  about  egress  port  state  

•  N  Rx  to  1  Tx  @  same  @me  è  Conges@on.  

N x

Tx

N x

Rx

Rx rate == Tx rate

Page 11: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Queuing  architectures  for  switch  fabric  

•  Output  Queuing  –  OQ  –  not  used  due  to  technological  limita@ons.  

•  Input  Queuing  –  IQ  

•  Virtual  Output  Queuing  -­‐  VOQ  

Page 12: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

IQ  

•  <100%  efficiency  •  Queue  fan-­‐out  need  to  be  over  2  x  desired  port  traffic  to  get  99%+  efficiency  

P1 OQ

P2 OQ

Pn OQ

Port P1 Rx

Port P2 Rx

Port Pn Rx

Port P1 Tx

Port P2 Tx

Port Pn Tx

Page 13: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

IQ  Flow-­‐control  •  Asynchronous  –  each  

egress  is  independent  •  Ingress  PFE  sends  Request  

–  Each  has  data  size  –  Only  for  packet  at  head  of  queue  

•  egress  PFE  answer  w/  Grant  –  when  egress  Fabric  port  is  free  

•  Egress  schedules  grants  –  Prevent  starva@on  –  E.g.  RoundRobin  or  fair-­‐share  

P1 OQ

P2 OQ

Pn OQ

Port P1 Rx

Port P2 Rx

Port Pn Rx

Port P1 Tx

Port P2 Tx

Port Pn Tx

RQ

GR

RQ

HoL blocking Can’t be send, even P2 Tx is free

Page 14: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

VOQ  •  Variant  Input  Queuing  –  “dedicated  Input  queue  for  each  output  port”  

•  No  need  for  over-­‐speed  •  Flow-­‐Control  and  scheduling  –  Extension  to  IQ  –  Ingress  PFE  can  send  requests  to  mul@ple  egresses  simultaneously  

   

Pn VOQ

Pn VOQ

Pn VOQ

Pn VOQ

Port P1 Rx

Port P2 Rx

Port Pn Rx

Port P1 Tx

Port P2 Tx

Port Pn Tx

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

RQRQ

RQ

RQRQ

RQ

Page 15: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

VOQ  •  Variant  Input  Queuing  –  “dedicated  Input  queue  for  each  output  port”  

•  No  need  for  over-­‐speed  •  Flow-­‐Control  and  scheduling  –  Extension  to  IQ  –  Ingress  PFE  can  send  grants  to  mul@ple  egresses  simultaneously  

   

Pn VOQ

Pn VOQ

Pn VOQ

Pn VOQ

Port P1 Rx

Port P2 Rx

Port Pn Rx

Port P1 Tx

Port P2 Tx

Port Pn Tx

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

GR

GR

GR

Page 16: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

VOQ  •  Variant  Input  Queuing  –  “dedicated  Input  queue  for  each  output  port”  

•  No  need  for  over-­‐speed  •  Flow-­‐Control  and  scheduling  –  Extension  to  IQ  –  Ingress  PFE  can  send  grants  to  mul@ple  egresses  simultaneously  

•  No  HoL  blocking  

Pn VOQ

Pn VOQ

Pn VOQ

Pn VOQ

Port P1 Rx

Port P2 Rx

Port Pn Rx

Port P1 Tx

Port P2 Tx

Port Pn Tx

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

Page 17: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

The  router    

•  Mul@stage  –  Ingress  mux  to  fabric  

•  No  conges@on  –  Fabric  switch  –  Egress  demux  to  ports  

•  hiSpeed  (fab)  to  low  speed  port  •  Conges@on  points  

–  Fabric-­‐out  (manyà  one)  –  Egress  mux  (fast  à  slow)  –  Need  queuing  

P1 Rx

P2 Rx

Pn Rx

P1 Tx

P2 Tx

Pn Tx

Switch Fabricswitch

Ingress PFEASIC/NPU

Fabric ports

Router ingressInterfaces

Buffermemory

egress PFEASIC/NPU

Buffermemory

Overload possible. Hi-speed fabric port

à low-speed Interface

Overload possible. Multiple Rx ports

sends to single Tx port

Switch Fabric Line Card

ingress part Line Card

egress part

Page 18: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Two  approaches  Buffer  twice  –  CIOQ  systems   Buffer  Once    -­‐  VOQ  systems  

Combined  Input  Output  Queuing    •  Buffer  before  fabric;  de-­‐queue  when  

fabric  egress  port  is  available  (empty)  –  fabric  VOQ  (or  IQ)  

•  Buffer  before  egress  interface;  de-­‐queue  when  interface  is  available  (empty)  –  OQ  

Virtual  Output  Queuing  •  Buffer  before  fabric;  

De-­‐queue  when  all  way  down  to  egress  interface  is  available  (empty)  –  end-­‐to-­‐end  system  VOQ  

•  Simpler  to  Implement  •  Higher  scalability  [O(n)]  •  Requires  more  memory  

•  Space  (size)  •  2  x  bandwidth  

•  Bigger  system  residency  @me  and  JiPer  

•  Requires  a  lot  of  queues  -­‐  complex  queue  management  @  scale  [O(n2)]  

•  Requires  less  memory  •  Space  (size)  •  1  x  bandwidth  

•  Lower  residency  @me  (latency  inside  router)  •  Lower  power  requirements  

Page 19: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Buffer  twice  –  latency  

•  Queues  ==  Buffer  ==  accommodate  burst  and  delay  it  

•  Burst  absorp@on  capability  depends  on  type  of  burst    -­‐  at  which  point  conges@on  appears  –  Max:  Σ  (VOQ  size,  OQ  size)  –  VOQ_size  or  OQ  _size  if  

conges@on  in  one  point  only.  •  Max  latency:  Σ  (VOQ  size,  OQ  size)  

•  Wait,  there  will  be  example  

Pn RX

IF0 Tx

IF1 Tx

IFn Tx Pn VOQ

Pn VOQ

Pn VOQ

Pn VOQ

Port P1 Rx

Port P2 Rx

Port Pn Rx

Port P1 Tx

Port P2 Tx

Port Pn Tx

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

P2 VOQ P1 VOQ

RQRQ

RQ

RQRQ

RQ

Poten@al  conges@on  point  

Page 20: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

P1 Rx

P2 Rx

Pn Rx

P1 Tx

P2 Tx

Pn Tx

Switch Fabricswitch

Ingress PFEASIC/NPU

Fabric ports

Router ingressInterfaces

Buffermemory

egress PFEASIC/NPU

Buffermemory

Buffer  twice  –  bandwidth    Fabric  Scheduler  and  flow-­‐control  

•  Which  ingress  PFE  get  GRANT  next.  (e.g.  fair-­‐share)  from  given  egress  PFE  

•  Monitor  Fabric  Egress  and  stop  giving  GR  from  queue  on  Fabric  ingress  that  suppose  to  egress  fabric  via  congested  port  

•  Packet  received  from  Fabric  are    –  stored  in  egress  port  

output  queue.    –  Or  dropped  if  queue  is  

full.  

•  Fabric queuing and flow-control independent form egress port queuing and scheduling. •  Router behavior – residency time (latency), jitter, drop rate depends on both.

Ingress PFEASIC/NPU

Pn VOQ

P2 VOQ P1 VOQ

egress PFEASIC/NPU

IF0 OQ

IF1 OQ

IFn OQ

IF0IF1

IFn

Ingress PFEASIC/NPU

Pn VOQ

P2 VOQ P1 VOQ

Fabric scheduler

Page 21: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

End-­‐to-­‐End  VOQ  system  

P1 Rx

P2 Rx

Pn Rx

P1 Tx

P2 Tx

Pn Tx

Switch Fabricswitch

Ingress PFEASIC/NPU

Fabric ports

Router ingressInterfaces

Buffermemory

egress PFEASIC/NPU

Buffermemory

•  For  queuing  purpose,  switch  fabric  and  all  de-­‐mux  –  seen  as  single  switch  –  N  inputs  (Rx)  à  M  output  (Tx)  –  N  x  Rx  speed  ==  M  x  Tx  speed  –  M  >>  N;  

Page 22: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Buffer  once  -­‐  latency  

Pn IFn VOQ

Pn IFn VOQ

Pn IFn VOQ

Pn IFn VOQ

Port P1 Rx

Port P2 Rx

Port Pn Rx

Port P1 Tx

Port P2 Tx

Port Pn Tx

P1 IFn VOQ P1 IF0 VOQ

P1 IFn VOQ P1 IF0 VOQ

P1 IFn VOQ P1 IF0 VOQ

P1 IFn VOQ P1 IF0 VOQ

IF0 TxIF1 Tx

IFn Tx

RQRQ

RQ/GR

RQ

RQ/GR

RQ/GR

Pn IFn VOQ

Pn IFn VOQ

Pn IFn VOQ

Pn IFn VOQ

Port P1 Rx

Port P2 Rx

Port Pn Rx

Port P1 Tx

Port P2 Tx

Port Pn Tx

P1 IFn VOQ P1 IF0 VOQ

P1 IFn VOQ P1 IF0 VOQ

P1 IFn VOQ P1 IF0 VOQ

P1 IFn VOQ P1 IF0 VOQ

IF0 TxIF1 Tx

IFn Tx

RQRQ

RQ/GR

RQ

RQ/GR

RQ/GR

•  Queues  ==  Buffer  ==  accommodate  burst  and  delay  it  

•  Burst  absorp@on  capability  DO  NOT  depends  on  type  of  burst    (regardless  at  which  point  conges@on  appears)  –  Max:  VOQ  size  –  Min:  VOQ  size  

•  Max  latency:  Σ  (VOQ  size)  

•  Wait,  there  will  be  example  Poten@al  conges@on  point  

Page 23: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Buffer  once  –  bandwidth  Fabric  Scheduler  and  flow-­‐control  

•  Fabric  Queuing  -­‐  VOQ  (per  egress  interface)  

•  Fabric  scheduler  and  Flow-­‐Control  –  Which  ingress  [PFE,  VOQ]  

get  GRANT  next.  (e.g.  fair-­‐share)  from  given  egress  PFE  

–  Monitor  egress  interface  and  stop  giving  GR  from  queue  on  Fabric  ingress  if  egress  interface  is  not  free  

•  Packet  received  from  Fabric  is  immediately  send  out  by  egress  interface  

•  Fabric queuing and flow-control depends on egress interface only.

P1 Rx

P2 Rx

Pn Rx

P1 Tx

P2 Tx

Pn Tx

Switch Fabricswitch

Ingress PFEASIC/NPU

Fabric ports

Router ingressInterfaces

Buffermemory

egress PFEASIC/NPU

Buffermemory

Fabric scheduler

Ingress PFEASIC/NPU

Pn IF0 VOQ

P2 IF0 VOQ P1 IF0 VOQ

Pn IF1 VOQ

P2 IF1 VOQ P1 IF1 VOQ

Pn IFn VOQ

P2 IFn VOQ P1 IFn VOQ

Ingress PFEASIC/NPU

Pn IF0 VOQ

P2 IF0 VOQ P1 IF0 VOQ

Pn IF1 VOQ

P2 IF1 VOQ P1 IF1 VOQ

Pn IFn VOQ

P2 IFn VOQ P1 IFn VOQ

egress PFEASIC/NPU

IF0IF1

IFn

Page 24: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

System  characteris@c  vs.  Queuing  architecture  

CIOQ   CIOQ  (w/  fabric  VOQ)  

E2E  VOQ  

Low  residency  @me   ✖   ✖   ✔  

High  load   ✖   ✔   ✔  

Low  power  footprint   ✖   ✖   ✔  

High  number  of  interfaces  (each  with  independent  queuing.  E.g.  BNG,  BE)  

✔   ✔   ✖  

mul@-­‐chassis  systems   ✔   ✔   ✔  

Examples*   C7500,    Early  C7600  

Juniper  MX,  Cisco  ASR9k,    CRS-­‐X  ALU  7750/7950*  

Juniper  PTX  Cisco  NCS600    

*  Please  contact  me  if  you  want  to  update,  correct  add  more  –  [email protected]  

Page 25: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

UNINTUITIVE  BEHAVIOR  -­‐  BANDWIDTH  

Page 26: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Intui@ve  drop  behavior  

F1  150G  

F2  50G  

F3  200G  

F4  100G  

F5  200G  

F6  1G  

F1  –  F5  

F6  

100GE  

100GE  

700G

 600/700  è  86%  loss  expected  

•  All  traffic  is  BE          

Intui've  (loss)?   Observed  2  (router  X)   Observed  2  (router  Y)  

Gbps   Loss  %   Gbps   Loss  %   Gbps   Loss  %  

F1   21   86%  

F2   7   86%  

F3   29   86%  

F4   14   86%  

F5   29   86%  

F6   1   0%  

Page 27: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Un-­‐intui@ve  drop  behavior  of  router  “X”  

F1  150G  

F2  50G  

F3  200G  

F4  100G  

F5  200G  

F6  1G  

F1  –  F5  

F6  

100GE  

100GE  

700G

 

Intui've  (loss)?   Observed  2  (router  X)  

Gbps   Loss  %   Gbps   Loss  %   Gbps   Loss  %  

F1   21   86%   25   82%  

F2   7   86%   8   82%  

F3   29   86%   22   88%  

F4   14   86%   11   88%  

F5   29   86%   33   83%  

F6   1   0%   0   52%  

•  All  traffic  is  BE  •  Why  losses  in  F6?  •  Why  unequal  losses  in  F1-­‐

F6  

Page 28: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Un-­‐intui@ve  drop  behavior  of  router  “Y”  

F1  150G  

F2  50G  

F3  200G  

F4  100G  

F5  200G  

F6  1G  

F1  –  F5  

F6  

Intui've  (loss)?   Observed  2  (router  X)   Observed  2  (router  Y)  

Gbps   Loss  %   Gbps   Loss  %   Gbps   Loss  %  

F1   21   86%   25   82%   26   83%  

F2   7   86%   8   82%   9   83%  

F3   29   86%   22   88%   22   89%  

F4   14   86%   11   88%   11   89%  

F5   29   86%   33   83%   33   84%  

F6   1   0%   0   52%   1   0%  

100GE  

100GE  

•  All  traffic  is  BE  •  Why  losses  in  F6?  •  Why  unequal  losses  in  F1-­‐

F6  •  No  losses  in  F6  on  router  Y  

700G

 

Page 29: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Know  your  hardware  

•  Router  X  -­‐>  CIOQ  •  Router  Y  -­‐>  VOQ  •  Fabric  port  -­‐>  300Gbps  

•  Fair-­‐share  fabric  scheduler  

F1  150G  

F2  50G  

F3  200G  

F4  100G  

F5  200G  

F6  10G  

F1  –  F5  

F6  PFE4  

Switch  fabric  

PFE3  

PFE2  

PFE1  

IF  1  (100GE)  

IF2  (100GE)  

300+  Gbps  fabric  port  

Page 30: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Router  X  –  CIOQ  -­‐  behavior  •  Flow  F1&F2  shares  same  buffer  

(queue)  on  PFE1.  Same  for  F3&F4  @  PFE2  and  for  F5&F6  @  PFE3  

•  There  is  3  ingress  PFE  that  want  to  talk  to  PFE4  –  Fabric  Scheduling  gives  100Gbps  to  each  ingress  PFE  

offered   From  fabric  @  PFE4   On  egress  interface  

Gbps   Gbps   Fabric  Loss  %  (F_loss)   Gbps   Cumula've  Loss  %  

F1   150   75   50%  

F2   50   25   50%  

F3   200   67   67%  

F4   100   33   67%  

F5   200   95.2   52%  

F6   10   4.8   52%  

}  100G  

}  100G  

}  100G  

F1  

F2  

F3  

F4  

F5  

F6  

F1  –  F5  

F6  PFE4  

Switch  fabric  

PFE3  

PFE2  

PFE1  

IF  1  (100GE)  

IF2  (100GE)  

Page 31: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Router  X  –  CIOQ  -­‐  behavior  •  Flows    F1  –  F5  (295Gbps)  are  queued  

in  OQ  of  IF1,  and  Tx  @  100Gbps.  (65%  loss  –  Egress  interface  loss;  E-­‐loss)  

•  Flows    F6  (10Gbps)  are  queued  in  OQ  of  IF2,  and  tx  @  100Gbps.  (0%  loss)  

offered   From  fabric  @  PFE4   On  egress  interface  

Gbps   Gbps   Fabric  Loss  %  (F_loss)   Gbps   Cumula've  Loss  %  

F1   150   75   50%   25   82%  

F2   50   25   50%   8   82%  

F3   200   67   67%   22   88%  

F4   100   33   67%   11   88%  

F5   200   95.2   52%   33   83%  

F6   10   4.8   52%   4.8   52%  

Cumula've  loss  for  F1-­‐F5:  F_loss  +  (1-­‐F_loss)*E_loss  

}  100G  

}  100G  

}  100G  

F1  

F2  

F3  

F4  

F5  

F6  

F1  –  F5  

F6  PFE4  

Switch  fabric  

PFE3  

PFE2  

PFE1  

IF  1  (100GE)  

IF2  (100GE)  

Page 32: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

CLI  example  

NPC3(eabu-sol-eng-be-mx480-2 vty)# sh cos halp fabric queue-stats 4 !!PFE index: 3 CCHIP 0 Low prio Queue: 4 !Queued : ! Packets : 4734895792 62812 pps! Bytes : 5734975634075 25125000 Bps !Transmitted : ! Packets : 0 31406 pps! Bytes : 0 12562500 Bps ! Tail-dropped pkts : 0 31406 pps! Tail-dropped bytes: 0 12562500 Bps ![...] !

PFE  3  

Des@na@on  PFE  (PFE4)  

50%  loss  

201  Gbps  

Page 33: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Router  Y  –  VOQ  -­‐  behavior  •  Flow  F1&F2  shares  same  buffer  (queue)  on  

PFE1.  Same  for  F3&F4  @  PFE2  and  for  F5  @  PFE3.  

•  Flow  F6  has  separate  buffer  on  PFE  3.  •  There  is  3  ingress  PFE  that  want  to  talk  to  

egress  interface  IF1  (100GE)  –  Fabric  Scheduling  gives  33Gbps  to  each  

ingress  PFE  for  Egress  Interface  IF1  VOQ  offered   From  fabric  @  PFE4   On  egress  interface  

Gbps   Gbps   Fabric  Loss  %  (F_loss)   Gbps   Cumula've  Loss  %  

F1   150   25   83%  

F2   50   8   83%  

F3   200   22   89%  

F4   100   11   89%  

F5   200   33   83%  

F6   10   10   0%  

}  33G  

}  33G  

}  33G  }  100G  

F1  

F2  

F3  

F4  

F5  

F6  

F1  –  F5  

F6  PFE4  

Switch  fabric  

PFE3  

PFE2  

PFE1  

IF  1  (100GE)  

IF2  (100GE)  

Page 34: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Router  Y  –  VOQ  -­‐  behavior  •  There  is  1  ingress  PFE  that  want  to  talk  to  

egress  interface  IF2  (100GE)  –  Fabric  Scheduling  gives  100Gbps  to  only  one  

ingress  PFE  (PFE3)  for  Egress  Interface  IF2  VOQ  

•  F6  do  not  consume  it’s  share  in  full.  Only  10Gbps.  

offered   From  fabric  @  PFE4   On  egress  interface  

Gbps   Gbps   Fabric  Loss  %  (F_loss)   Gbps   Cumula've  Loss  %  

F1   150   25   83%   25   83%  

F2   50   8   83%   8   83%  

F3   200   22   89%   22   89%  

F4   100   11   89%   11   89%  

F5   200   33   83%   33   83%  

F6   10   10   0%   10   0%  

Cumula've  loss  for  F1-­‐F5:  F_loss  

}  33G  

}  33G  

}  33G  }  100G  

F1  

F2  

F3  

F4  

F5  

F6  

F1  –  F5  

F6  PFE4  

Switch  fabric  

PFE3  

PFE2  

PFE1  

IF  1  (100GE)  

IF2  (100GE)  

Page 35: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

CLI  example  

SNGFPC1(Thorax-re0 vty)# debug cos halp qlen tq 3 voq 2048 !<snip> !! VOQ| AQID| qlen| qlenold| tabw| ntabw| maxrate(Mbps)| DBB Time(us)| !================================================================================ ! 560| 29771| 2752| 2304| 64| 14| 1613| 111813| !================================================================================ !!Number of Samples Collected = 1 ! Parm | Min| Avg| Max| !=============================== ! qlen | 2752| 2752| 2752| ! tabw | 64| 64| 64| !ntabw | 14| 14| 14| !qdrain| 33000| 33000| 33000| !FreePg| 259387| 259387| 259387| ! UM | 0| 0| 0| !

VOQ  of  egress  IF  (100GE)  

33Gbps  drain-­‐rate  

Ingress  PFE  3  (200Gbps  toward  egress  interface)  

Page 36: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Accept  your  router  personality  

•  Behavior  is  clear  now  J  -­‐  caused  by  Fabric  scheduler  –  Non  of  vendor  (AFAIK)  allows  for  Fabric  scheduler  tuning.  

–  Have  to  live  with  it  as  it  is.  F1  150G  

F2  30G  

F3  200G  

F4’’  33G  

F5  200G  

F6  1G   PFE3  

PFE2  

PFE1  

F4’  33G  

F4’’’  33G  

Mi@ga@on:  Load  PFE  fairly  •  you  get  fair  results  among  all  flows  •  Other  goodies  behind  –  blast  radius  

Page 37: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

ROUTER  BEHAVIOR  CASE  STUDY  -­‐  LATENCY  

Page 38: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Intui@ve  latency  

F1  200G  

F2  100G  

F3  100G  

F1  –  F3  100GE  

400G

 400Gbps  -­‐-­‐>  100Gbps  Buffer  100%  -­‐>  100ms  latency  And  75%  losses  

 Expected  (loss)?   Observed  1  

(router  X)   Observed  2  (router  Y)  

ms   ms   ms  

F1   100   200   100  

F2   100   200   100  

F3   100   100   100  

Page 39: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Know  your  hardware  

•  Router  X  -­‐>  CIOQ  –  100ms  VoQ  before  fabric  

–  100ms  OQ  

•  Router  Y  -­‐>  VOQ  •  100ms  VoQ  before  fabric  

•  Fabric  port  -­‐>  300Gbps  

F1  200G  

F2  100G  

F3  100G  

F1  –  F3  

PFE4  

Switch  fabric  

PFE3  

PFE2  

PFE1  

IF  1  (100GE)  

300+  Gbps  fabric  port  

Page 40: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Router  X  -­‐  CIOQ  •  Flow  F1&F2  shares  same  VoQ  buffer  

(queue)  on  PFE1.  Flow  F3  is  alone  on  PFE2  

•  There  is  2  ingress  PFE  that  want  to  talk  to  PFE4  –  Fabric  Scheduling  guarantee  150Gbps  to  each  ingress  PFE  

•  Flows    F1-­‐F2  (300Gbps)  are  queued  in  VOQ  of  PFE1,  and  Tx  @  200Gbps  (150Gbps  +  le�over).  Latency  is  100ms  

offered   Fabric  component   Egress  Interface   Total  Gbps   ms  

F1   200   100  (33%  loss)  

F2   100   100  (33%  loss)  

F3   100  

F1  F2  

F3  

F1-­‐F3  

PFE4  

Switch  fabric  

PFE3  

PFE2  

PFE1  

IF  1  (100GE)  

Page 41: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Router  X  -­‐  CIOQ  •  Flows    F3  (100Gbps)  are  queued  in  VOQ  of  PFE1,  and  Tx  @  100Gbps    à  no  buffering  

•  Flows  F1-­‐F3  (300Gbps)  are  queued  in  OQ  of  IF1,  and  Tx  @  100Gbps.  Egress  interface  latency  is  100ms  

F1  F2  

F3  

F1-­‐F3  

PFE4  

Switch  fabric  

PFE3  

PFE2  

PFE1  

IF  1  (100GE)  

offered   Fabric  component   Egress  Interface   Total  Gbps   ms   Ms   ms  

F1   200   100  (33%  loss)   100  (66%  loss)   200  (77%  loss)  

F2   100   100  (33%  loss)   100  (66%  loss)   200  (77%  loss)  

F3   100   0  (0%  loss)   100  (66%  loss)   100  (66%)  

Page 42: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Router  Y  -­‐  VOQ  •  Flow  F1&F2  shares  same  VoQ  buffer  (queue)  on  PFE1.  Flow  F3  is  alone  on  PFE2  

•  There  is  2  ingress  PFE  that  want  to  talk  to  PFE4  – Fabric  Scheduling  guarantee  50Gbps  to  each  ingress  PFE  

F1  F2  

F3  

F1-­‐F3  

PFE4  

Switch  fabric  

PFE3  

PFE2  

PFE1  

IF  1  (100GE)  

offered   Fabric  component   Egress  Interface   Total  Gbps   ms   ms   sm  

F1   200   100  

F2   100   100  

F3   100   100  

Page 43: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Router  Y  -­‐  VOQ  •  Flows    F1-­‐F2  (300Gbps)  are  queued  in  VOQ  of  IF1,  and  Tx  @  50Gbps.  Latency  is  100ms  

•  Flows    F3  (100Gbps)  are  queued  in  VOQ  of  IF1,  and  Tx  @  50Gbps  Latency  is  100ms  

F1  F2  

F3  

F1-­‐F3  

PFE4  

Switch  fabric  

PFE3  

PFE2  

PFE1  

IF  1  (100GE)  

offered   Fabric  component   Egress  Interface   Total  Gbps   ms   ms   sm  

F1   200   100  (83%  loss)   0   100  (83%  loss)  

F2   100   100  (83%  loss)   0   100  (83%  loss)  

F3   100   100  (50%  loss)   0   100  (50%  loss)  

Page 44: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

PFE4  

Switch  fabric  

PFE3  

PFE2  

PFE1  

Other  surprising  behavior  –  router  W  

•  All  traffic  is  BE    

F1  100G  

F3  100G  

F4  100G  

F5  100G  

F7  100G  

F1,  F4,  F6  

F3  

Observed  on  egress  

Gbps   Loss  %  

F1   33   67%  

F2   50   50%  

F3   100   0%  

F4   33   67%  

F5   25   75%  

F6   33   67%  

F7   25   75%  

100GE  

100GE  

F2  100G  

F6  100G  

F2,  F5,  F7  

Page 45: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Homework  

•  What  Queuing  architecture  router  W  is?  •  Explain  behavior.  

•  Answers:  [email protected]  – Deadline  –  11pm  today.  

Page 46: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Summary  

•  Know  your  router  anatomy  •  System  queuing  architecture  impacts  power  consump@on  and  system  scaling  capabili@es.  

•  System  queuing  architecture  impact  residency-­‐@me    

•  System  queuing  architecture  may  be  a  reasoned  for  non-­‐intui@ve  traffic  loss  paPern.  –  (Re)Assign  ports  to  roles  smart  way.  –  Trying  to  solve  of  non-­‐exis@ng  problem  cost  @me  and  headache  of  wri@ng  a  incident  report  –  avoid  it.  

Page 47: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)
Page 48: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

BACKUP  SLIDES.  

Page 49: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

From  building  blocks  to  centralized  router  

•  Single  switching  element  •  No  Switch  Fabric  •  N  x  N  Interfaces  •  Interfaces  may  have  different  speeds  •  Memory  used  to  build  egress  

interface  queues  •  Different  memory  op@ons  

–  On-­‐chip  shared  by  mux  and  de-­‐mux  •  Very  fast  (SRAM)  –  ~10Tbps+  •  Small  and  costly  (10’s  MB)  

–  Off-­‐chip  shared  by  mux  and  de-­‐mux  •  Deep  queues/buffers  (GB)  •  Slower  (DRAM)  –  ~1Tbps  

–  Off-­‐chip  memory  limits  PFE  performance.  

1 x

Tx

N x

Rx

N x

Tx

1 x

Rx

N x

Tx

N x

Rx

Page 50: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Combined  Input  Output  Queuing  

IQ  requairments  for  loss-­‐less:    Fabric  Port  speed  >>  Σ (ASIC  egress  interfaces)  

Pn RX

IF0 Tx

IF1 Tx

IFn Tx

P1 OQ

P2 OQ

Pn OQ

Port P1 Rx

Port P2 Rx

Port Pn Rx

Port P1 Tx

Port P2 Tx

Port Pn Tx

Page 51: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

OQ  •  Fabric  switch  is  not  a  

really  switch  here.  •  Need  very  high  speed  (N  

x  )  fan-­‐in  to  buffer.  –  If  switch  port  is  600Gbps,  and  we  have  100  ports  à    

–  60T  of  raw  bandwidth  into  buffers  !  

•  100%  efficiency  

•  Good  on  paper  only  –  extremely  expensive  –  Bound  by  technology  

P1 OQ

P2 OQ

Pn OQ

Port P1 Rx

Port P2 Rx

Port Pn Rx

N x Port Speed

N x N x Port Speed

Port P1 Tx

Port P2 Tx

Port Pn Tx

Page 52: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Buffer  once  –  per  egress  interface  VOQ  Theory  vs.  prac@ce  

•  In  stat-­‐mux  system  it  is  unpredictable  when  egress  interface  becomes  free.  

•  Grant  signaling  delay  affect  deficiency  

•  Need  for  shallow  buffer  a�er  fabric  (egress  PFE)  to  compensate  delay.  –  Need  just  ~10  usec.  

•  Similar  to  CIOQ  but:  –  Fabric  flow-­‐control  ensure  

that  OQ  never  overflow.  –  Try  to  keep  IQ  always  full,  

never  empty  (if  data  are  in  VOQ)  

–  Egress  interface  free/busy  indirectly  controls  fabric  flow-­‐control  

Egr

ess

Inte

rface

bus

y

RQ

GR

DATA

Egr

ess

Inte

rface

bus

y

Data in VOQ Data in shallow output buffer

Egr

ess

Inte

rface

bu

sy

Data in shallow output buffer

Page 53: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Buffer  once  –  per  egress  interface  VOQ  Theory  vs.  prac@ce  

•  In  stat-­‐mux  system  it  is  unpredictable  when  egress  interface  becomes  free.  

•  Grant  signaling  delay  affect  deficiency  

•  Need  for  shallow  buffer  a�er  fabric  (egress  PFE)  to  compensate  delay.  –  Need  just  ~10  usec.  

•  Similar  to  CIOQ  but:  –  Fabric  flow-­‐control  ensure  

that  OQ  never  overflow.  –  Try  to  keep  IQ  always  full,  

never  empty  (if  data  are  in  VOQ)  

–  Egress  interface  free/busy  indirectly  controls  fabric  flow-­‐control  

Egr

ess

Inte

rface

bus

y

RQ

GR

DATA

Egr

ess

Inte

rface

bus

y

Data in VOQ Data in shallow output buffer

Egr

ess

Inte

rface

bu

sy

Data in shallow output buffer

Page 54: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Too  shallow  shallow  output  buffer  •  Latency  

–  3  x  fabric  one-­‐way  latency  is  worst  case  (RQ-­‐GR-­‐DATA)  

–  RQ  scheduler  on  egress  –  GR  scheduler  on  ingress  

•  RQ  latency  –  Can’t  be  compensated  –  Sta@s@cally  minor  problem  –  

asynchronous.  RQ  could  be  send  while  egress  interface  handles  other  data  

•  RQ/GR  scheduler  -­‐  Can’t  be  compensated  

•  If  shallow  buffer  <  2x  latency  –  inefficient  egress  IF  u@liza@on  

Egr

ess

Inte

rface

bus

y

RQ

GR

DATA

Egr

ess

Inte

rface

bu

sy

Data in VOQ

Data in shallow output buffer

Egr

ess

Inte

rface

bu

sy

RQ

la

tenc

y lo

ss

Page 55: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

Impact  on  PFE  design  

Page 56: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

PFE  complexity  •  PFE  for  per-­‐egress  IF  VOQ  system  !=  PFE  for  CIOQ  system  

•  For  system  of  4k  IF  and  8  QoS  Classes  – VOQ  PFE  need  support  32.000  queues  –  CIOQ  PFE  need  1.040  queues  

•  If  PFE  support  400k  queues  – VOQ  system  can  support  50k  IF  –  CIOQ  system  can  support  1.000.000’s  IF  (from  queue  scaling  perspec@ve  only.  Other  limits  apply)  

•  CIOQ  PFE  has  typically  less  queues  but  much  more  of  other  func@onali@es.  

Page 57: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

PFE  performance  •  To  handle  4  x  100GE  interfaces,  PFE  

need:  –  CIOQ:  

•  more  then  3.6Tbps  of  PFE  I/O.  •  All  packet  goes  to  and  from  memory  

twice  –  4  memory  accesses.  –  VOQ    

•  more  then  2.4Tbps  (25%  less)of  PFE.  •  All  packet  goes  to  and  from  memory  

once  –  2  memory  accesses.  –  Memory  I/O  BW  need  to  be  

oversized    -­‐  even  bePer  reduc@on  (~30%)  

–  Each  memory  access  causes  latency  •  Less  I/O  ==  saved  gates  

–  Have  more  VOQ  (bigger  system),  OR  –  lower  cost  and  power,  OR  –  higher  performance,  OR  

PFE

Memory DRAM

Mem I/O

Fabr

ic I/

O

Ext

erna

l IF

I/O

400

400

400

400

•  1600 (CIOQ) or

•  800 (VOQ)

Page 58: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

CoS  vs.  Queuing  •  Queuing  as  discussed  so  fare  

–  manage  conges@on/overload  –  Assumes  all  traffic  is  same  class  

•  CoS  –  Traffic  has  different  classes  –  Each  class  need  other  treatment  

•  CoS  +  Queueing  =  QoS  –  Scheduler(s)  take  into  account  2  things  

•  Class  of  traffic  •  Source  instance  (e.g.  ingress  PFE)  

–  What  was  1  Queue  becomes  set  of  parallel  queues.  –  Classifier  needed  -­‐>  put  traffic  to  this  or  other  queue  (out  of  scope)  

Class n

Class 1Class 0

Class n

Class 1Class 0

egress 0egress 1

egress n

egress PFE 0egress PFE 1

egress PFE n

IQ VoQ

no C

oS

Co

S(4

cla

sses

)

Queueing vs. QoS(ingress PFE)

Page 59: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

PFE  complexity  •  PFE  for  per-­‐egress  IF  VOQ  system  !=  PFE  for  CIOQ  system  

•  For  system  of  4k  IF  and  8  QoS  Classes  –  VOQ  PFE  need  support  32.000  queues  –  CIOQ  PFE  need  1.040  queues  

•  If  PFE  support  400k  queues  –  VOQ  system  can  support  50k  IF  –  CIOQ  system  can  support  1.000.000’s  IF  (from  queue  scaling  perspec@ve  only.  Other  limits  apply)  

•  CIOQ  PFE  has  typically  less  queues  but  much  more  of  other  func@onali@es.  

Page 60: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

CIOQ-­‐  Fabric  •  CIOQ  •  Fabric  VOQ  

–  Flow-­‐Controll  •  per  Fabric  Egress  •  By  Request  –  Grant  protocols.  •  Do  not  depends  on  Egress  

interfaces  and  OQ  state.  –  2  Classes  Hi-­‐/Low-­‐  priority  

•  classifica@on  –  Scheduling  

•  ingress  PFE  fairness  1st  •  CIR-­‐bound  Priority  scheduling.    

–  Delay  Bandwidth  Buffer:  •  Scheduler  logic  seats  on  egress  

PFE;  Memory  is  on  ingress  side  of  fabric.  

root  

Ingress  PFE1  

Hi-­‐Priority  Queue  

Low-­‐priority  Queue  

Ingress  PFE2  

Hi-­‐Priority  Queue  

Low-­‐priority  Queue  

Ingress  PFEn  

Hi-­‐Priority  Queue  

Low-­‐priority  Queue  

SWITCH FABRIC

Page 61: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

CIOQ  –  Egress  Interface  •  CIOQ  •  Egress  (logical)  Interface  Queues  

–  8  Queues  per  (logical)  Interface    –  10M  queues  per  system  –  Scheduling  

•  5  priori@es  level,  4  RED  profiles  per  queue.  Configurable.  

•  2  or  4  scheduling  hierarchy  level  w/  priority  propaga@on  

–  Delay  Bandwidth  Buffer:  ______  •  Scheduler  logic  and  Memory  seats  

on  egress  PFE  Egress  

Interface  1  

logical  interface  set  

Logical  interface  

Logical  interface  

logical  interface  set  

Logical  interface  

Queue  0  

Queue  1  

Queue  2  

Queue  3  

Queue  4  

Queue  5  

Queue  6  

Queue  7  

Logical  interface  

root  

Ingress  PFE1  

Hi-­‐Priority  Queue  

Low-­‐priority  Queue  

Ingress  PFE2  

Hi-­‐Priority  Queue  

Low-­‐priority  Queue  

Ingress  PFEn  

Hi-­‐Priority  Queue  

Low-­‐priority  Queue  

clas

sific

atio

n

Page 62: Strategies)of)packetbuffering) inside)Routers) · Strategies)of)packetbuffering) inside)Routers))) Rafal)Jan)Szarecki)#JNCIE136) Solu@on)Architect,)Juniper)Networks)

VOQ  –  integrated  scheduler  •  Flow-­‐Control  

–  per  egress  (logical)  interface  –  By  Request  –  Grant  protocols.  –  Depends  on  Egress  interfaces  OQs  (Output  Queues)  

state.  •  Queues  

•  8  per  egress  (logical)  interface  •  390.000  per  system  (HW  limit)  

•  Scheduling  –  4  priori@es  level,  4  RED  profiles  per  queue.  

Configurable.  –  2  or  3  scheduling  hierarchy  level  w/  priority  

propaga@on  •  Delay  Bandwidth  Buffer:  

–  Shared  memory  –  100ms  upper  bound  –  40ms  worst  case.  

•  Scheduler  logic  seats  on  egress  PFE;  Memory  is  on  ingress  side  of  fabric  

Egress  Interface  

Logical  interface  

Logical  interface  

Logical  interface  

Ingress  PFE1  

VOQ_Class_0  

VOQ_Class_1  

VOQ_Class_2  

VOQ_Class_3  

VOQ_Class_4  

VOQ_Class_5  

VOQ_Class_6  

VOQ_Class_7  

Ingress  PFE2  

VOQ_Class_0  

VOQ_Class_1  

VOQ_Class_2  

VOQ_Class_3  

VOQ_Class_4  

VOQ_Class_5  

VOQ_Class_6  

VOQ_Class_7  

Ingress  PFE3  

Ingress  PFEn  

Logical  interface  

SWITCH FABRIC


Configuring Routers

Configuring Routers

Documents
Manual Routers

Manual Routers

Documents
Conf routers

Conf routers

Technology
Thinking inside the box data permission strategies - Wednesday 18 November

Thinking inside the box data permission strategies - Wednesday 18 November

Marketing
Inside Strategies for Diagnosing Abiotic and Biotic Problems

Inside Strategies for Diagnosing Abiotic and Biotic Problems

Documents
The Agenda And Strategies Inside The Church - rhm-net.org Point Shows/Jezebel and Bitteroot... · The Agenda And Strategies Inside The Church. The Church In Thyatira (odor of affliction)

The Agenda And Strategies Inside The Church - rhm-net.org Point Shows/Jezebel and Bitteroot... · The Agenda And Strategies Inside The Church. The Church In Thyatira (odor of affliction)

Documents
Inside the Router Routers are computers Router CPU and Memory Internetwork Operating System Router Bootup Process Router Ports and Interfaces Routers and

Inside the Router Routers are computers Router CPU and Memory Internetwork Operating System Router Bootup Process Router Ports and Interfaces Routers and

Documents
CS 3035/GZ01: Networked Systems Kyle Jamieson Department of Computer Science University College London Inside Internet Routers

CS 3035/GZ01: Networked Systems Kyle Jamieson Department of Computer Science University College London Inside Internet Routers

Documents
festool routers

festool routers

Documents
Routers, Oh My! Hubs, Switches, and Routers, Oh My!

Routers, Oh My! Hubs, Switches, and Routers, Oh My!

Documents
INTERACTIVE DVD INSIDE! 4-Real-Worl STRATEGIES Tips TO …the-eye.eu/public/Books/retrowith.in/Dragon Magazine... · 2018. 8. 22. · interactive dvd inside! 4-real-worl strategies

INTERACTIVE DVD INSIDE! 4-Real-Worl STRATEGIES Tips TO …the-eye.eu/public/Books/retrowith.in/Dragon Magazine... · 2018. 8. 22. · interactive dvd inside! 4-real-worl strategies

Documents
Chapter 5 WANs and Routers/ Introduction to Routers

Chapter 5 WANs and Routers/ Introduction to Routers

Documents
Routers planers

Routers planers

Education
Switches & Routers

Switches & Routers

Documents
DECEMBER Instructional Design Team 2013 Strategies 4 Sakai Stuff-ers 5 Inside Story Inside Story Inside Story INSIDE THIS ISSUE: 2 ID Team Support Portal The ID Team Support Portal

DECEMBER Instructional Design Team 2013 Strategies 4 Sakai Stuff-ers 5 Inside Story Inside Story Inside Story INSIDE THIS ISSUE: 2 ID Team Support Portal The ID Team Support Portal

Documents
Bridges Routers

Bridges Routers

Documents
Inside Internet Routers

Inside Internet Routers

Documents
Inside Sales Strategies for MSPs

Inside Sales Strategies for MSPs

Technology
Thinking inside the box: data permission strategies - 18 November

Thinking inside the box: data permission strategies - 18 November

Marketing
TCP & Routers

TCP & Routers

Documents
Orbi routers

Orbi routers

Others
Securing Your Home Routers - Trend Micro Internet … | Securing Your Home Routers: Understanding Attacks and Defense Strategies Figure 1: A trading website that displays a list of

Securing Your Home Routers - Trend Micro Internet … | Securing Your Home Routers: Understanding Attacks and Defense Strategies Figure 1: A trading website that displays a list of

Documents
Mobile Strategies for Recuitement and Onboarding - Webinar by JWT INSIDE & Mirum

Mobile Strategies for Recuitement and Onboarding - Webinar by JWT INSIDE & Mirum

Business
Inside this issue: 1 Learning Strategies - Ministry of … Strategies Inside this issue: Leaning Strategies 1 Culture Coner 2 Common Mistakes in English Made by Thais 2-3 Here’s

Inside this issue: 1 Learning Strategies - Ministry of … Strategies Inside this issue: Leaning Strategies 1 Culture Coner 2 Common Mistakes in English Made by Thais 2-3 Here’s

Documents
Routers & Switches

Routers & Switches

Engineering
No Time for Testing: Strategies to Keep Testing Inside Your Sprints

No Time for Testing: Strategies to Keep Testing Inside Your Sprints

Software
Inside the MOOC – An argumentation analysis of MOOC Implementation strategies

Inside the MOOC – An argumentation analysis of MOOC Implementation strategies

Education
Inside the Big Oil Playbook: Strategies and Tactics Used in the

Inside the Big Oil Playbook: Strategies and Tactics Used in the

Documents
Routers Configuration

Routers Configuration

Documents
people.uwplatt.edupeople.uwplatt.edu/~yangq/CSSE411/csse411-materials/f13/... · Web viewServers, routers, switches, and any other important hardware is normally found inside a closet

people.uwplatt.edupeople.uwplatt.edu/~yangq/CSSE411/csse411-materials/f13/... · Web viewServers, routers, switches, and any other important hardware is normally found inside a closet

Documents