Potential and Challenges for Silicon Photonics in …...Workshop on Silicon Photonics for High Performance Computing, Fort Collins, CO, May 2018. • C. L. Schow, “Ultra-Low Power

INTREPID UpdateARPA-E Annual ReviewOctober 31th, 2019

Presenters:Clint Schow: Overview and progress updatesKatharine Schmidtke: Data center perspective and T2MAdel Saleh: Network architecture and modeling

Representing contributions from the entire INTREPID team

INTREPID Project Objectives

2

‣ Making Coherent Optics in the Data Center a Reality– Demonstrate energy efficient coherent engines compatible with

package integration– Establish path to low power links at 200Gb/s/λ and beyond– Leverage large sensitivity gains to enable new architectures

based on photonic routing and switching‣ Unique approach: Use Optical Phased Locked Loops (OPLLs) to

eliminate DSP for orders of magnitude reduction in power‣ Overturn the conception that coherent = high power consumption‣ Explore and model novel architectures enabled by the expanded link

budget of coherent receivers‣ Phase 2 Goal: Pave the way for transition to widespread deployment

and commercial success

Phase 1 INTREPID Team: UCSB and Facebook

3

‣ Highly multidisciplinary, expertise spanning many fields– Datacenter operation, network design and management– Requirements and roadmap for optics at cloud-scale– Global optical supply chain, expertise and ability to drive– Electronic/photonic integration and co-design– Low-power, high-speed circuit design– Photonic integrated circuits: design and fabrication– High-speed packaging, assembly, and characterization– Network architecture design and modeling

‣ Phase 2 Team will add a partner with extensive engineering and manufacturing resources to enable technology transition

Facebook: Katharine Schmidtke (co-PI), Hans-Juergen Schmidtke, Ariel Hendel, James Stewart, Todd Hollmann, Jimmy WilliamsUCSB Faculty: Clint Schow (co-PI), James Buckwalter, Larry Coldren, Jonathan Klamkin, Adel SalehUCSB Students and Post-Docs: Hector Andrade, Takako Hirokawa, Navid Hosseinzadeh, Junqian Liu, Aaron Maharry, Thomas Meissner, Stephen Misak, Luis Valenzuela, Yujie Xia, Shireesh Bhat, Fabrizio Gambini, Sergio Pinna

INTREPID Highlights from Phase 1Technology Development: Hardware• Targets: 200 Gb/s analog coherent links

50 Gb/s NRZ VCSEL links• Si & InP PICs, SiGe electronics, VCSEL arrays,

high-speed packaging

Technology to Market• Top-down approach, driven by operational needs at cloud-scale

– Prioritizing critical metrics for market success: power, reliability,cost, volume manufacturing

– Exploring business models for ecosystem development– Developing and publishing open interfaces

• Lightcounting market research study on co-packaged optics• Co-Packaged Optics Collaboration (CPO) led by Facebook and Microsoft

Network Architecture• Developed folded Clos data center architectures with AWGRs, end-of-row

(EoR) switching for increased scalability, economy and energy efficiency• Built a Python simulation platform to investigate a wide range of

architectures/technologies• Devised various energy efficient, optical-switching-based architectures,

including a cost-effective, energy-efficient data center growth strategy using quasi-static optical switches

High-Speed Electronics and Packaging

InP PICs: Coherent RX and TX

Si PICs: Coherent RX

Coherent Link Modeling~5pJ/bit possible

50Gbd/s, dual-pol, QPSK

Tunable laser/LO

INTREPID Review, September 2019

ToR Switches with Server and Fabric InterfacesAnalog Coherent WDM links• “Up”: all switch-to-switch connections, up to 2km over SMF• OPLL (Optical Phased Locked Loop) No/very little DSP• Expanded link budgets to enable photonic routing/switching• 1310nm: uncooled lasers, dispersion not an issue• Target: 800Gb/s/fiber = 4λ @ 200Gb/s/λ (dual-pol QPSK, 50 Gbd/s)

Multimode VCSEL links• “Down”: ToR-server connections (30m)• Currently all copper today challenging cost point • 50G (Phase 1)100G (Phase 2)

Photonic interfaces integrated in first-level switch packages:

• Modular: common interface to switch ASIC• Optimum I/O architecture between ASIC and

optics to maximize efficiency and performance

5INTREPID Review October 2018

Phase 2: Focus on analog coherent transceivers

High-Speed Packaging and Test

6INTREPID, ENLITENED SPECIAL SESSION, OFC 2019

TX with custom OptiGOT VCSEL

Limiting Amplifier

OPLLTestbed

TDR of Connector Launch

Fully Functional InP Coherent RX PIC


Signal in

Local Oscillator

Back mirror Gain

Front Mirror

PhaseSOA

SOA

2 X 4 MMIPDs

Layout

Fabricated Device1 mm

Gen 1 PICs functional• LO lasers tunable over 26 nm• PD BW >20 GHz• PD responsivity ~1 A/WGen 2 Improvements:• Epi modification to improve laser efficiency• Semi-insulating substrate for higher speed

and easier packaging

Schematic LO tuning response

Photodiode BW

Photodiode spectral response

COTS OPLL Testbeds


Gen 1

4.34 4.35 4.36 4.37 4.38 4.39 4.4

Frequency (GHz)

-60

-55

-50

-45

-40

-35

Mag

nitu

de (d

Bm)

Stable FWHM = 6.336 MHz

Frequency locking (TIA output):

Gen 2: Oct 2019

• OPLL testbed for PIC verification and testing• COTS TIAs and XOR to be replaced with single custom ASIC

(first-gen OPLL chip received 10/18/2019)

Loop OFF39 MHz LW

Loop ONLW: 6 MHz

DP-QPSK Dual-Pol OPLL Receiver

• Multiple designs of two-channel RX and two-channel RX with OPLL in fab

• Global Foundries GFUS8XP process• Novel design consolidating functional blocks

expected to reduce power dissipation up to 2X– 1.69 pJ/bit for OPLL chip


Chip 1

Chip 2

Generalized Block Diagram of QPSK Costas Loop Architecture

Used in Receiver OPLL Integrated Circuit

1st Generation OPLL RX

Received 10/18/2019


Functional Gen1 Si Coherent Receiver

• All functional blocks verified• Now mounted on high-speed

test board for full coherent RX characterization

30Gb/s

BW>40GHz

Photodiodes

4x4 MMIPC

Schematic: Dual-Pol Coherent RX

PIC fabricated in Global Foundries 9WG process

Gen2 Silicon PICs


• GF 9WG Process• Expected Mid-November 2019• Devices:

• 3 coherent receivers:• Splits on 2 heater types and 4x4 MMI

vs directional couplers• 2 IQ Modulators

• Traveling wave• Segmented

• 1 MZM + monolithically integrated driver

Loaded electrode Phase shifter Full TW MZM

47 GHz 43 GHz10 GHzdue to velocity mismatch

Traveling wave modulator: Gen1 Gen26mm x 6 mm tile

Coherent Link Modeling

-10 to 40 dBm -10 to 40 dBm

Sensitivity:50 uA

Traveling wave or Segmented• Length• 𝑉𝑉π𝐿𝐿π• α𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜• Differential drive• Total capacitance

η𝑜𝑜𝑜𝑜𝑙𝑙𝑙𝑙𝑙𝑙 = 20%

η𝐿𝐿𝐿𝐿 = 20%

13 dB

20% efficiency

150 mW per TIA

204 mW for OPLL circuit R = 1.0 A/W

𝑉𝑉𝑆𝑆𝑜𝑜𝑆𝑆𝑆𝑆𝑜𝑜𝑜𝑜

Assumed:• BER 10−12• 56 GBd


1. Find optimal LO and TX laser operating point on blue curve.2. Find energy per bit (EPB) for corresponding TX laser power along orange curve.

Full-Link Energy Efficiency


Si TW-MZM

1

2

InP TW-MZM

56 GBdBER 10−12

• Si photonic receiver—support for dual-pol• Significant efficiency advantage for traveling-wave modulators

compared to segmented designs• Significant efficiency advantage for InP vs Si modulators

Talks and PublicationsPlenary Talks• L. A. Coldren, " Photonic Integrated Circuits for Coherent Communication and Sensing," MOC '18,

Plenary talk, Taipei, October 16, 2018 (Micro-optics Conf.)• C. Schow, “Opportunities for Si Photonics in Next-Generation Data Centers” IEEE Components and

Packaging Symposium Japan, Plenary talk, Kyoto, Japan 2018.Invited Talks and Workshop Presentations• C. L. Schow, PANEL SESSION 1: Optics Invading Copper? The Future of Backplane Communication, CSICS,

Miami, October 2017.• C. L. Schow, “Low Power Analog Coherent Links for Next-Generation Datacenters, Next-Generation Optical

Networks (NGON), San Francisco, CA, October 2017 .• C. L. Schow, “Can Coherent Optics Invade the Datacenter, and then HPC Systems?,” North American

Workshop on Silicon Photonics for High Performance Computing, Fort Collins, CO, May 2018.• C. L. Schow, “Ultra-Low Power Short-Reach Interconnects for 100G and Beyond,” 6th International

Symposium for Optical Interconnect in Data Centres, ECOC, Rome, Italy, Sept. 2018.• C. L. Schow, “Low Power Analog Coherent Links for Next-Generation Datacenters,” JIEP OPT Workshop,

Tokyo, Japan, November 2018.• C. L. Schow and K. Schmidtke, “INTREPID: Developing Power Efficient Analog Coherent Interconnects to

Transform Data Center Networks,” Open Compute Project (OCP) Global Summit 2019, San Jose, CA, March 2019.

• C. L. Schow and K. Schmidtke, “INTREPID: Developing Power Efficient Analog Coherent Interconnects to Transform Data Center Networks,” Optical Fiber Communication (OFC) Conference 2019, Paper M4D.9, San Diego, CA, USA, Mar. 2019.

• C. L. Schow, “Beyond 400G for Hyperscale Data Centers,” Optical Fiber Communication (OFC) Conference 2019, panelist, San Diego, CA, USA, Mar. 2019.


Talks and Publications (continued)Invited Talks (continued)• C. L. Schow, “Low Power Analog Coherent Links for Next-Generation Datacenters,” Conference on Lasers and

Electro-Optics (CLEO) 2019, paper STh4N.3, San Jose, CA, USA, May 2019.• C. L. Schow, “Low Power Coherent Links to Enable New System Architectures,” 2nd North American

Workshop on Silicon Photonics for High Performance Computing, Estes Park, CO, USA, May 2019.• J. Buckwalter, “Picojoule-per-Bit Coherent Links through Co-Design of Photonic and Electronic ICs,” European

Conference on Optical Communication (ECOC), paper Tu.1.E.1, Dublin, Ireland, Sept. 2019.

Journal and Conference Papers• S. Arafin, L. A. Coldren, "Advanced InP Photonic Integrated Circuits for Communication and Sensing," JSTQE,

24 (1) Jan/Feb, 2018. Invited Paper• H. Andrade et al., “Monolithically-Integrated 50Gbps 2pJ/bit Photoreceiver with Cherry-Hooper TIA in 250nm

BiCMOS Technology,” Optical Fiber Communication (OFC) Conference 2019, Paper M3A.5, San Diego, CA, USA, Mar. 2019.

• T. Hirokawa, S. Pinna, J. Klamkin, J. Buckwalter, C. L. Schow, “Energy Efficiency Analysis of Coherent Links for Datacenters,” IEEE Optical Interconnects Conference, paper TuC1, Santa Fe, NM, April 2019.

• A. Maharry, H. Andrade, T. Hirokawa, J. F. Buckwalter, and C. L. Schow, “A Novel Architecture for a Two-Tap Feed-Forward Optical or Electrical Domain Equalizer Using a Differential Element,” IEEE Photonics Conference, paper TuH2.2, San Antonio, TX, Oct. 2019.

Workshops Organized• “Si Photonics for the data centre” ECOC, Gothenberg, Sweden, September 2017.• “Has the time come for coherent optics in the data centres?” ECOC, Rome, Italy, 2018.• “What Will Drive the Transition to Coherent Intra-Data-Center Optics?” OFC, San Diego, CA, 2020


Coherent Optics is Coming to Datacenters


2018

2019

2020

Tech to Market

Katharine Schmidtke

Could-Scale Drives the Market

Global Sales of Ethernet Transceivers by Application

Source: LightCounting

$ Bi

llion

0

4

8

2

6

Forecast

T2M Critical Metrics

• Power consumption

• Reliability

• Cost

• Manufacturing volume


Business Models for Co-Packaged

• Switch driven

• Photonics driven

• OEM/ODM driven

• Customer driven


Building an Ecosystem: Need for Open Interfaces

• INTREPID interface standards shared with the industry through open forums such as OCP, TIP or other industry MSA group


• Build ecosystem to enable innovation• JDF Collaboration with Microsoft

• Three discussion documents:• Optical Module (Released)• Optional External Light Source• Co-Packaged Optics Assembly

• http://www.facebook.com/CoPackagedOpticsCollaboration

Co-Packaged Optics (CPO)

http://www.facebook.com/CoPackagedOpticsCollaboration

Network Architecture

Adel Saleh

Photonic Integration of Switching ASIC using INTREPID’sCo-Packaged, Coherent, WDM Optics – Schematic View


• Analog Coherent, Dual-Pol, QPSK with WDM over SM Fiber– Basic Symbol Rate: 𝝆𝝆 = 𝟓𝟓𝟓𝟓 Gbaud(same as bit rate per server &bit rate per switch ASIC port)

– Multiplexing Factor: 𝝁𝝁 = 𝟐𝟐 × 𝟐𝟐 = 𝟒𝟒 b/symbol

– Bit Rate per Wavelength: 𝒓𝒓 = 𝝁𝝁𝝆𝝆 = 𝟐𝟐𝟓𝟓𝟓𝟓 Gb/s

– Number of Wavelengths per fiber: 𝝂𝝂 = 𝟒𝟒 (up to 𝟏𝟏𝟏𝟏)

– Bit Rate per Fiber: 𝑹𝑹 = 𝝂𝝂𝒓𝒓 = 𝟖𝟖𝟓𝟓𝟓𝟓 (up to 𝟑𝟑𝟐𝟐𝟓𝟓𝟓𝟓) Gb/s

• For the 𝟓𝟓𝟏𝟏.𝟐𝟐 Tb/s Target ASIC– ASIC radix, 𝒏𝒏 = 𝑻𝑻/𝝆𝝆 = 𝟏𝟏𝟓𝟓𝟐𝟐𝟒𝟒– Switch Radix, 𝑵𝑵 = 𝒏𝒏/𝝁𝝁𝝂𝝂 = 𝟏𝟏𝟒𝟒 (to 𝟏𝟏𝟏𝟏)

• # of Supported Servers ∝ 𝒏𝒏𝑵𝑵𝑳𝑳−𝟏𝟏, where 𝑳𝑳 is the number of switching levels

bit rate (b/s)𝝆𝝆 𝒓𝒓 = 𝝁𝝁𝝆𝝆 𝑹𝑹 = 𝝂𝝂𝒓𝒓

𝑵𝑵 Integrated Ports𝑹𝑹 b/s per Port

WD

M M

UX/

DM

X

Mx/D

xMx

/Dx

Mx/D

x𝝁𝝁

𝝂𝝂

𝝁𝝁

𝝁𝝁

SMF

Modulation& Dual-Pol MUX/DMX

Target Switching ASIC Throughput, 𝑻𝑻 = 𝟓𝟓𝟏𝟏.𝟐𝟐 Tb/s

Conventional, 3-Level, ToR-Based Data Center with 800 Gb/s per Fiberwith an Oversubscription Ratio (OSR) = 1, it Supports 131,072 Servers


• Number of the 51.2 Tb/s Spine and Aggregation Switches = 384• Number of 6.4 Tb/s ToR switches = 2,048• If a higher OSR is used, which is often done to reduce cost, the resulting

bandwidth constriction can reduce the system utilization for multi-rack communications The above 3-D representation is similar to that used by Alexey Andreyev,

https://code.fb.com/production-engineering/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/ Nov. 14, 2014

https://code.fb.com/production-engineering/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/

EoR-Based Data Center with 4-λ, 800 Gb/s per Fiber having only two electronic switching levels and one intermediate layer of 4 x 4 AWGRs

with an OSR = 1, it also Supports 131,072 Servers


• Number of the 51.2 Tb/s spine and aggregation switches = 384 – same as before• The 2048 ToR switches have been replaced by the same number of 4x4 AWGRs• thus saving cost, power and latency (due to reducing the number of switching levels)• Also, if a higher OSR is used, the system utilization within an EoR cluster of racks is

markedly increased because of replacing the single-rack ToRs with multi-rack EoRs




• Same number of 51.2 Tb/s switches as before, but now with 512 8x8 AWGRs• Same cost, power and latency savings as previous design, and same improvement in

the overall system utilization within an EoR cluster of racks if a higher OSR is used• An improvement over the previous design is that the degree of integration is doubled,

thus the number of fibers per switch is halved, without reducing the number of servers




• The degree of integration is again doubled by using 16 λs and 16x16 AWGRs• All the previous performance improvements are maintained, but the number of fibers

per switch is halved again, without reducing the number of supported servers … But,• is it more feasible to keep increasing the degree of transceiver integration and reduce

their number, or to co-package a larger number of simpler transceivers with the ASIC ?

Integration versus Scalability of Data CentersComparing ToR-Based, 3-Level, and EoR/AWGR-Based, 2-Level, Designs


Integration versus Scalability of Data CentersAdding EoR-Based, 2-, 3- and 4-Level Designs


Outlook/Next Steps

‣ Finish assembly and characterization of Phase 1 VCSEL and analog coherent links

– 8XP ICs received 10/18/2019– 9WG Si photonics expected 11/12/2019

‣ Meet Phase 1 metrics‣ Update and refine link performance and power modeling‣ Transition to Phase 2‣ Carry out IP-based simulations comparing the performance

of ToR-, EoR- and AWGR-based data center designs‣ Assess the cost, performance and energy efficiency of data

centers utilizing quasi-static and/or dynamic optical switches

31

Documents

Potential and Challenges for Silicon Photonics in …...Workshop on Silicon Photonics for High Performance Computing, Fort Collins, CO, May 2018. • C. L. Schow, “Ultra-Low Power