10Gbps transfers

Presenter: Christian Charette, John Tkaczewski

Duration: 30 minutes

May Webinar: 10Gbps transfers

Presenter Introduction

| © Copyright FileCatalyst, 20132

John Tkaczewski• Co-Founder and President of Unlimi-Tech

Software

• Oversees creative and technical vision for FileCatalyst Webmail and FileCatalyst Workflow

• Blog: http://www.filecatalyst.com/blog

Christian Charette• Director of Software Development• Project lead on 10 Gbps research

Agenda

• FileCatalyst Direct Overview • Quick refresher why TCP doesn’t work in high speed networks• 10Gbps test infrastructure

– Ie: what are we using to perform the demo?

• Watch the data fly!• Questions (please type them in as we go)


FileCatalyst Direct Overview


• Software architecture is Client-Server.• One side acts as the main file repository (FileCatalyst Server)• One or many clients connect to server, and push/pull data• Clients include:

– automated interfaces (API, CLI, Javascript, C++)– web components (applets)– feature rich applications (HotFolder, Express, mobile)

• Primarily written in Java, works on every major platform (Windows, Linux, Mac OSX, iOS, Android, and more)

• Transport protocol utilizes proprietary UDP, with error correction and congestion control built into the application layer (instead of relying on TCP network layer)

Putting 10 Gbps in context


• 10 Gbps transfers an average BlueRay disk (26GB) in under 30 seconds

• Cost of 10 Gbps dedicated lines is dropping – becoming cost effective to many companies (~10K/month US to UK)

• 1 Gbps fiber connections becoming available to residential consumers

TCP vs UDP for packet loss & latency


• Quick refresher: TCP can’t cut it. • Data provided as part of Photonics North 2012 presentation, FTP data uses FileZilla client

connecting to PureFTPD



• TCP cannot handle packet loss -- even for networks with small latency.• High latency networks fall apart under real world conditions.



• You cannot transfer files at 10 Gbps with TCP. It just doesn’t work.

10 Gbps transfers


• 10 Gbps research project started in May 2011.• Project completed 6 months later with Nov 2011 release of

FileCatalyst Direct v3.0, able to transfer at full 10gbps speed.• 1½ years of successful deployments of 2-10 Gbps in the field.• Tested in 2012 by independent researchers (ex-CEO of

competitor Tixel GmbH), validated as most stable transfer results between 5 managed transport solutions.Comparison of Contemporary Solutions for High Speed Data Transport on WAN 10 Gbit/s Connections, Dmitry Kachan, Eduard Siemens, Vyacheslav Shuvalov

What are we using to run this demo?


FileCatalyst HotFolderAres x.x.x.58

FileCatalyst ServerHermes x.x.x.59

What are we using to run these tests?


• Machines utilize standard components circa 2011.• Each machine costs approx. $5000 to put together.

– Intel Westmere 5600 series (used in demo testbed)– AMD Magny-Cours 6100 Opterons


Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards

Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2






• Machine has 48GB total, 36GB reserved for RAMFS (so 12GB for OS + FileCatalyst).• For the purpose of these tests, going to read/write to RAM drive. • Why? Want to remove disks as part of the equation for demo to simplify things

(everybody has different IO solutions in place). • If your disks can handle the speeds, our software can transfer it.









• In demo, to simulate WAN environment, a Linktropy box from Apposite Technologies is used– Control link speed, packet loss, and link latency







WAN Emulator Apposite Technologies

So what are we going to see?











• (1) Going to test most basic scenario: – Transfer static file from point A to B– 30GB file… slightly larger than the average Blue Ray disk.










• (2) Going to test standard live media scenario: – Growth files (5) simulate live video feeds saved to system– Move files in real-time to destination as files are output










• Configurations used for single file transfer @ 10gbps:– Server & HotFolder each using 4GB of RAM– MTU 9000 (packet 8972 bytes)– Blocks: 10MB– Number of sender threads: 50– Sender sockets: 5– Receiver sockets: 3– RTT congestion control (because test scenario utilizes packet loss)– Congestion control set to 7 (more aggressive)

• Linktropy setup to simulate LA -> NY transfers– Latency of 60 ms– 0.1% packet loss

DEMOS

DEMOS


Upcoming Events:

- NYC Digital Asset Managers, (May 29th)

- Broadcast Asia, Singapore (June 18-21)

- IBC, Amsterdam (Sept 13-17)

- Interop New York (Oct 2-3)

- Broadcast India, Mumbai (Oct 9-11)

And of course don’t forget our monthly webinar....

Thank you for attending our May webinar!

Questions? Please type in your questions.