netkit ftpd/ftp migration

Part 4Yufei

10/08/2010

• Improvement

• Separate Executer

• Buffer Data Strature

• Bottleneck

New result

File Size

RDMAsequential

TCPRDMA

circled- buffer(30 x 50MB)

1GB1.87 secs (5.5e+05

Kbytes/sec)

1.8 secs (5.7e+05

Kbytes/sec)

1.56 secs(6.6e+05

Kbytes/sec)

100GB

444 secs (2.3e+05

Kbytes/sec)

311 secs (3.3e+05

Kbytes/sec)

306 secs(3.3e+05

Kbytes/sec)

Improvement• Independent Data Loading Module

– Reader and Writer

• Independent Data Transfer Module– Sender and Receiver

• Circled Buffer– Free list

– Sender list

– Writer list

Client open new connection, then login to the server

Child use the established connection to transfer

COMMANDs and REPLIES information with Client

USER, PASS,PORT, PASV,RETR, STOR

communication channel

data transfer channel

P P

Previous Type

Client open new connection, then login to the server

Child use the established connection to transfer

COMMANDs and REPLIES information with Client

USER, PASS,PORT, PASV,RETR, STOR

communication channel

data transfer channel

T-sender

T-reader

T-recver

T-writer

Separate Executer

Client open new connection, then login to the server

Child use the established connection to transfer

COMMANDs and REPLIES information with Client

USER, PASS,PORT, PASV,RETR, STOR

communication channel

data transfer channel

T-sender

T-reader

T-recver

T-writer

BUFFER BUFFER

Buffer Resource

T-sender

T-reader

T-recver

T-writer

free list

sender list

free list

writer list

RDMA_READRDMA_WRITE

Buffer List

T-sender

T-reader

free list

sender list

pthread_mutex_t

Thread Synchronization

pthread_cond_tnot alarm

Buffer Block Data Strature

next

work request record

rdma single SGE

rdma_mr

rdma_mr

CTL

• CTL(control block) is used to indicate the length of the real transfer data, the EOF message of the transfer

How to teardown the connection-- Domino method

• Reader met the EOF of the source file– reader -> sender -> recver -> writer

• Generate a new package whose body length is ZERO– reader read fail or EOF, generate ZERO

– sender check the CTL block of the buffer• If ZERO, quit

– recver check the CTL block of the buffer• If ZERO, quit

– writer check the CTL block of the buffer• If ZERO, quit

Find the bottleneck

T-sender

T-reader

T-recver

T-writer

free list

sender list

free list

writer list

RDMA_READRDMA_WRITE

Who is always WAITing?

Find the bottleneck• Different environments(CPU, Memory,

Network, Storage) have different bottlenecks. Buffer is a method to balance the difference of each modules. But we could improve some expensive cost procedures, such as– Expensive information exchange steps

Expensive control information exchange

Data Source Data SinkHello, I have some data to

transfer

Ok, please write the data to this address

I am writing

The writing complete

I get the data, please do the next step

Q & A