Smirnov twisted-python

  • View
    11.666

  • Download
    0

Embed Size (px)

Transcript

Twisted (Python)

/ /

""

CPU time/wall time

Twisted Framework

Python

: async IO (select/poll)

Deferred

:DNS, HTTP, IMAP, memcached, AMQP,

Twisted

,

- (memcached, nginx, haproxy, )

/ 1

Python

Python -

10x C

90% 10%

Cython, Nukitka, Unladen Swallow,

kcachegrind

Python

gc, heapy

: __slots__, Cython, ...

/

, NUMA,

NUMA

CPU

Interconnection

CPUMemory

CPU

CPUMemoryCPU binding

Node binding

Local alloc

+10-15%

DB1DB2App1App2App3Proxy

DB1DB2App1App2App3Proxy

CacheCacheCache

App1CacheApp2CacheApp3Cache

Message bus

AMQP:

:fanout ( )

origin ( )

RabbitMQ

txAMQP

ZeroMQ

App2App3Proxy

App1

Message bus

IPIPIP

Key-value

Memcached memcached, MemcacheDB,
MemcacheQ, membase, ...

txRedis

txMongo

...

pickling

Object(id=199)key1=v1key2=v2...key-valuestorageObject_199 pickledObject_378 pickledObject _988 pickled

user_199_posts

(id, date)

(197, 11:03)

(180, 10:45)

(111, 9:50)

(120, 9:01)

(98, 8:15)

(60, 7:05)

(42, 7:04)

(37, 7:01)

text="Nice shoes..."author="john"tags=['dev','schedule']post_197

text="Nice shoes..."author="john"tags=['dev','schedule']post_180

...post_111

...post_120

originMemcached

origins[]"" increment/add

append/add

originsAggApp1App2stats_1stats_2Memcached

Twisted: manhole + telnet/ssh

$ telnet localhost 2000

C100k+

Firewall stateless mode

twistd: reactor=epoll (kqueue)

?

"" ( .. )?

""

memcached, MongoDB, RabbitMQ, cPickle, ...

JSON-RPC, "hello world"890 req/s @ 0.022 sec

530 req/s @ 0.006 sec

()JSON-RPC: 710 req/sec

memcachedX: 11833 req/sec

AMQP: 800 req/sec

connections: 600K+ (670 reconnects/sec)

web- Qik

me@smira.ru, smira@qik.com

http://www.smira.ru/, @smira

24.10.10

24.10.10