Upload
raghavanvin
View
233
Download
0
Embed Size (px)
Citation preview
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 1/200
Plan
5min - Introductions
7min - Network
7min - reactor intro
15min - uppercase
10min - proxy1
10min - deferred
10min - proxy2
15m - BREAK
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 2/200
Twistedan introductory training
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 3/200
About me- @orestis
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 4/200
About youRaise your hand if...
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 5/200
Some setup TODO
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 6/200
Test
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 7/200
Network programming
Server listens
on a port
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 8/200
Network programming
Server listens
on a port
Client connects
to port
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 9/200
Service request
Server reads/
writes data
Client reads/
writes data
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 10/200
While servicing...
Server reads/
writes data
Client reads/
writes data
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 11/200
While servicing...
Server reads/
writes data
Client reads/
writes data
Client connects
to port
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 12/200
Timeout!
Server reads/
writes data
Client reads/
writes data
Client connects
to port
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 13/200
Timeout!
Server reads/
writes data
Client reads/
writes data
Client connects
to port
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 14/200
Must not block!
Server listens
on a port
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 15/200
Must not block!
Server listens
on a port
Client connects
to port
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 16/200
Handle request elsewhere
Server listens
on a port
Client connects
to port
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 17/200
Handle request elsewhere
Server listens
on a port
Client reads/
writes data
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 18/200
Handle request elsewhere
Server listens
on a port
Client reads/
writes data
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 19/200
Handle request elsewhere
Fork a new process
Have worker threads
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 20/200Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 21/200Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 22/200Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 23/200
Twisted?
Twisted is a networking engine written in Python,
supporting numerous protocols. It contains a web
server, numerous chat clients, chat servers, mail
servers, and more.
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 24/200
Twisted!
Text
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 25/200
Twisted!
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 26/200
Twisted
Twisted is a networking engine written in Python,
supporting numerous protocols. It contains a web
server, numerous chat clients, chat servers, mail
servers, and more.
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 27/200
twisted.internet Asynchronous I/O and Events.
Saturday, 18 June 2011
i d i !!!
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 28/200
twisted.internet !!!
Saturday, 18 June 2011
i d i !!!
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 29/200
twisted.internet !!!
defer
endpoints
error
protocol
reactor
task
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 30/200
twisted.internet reactor“the loop which drives applications using Twisted”
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 31/200
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 32/200
Most of the
time we arewaiting
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 33/200
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 34/200
What if we
could...
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 35/200
Eliminate Blocking?
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 36/200
Eliminate Blocking?
Reactor loop Callback functions
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 37/200
Reactor loop
Event happens
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 38/200
Twisted reactor loop
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 39/200
Listen on a port, then do
something
Saturday, 18 June 2011
Li h d
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 40/200
Listen at a port, then do
something
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 41/200
Handle request elsewhere
factory listens
on a port
for each connection, a protocol
instance is created
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 42/200
Handle request elsewhere
factory listens
on a port
for each connection, a protocol
instance is created
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 43/200
Twisted Uppercase ServerServer that returns the data, uppercased
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 44/200
Twisted Uppercase Server
upperserver.pySaturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 45/200
Twisted Uppercase Server
reactor.listenTCP(8000, factory)reactor.run()
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 46/200
Twisted Uppercase Server
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 47/200
A better client multiclient.py
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 48/200
A better client (output)
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 49/200
Questions so far?Exercise coming up!
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 50/200
Count connected clients
Announce number of connected clients when
connecting
HINT: Protocols have a “factory” instance attribute
Exercise 1
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 51/200
Counting uppercase server
upperserver_ex.pySaturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 52/200
An even better client
threadedclient.pySaturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 53/200
A brief recap
reactor runs forever
a factory instance is tied to a specific port
protocol instances are created for each client
implement specific methods to add functionality
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 54/200
Twisted Proxy Server (v1)
Client sends an URL followed by a newline
The server returns the contents of that URL
Connection is closed
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 55/200
“followed by a newline”
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 56/200
twisted.protocols
amp basic dict finger ftp
gps htb ident loopback
memcache mice pcppolicies portforward postfix
shoutcast sip socks
stateful telnet tls wire
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 57/200
twisted.protocols.basic
NetstringReceiver LineOnlyReceiver
LineReceiver IntNStringReceiverInt32StringReceiver Int16StringReceiver
Int8StringReceiver StatefulStringProtocol
FileSender
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 58/200
twisted.protocolsDon’t reinvent the wheel!
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 59/200
Twisted Proxy Server (v1)
proxy1.pySaturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 60/200
Let’s time it!
timingclient.pySaturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 61/200
Let’s time it! (output)
Client
Server
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 62/200
Something’s wrong!
Individual requests
0.770982027054
0.566583156586
1.45738196373
1.01898193359
3.81392908096
Threaded client
3.81683182716
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 63/200
Eliminate Blocking?
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 64/200
But in this case...
Waiting!
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 65/200
The culprit
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 66/200
The culprit
You didn’t think it’d be that easy, right?
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 67/200
The callbacks must becooperative
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 68/200
The callbacks must be
cooperative
When accessing the network, return control backto the loop
The loop will call your code when the network is ready
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 69/200
Network programming
Server listens
on a port
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 70/200
Network programming
Server listens
on a port
Client connects
to port
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 71/200
Network programming
Server listens
on a port
Client connects
to port
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 72/200
High-level panini stall
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 74/200
Low-level panini stall
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 75/200
Low-level panini stall
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 76/200
Waste of time!
We are idling the CPU!
Nothing else can run!
How selfish of us!
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 77/200
Solution: Callbacks!
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 78/200
Callbacks, you know...
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 80/200
High-level panini stall
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 81/200
Callbacks can be messy
Add error handling?
Pass the result around?
Cancel the original request?
Consistent API?
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 82/200
Introducing Deferredtwisted.internet.defer
Saturday, 18 June 2011
A D f d i
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 83/200
A Deferred is...
A promise of a result...
A result that will appear in the future...
A result you can pass around...
Something you can attach callbacks to.
Saturday, 18 June 2011
D f d P i i
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 84/200
Deferred Panini
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 85/200
Deferreds are EverywhereGet used to them!
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 86/200
Twisted has re-implementations of most of
the stdlib. They had to do it - not a case of NIH!
Saturday, 18 June 2011
S
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 87/200
So....
Saturday, 18 June 2011
In context
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 88/200
In context...
Saturday, 18 June 2011
Python reminder:
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 89/200
Python reminder:
Saturday, 18 June 2011
Python reminder:
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 90/200
Python reminder:
Saturday, 18 June 2011
A tidier way
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 91/200
A tidier way
Saturday, 18 June 2011
Twisted Proxy Server (v2)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 92/200
y ( )
pr ox y
2 . p y
Saturday, 18 June 2011
L t’ ti it! ( t t)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 93/200
Let’s time it! (output)
Client
Server
Saturday, 18 June 2011
M h b tt !
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 94/200
Much better!
Individual requests
0.486361026764
0.850247859955
0.998661994934
1.58235692978
3.917627811433
Threaded client
1.5850892067
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 95/200
The callbacks must be
cooperative
When accessing the network, return control backto the loop
The loop will call your code when the network is ready
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 96/200
never call blocking functionsreturn control to the loop
Saturday, 18 June 2011
Exercise 2a
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 97/200
Implement a caching proxy server!
Save response data in plain dict
Lookup response data
QUESTION: Where should you store the dict?
Exercise 2a
Saturday, 18 June 2011
Caching Proxy Server (v1)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 98/200
pr ox y 2 _
e x 1 . p y
Saturday, 18 June 2011
Cool Deferred Features:
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 99/200
Chaining
Saturday, 18 June 2011
Cool Deferred Features:
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 100/200
Deferring
Saturday, 18 June 2011
Chaining and Deferring
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 101/200
Chaining and Deferring
Saturday, 18 June 2011
Chaining and Deferring
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 102/200
Chaining and Deferring
Saturday, 18 June 2011
Chaining and Deferring
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 103/200
Chaining and Deferring
Saturday, 18 June 2011
Exercise 2b
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 104/200
Put those cool features into use!
One callback to get a page
One callback to store it to cache
One callback to write to transport
HINT: Use defer.succeed(data) to return a “ready”Deferred
Exercise 2b
Saturday, 18 June 2011
Caching Proxy Server (v2)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 105/200
pr ox y 2 _
e x 2 . p y
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 106/200
BREAKWrite questions on whiteboard
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 107/200
Writing clientsthe Twisted way
Saturday, 18 June 2011
Remember this?
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 108/200
Remember this?
Saturday, 18 June 2011
Remember this?
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 109/200
Remember this?
Saturday, 18 June 2011
Remember this?
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 110/200
Remember this?
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 111/200
Return control to the loopIn clients, too
Saturday, 18 June 2011
Twisted reactor loop
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 112/200
Twisted reactor loop
Saturday, 18 June 2011
Connect to a host, then do
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 113/200
something
Saturday, 18 June 2011
Connect to a host, then do
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 114/200
something
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 115/200
Let’s write a twisted clientRun the uppercase server
Saturday, 18 June 2011
Twisted Simple Client (v1)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 116/200
Twisted Simple Client (v1)
simpleclient.pySaturday, 18 June 2011
Simple Client v1 (output)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 117/200
Simple Client v1 (output)
Saturday, 18 June 2011
Observations
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 118/200
Observations
Data returns with random order
We cannot access the returned data
Loop never stops
Performance?
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 119/200
We need to gather resultshmm, what should we use?
Saturday, 18 June 2011
Twisted Simple Client (v2)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 120/200
Twisted Simple Client (v2)
simpleclient2.pySaturday, 18 June 2011
Simple Client v2 (output)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 121/200
Simple Client v2 (output)
Saturday, 18 June 2011
Observations
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 122/200
Observations
Data returns with random order
We cannot access the returned data
Loop never stops
Performance?
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 123/200
Stop the loop when
everything is finishedwait until all Deferreds have fired
Saturday, 18 June 2011
Introducing DeferredList
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 124/200
Introducing DeferredList
A list of Deferreds! You create it with a list of Deferreds
When all the Deferreds have finished, its callback fires.
Saturday, 18 June 2011
Introducing DefferedList
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 125/200
Introducing DefferedList
Saturday, 18 June 2011
Twisted Simple Client (v3)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 126/200
Twisted Simple Client (v3)
simpleclient3.pySaturday, 18 June 2011
Simple Client v3 (output)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 127/200
Simple Client v3 (output)
Saturday, 18 June 2011
Observations
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 128/200
Observations
Data returns with random order
We cannot access the returned data
Loop never stops
Performance?
Saturday, 18 June 2011
Twisted Simple Client (v4)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 129/200
Twisted Simple Client (v4)
simpleclient4.pySaturday, 18 June 2011
Simple Client v4 (output)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 130/200
Simple Client v4 (output)
against the proxy server
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 131/200
Single threaded
performance! As good as threaded performance, without thecomplexity
Saturday, 18 June 2011
Exercise 3
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 132/200
Write a memcached SET command-line script
Exercise 3
Send:
Receive:
Usage: must wait to store and return!
Test:
Saturday, 18 June 2011
Memcached SET client
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 133/200
Memcached SET client
Saturday, 18 June 2011
Exercise 4
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 134/200
Make this reusable!
Exercise 4
Saturday, 18 June 2011
Exercise 4
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 135/200
e c se
Saturday, 18 June 2011
Exercise 5
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 136/200
Write an HTTP GET command-line script
Saturday, 18 June 2011
Exercise 5
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 137/200
Saturday, 18 June 2011
A brief recap on writing
li t
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 138/200
clients
Figure out the protocol
Write the protocolCreate a factory, set instance variables
Access the variables from the protocol
Connect the factory to a host & socket
Saturday, 18 June 2011
That’s tedious!
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 139/200
Sharing state between protocols can be usefulMany times we don’t need it
There’s another way to do this.
Saturday, 18 June 2011
t.i.protocol.ClientCreator
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 140/200
p
Change your protocol to have __init__
Create it with a protocol class and argsConnect it to a host:port
Attach a callback
When the protocol is instantiated, callback is fired
Saturday, 18 June 2011
ClientCreator
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 141/200
Saturday, 18 June 2011
Exercise 6a
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 142/200
Rewrite the HTTP GET protocol to work with
t.i.p.ClientCreator
Saturday, 18 June 2011
Exercise 6b
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 143/200
Rewrite the HTTP GET protocol to work with
t.i.p.ClientCreator
...and the plain factory way
HINT: Implement Factory.buildProtocol to customise
the way...
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 144/200
twisted.web“the early days”
Saturday, 18 June 2011
twisted.web NOP
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 145/200
Saturday, 18 June 2011
twisted.web
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 146/200
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 147/200
Let’s build something coolIt’s going to be easy
Saturday, 18 June 2011
Website status dashboard
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 148/200
Monitor a list of websites
Shows their status and response time
Notify people via email when a site becomes slow
Notify people via email when a site goes offline and
comes back up
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 149/200
How should we go about it?Discuss
Saturday, 18 June 2011
curl -N
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 150/200
monitor1.py
Saturday, 18 June 2011
Timeouts!
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 151/200
Abort the attempt after 30 seconds
How should we specify that? An exception?
Saturday, 18 June 2011
Introducing errbacks
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 152/200
Like callbacks, but for error condition
Called explicitly - d.errback(reason)
Called implicitly, when a callback function raises
Saturday, 18 June 2011
Errback example (v1)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 153/200
errback.py
Saturday, 18 June 2011
Errback example (v2)
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 154/200
errback_wrong.pySaturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 155/200
defer.setDebugging(True)
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 156/200
There can only be one calleither one callback or one errback
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 157/200
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 158/200
Saturday, 18 June 2011
Calling code
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 159/200
Saturday, 18 June 2011
Calling code
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 160/200
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 161/200
Relax, it’s easier than it
looks
Saturday, 18 June 2011
Calling code
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 162/200
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 163/200
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 164/200
addCallbacks?
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 165/200
Saturday, 18 June 2011
Scenario: Download Google
Doodle Alt Text
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 166/200
Doodle Alt Text
getPage(“ ”)
On error, print “ERROR: Google down”
Try to find doodle text
On error, print “ERROR: No doodle found”
Finally, print doodle text
Saturday, 18 June 2011
Scenario: Download Google
Doodle Alt Text
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 167/200
Doodle Alt Text
Saturday, 18 June 2011
Scenario: Download GoogleDoodle Alt Text
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 168/200
Doodle Alt Text
Saturday, 18 June 2011
getPage
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 169/200
Saturday, 18 June 2011
Scenario: Download GoogleDoodle Alt Text
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 170/200
Doodle Alt Text
Saturday, 18 June 2011
getPage
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 171/200
Saturday, 18 June 2011
getPage
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 172/200
Saturday, 18 June 2011
getPage
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 173/200
Saturday, 18 June 2011
getPage
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 174/200
Saturday, 18 June 2011
GET with timeout
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 175/200
httpget2.py
sudo python slowserver.pySaturday, 18 June 2011
curl -N
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 176/200
monitor2.pySaturday, 18 June 2011
Exercise 7
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 177/200
Differentiate between different errors:
twisted.names.dns.DomainError
twisted.error.TimeoutError
Everything else
Have a final callback that writes results to request
HINT: Use Failure.trap(error_class)
Saturday, 18 June 2011
curl -N
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 178/200
monitor3.pySaturday, 18 June 2011
getHostByName
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 179/200
Saturday, 18 June 2011
curl -N
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 180/200
monitor3.pySaturday, 18 June 2011
getHostByName
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 181/200
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 182/200
Let’s drink from the firehoseremove the site limit
Saturday, 18 June 2011
Doesn’t scale!
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 183/200
You just DOSed your operating system
Twisted will happily open the connections
You need to ensure you don’t overload the system
Saturday, 18 June 2011
Many issues
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 184/200
We overload the DNS
We saturate the network
We exhaust the open file limit
We do this for every request!
Saturday, 18 June 2011
DeferredSemaphore
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 185/200
Saturday, 18 June 2011
DeferredSemaphore
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 186/200
Example: You do not want to have more than 5simultaneous DNS queries and 10 GETs
Saturday, 18 June 2011
Seems to work, but...
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 187/200
We are creating the workload while we are operating
on it
We seem to be blocking - everything is run inside a
giant callback chain
Saturday, 18 June 2011
The callbacks must be
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 188/200
cooperativeWhen accessing the network, return control back
to the loop
The loop will call your code when the network is ready
Must do as little work as possible
Doesn’t eliminate CPU-bound delays!
Saturday, 18 June 2011
Seems to work, but...
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 189/200
In our case, the limit.release is triggering a callback
which triggers a deferred which triggers a callbackimmediately
reactor doesn’t get a chance to breathe
Saturday, 18 June 2011
twisted.internet.task
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 190/200
Saturday, 18 June 2011
cooperate
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 191/200
Saturday, 18 June 2011
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 192/200
Saturday, 18 June 2011
DeferredQueue
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 193/200
Saturday, 18 June 2011
Writing clients
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 194/200
timeouts
error handling
Saturday, 18 June 2011
Using twisted from clientcode
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 195/200
show sendmail
show spawnProcess
show deferToThread/callInThread
Saturday, 18 June 2011
twisted codebase
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 196/200
interfaces/adapters/components/plugins
Saturday, 18 June 2011
More deferred goodness
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 197/200
cancellable deferreds
Saturday, 18 June 2011
Perspective broker
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 198/200
just simple mention
Saturday, 18 June 2011
manholes?
8/2/2019 Twisted Training
http://slidepdf.com/reader/full/twisted-training 199/200
Saturday, 18 June 2011
testing