Upload
confoo
View
3.012
Download
3
Embed Size (px)
Citation preview
MarrowMeta–Framework for Python 2.6+ and 3.1+
Alice Zoë Bevan–McGregor
Overview
ConfigurationYAML-Based Application Configuration
Introspective ScriptingNon-Imperative Command-Line Parsing
BlueprintTemplate–Derived Directory Trees
Interactive & Command–Line Interrogation
Streaming TemplatesA Python Micro–Language
Server InterfaceModified Tornado IOLoop and IOStream
Server and Protocol Wrappers
HTTP/1.1 WSGI 2 ServerHighly Performant Pure Python HTTP/1.1 Implementation
Object WrappersPEP 444 Request / Response Objects
HTTP Status Code Exception Applications
Middleware / FiltersCompression, Sessions, etc.
Performance & OptimizationsTime for Timeit
CompatibilityPython 2.6+ and 3.1+
Configuration
marrow.config
Unfortunately…
Least Developed(So far.)
Paste Deploy
1 [server] 2 use = marrow.server.http:HTTPServer 3 host = 127.0.0.1, ::1 4 port = 8080, 8088 5 6 [mapping] 7 / = root 8 9 [app:root] 10 use = marrow.server.http.testing:hello 11 name = ConFoo
INI = Evil
Typecasting
1 [server] 2 use = marrow.server.http:HTTPServer 3 host = 127.0.0.1, ::1 4 port = 8080, 8088 5 6 [mapping] 7 / = root 8 9 [app:root] 10 use = marrow.server.http.testing:hello 11 name = ConFoo
String to List
String to Integer
YAML to the Rescue
1 version: 1 2 3 server: 4 use: marrow.server.http:HTTPServer 5 host: ["127.0.0.1", "::1"] 6 port: [8080, 8088] 7 8 mapping: 9 /: *root 10 11 root: &root 12 use: marrow.server.http.testing:hello 13 name: ConFoo
References
1 version: 1 2 3 server: 4 use: marrow.server.http:HTTPServer 5 host: ["127.0.0.1", "::1"] 6 port: [8080, 8088] 7 8 mapping: 9 /: *root 10 11 root: &root 12 use: marrow.server.http.testing:hello 13 name: ConFoo
Direct Object Access(Entry points are for chumps.)
1 version: 1 2 3 server: 4 use: marrow.server.http:HTTPServer 5 host: ["127.0.0.1", "::1"] 6 port: [8080, 8088] 7 8 mapping: 9 /: *root 10 11 root: &root 12 use: marrow.server.http.testing:hello 13 name: ConFoo
Logging
16 logging: 17 formatters: 18 brief: 19 format: '%(levelname)-8s: %(name)-15s: %(message)s' 20 handlers: 21 console: 22 class: logging.StreamHandler 23 formatter: brief 24 level: INFO 25 stream: ext://sys.stdout 26 loggers: 27 foo: 28 level: ERROR 29 handlers: [console] 30 root: 31 level: DEBUG 32 handlers: [console]
Scripting
marrow.script
Existing Solutions
sys.argv(painful)
sys.argv(inconsistent)
getopt(old–school)
optparse(old–school)
optparse(deprecated)
argparse(new old–school)
Paste Script(fancy)
Paste Script(entry point magic)
Paste Script(paster <name> […])
Paste Script(context–aware)
Commonality?
Commonality?(un–Pythonic…)
Commonality?(…hideous, hideous, imperative parser objects…)
1 import optparse 2 3 if __name__=="__main__": 4 parser = optparse.OptionParser("usage: %prog [options]
arg1 arg2") 5 parser.add_option("-H", "--host", dest="hostname", 6 default="127.0.0.1", type="string", 7 help="specify hostname to run on") 8 parser.add_option("-p", "--port", dest="portnum", 9 default=80, type="int", 10 help="port number to run on") 11 (options, args) = parser.parse_args() 12 if len(args) != 2: 13 parser.error("incorrect number of arguments") 14 hostname = options.hostname 15 portnum = options.portnum
1 import argparse 2 3 parser = argparse.ArgumentParser(description='Process some integers.') 4 parser.add_argument('integers', metavar='N', type=int, nargs='+', 5 help='an integer for the accumulator') 6 parser.add_argument('--sum', dest='accumulate', action='store_const', 7 const=sum, default=max, 8 help='sum the integers (default: find the max)') 9 10 args = parser.parse_args() 11 print(args.accumulate(args.integers))
Most needed?
Simplicity
Arguments ➢ Variables
1 # encoding: utf-8 2 3 def ultima(required, value=None, name="world",
switch=False, age=18, *args, **kw): 4 print "Hello %s!" % (name, ) 5 6 7 if __name__ == '__main__': 8 __import__('marrow.script').script.execute(ultima)
Usage: ultima.py [OPTIONS] [--name=value...] <required> [value...]
OPTIONS may be one or more of:
-a, --age=VAL Override this value. Default: 18 -h, --help Display this help and exit. -n, --name=VAL Override this value. Default: 'world' -s, --switch Toggle this value. Default: False -v, --value=VAL Set this value.
Additional Detail
__doc__ = Help Text
Decorators
@annotateArgument Typecasting & Validation Callbacks
@describeHelp Text
@shortArgument Abbreviations
@callbacksSimple Callbacks
optparse Example
1 # encoding: utf-8 2 3 import marrow.script 4 5 6 @marrow.script.describe( 7 host = "specify a hostname to run on", 8 port = "port number to run on" 9 ) 10 def serve(arg1, arg2, host="127.0.0.1", port=80): 11 pass 12 13 14 if __name__ == '__main__': 15 marrow.script.execute(serve)
Sub–Commands
Method = Command
__init__ + method
Blueprint
marrow.blueprint
Paste Script
Á La Carte Templates
Best way to describe it…
1 class PackageBlueprint(Blueprint): 2 """Create an installable Python package.""" 3 4 base = 'marrow.blueprint.package/files' 5 engine = 'mako' 6 7 settings = [ 8 Setting( 9 'name', 10 "Project Name", 11 "The name to appear on the Python Package Index, e.g. CluComp.", 12 required=True 13 ), 14 Setting( 15 'package', 16 "Package Name", 17 "The name of the Python package, periods indicating namespaces, e.g. clueless.compiler.", 18 required=True 19 ), 20 # ... 21 ] 22 23 manifest = [ 24 # ... 25 File('setup.py'), 26 File('setup.cfg'), 27 Folder('tests', children=[ 28 File('.keep', 'keep') 29 ]), 30 package 31 ]
1 def package(settings): 2 def recurse(name): 3 head, _, tail = name.partition('.') 4 5 return [ 6 Folder(head, children=[ 7 File('__init__.py', 'namespace.py' if tail else 'init.py') 8 ] + (recurse(tail) if tail else [])) 9 ] 10 11 return recurse(settings.package)
class Setting
target
title
help
required
validator
condition
values
default
cast
hidden
class File
target
source
condition
data
class Folder
≈ File- data
Class Inheritance
pre/post Callbacks
Interactive Questioning
Command–Line Answers(marrow.script + **kw ;)
Templating
marrow.tags
Streaming
yield
Enter / Exit
Text / Flush
HTML5
High–Level
Widgets
Python ±
1 #!/usr/bin/env python 2 # encoding: utf-8 3 4 from __future__ import unicode_literals 5 6 from marrow.tags.html5 import * 7 8 from master import SITE_NAME, site_header, site_footer 9 10 11 def welcome(): 12 return html [ 13 head [ title [ 'Welcome!', ' — ', SITE_NAME ] ], 14 flush, # allow the browser to start downloading static resources early 15 body ( class_ = "nav-welcome" ) [ 16 site_header, 17 p [ 18 'Lorem ipsum dolor sit amet, consectetur adipisicing elit…' 19 ], 20 site_footer 21 ] 22 ] 23 24 25 if __name__ == '__main__': 26 with open('welcome.html', 'w') as fh: 27 for i in welcome().render('utf8'): 28 fh.write(i)
1 login = Form('sign-in', class_="tabbed", action='/users/action:authenticate', children=[ 2 HiddenField('referrer'), 3 FieldSet('local', "Local Users", TableLayout, [ 4 TextField('identity', "User Name", autofocus=True), 5 PasswordField('password', "Password") 6 ]), 7 FieldSet('yubikey', "Yubikey Users", TableLayout, [ 8 TextField('identity', "User Name"), 9 PasswordField('password', "Password"), 10 PasswordField('yubikey', "Yubikey") 11 ]), 12 FieldSet('openid', "OpenID Users", TableLayout, [ 13 URLField('url', "OpenID URL") 14 ]) 15 ], footer=SubmitFooter('form', "Sign In"))
Guts
1 class Tag(Fragment): 2 def __call__(self, data_=None, strip=NoDefault, *args, **kw): 3 self = deepcopy(self) 4 5 self.data = data_ 6 if strip is not NoDefault: self.strip = strip 7 self.args.extend(list(args)) 8 self.attrs.update(kw) 9 10 return self
12 def __getitem__(self, k): 13 if not k: return self 14 15 self = deepcopy(self) 16 17 if not isinstance(k, (tuple, list)): 18 k = [k] 19 20 for fragment in k: 21 if isinstance(fragment, basestring): 22 self.children.append(escape(fragment)) 23 continue 24 25 self.children.append(fragment) 26 27 return self
29 def __unicode__(self): 30 """Return a serialized version of this tree/branch.""" 31 return ''.join(self.render('utf8')).decode('utf8') 32 33 def enter(self): 34 if self.strip: 35 raise StopIteration() 36 37 if self.prefix: 38 yield self.prefix 39 40 yield u'<' + self.name + u''.join([attr for attr in quoteattrs(self, self.attrs)]) + u'>' 41 42 def exit(self): 43 if self.simple or self.strip: 44 raise StopIteration() 45 46 yield u'</' + self.name + u'>'
48 def render(self, encoding='ascii'): 49 # ... 50 51 for k, t in self: 52 if k == 'enter': 53 # ... 54 continue 55 56 if k == 'exit': 57 # ... 58 continue 59 60 if k == 'text': 61 # ... 62 continue 63 64 if k == 'flush': 65 yield buf.getvalue() 66 del buf 67 buf = IO() 68 69 yield buf.getvalue()
71 def __iter__(self): 72 yield 'enter', self 73 74 for child in self.children: 75 if isinstance(child, Fragment): 76 for element in child: 77 yield element 78 continue 79 80 if hasattr(child, '__call__'): 81 value = child(self) 82 83 if isinstance(value, basestring): 84 yield 'text', unicode(value) 85 continue 86 87 for element in child(self): 88 yield element 89 90 continue 91 92 yield 'text', unicode(child) 93 94 yield 'exit', self
29 def __unicode__(self): 30 """Return a serialized version of this tree/branch.""" 31 return ''.join(self.render('utf8')).decode('utf8')
96 def clear(self): 97 self.children = list() 98 self.args = list() 99 self.attrs = dict() 100 100 def empty(self): 101 self.children = list()
Server Interface
Asynchronous IO
Callbacks
Low–Level
marrow.io
Py3K Tornado + PatchesIOLoop + IOStream
Apache License
1 # encoding: utf-8 2 3 """An example raw IOLoop/IOStream example. 4 5 Taken from http://nichol.as/asynchronous-servers-in-python by Nicholas Piël. 6 """ 7 8 import errno, functools, socket 9 10 from marrow.util.compat import exception 11 from marrow.io import ioloop, iostream 12 13 def connection_ready(sock, fd, events): 14 while True: 15 connection, address = sock.accept() 16 17 connection.setblocking(0) 18 stream = iostream.IOStream(connection) 19 stream.write(b"HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n", stream.close) 20 21 if __name__ == '__main__': 22 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) 23 sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 24 sock.setblocking(0) 25 sock.bind(("", 8010)) 26 sock.listen(5000) 27 28 io_loop = ioloop.IOLoop.instance() 29 io_loop.set_blocking_log_threshold(2) 30 callback = functools.partial(connection_ready, sock) 31 io_loop.add_handler(sock.fileno(), callback, io_loop.READ) 32 33 try: 34 io_loop.start() 35 except KeyboardInterrupt: 36 io_loop.stop()
1 # encoding: utf-8 2 3 """An example raw IOLoop/IOStream example. 4 5 Taken from http://nichol.as/asynchronous-servers-in-python by Nicholas Piël. 6 """ 7 8 import errno, functools, socket 9 10 from marrow.util.compat import exception 11 from marrow.io import ioloop, iostream 12 13 def connection_ready(sock, fd, events): 14 while True: 15 connection, address = sock.accept() 16 17 connection.setblocking(0) 18 stream = iostream.IOStream(connection) 19 stream.write(b"HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n", stream.close) 20 21 if __name__ == '__main__': 22 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) 23 sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 24 sock.setblocking(0) 25 sock.bind(("", 8010)) 26 sock.listen(5000) 27 28 io_loop = ioloop.IOLoop.instance() 29 io_loop.set_blocking_log_threshold(2) 30 callback = functools.partial(connection_ready, sock) 31 io_loop.add_handler(sock.fileno(), callback, io_loop.READ) 32 33 try: 34 io_loop.start() 35 except KeyboardInterrupt: 36 io_loop.stop()
1 # encoding: utf-8 2 3 """An example raw IOLoop/IOStream example. 4 5 Taken from http://nichol.as/asynchronous-servers-in-python by Nicholas Piël. 6 """ 7 8 import errno, functools, socket 9 10 from marrow.util.compat import exception 11 from marrow.io import ioloop, iostream 12 13 def connection_ready(sock, fd, events): 14 while True: 15 connection, address = sock.accept() 16 17 connection.setblocking(0) 18 stream = iostream.IOStream(connection) 19 stream.write(b"HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n", stream.close) 20 21 if __name__ == '__main__': 22 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) 23 sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 24 sock.setblocking(0) 25 sock.bind(("", 8010)) 26 sock.listen(5000) 27 28 io_loop = ioloop.IOLoop.instance() 29 io_loop.set_blocking_log_threshold(2) 30 callback = functools.partial(connection_ready, sock) 31 io_loop.add_handler(sock.fileno(), callback, io_loop.READ) 32 33 try: 34 io_loop.start() 35 except KeyboardInterrupt: 36 io_loop.stop()
1 # encoding: utf-8 2 3 """An example raw IOLoop/IOStream example. 4 5 Taken from http://nichol.as/asynchronous-servers-in-python by Nicholas Piël. 6 """ 7 8 import errno, functools, socket 9 10 from marrow.util.compat import exception 11 from marrow.io import ioloop, iostream 12 13 def connection_ready(sock, fd, events): 14 while True: 15 connection, address = sock.accept() 16 17 connection.setblocking(0) 18 stream = iostream.IOStream(connection) 19 stream.write(b"HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n", stream.close) 20 21 if __name__ == '__main__': 22 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) 23 sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 24 sock.setblocking(0) 25 sock.bind(("", 8010)) 26 sock.listen(5000) 27 28 io_loop = ioloop.IOLoop.instance() 29 io_loop.set_blocking_log_threshold(2) 30 callback = functools.partial(connection_ready, sock) 31 io_loop.add_handler(sock.fileno(), callback, io_loop.READ) 32 33 try: 34 io_loop.start() 35 except KeyboardInterrupt: 36 io_loop.stop()
1 # encoding: utf-8 2 3 """An example raw IOLoop/IOStream example. 4 5 Taken from http://nichol.as/asynchronous-servers-in-python by Nicholas Piël. 6 """ 7 8 import errno, functools, socket 9 10 from marrow.util.compat import exception 11 from marrow.io import ioloop, iostream 12 13 def connection_ready(sock, fd, events): 14 while True: 15 connection, address = sock.accept() 16 17 connection.setblocking(0) 18 stream = iostream.IOStream(connection) 19 stream.write(b"HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nPong!\r\n", stream.close) 20 21 if __name__ == '__main__': 22 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) 23 sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 24 sock.setblocking(0) 25 sock.bind(("", 8010)) 26 sock.listen(5000) 27 28 io_loop = ioloop.IOLoop.instance() 29 io_loop.set_blocking_log_threshold(2) 30 callback = functools.partial(connection_ready, sock) 31 io_loop.add_handler(sock.fileno(), callback, io_loop.READ) 32 33 try: 34 io_loop.start() 35 except KeyboardInterrupt: 36 io_loop.stop()
Not fun!
High–Level
marrow.server
1 # encoding: utf-8 2 3 """A simplified version of the raw example.""" 4 5 6 from marrow.server.base import Server 7 from marrow.server.protocol import Protocol 8 9 10 11 class HTTPResponse(Protocol): 12 def accept(self, client): 13 client.write( b"HTTP/1.0 200 OK\r\nContent-Length: 7\r\n\r\nPong!\r\n", client.close) 14 15 16 if __name__ == '__main__': 17 Server(None, 8010, HTTPResponse).start()
functools.partial(Pass the client object to your callbacks.)
Your BFF
Single–Thread Async
Futures–Based Threading
Multi–Process(incl. processor detection)
r".+"
PEP 444
Warning!
PEP 444Hurts babies!
PEP 444Is highly addictive?
PEP 444Is a draft of one possible WSGI 2 solution.
WSGI
Complete Rewrite
Simplified
Consistent
1 def hello(environ): 2 return b'200 OK', [ 3 (b'Content-Type', b'text/plain')], 4 (b'Content-Length', b'12') 5 ], b"Hello world!"
Distinctions
Byte String
Unicode String
Native String
RFC–Style
Applications…
An application is any function, method, or instance with a __call__ method. Applications must:
1. Be able to be invoked more than once. If this can not be guaranteed by the application implementation, it must be wrapped in a function that creates a new instance on each call.
2. Accept a single positional argument which must be an instance of a base Python dictionary containing what is referred to as the WSGI environment. The contents of this dictionary are fully described in the WSGI Environment section.
3. Return a 3-tuple of (status, headers, body) where:1.status must contain the HTTP status code and reason phrase of the response. The status code and reason must be present, in that
order, separated by a single space. (See RFC 2616, Section 6.1.1 for more information.)2.headers must be a standard Python list containing 2-tuples of (name, value) pairs representing the HTTP headers of the response. Each
header name must represent a valid HTTP header field name (as defined by RFC 2616, Section 4.2) without trailing colon or other punctuation.
3.body must be an iterable representing the HTTP response body.4.status and the name of each header present in headers must not have leading or trailing whitespace.5.status, and the contents of headers (both name and value) must not contain control characters including carriage returns or linefeeds.6.status, headers, and the chunks yielded by the body iterator should be returned as byte strings, though for implementations where
native strings are unicode, native strings may be returned. The server must encode unicode values using ISO-8859-1.7. The amount of data yielded by the body iterable must not exceed the length specified by the Content-Length response header, if
defined.8. The body iterable may be a native string, instead of a native string wrapped in a list for the simple case, but this is not recommended.
Additionally, applications and middleware must not alter HTTP 1.1 "hop-by-hop" features or headers, any equivalent features in HTTP 1.0, or any headers that would affect the persistence of the client's connection to the web server. Applications and middleware may, however, interrogate the environment for their presence and value. These features are the exclusive province of the server, and a server should consider it a fatal error for an application to attempt sending them, and raise an error if they are supplied as return values from an application in the headers structure.
Servers…
A WSGI 2 server must:
1. Invoke the application callable once for each request it receives from an HTTP client that is directed at the application.2. Pass a single positional value to the application callable representing the request environment, described in detail in the WSGI Environment
section.3. Ensure that correct response headers are sent to the client. If the application omits a header required by the HTTP standard (or other relevant
specifications that are in effect), the server must add it. E.g. the Date and Server headers.1. The server must not override values with the same name if they are emitted by the application.
4. Raise an exception if the application attempts to set HTTP 1.1 "hop-by-hop" or persistence headers, or equivalent headers in HTTP 1.0, as described above.
5. Encode unicode data (where returned by the application) using ISO-8859-1.6. Ensure that line endings within the body are not altered.7. Transmit body chunks to the client in an unbuffered fashion, completing the transmission of each set of bytes before requesting another one.
(Applications should perform their own buffering.)8. Call the close() method of the body returned by the application, if present, upon completion of the current request. This should be called
regardless of the termination status of the request. This is to support resource release by the application and is intended to complement PEP 325's generator support, and other common iterables with close() methods.
9. Support the HTTP 1.1 specification where such support is made possible by the underlying transport channel, including full URL REQUEST_URI, pipelining of requests, chunked transfer, and any other HTTP 1.1 features mandated in the relevant RFC.
Additionally,
1. HTTP header names are case-insensitive, so be sure to take that into consideration when examining application-supplied headers.2. The server may apply HTTP transfer encodings or perform other transformations for the purpose of implementing HTTP features such as
chunked transfer.3. The server must not attempt to handle byte range requests; the application can optimize this use case far more easily than a server. (For example
an application can generate the correct body length vs. generating the whole body and having the server buffer and slice it.)4. Servers must not directly use any other attributes of the body iterable returned by the application.
More Demanding
(Optional = Never)
HTTP/1.1
Chunked Encoding
(Request)
(Response)
Expect/Continue
Pipelining / Keep–Alive
HTTP/1.1 Server
4.5KR/sec(Single process, single thread.)
C10K(4 processes, single thread.)
10KR/sec(4 process, single thread, lower concurrency.)
Pure Python!
(171 Opcodes)
PEP 444
Async
Threading
Futures!
Multi–Process
Explicit
Processor Detection
Request Cycle
Socket Accept
Protocol .accept()
Read Headers
Pre–Buffer Body
Dispatch to Worker
Emit Status & Headers
Stream Body
Keep–Alive
wsgi.errors ➢ logging
Object Wrappers
marrow.wsgi.objects
Request / Response
Exceptions
WebOb
1 #!/usr/bin/env python 2 # encoding: utf-8 3 4 from __future__ import unicode_literals 5 6 from pprint import pformat 7 8 from marrow.server.http import HTTPServer 9 from marrow.wsgi.objects.decorator import wsgify 10 11 12 @wsgify 13 def hello(request): 14 resp = request.response 15 resp.mime = "text/plain" 16 resp.body = "%r\n\n%s\n\n%s" % (request, request, pformat(request.__dict__)) 17 18 19 if __name__ == '__main__': 20 import logging 21 logging.basicConfig(level=logging.DEBUG) 22 23 HTTPServer(None, 8080, application=hello).start()
Request
Dict Proxy
WSGI Environment
Singleton
Accessor Objects
… 24 class Request(object): … 28 body = RequestBody('wsgi.input') 29 length = Int('CONTENT_LENGTH', None, rfc='14.13') 30 mime = ContentType('CONTENT_TYPE', None) 31 charset = Charset('CONTENT_TYPE') …
… 102 def __getitem__(self, name): 103 return self.environ[name] 104 105 def __setitem__(self, name, value): 106 self.environ[name] = value 107 108 def __delitem__(self, name): 109 del self.environ[name] …
1 class ReaderWriter(object): 2 default = NoDefault 3 rw = True 4 5 def __init__(self, header, default=NoDefault, rw=NoDefault, rfc=None): 6 pass # save arguments 7 8 def __get__(self, obj, cls): 9 try: 10 return obj[self.header] 11 12 except KeyError: 13 pass 14 15 if self.default is not NoDefault: 16 if hasattr(self.default, '__call__'): 17 return self.default(obj) 18 19 return self.default 20 21 raise AttributeError('WSGI environment does not contain %s key.' % (self.header, )) 22 23 def __set__(self, obj, value): 24 if not self.rw: 25 raise AttributeError('%s is a read-only value.' % (self.header, )) 26 27 if value is None: 28 del obj[self.header] 29 return 30 31 obj[self.header] = value 32 33 def __delete__(self, obj): 34 if not self.rw: 35 raise AttributeError('%s is a read-only value.' % (self.header, )) 36 37 del obj[self.header]
Filtering
Ingress
Egress
“Light-Weight Middleware”
38 class CompressionFilter(object): 39 def __init__(self, level=6): 40 self.level = level 41 42 super(CompressionFilter, self).__init__() 43 44 def __call__(self, request, status, headers, body): 45 """Compress, if able, the response. 46 47 This has the side effect that if your application does not declare a content-length, this filter will. 48 """ 49 50 # TODO: Remove some of this debug logging; it'll slow things down and isn't really needed. 51 52 if request.get('wsgi.compression', True) is False: 53 log.debug("Bypassing compression at application's request.") 54 return status, headers, body 55 56 if request.get('wsgi.async') and hasattr(body, '__call__'): 57 log.debug("Can not compress async responses, returning original response.") 58 return status, headers, body 59 60 if b'gzip' not in request.get('HTTP_ACCEPT_ENCODING', b''): 61 log.debug("Browser support for GZip encoding not found, returning original response.") 62 return status, headers, body
38 class CompressionFilter(object): 39 def __init__(self, level=6): 40 self.level = level 41 42 super(CompressionFilter, self).__init__() 43 44 def __call__(self, request, status, headers, body): 45 """Compress, if able, the response. 46 47 This has the side effect that if your application does not declare a content-length, this filter will. 48 """ 49 50 # TODO: Remove some of this debug logging; it'll slow things down and isn't really needed. 51 52 if request.get('wsgi.compression', True) is False: 53 log.debug("Bypassing compression at application's request.") 54 return status, headers, body 55 56 if request.get('wsgi.async') and hasattr(body, '__call__'): 57 log.debug("Can not compress async responses, returning original response.") 58 return status, headers, body 59 60 if b'gzip' not in request.get('HTTP_ACCEPT_ENCODING', b''): 61 log.debug("Browser support for GZip encoding not found, returning original response.") 62 return status, headers, body
38 class CompressionFilter(object): 39 def __init__(self, level=6): 40 self.level = level 41 42 super(CompressionFilter, self).__init__() 43 44 def __call__(self, request, status, headers, body): 45 """Compress, if able, the response. 46 47 This has the side effect that if your application does not declare a content-length, this filter will. 48 """ 49 50 # TODO: Remove some of this debug logging; it'll slow things down and isn't really needed. 51 52 if request.get('wsgi.compression', True) is False: 53 log.debug("Bypassing compression at application's request.") 54 return status, headers, body 55 56 if request.get('wsgi.async') and hasattr(body, '__call__'): 57 log.debug("Can not compress async responses, returning original response.") 58 return status, headers, body 59 60 if b'gzip' not in request.get('HTTP_ACCEPT_ENCODING', b''): 61 log.debug("Browser support for GZip encoding not found, returning original response.") 62 return status, headers, body
Exit Early
Stream Process
Flat Stack
Performance & Optimization
timeit FTW
s="Content-Type: text/html\r\n"^
Split or Partition?
a,b = s.split(":", 1)
a,b = s.split(":", 1)• Python 2.7: 0.665 • Python 3.1: 0.909
a,b = s.split(":", 1)
a,b = s.split(":")• Python 2.7: 0.665 • Python 3.1: 0.909
a,b = s.split(":", 1)
a,b = s.split(":")• Python 2.7: 0.631 • Python 3.1: 0.837
• Python 2.7: 0.665 • Python 3.1: 0.909
a,b = s.split(":", 1)
a,b = s.split(":")• Python 2.7: 0.631 • Python 3.1: 0.837
• Python 2.7: 0.665 • Python 3.1: 0.909
a,c = s.partition(":")[::2]
a,b = s.split(":", 1)
a,b = s.split(":")• Python 2.7: 0.631 • Python 3.1: 0.837
• Python 2.7: 0.665 • Python 3.1: 0.909
a,c = s.partition(":")[::2]• Python 2.7: 0.642 • Python 3.1: 0.690
a,b = s.split(":", 1)
a,b = s.split(":")• Python 2.7: 0.631 • Python 3.1: 0.837
• Python 2.7: 0.665 • Python 3.1: 0.909
a,c = s.partition(":")[::2]• Python 2.7: 0.642 • Python 3.1: 0.690
a,b,c = s.partition(":")
a,b = s.split(":", 1)
a,b = s.split(":")• Python 2.7: 0.631 • Python 3.1: 0.837
• Python 2.7: 0.665 • Python 3.1: 0.909
a,c = s.partition(":")[::2]• Python 2.7: 0.642 • Python 3.1: 0.690
a,b,c = s.partition(":")• Python 2.7: 0.407 • Python 3.1: 0.429
s="Content-Type: text/html\r\n"
.upper() or .lower()?
"Content-Type: text/html\r\n".upper()
"Content-Type: text/html\r\n".upper()• Python 2.7: 0.479 • Python 3.1: 0.469
"Content-Type: text/html\r\n".upper()
"CONTENT-TYPE: text/html\r\n".upper()
• Python 2.7: 0.479 • Python 3.1: 0.469
"Content-Type: text/html\r\n".upper()
"CONTENT-TYPE: text/html\r\n".upper()
• Python 2.7: 0.417 • Python 3.1: 0.616
• Python 2.7: 0.479 • Python 3.1: 0.469
"Content-Type: text/html\r\n".upper()
"CONTENT-TYPE: text/html\r\n".upper()
• Python 2.7: 0.417 • Python 3.1: 0.616
• Python 2.7: 0.479 • Python 3.1: 0.469
"CONTENT-TYPE: TEXT/HTML\r\n".upper()
"Content-Type: text/html\r\n".upper()
"CONTENT-TYPE: text/html\r\n".upper()
• Python 2.7: 0.417 • Python 3.1: 0.616
• Python 2.7: 0.479 • Python 3.1: 0.469
"CONTENT-TYPE: TEXT/HTML\r\n".upper()
• Python 2.7: 0.291 • Python 3.1: 0.407
"Content-Type: text/html\r\n".upper()
"CONTENT-TYPE: text/html\r\n".upper()
• Python 2.7: 0.417 • Python 3.1: 0.616
• Python 2.7: 0.479 • Python 3.1: 0.469
"CONTENT-TYPE: TEXT/HTML\r\n".upper()
• Python 2.7: 0.291 • Python 3.1: 0.407
"Content-Type: text/html\r\n".lower()
"Content-Type: text/html\r\n".upper()
"CONTENT-TYPE: text/html\r\n".upper()
• Python 2.7: 0.417 • Python 3.1: 0.616
• Python 2.7: 0.479 • Python 3.1: 0.469
"CONTENT-TYPE: TEXT/HTML\r\n".upper()
• Python 2.7: 0.291 • Python 3.1: 0.407
"Content-Type: text/html\r\n".lower()• Python 2.7: 0.319 • Python 3.1: 0.497
a="foo"; b="bar"
Efficient Concatenation?
", ".join((a, b))
", ".join((a, b))• Python 2.7: 0.405 • Python 3.1: 0.319
", ".join((a, b))
a + ", " + b• Python 2.7: 0.405 • Python 3.1: 0.319
", ".join((a, b))
a + ", " + b• Python 2.7: 0.257 • Python 3.1: 0.283
• Python 2.7: 0.405 • Python 3.1: 0.319
Determine Presence
b.find(a)
b.find(a)• Python 2.7: 0.255 • Python 3.1: 0.448
b.find(a)
a in b• Python 2.7: 0.255 • Python 3.1: 0.448
b.find(a)
a in b• Python 2.7: 0.104 • Python 3.1: 0.119
• Python 2.7: 0.255 • Python 3.1: 0.448
a="foo.bz"
Test Filename Extension
a.endswith(".bz")
a.endswith(".bz")• Python 2.7: 0.338 • Python 3.1: 0.515
a.endswith(".bz")
a[-3:] == ".bz"• Python 2.7: 0.338 • Python 3.1: 0.515
a.endswith(".bz")
a[-3:] == ".bz"• Python 2.7: 0.229 • Python 3.1: 0.312
• Python 2.7: 0.338 • Python 3.1: 0.515
uri="/foo/bar/baz"
Absolute Path?
uri.startswith("/")
uri.startswith("/")• Python 2.7: 0.324 • Python 3.1: 0.513
uri.startswith("/")
uri[0] == "/"• Python 2.7: 0.324 • Python 3.1: 0.513
uri.startswith("/")
uri[0] == "/"• Python 2.7: 0.133 • Python 3.1: 0.146
• Python 2.7: 0.324 • Python 3.1: 0.513
(Negative case identical.)
Compatibility
marrow.util.compat
formatdaterfc822 vs. email.utils
range vs. xrange
str vs. bytes
unicode vs. str
“foo” vs. b“foo” vs. u“foo”
from __future__ import unicode_literals
No implicit conversion!
(Stop that!)
StringIO vs. BytesIO
Exception Handling
… 216 def _handle_read(self): 217 try: 218 chunk = self.socket.recv(self.read_chunk_size) 219 220 except socket.error: 221 e = exception().exception 222 if e.args[0] in (errno.EWOULDBLOCK, errno.EAGAIN): 223 return …
Questions?
Chasing Corporate care of Air Reviewmyspace.com/airreview
Core DevelopersAlice Bevan-McGregorAlex Grönholm
ResourcesHTTP: The Definitive Guide, David Gourley & Brian Totty, O’Reilly PressPorting to Python 3, Lennart Regebro, CreateSpace
Relevant SpecificationsPEP 333 WSGI 1.0PEP 391 Dict Logging ConfigurationPEP 444 WSGI 2.0PEP 3148 FuturesPEP 3333 WSGI 1.1
RFC 1945 — HTTP 1.0RFC 2616 — HTTP 1.1
Get GA to GA for PyCon!http://pledgie.com/campaigns/14434