Behind the Curtain

Testing This

TESTING TESTING

TESTING TESTING

Thursday, April 21, 2011

ZOMG!


GOOD MORNING!


Aaron Patterson


@tenderlove


WWFMD?Thursday, April 21, 2011

ITWFMWD?Thursday, April 21, 2011


AT&T, AT&T logo and all AT&T related marks are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies.


Miami Vice Principal Señor Facebook

Integration Engineering Manager


Elect


12%

61%

24%

2%

Inspirational Entertaining Informative Wasted

Our Time Together



KEYNOTE


10%

50%

20%

20%

Inspirational Entertaining Informative Wasted

Our Time Together



20% More Transitions


Living Behindthe Curtain 6 subjects, less

than 10minutes on each of them

I enjoy dealing with behinds the scenes information


DATABASE

APPLICATION

CLIENT


DATABASE

APPLICATION

CLIENT


DATABASE

APPLICATION

CLIENT


DATABASE

APPLICATION

CLIENT


Code as Data


Politics


"Open Sourceis a Democracy"

We don't vote on commits. It would be too inefficient.


OSS is an Oligarchy


Benevolent Dictator


Tyrannical Leader

Small projects have few people, so it is efficient

Large projects have too many people so it becomes inefficient to individually consider all feedback



Care and Feeding of Your Tyrant


Cheetos


Feed them Quality Bugs

Good reproduction steps


Feed them Patches with Tests


leave build systems

Gemspec,JewlerHoe,etc


leave test frameworks


Use +1's Carefully




320 comments


210 plus one



FFFFFFFUUUUUUUUUUUU



Superficial


Merit


Quality


How to fix?

We'll discuss this in closing.


Encodingsthe silent killer


Never a miscommunication

Data goes in, Data goes out


Until there is


`encode': "\xE9" from ASCII-8BIT to UTF-8(Encoding::UndefinedConversionError)


You can't explain that

Encoding Error?


String Tagging


"hello! <3".encoding # => #<Encoding:UTF-8>

Strings with unknown encoding are marked as binary.


socket.read(4).encoding # => #<Encoding:ASCII-8BIT>


Boundaries

To understand these bugs, we must find boundaries


App Code


App Code

Web Server

Database

NoSQL File System

SadnessBelt


#<Encoding:ASCII-8BIT>is usually a bug!

Databases know the stored encoding

Post contains encoding / spec declares default


# encoding: utf-8

name = 'たこ焼き仮面'

user = User.create!(:name => name)user.reloaduser.name + name


# encoding: utf-8

name = 'たこ焼き仮面'

user = User.create!(:name => name)user.reloaduser.name + name

incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)


>> name = "Aaron Patterson">> name.encoding=> #<Encoding:UTF-8>>> User.create!(:name => name)>> u = User.find :first>> u.name.encoding=> #<Encoding:ASCII-8BIT>


>> name = "Aaron Patterson">> name.encoding=> #<Encoding:UTF-8>>> User.create!(:name => name)>> u = User.find :first>> u.name.encoding=> #<Encoding:ASCII-8BIT>


name = 'aaron patterson'user = User.create!(:name => name)user.reloaduser.name + name

Ruby attempts to convert the binary for you


name = 'aaron patterson'user = User.create!(:name => name)user.reloaduser.name + name

no error

Ruby attempts to convert the binary for you


ASCII-8BIT + othershould raise


ASCII-8BIT + othershould raise

Aaron'sOpinionCorner!


Resources Know Encoding


Prepared Statements


SELECT * FROM "users" WHERE id = 1


♥Parse the Query♥Formulate an Execution Plan♥Return results

Normal Queries


Application

Database


Application

Database

Result Set


Application

Database


Application

Database

query: 1234


Application

Database


Application

Database

Result Set


♥Parse the Query♥Formulate an Execution Plan

Preparation


Execution

♥Return Results


Data transfer decreases


Query Planner Improves


Security


SQLite3 Issues :'(

Talk about problem inserting rows with session data

Could insert session data, but could not find it


INSERT INTO "sessions"("data", "session_id")VALUES (?, ?)

['data', 'BAh7BjoIZm9vSSIIYmF6BjoGRUY=']['session_id', 'dae0fb8a9c34e6980c9d0fae33a8fff6']


db = SQLite3::Database.new ':memory:'db.trace { |sql| puts sql }


INSERT INTO "sessions"("data", "session_id")

VALUES (

'BAh7BjoIZm9vSSIIYmF6BjoGRUY=',

x'3633303337623066376536613130343132343765623763626631616439303631')


INSERT INTO "sessions"("data", "session_id")

VALUES (

'BAh7BjoIZm9vSSIIYmF6BjoGRUY=',

x'3633303337623066376536613130343132343765623763626631616439303631')


db.trace { |sql| puts sql }

stmt = db.prepare('INSERT INTO "sessions" (session_id)VALUES (?)')

stmt.bind_param 1, 'fuu'stmt.execute

Made a sample program to reproduce the problem


INSERT INTO "sessions"(session_id) VALUES ('fuu')

Resulting Query


stmt.bind_param 1, 'fuu'.encode('BINARY')stmt.execute


INSERT INTO "sessions" (session_id)VALUES (x'667575')


INSERT INTO "sessions" (session_id)VALUES (x'667575')

Why did sqlite3 do this?

Talk about column affinity


Why is it stored as Binary?


Why is the session id tagged binary?


>> x = OpenSSL::Random.random_bytes(10)=> "G\x93\xFC\xB2\xCE\xC0\xEC\xBB\xA7W">> x.encoding=> #<Encoding:ASCII-8BIT>


>> y = x.unpack('H*')=> ["4793fcb2cec0ecbba757"]>> y.first.encoding=> #<Encoding:ASCII-8BIT>


def generate_sid ActiveSupport::SecureRandom.hex(16)end


def generate_sid sid = ActiveSupport::SecureRandom.hex(16) if sid.respond_to?(:encode!) sid.encode!('UTF-8') end sidend

Should hex have tagged it?

Should have generate_sid tagged it?


Find the Source


ARel and ActiveRecord


Fear of SQL


Why?


We must learn SQL

But not specifically SQL

Working with sets

Can fetch correct data.


We must learn to work with Sets


Avoid SQL for:


code reuse


security


independence

from databases


What is ARel?


AST

Represents SQL as a Tree

Represents the IDEA of a SQL statement


SQL Compiler


Representsan IDEA

Only until we invoke the compiler, does the AST become a query


Relationship to ActiveRecord


User.where('something')


class User scope :heart where(:name => '<3')end

User.heart.select('name')


ActiveRecord

Our Application

ARel Database


ActiveRecord

Our Application

.to_a

ARel Database


ActiveRecord

Our Application

ARel Database

to_sql


ActiveRecord

Our Application

ARel Database

SELECT * ...


ActiveRecord

Our Application

ARel Database

[{: name => '<3'}]


ActiveRecord::Relation


shortens to...


ARel


ARel != ARel


:'(


SQL Compiler

Future we can have optimizers, compiler cache, etc


Interpreter?


module Arel module Visitors class Mongo < Arel::Visitors::Visitor attr_reader :db

def initialize db; @db = db; end Query = Struct.new(:collection_name, :fields, :conditions)

private

def visit_Arel_Nodes_SelectStatement o o.cores.map { |c| visit_Arel_Nodes_SelectCore c }.map { |query| collection = db.collection query.collection_name fields = query.fields selector = Hash[query.conditions] opts = {} opts[:fields] = fields unless fields.empty? opts[:limit] = o.limit.expr.to_i if o.limit collection.find(selector, opts).to_a }.flatten end

def visit_Arel_Nodes_SelectCore o fields = o.projections.map { |proj| visit(proj) }.compact conditions = o.wheres.map { |condition| visit(condition) }.flatten(1) Query.new(visit(o.source), fields, conditions) end

def visit_Arel_Nodes_And o o.children.map { |child| visit child } end

def visit_Arel_Nodes_Equality o [visit(o.left), visit(o.right)] end

def visit_Arel_Attributes_Attribute o return if o.name == '*'; o.name end

def visit_Arel_Nodes_JoinSource o visit o.left end

def visit_Arel_Table o o.name end def literal o; o; end alias :visit_String :literal alias :visit_Fixnum :literal end endend


module Arel module Visitors class Mongo < Arel::Visitors::Visitor attr_reader :db

def initialize db; @db = db; end Query = Struct.new(:collection_name, :fields, :conditions)

private

def visit_Arel_Nodes_SelectStatement o o.cores.map { |c| visit_Arel_Nodes_SelectCore c }.map { |query| collection = db.collection query.collection_name fields = query.fields selector = Hash[query.conditions] opts = {} opts[:fields] = fields unless fields.empty? opts[:limit] = o.limit.expr.to_i if o.limit collection.find(selector, opts).to_a }.flatten end

def visit_Arel_Nodes_SelectCore o fields = o.projections.map { |proj| visit(proj) }.compact conditions = o.wheres.map { |condition| visit(condition) }.flatten(1) Query.new(visit(o.source), fields, conditions) end

def visit_Arel_Nodes_And o o.children.map { |child| visit child } end

def visit_Arel_Nodes_Equality o [visit(o.left), visit(o.right)] end

def visit_Arel_Attributes_Attribute o return if o.name == '*'; o.name end

def visit_Arel_Nodes_JoinSource o visit o.left end

def visit_Arel_Table o o.name end def literal o; o; end alias :visit_String :literal alias :visit_Fixnum :literal end endend

Mongo Interpreter64 loc



gist.github.com/845782


https://gist.github.com/845782

https://gist.github.com/845782

Custom DSL


class Select < Struct.new(:columns) def self.* other other.select = new(Arel.sql('*')) other endend

Responds to *


Responds to WHERE

class From < Struct.new(:table, :conditions) def WHERE conditions self.conditions = conditions Where.new(self) endend


Responds to to_s

class Where < Struct.new(:from, :select) def to_s Arel::Table.engine = Arel::Sql::Engine.new( FakeRecord::Base.new) table = Arel::Table.new from.table table.project(select.columns).where( from.conditions.map { |k,v| table[k].eq v }).to_sql endend


Bootstrap Methods

SELECT = Selectdef FROM table From.new tableend


x = SELECT * FROM("users") .WHERE(:id => 10)puts x



Write SQL Rubyto avoid SQL


Streaming

Talked about data from the back end, talk a bit about data to the front end


Twitter Streaming API


Chunked Responses

Needed for persistent connections


Normal TimelineBuffers data, then sends a response


Processing ERb



Processing ERb

Download Asset

Download Asset



Chunked TimelineSend data as soon as it is available


Processing ERb



Processing ERb

Download Asset

Download Asset



Implementation


Rack API


status, headers, body = app.call(env)


status.to_i


headers.is_a?(Hash)


body.responds_to?(:each)


Inside Rack

body.each do |chunk| output(chunk)endbody.close rescue nil

Calls each,

possibly calls close


Sample Applicationclass MyApp def call(env) # some computation body = 'hello' # more computation body << ' world'

[200, { 'X-Hello' => 'World' }, [body]] endend


Delay work until eachclass FooBody def each yield "hello " sleep(10) # simulate work yield "world!" endend


Problems ☹

Database connections were lost


Middleware

Database connections are managed in middleware


"There are two hard problems in CS: cache invalidation

and naming things"-- Phil Karlton


class DbCache < Struct.new(:app) def call(env) # init cache status, headers, body = app.call(env) # clear cache [status, headers, body] endend


class BodyProxy < Struct.new(:delegate) def each delegate.each { |x| yield x } end

def close delegate.close rescue nil # clear cache endend


Update Middleware, Database Works! ☺


But Query Cache was broken. ☹

Same problem in query cache as the database handle cache.


Middleware is coupled to the callstack


~ 24 middleware


~ 5 required a proxy


Types of Middleware

♥Generators♥Filters♥Lifecycle Handlers

Rack jams these to one API.

Approach seems naive when examining usage

Tied to callstack


Solutions!


Embrace The Differences!


class Listener def created(event) end

def destroyed(event) endend

Lifecycle Listeners


Generatorsclass Resource def service(request, response) 10.times { |i| response.body.write "hello #{i}" } response.body.close endend

Synchronous or Asynchronous

Return value is ignored


Filters

class Filter def filter(request, response, chain) chain.filter(request, response) endend


♥Synchronous

♥Asynchronous

♥Freedom from the Callstack

♥Thread Safety


I stole this API.


TEE HEE!!!!!


fixing our tyrants


Facebook + Github


Get Involved


Gain Trust

Only fix the immediate problem


Then Persuade




Questions?


Technology

Behind the Curtain