oslo.versionedobjects:Deep Dive

Dan Smith

What is it?

● Object model to solve problems we had in nova

● Each RPC-visible change is versioned● Remotable, serializable● Backportable

What is it? (versioning)

● Attempt to get away from dicts (of madness)● Fields are defined and typed, with

serialization routines● Defined at the edges, no embedding code or

class names in the serialized form● Controlled evolution of object/RPC schema

independent of DB schema over time

What is it? (remotable)

● You implement Indirection API (nova-conductor implements for nova)

● @remotable methods get called via indirection

● Self-serializing

obj.do_thing()

def do_thing()

def do_thing()

DB

Indirection Service

@remotable

When indirectionis active

● Remotable methods wrapped with decorator

● Calls may avoid the local implementation and make the call run through the indirection service

● Same implementation both places!● Copy of the object (including version)

is sent● Object (may be) modified on remote,

changes copied back to the caller

Remotable Methods

What is it not?

● Amazingly novel● ...going to solve world hunger● ...going to solve your problems (alone)● A lemming

from oslo.versionedobjects import basefrom oslo.versionedobjects import base

class MyObject(base.VersionedObject): # Version 1.0: Initial # Version 1.1: Added ‘name’ field VERSION = ‘1.1’ fields = { ‘id’: fields.IntegerField(nullable=False), ‘name’: fields.StringField(nullable=True), ‘metadata’: fields.DictOfStringsField(nullable=True), }

@base.remotable_classmethod def get_by_id(cls, context, id): db_thingy = db.get_thingy(context, id) obj = cls(context=context, id=db_thingy[‘id’], name=db_thingy.get(‘thingy_name’), metadata=db.get_metadata_for(context, id)) obj.obj_reset_changes() return obj

Careful, humans will put non-versioned stuff in here and cause problems (as humans do)!

class MyObject(base.VersionedObject): # … continued

@base.remotable def save(self): changes = self.obj_what_changed() db_updates = {} if ‘id’ in changes: raise Exception(‘Can\’t change the id!’) if ‘name’ in changes: db_updates[‘thingy_name’] = self.name

with db.transaction(): if ‘metadata’ in changes: db.update_metadata_for(self.context, self.id, self.metadata) if db_updates: db.update_thingy(self.context, self.id, db_updates) self.obj_reset_changes()

Lazy-Loading

● Attributes can be set or unset, independent of nullability (i.e. “can be None”)

● Loadable attributes handled by a method● In practice, only some (heavy) attributes are

loadable

class MyObject(base.VersionedObject): # … continued

# Not remotable! def obj_load_attr(self, attrname): if attrname == ‘metadata’: obj = self.get_by_id(self.context, self.id) self.metadata = obj.metadata self.obj_reset_changes() else: # Fail as not-lazy-loadable super(MyObject, self).obj_load_attr(attrname)

Serialization

● Convert object to/from serialized form● Field implementations are responsible for

serializing their contents (not your object)● Serializer object actually produces the result● Nova uses JSON, but you can do whatever● Object mentions no classnames or code,

relies on the model to deserialize

Serialization (continued)

● Most services will need to have a request context for all operations

● Object stores this for ease and security● Each RPC trip re-attaches the context of the

RPC call, so context is not serialized!● Can be anything or ignored if you don’t need

it

obj = MyObj(id=123, name=’foo’, metadata={‘speed’: ‘88mph’})prim = obj.obj_to_primitive()string = json.dumps(prim)reprim = json.loads(string)reobj = VersionedObject.obj_from_primitive(reprim, context)

class MyRPCClientAPI(object): def __init__(self): # ... serializer = base.VersionedObjectSerializer() self.client = self.get_client(target, version_cap, serializer)

class MyRPCServerAPI(service.Service):

def start(): # ... serializer = base.VersionedObjectSerializer() self.rpcserver = rpc.get_server(target, endpoints, serializer) self.rpcserver.start()

Versioning

● Requires diligence (with some tools to help catch offenders)

● Backwards-compatible changes● Recipients should honor old versions● Senders can pre-backport to older versions● Nova does this automatically, but simply

pinning versions is an easy way

class MyObject(base.VersionedObject): # … continued

def obj_make_compatible(self, primitive, version): if version == ‘1.0’: del primitive[‘name’]

obj = MyObject(id=123, name=’foo’, metadata={‘speed’: ‘88mph’})prim = obj.obj_to_primitive(target_version=’1.0’)# Now prim won’t have the ‘name’ fieldoldobj = VersionedObject.obj_from_primitive(prim, context)self.assertFalse(oldobj.obj_attr_is_set(‘name’))

Why do this?

● Get a better handle on what you’re sending over RPC and what happens if you change

● Decouples the database from services, allowing DB schema upgrades independent of object schemas

● Easy serialization without sacrificing rich typing and bundled methods

Why not do this?

● Because it will magically solve upgrade problems (it won’t)

● Because you have a better way● Because it doesn’t fit your model or process● Because the name is too boring