Lessons from the Trenches

Dan BishopRavenHQ

dbishop@ravenhq.comgithub.com/bishbulbtwitter.com/bishbulb

Typical Project Lifecycle

Technical Discovery

Convince Stakeholde

rsPrototype Iterate Launch

MVP Iterate

Life is good! Life is good, right?

Reality Sets In• Business is not “finished” with the MVP solution• Users are never satisfied• User hacks are conceived and proliferated• Small gaps grow into chasms• Production support issues start coming in• Things aren’t shiny anymore

Data Conversions• Properties are added and removed• Data types change• Class names and namespaces change• Documents are split or merged• Indexes change

Data Conversions – Adding Properties

public class OrderDelivery {public string Id { get; set; } public string Address { get; set; }public DateTime DeliveryDate { get;

set; }}

{"Address": "100 First St, NY, NY",

"DeliveryDate": "2016-05-30T12:30:00"}

public class OrderDelivery {public string Id { get; set; } public string Address { get; set; }public DateTime DeliveryDate { get;

set; }public bool Expedited { get; set; }

}

Data Conversions – Adding Propertiespublic class OrderDeliveriesIndex : AbstractIndexCreationTask<OrderDelivery>{ public OrderDeliveriesIndex() { Map = docs => from doc in docs select new { doc.Address, doc.DeliveryDate, doc.Expedited }; }}

Data Conversions – Adding Properties

var expeditedDeliveries = session .Query<OrderDelivery, OrderDeliveriesIndex>() .Where(x => x.Expedited);

if (expeditedDeliveries.Count > 10) { // email delivery manager}

Javascript Patchingvar result = documentStore.DatabaseCommands.UpdateByIndex("Raven/DocumentsByEntityName", new IndexQuery { Query = "Tag: OrderDeliveries" }, new ScriptedPatchRequest { Script = "this.Expedited = true;" }, new BulkOperationOptions { AllowStale = false });result.WaitForCompletion();

Javascript Patchingvar result = documentStore.DatabaseCommands.UpdateByIndex("Raven/DocumentsByEntityName", new IndexQuery { Query = "Tag: OrderDeliveries" }, new ScriptedPatchRequest { Script = @" var customerLookup = JSON.parse('customers'); this.Expedited = customerLookup[this.CustomerId].IsPreferred;", Values = new Dictionary<string, object> { { "customers", JsonConvert.SerializeObject(customers) } } });result.WaitForCompletion();

Javascript Patching• Patch scripts always run against an index• Make patch scripts idempotent• Consider rollback strategy• Split changes into multiple releases

• Debugging• Test in the Studio, use output() function• View patch progress in RavenDB 3.5

Data Conversions – Removing Properties

• Safe by default• Properties in the underlying document are retained

documentStore.Conventions.PreserveDocumentPropertiesNotFoundOnModel = true;

• Clutter by default• Keep track of the properties being removed• Run Javascript patch scripts in the next release to

remove the properties

You Want All the Data?• Raven intentionally makes this hard with the

normal APIs• Indexes can be stale, query results limited to

1024 results• Streaming to the rescue

var results = session.Advanced.Stream<OrderDelivery>("orderdeliveries-");

session.Advanced.LoadStartingWith<OrderDelivery>("orderdeliveries-", start: 0, pageSize: 1000);

var query = session.Query<OrderDelivery, OrderDeliveriesIndex>() .Where(x => !x.Delivered);var results = session.Advanced.Stream(query);

• Other tips

session.Advanced.MaxNumberOfRequestsPerSession = int.MaxValue;

Semantic Document Identifiers• Identifiers that have embedded data/context• employees/1/deliveryroutes/2016-06-01

• Prevents logical duplicates• Enables multi-load with one round-trip• employees/1/preferences• employees/1/workschedule

•deliveryroutes/2016-06-01/employees/1• Enables you to bypass indexes altogether• Example: Get all delivery routes for a particular

day

LoadDocumentManager

Employee1

Employee2

EmployeeN

public EmployeesIndex(){ Map = employeees => from employee in employeees let manager = LoadDocument<Manager>(employee.ManagerId) select new { employee.Id, ManagerName = manager.Name };}

LoadDocumentManager

CompanyVehicle

public CompanyVehiclesIndex(){ Map = vehicles => from vehicle in vehicles let manager = LoadDocument<Manager>(vehicle.ManagerId) select new { vehicle.Id, ManagerName = manager.Name };}

LoadDocument• One to many relationships could cause problems• Only when updating the document that has many

other documents referencing it• One to one relationships are fine• Look for alternatives• Transformers• Data duplication

Auto vs Static Indexes• Auto indexes are great for prototyping• Consider the implications<appSettings> <add key="Raven/CreateAutoIndexesForAdHocQueriesIfNeeded" value="false" /></appSettings>

• Indexes are resource intensive• Be aware of the indexes you have

WaitForNonStaleResults• Gives you up-to-date results from an index

• Dealing with CRUD views• Make the writer wait for indexes, not the reader

var results = session .Query<OrderDelivery, OrderDeliveriesIndex>() .Customize(x => x.WaitForNonStaleResultsAsOfNow());

Consistency Issues• ACID provides specific guarantees• Distributed Transactions• DTC commit and rollback happen in a separate thread

from updates• Rather than locking the underlying documents, Raven will

allow you to access them but will return an HTTP 203 Non-Authoritative response

• Scenario: NServiceBus processing chain of messagesUpdateUserProfile ConfirmPhoneNumber

session.Advanced.AllowNonAuthoritativeInformation = false;

Replication Failover

Create Order

Upgrade to Expedited Delivery

Schedule Delivery Failover to secondary node

• Consider multi-transaction business workflows• What happens if the Raven client fails over to

another node midway through the processing sequence?

Replication Failover• Consider the risk/reward of allowing automatic

failover • Manual failover is an option• Secondary nodes are treated as “hot standby”

nodes• Mix-and-match failover behaviors• Consider SLAs and business needs when

deciding on failover strategies• These tradeoffs are not specific to RavenDB• All distributed systems have similar challenges

Questions