Alt.Net Presentation

Stories from the trenchesIt’s just a bloody one way synchronisation between two systems. How hard can it be?

Melbourne Alt.NetMercury Project

23 February 2016

Shohre Mansouri1

how the battle started

2

3

First Attack

4

Exsiting wcf methods for useShipmentId[] GetUpdatedShipments(UpdateAtFrom, UpdateAtTo)

Shipment GetShipment(ShipmentId)

5

Use “UpdateAt” as means of getting the updates

Every 2 minutes:GetUpdatedShipmentsCommand(From, DateTime.UtcNow)Handler sends: GetShipmentCommand(ShipmentId)

ShipmentId UpdateAt

Shipment1 12:00:01

Shipment2 12:00:02

Shipment3 12:00:02

Shipment4 12:00:03

Shipment5 12:00:04

6

Use “UpdateAt” as means of getting the updates

Every 2 minutes:GetUpdatedShipmentsCommand(From, DateTime.UtcNow)Every 1 minute:GetUpdatedShipmentsFromDatabaseCommand() -> GetShipment(ShipmentId)

ShipmentId UpdateAt

Shipment1 12:00:01

Shipment1 12:00:02

Shipment2 12:00:02

Shipment1 12:00:03

Shipment1 12:00:04

7

No Unbounded sets

Every 2 minutes:GetUpdatedShipmentsFromWcfCommand(From, DateTime.UtcNow, PageSize)Every 1 minute:GetUpdatedShipmentsFromDatabaseCommand(PageSize)

ShipmentId UpdateAt

Shipment1 12:00:01

Shipment1 12:00:02

Shipment2 12:00:02

Shipment1 12:00:03

Shipment1 12:00:04

8

Sample

UpdatedAt => From && UpdatedAt < To

ShipmentId UpdateAt

1 Shipment3 12:00:01 x2 Shipment1 12:00:02 x x3 Shipment2 12:00:02 x x4 Shipment1 12:06:04 X

5 Shipment2 12:06:06 X X

6 Shipment3 12:08:00 X

From To (Now) PageSize

12:00:00 12:02:00 3

12:00:02 12:04:00 3

12:04:00 12:06:00 3

12:06:00 12:08:00 3

12:08:00 12:10:00 3

12:10:00 12:12:00 3

9

Sample ShipmentId UpdateAt

1 Shipment3 12:00:01 x x x X2 Shipment1 12:00:01 x x x X3 Shipment2 12:00:01 x x x X4 Shipment1 12:00:01

5 Shipment2 12:00:01

6 Shipment3 12:06:06

From To (Now) PageSize

12:00:00 12:02:00 3

12:00:01 12:04:00 3

12:00:01 12:06:00 3

12:00:01 12:08:00 3

12:00:01 12:10:00 3

UpdatedAt > From && UpdatedAt < To

10

There was a bug!

11

This other WCF methodGetUpdatedShipments(FromVersion, PageSize)

Version is a sequential integer assigned to each update!

12

Use Version as means of getting the updates

Every 2 minutes:GetUpdatedShipmentsCommand(StartVersion, PageSize)Every 1 minute:GetUpdateShipmentFromDatabaseCommand(PageSize)

ShipmentId Version

Shipment1 100

Shipment1 104

Shipment2 106

Shipment1 111

Shipment1 112

13

Shipments have filesConsistency of the shipments and document!

Saga: For handling long-running business processes

“NServiceBus employs event-driven architectural principles to bake fault-tolerance and scalability into these processes.”

14

Added Blob storage And saga

15

DTC

is o

n!

Shipments have documentspublic class GetShipmentSagaData : IContainSagaData{

public virtual Guid Id { get; set; }public virtual Guid UpdatedShipmentCorrelationId { get; set; }public virtual string Originator { get; set; }public virtual string OriginalMessageId { get; set; }public virtual string ShipmentId { get; set; }

// The list of all documents that need to be uploaded:public virtual IList<DocumentKey> ShipmentDocumentKeys { get; set; }

// The documents that have been uploaded (successfully, or unsucessfully)public virtual IList<CompletedShipmentDocument> CompletedDocumentsOnSaga { get; set; }

public virtual ShipmentSagaData ShipmentData { get; set; }

}16

Keeping up-to-Date Vs InitialisingGetUpdatedShipmentsFromWcfCommand(version: 110, pageSize: 10)

GetUpdatedShipmentsFromWcfCommand(130, 10)

GetUpdatedShipmentsFromWcfCommand(146, 10)

GetUpdatedShipmentsFromWcfCommand(250, 10)

GetUpdatedShipmentsFromWcfCommand(295, 10)

GetUpdatedShipmentsFromWcfCommand(300, 10)

2 minutes later: GetUpdatedShipmentsFromWcfCommand(300, 10)

4 minutes later: GetUpdatedShipmentsFromWcfCommand(300, 10) 17

Keeping up-to-Date Vs Initialising

New -> Polling -> Complete and persist the next New

Version Polled Status Timestamp

100 Complete 4 minutes ago

112 Complete 2 minutes ago

112 New 1 second ago

18

Messages were lost!Azure service bus is not

Distributed transactional!

19

Stuck polling sessionsVersion Polled Status Timestamp

100 Complete Last night - 4 minutes

112 Complete Last night - 2 minutes

112 Polling Last night

20

OutBoxAdd another graph here!

21

NService Bus OutBoxNow we need an Outbox!

22

Code of outboxBusConfiguration busConfiguration = new BusConfiguration();

busConfiguration.EnableOutbox();

<appSettings> <add key="NServiceBus/Outbox" value="true" /></appSettings>

23

OutBox CaveatsBoth the business data and deduplication data need to share the same database.

So we moved NSB Saga to SQL server!

24

Finaly!

25

DTC

is o

ff!

Stuck polling sessionsThis time with messages in the error queue...

Version Polled Status Timestamp

100 Complete Last night - 4 minutes

112 Complete Last night - 2 minutes

112 Polling Last night

26

Got rid of statusVersion Polled Timestamp

100 4 minutes ago

112 2 minutes ago

112 Now - 1 second

27

We lost the battle but won the war

28

Only one thing is leftInitialising!

But I know it is going to be bloody easy.

29

Questions?

30