CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8

IRIS-Lab / IEETA / DETI / UAUniversity of Aveiro, Portugal

CAMBADA Realtime DatabaseThe 2nd Generation

Ricardo [email protected]

MSL Workshop 2017 – 16 November – Eindhoven, Netherlands

Disclaimer

The RtDB 2 is a M.Sc. Thesis work by Diogo SilvaSupervised by professors Artur Pereira and Nuno Lau

CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands

Disclaimer

2

• Introduction– The CAMBADA RtDB– Motivation– Objectives

• The RtDB v2– Architecture– Design Choices– The new API

• Back-Compatibility With RtDB 1• Results• Conclusion and Future Work

Presentation Outline

3CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands

Introduction

CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands 4

• RtDB is an an open-source module that handles inter-robot and inter-process communication in CAMBADA

• It allows the replication of data among different agents on the team in a key-value fashion

• Any agent can retrieve information about any teammate at any time


IntroductionThe CAMBADA RtDB v1

5

• Blackboard logic


IntroductionThe CAMBADA RtDB v1

6

int DB_init(void)

void DB_free(void)

int DB_put(int _id, void *_value)

int DB_get(int _from_agent, int _id, void *_value)


IntroductionRtDB v1 API

7

• RtDB v1 Limitations– Simple raw data structure binary storage in shared

memory• Inability to store dynamic items

– The user has to define a maximum size– Always store/broadcast all item capacity, although sometimes

only a portion is useful– Static Configuration Scheme

• A configuration file is used to define all local/shared structures

– Inability to dynamically (programmatically) create items in the RtDB

– Unreliable when agents are using different structures for the same key

– No compression on broadcast


IntroductionMotivation

8

– Maintain blackboard logic– Dynamic data structures

• Allow dynamic containers (arrays) like C++ std::vector and alike– Dynamic Configuration Scheme

• Ability to add new items at will (by code)– Structure Changes Resilience

• Agents should be resilient to different schemes and data structures in the network

– Structures with new items coming should be parsed with the known items (new attributes should be ignored)

– Structures lacking attributes should also be parsed (missing attributes should retain a default value)

– No re-compilation when scheme changes• Time is precious during the competitions

– Smooth integration with the previous RtDB version• We wanted to validate the new version on the competition, but also an easy

way to fallback to the previous version if something went wrong– Duration of operations should be negligible


IntroductionRequirements

9

The RtDB v2



The RtDB v2Storing and sending

11

Raw Data Structure

LocalStorage



12

Raw Data Structure

LocalStorage

ProblemsNot tailored for dynamic structuresNot resilient to differences in data structures between processes/robots



13

Raw Data Structure

Serialized Data

LocalStorage



14

Raw Data Structure

Serialized Data

LocalStorage

WiFi - Send/Receive



15

Raw Data Structure

Serialized Data

LocalStorage

WiFi - Send/Receive

ProblemGreater bandwidth usage



16

Raw Data Structure

Serialized Data

Compressed and Serialized

Data

LocalStorage

WiFi - Send/Receive



17

Raw Data Structure

Serialized Data


Data

LocalStorage

WiFi - Send/Receive

• We evaluated several key-value storage libraries

• Requirements:– Free– Non-copyleft license– Active project (receives regular updates)– In-memory storage (for better performance)– Support for multi-process and multi-thread access– C++ API


Design ChoicesStorage

18



19

LMBDLevelDB

RedisRocksDB

BerkeleyDB

UnQLite

BangDB

Kyoto Cabinet

C-treeACE



20

LMBDLevelDB

RedisRocksDB

BerkeleyDB

UnQLite

BangDB

Kyoto Cabinet

C-treeACE

ExcludeCommercial



21

LMBDLevelDB

RedisRocksDB

BerkeleyDB

UnQLite

BangDB

Kyoto Cabinet

C-treeACEExcludeNo in-memory

ExcludeNo in-memory



22

LMBDLevelDB

RedisRocksDB

BerkeleyDB

UnQLite

BangDB

Kyoto Cabinet

C-treeACE

Exclude1 Writer process only

Exclude1 Writer process only



23

LMBDLevelDB

RedisRocksDB

BerkeleyDB

UnQLite

BangDB

Kyoto Cabinet

C-treeACE

ExcludeNo multi-process writing



24

LMBDLevelDB

RedisRocksDB

BerkeleyDB

UnQLite

BangDB

Kyoto Cabinet

C-treeACE

ExcludeNeeds additional ‘broker’ process



25

LMBDLevelDB

RedisRocksDB

BerkeleyDB

UnQLite

BangDB

Kyoto Cabinet

C-treeACE

ExcludeCopyleftlicense



26

LMBD

OpenLDAP 2.8 License (non-copyleft)

Active projectIn-memory storageMulti-processMulti-threadC++ API



27

Raw Data Structure

Serialized Data


Data

LocalStorage

WiFi - Send/ReceiveLMDB


Design ChoicesSerialization

28

Raw Data Structure

Serialized Data


Data

LocalStorage




29

CLCSchema Evolution

Binary Encoding

Text Encoding

Requires SchemaField

RenamedField

AddedField

RemovedType

Inheritance

Thrift Yes Yes Yes Yes No Yes Yes Yes

Protobuf Yes Yes Yes Yes Partial Yes Yes Yes

Avro Yes Yes Yes No Partial Yes No Yes

Msgpack Yes Yes Yes Yes Yes Yes No No

BSON Yes No Yes Yes No Yes No No

Flatbuffers Yes Yes Yes No No Yes Yes Yes

Cap’n Proto No Yes Yes No Partial Yes No Yes

Boost No Yes No No No Yes No No

Cereal No Yes No No No Yes Yes No



30

CLCSchema Evolution

Binary Encoding

Text Encoding

Requires SchemaField

RenamedField

AddedField

RemovedType

Inheritance

Thrift Yes Yes Yes Yes No Yes Yes Yes

Protobuf Yes Yes Yes Yes Partial Yes Yes Yes

Avro Yes Yes Yes No Partial Yes No Yes

Msgpack Yes Yes Yes Yes Yes Yes No No

BSON Yes No Yes Yes No Yes No No

Flatbuffers Yes Yes Yes No No Yes Yes Yes

Cap’n Proto No Yes Yes No Partial Yes No Yes

Boost No Yes No No No Yes No No

Cereal No Yes No No No Yes Yes No



31

Raw Data Structure

Serialized Data


Data

LocalStorage


MsgPack


Design ChoicesCompression

32

Raw Data Structure

Serialized Data


Data

LocalStorage


MsgPack

• We want to compress MsgPack packets

• “It's like JSON. but fast and small.”



33

{

“position”: [1.2, 4.5],

“velocity”: [0.2, 6.0],

“hasBall”: false,

“opponents”:

[

[4.3, 5.9],

[1.2, 2.0]

]

}



34

{

“position”: [1.2, 4.5],“velocity”: [0.2, 6.0],“hasBall”: false,“opponents”:

[

[4.3, 5.9],[1.2, 2.0]

]

}



35

{

“position”: [1.2, 4.5],“velocity”: [0.2, 6.0],“hasBall”: false,“opponents”:

[

[4.3, 5.9],[1.2, 2.0]

]

}



36

{

“position”: [???, ???],“velocity”: [???, ???],“hasBall”: ???,“opponents”:

[

[???, ???],[???, ???]

]

}

{


[

[???, ???],[???, ???]

]

}

{


[

[???, ???],[???, ???]

]

}

• Train a dictionary for dictionary-based compression using these dummy data packets

• lzbench1 tool was used to benchmark different frameworks inside the CAMBADA system

• zstd was finally selected for the best performance



37

1 https://github.com/inikep/lzbench



38

Raw Data Structure

Serialized Data


Data

LocalStorage


MsgPack zstd


Design ChoicesFinal Architecture

39

Raw Data Structure

Serialized Data


Data

LocalStorage


MsgPack zstd


RtDB v2 API

C

Global Functions

DB_put(int key …DB_get(int key …

DB_init()

DB_free()


RtDB v2 API

41

C C++

Global Functions


DB_init()

DB_free()


RtDB v2 API

42

C C++

Global Functions Object Methods


DB_init()

DB_free()


RtDB v2 API

43

C C++



DB_init() Constructor

DB_free() Destructor


RtDB v2 API

44

C C++



put(string key …get(string key …

DB_init() Constructor

DB_free() Destructor

Backwards Compatibility With RtDB 1


• By using a CMake custom option, we could easily revert to RtDB v1 at any time by just recompiling the code with an option– No additional actions on the code itself

• A set of macros were defined to maintain compatibility with the old API in the code

• An XML configuration file was used to ‘translate’ the old integer keys to strings


Back-Compatibility with RtDB

46

Results


• We tried the RtDB v2 for the 1st time in a competition on Portuguese Open 2017 (Coimbra, Portugal)– It ran successfully without hiccups– We never had to actually use the revert function– Still using static arrays

• We continued using it on RoboCup 2017 (Nagoya, Japan)– We used dynamic arrays (std::vector instead of static

arrays)– Again, the solution proved to be effective


Results

48


ResultsTime – PUT Operation

49

RtDB 1 = 0.86 µs average


ResultsTime – GET Operation

50

RtDB 1 = 0.94 µs average


ResultsCompression

51

RtDB 1 = N/A Compression: 45.2 µs avg. Decompression: 13.9 µs avg.

Conclusion


• The implementation of the RtDB v2 was a success!– All requirements were met!

• It’s Fast– The performance impact is negligible

• It’s Stable– Used in the two competitions CAMBADA has

participated in 2017


Conclusion

53

• Data can now be non-volatile– Database is stored in a memory-mapped file– You can reboot the computer and return to the

task right where you left it

• Cross-language compatibility• Data Safety• Sharing periodicity and phase


ConclusionSweet Extras

54

• More tools !!– Dictionary generator and RtDB realtime inspector



55

• More tools !!– Dictionary generator and RtDB realtime inspector



56

Release it open-sourceSoon at github.com/CAMBADA

Integrate process synchronizationWake next process on the pipeline after a particular PUT operation is performed

Integrate LoggingSerialized data scales better


The Next Steps

57

IRIS-Lab / IEETA / DETI / UAUniversity of Aveiro, Portugal

Thanks for your attention!/cambadamsl /cambadamsl robotica.ua.pt

Documents

CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8