Accessing IoT Data with Couchbase Server, Couchbase Mobile and Kaa: Couchbase Connect 2015

Accessing IoT data with Couchbase Server, Couchbase mobile, and Kaa

Andrew Kokhanovskyi

CTO, CyberVision, Inc.

@akokhanovskyikaaproject.org© 2015 CyberVision, Inc. All rights reserved.

kaaproject.org© 2015 CyberVision, Inc. All rights reserved.


Endpoints

Cloud

Web dashboardsMobile applications Monitoring systems

Data

Aggregated Data

Conceptual diagram


Server / Lite


➢ Fully-featured IoT middleware platform➢ 10 Kb RAM footprint (with C SDK)➢ Guaranteed data delivery and reliable local storage➢ Built-in transport security➢ Efficient data serialization➢ Horizontally scalable and fault tolerant➢ 100% open-source (Apache license 2.0)➢ Rapid application development using C / C++ / Java SDKs➢ Integration with popular device platforms

Why Kaa?


➢ Flexible document-oriented data model➢ Elastic scalability➢ Consistent high performance➢ Real-time big data analytics➢ Always-on availability➢ Open-source (Apache license 2.0)➢ Real-time Server synchronization with Couchbase Lite in user apps

Why Couchbase?

Architecture


Endpoints

Web dashboardsMobile applications Monitoring systems

Telemetrydata

Lots of telemetry data

Aggregated data

Server

Lite

Kaa cluster

Data flow: Kaa perspective

© 2015 CyberVision, Inc. All rights reserved.

Your application

KaaSDK

Sensors

Raw Data

Your device

Operations server node

Couchbaseappender

Kaaservices

Structured Data

Structured Data

kaaproject.org

Mobile application

CouchbaseLite

Applicationlogic

Data flow: Couchbase perspective


Couchbase cluster

Documents(huge volume)

View

Documents(summary)

Structured Data

Sync Gateway

CouchbaseLite

Applicationlogic

Mobile application

kaaproject.org

The secret sauce:structured data e2e



Power plant

Power plant: telemetry data collection


kaa::KaaUserLogRecord voltageReport;voltageReport.timestamp = getCurrentTime();

samples.reserve(configuration.panelCount);for (std::int32_t i = 0; i < configuration.panelCount; ++i) { kaa_log::VoltageSample sample; sample.zoneId = solarPanels_[i].getZoneId(); sample.panelId = solarPanels_[i].getPanelId(); // Analog-to-digital converter factor sample.voltage = solarPanels_[i].getVoltage(); samples.push_back(sample);}voltageReport.samples = std::move(samples);

// upload log recordkaaClient_.addLogRecord(report);

Data record schema C++ client code snippet

kaaproject.org

{ "type": "record", "name": "VoltageReport", "namespace": "org.kaaproject.kaa.examples.powerplant", "fields": [ {"name": "timestamp", "type": "long"}, {"name": "samples", "type": { "type": "array", "items": { "type": "record", "name": "VoltageSample", "fields": [ {"name": "zoneId", "type": "int"}, {"name": "panelId", "type": "int"}, {"name": "voltage", "type": "double"} ] }}}]}

© 2015 CyberVision, Inc. All rights reserved. kaaproject.org

Power plant: same client code in Java and pure C

Java snippet C snippet

Long timestamp = System.currentTimeMillis();

List<VoltageSample> samples = new ArrayList<>();for (int i = 0; i < KAA_DEMO_MAX_ZONE_ID; i++) {

Integer zoneId = i;Integer panelId = getPanelId(i);

Double voltage = getVoltage(panelId); VoltageSample sample = new VoltageSample(zoneId, panelId, voltage); samples.add(sample);}VoltageReport report = new VoltageReport(timestamp,

samples);kaaClient.addLogRecord(report);

kaa_user_log_record_t *log_record = kaa_logging_voltage_report_create();

log_record->timestamp = time(NULL) * 1000;log_record->samples = kaa_list_create();

for (size_t i = 0; i < KAA_DEMO_MAX_ZONE_ID; i++) { kaa_logging_voltage_sample_t *sample = kaa_logging_voltage_sample_create();

sample->zone_id = i; sample->panel_id = getPanelId(i); sample->voltage = getVoltage(sample->panel_id); kaa_list_push_back(log_record->samples, sample);}kaa_logging_add_record(log_collector, log_record);log_record->destroy(log_record);


Power plant: sample log document


Power plant: Couchbase view


Power plant: Couchbase Java daemon


database = manager.getDatabase(localDBName);URL dbURL = new URL(dbURLStr + gatewayName);pull = database.createPullReplication(dbURL);List<String> channels = Arrays.asList("totals");Authenticator auth = new BasicAuthenticator(username, password);pull.setAuthenticator(auth);pull.setChannels(channels);pull.setContinuous(true);pull.start();

sync gateway pull replication

View reportView = database.getView(VIEW_NAME);reportView.setMap(new Mapper() { @Override public void map(Map<String, Object> document, Emitter emitter) {

Long timestamp = (Long) document.get(TIMESTAMP_FIELD);

if (timestamp != null) { emitter.emit(timestamp, document);}

}}, "1");

local view

Power plant: Couchbase Lite Android

Video goes here


http://www.youtube.com/watch?v=sOStmVT6rXw


➢ 2 days implementation➢ Reliable telemetry data delivery➢ Average e2e latency < 1-2 sec➢ Horizontal scalability➢ Clear and simple integration➢ Collected endpoint data can be easily queried and analyzed using

Couchbase views➢ Couchbase Lite pull replication: synchronizing data to apps with ~ zero

effort

Results

Andrew Kokhanovskyi, CTO, [email protected]

kaaproject.orgcybervisiontech.com

THANK YOU FOR YOUR ATTENTION


Memory usage

Operations at 80% CPU utilization

1 node 2 nodes 4 nodes 16 nodes

Notifications / sec

45 k 86 k 173k 710 k

Configuration updates / sec

30 k 58 k 122 k 465 k

Events / sec 120 k 150 k 410 k 1 500 k

Endpoint count 100 k 200 k 400 k 800 k

RAM usage 2Gb 3Gb 5Gb 9.5Gb

Key performance indicators

kaaproject.org


Zookeeper quorum

Endpoints

Control servers

standby

Bootstrap servers

Operations servers

Fault-tolerance and horizontal scalability

kaaproject.org

active


Power plant: sync gateway configuration

{ "interface":":4984", "adminInterface":":4985", "log":["CRUD", "REST+", "Access"], "databases":{ "sync_gateway":{ "server":"http://localhost:8091", "bucket":"sync_gateway", "shadow": { "server": "http://localhost:8091", "bucket": "powerplant" }, "sync":`function(doc) { if (doc.zones != undefined) {channel("totals"); } }` } }}

Technology

Accessing IoT Data with Couchbase Server, Couchbase Mobile and Kaa: Couchbase Connect 2015