Couchbase usage at Symantec

Couchbase at Symantec

Gaurav ChandnaSr Principal Software Engineer

About Me

Gaurav ChandnaAt Symantec for 11 years.

Worked at AntiSpyware, Behavioral malware, Norton and now building platforms.

Engineering Manager previously

@xyxxy

2

Copyright © 2014 Symantec Corporation3

Agenda

1 Introduction

2 Time series

3 Relational

4 Q&A

Copyright © 2014 Symantec Corporation 4

Introduction


Environment

• Service Oriented Architecture

• Message Oriented Middleware

• Multi-tenant

• Multiple data centers


Message Oriented Architecture


Replication between data centers


Time Series Data




Message Broker

Service

ElasticSearch

Query & Visualization


Quick side bar: Elasticsearch


Elasticsearch

• Document oriented search engine

• Based on Apache Lucene project

• JSON based

• Distributed

• Scale out

• Multi-tenancy

• RESTful


Elasticsearch

• Structured search

• Unstructured search

• Analytics

• Combine


All good?


Well… kind of

• Elasticsearch doesn’t recommend using it as data store– Due to write coalescing

• Limited support for replication between data centers– Especially across unreliable networks


New pipeline

Message Broker

Service

Couchbase

ElasticSearch

Query & Visualization


Cross data center replication


Used to replicate Elasticsearch between data centers


It’s all good!


Just one more thing…

• Document stored on Couchbase After X days, the document gets purged

• XDCR will purge in Elasticsearch also

• Forked the Couchbase – Elasticsearch connector to disable deletion

• Looking to integrate those changes to Couchbase for community use


And we are finally good!


Data with relationships


Relationships are everywhere

Engineering

Gaurav

iPhone Mac

John

PC Android


Right here…

Meetup @Symantec

Gaurav

iPhone

Joshua

Android


In abstract terms

A

B

C D

C’

Y


You get information through queries

• Get all members of a particular group

• All groups ordered by number of people in the group

• List all members of a group, order by location

• List all groups where users have iPhone, ordered by total number of people

• And so on..


Engineering considerations

• Ability to add new types – Don’t make big changes when iWatch came out

• Ability to extend the types

• Add new relationships quickly


Table stakes

• Cross data center replication

• Performance

• Horizontal scaling

• Searching

• Querying capability


Initial implementation

• {

• ”type": ”person”,

• "name": ”Gaurav",

• "modified": "2015-04-17T00:39:45.468Z",

• "_id": "79014cbb-69ce-4459-8f4a-6edaf4b8313a",

• // other information

• }

• {

• ”type": ”department”,

• "name": ”Engineering",

• "_id": "79014cbb-69ce-4459-8f4a-6edaf4b8313a",

• // other information

• }


Initial version

• Store different objects with types as documents

• Create views in Couchbase– Type– Relationships between types– Specialize on special types

• Ensure persistence during operations

• Works well… to a point


So we needed a next generation


Second iteration

• {

• "_id":"Well_crafted_id”

• // Other meta information

• // Other associated information

• }

• Relationship information embedded in the _id

• Takes advantage of key access to document by id

• Can still build views on these documents

• Does require more hand crafting to get better performance

• Maintain flexibility

• Need to play around with different APIs to see which work best for use case

• More generic code


What about search?


Use Elasticsearch for search!

• Couchbase is primary data store

• Use Couchbase XDCR for one direction replication – Like time series data– But don’t disable deletes

Message Queue

Service

Couchbase

Elasticsearch


But…

• Elasticsearch prefers types to be well defined for indexing

• Code is generic for extensibility

• Couchbase – Elasticsearch plugin only supports 1 configuration

• Different services have different needs


• More modifications to the Elasticsearch – Couchbase connector

• Custom code to deduce types for compatibility with Elasticsearch

• Per bucket configurations so deletes for time series but not for relational data

• Keep dynamically defined types and the code generic


Final thoughts

• Data modeling is extremely important

• JSON Parser – experiment for your use case

• Experiment with the different API options for your needs

• Don’t be afraid to customize for your needs!

• Polyglot persistence – Matt’s presentation

Thank you!

Copyright © 2014 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.

This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

Gaurav [email protected]: @xyxxy

mailto:[email protected]

Thank you!

Copyright © 2014 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.

This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

We are hiringwww.symantec.com/careers


&Q A

Data & Analytics

Couchbase usage at Symantec