80
Using Elas*csearch and Couchbase Together to Build Large Scale Apps Uri Boness, Founder, Elas*csearch Dip* Borkar, Director, Products, Couchbase

Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Embed Size (px)

DESCRIPTION

Couchbase Server 2.0 allows for full-text search integration. In this webinar we examine how you can integrate your Couchbase Server 2.0 cluster with an Elasticsearch Cluster to provide enhanced querying capabilities and build large scale applications.

Citation preview

Page 1: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Using&Elas*csearch&and&Couchbase&Together&to&Build&Large&Scale&Apps&

Uri&Boness,&Founder,&Elas*csearch&

Dip*&Borkar,&Director,&Products,&Couchbase&

Page 2: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Introduction to Elasticsearch

Page 3: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

What is Elasticsearch?

Open source Apache 2 license•

multi-tenant, realtime anddistributed search & analytics

engine

Backed by Elasticsearch (the company)•

Proven technology in productionOver 2 million downloads

Page 4: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

What can Elasticsearch do?Unstructured search

find all companies in the “search” market

Structured searchfind all companies founded since 2000

Analyticsfind the average annual revenue of all companies

Combine allfind the average annual revenue of all companies foundedsince 2000 within the “search” market

Page 5: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

(near) real-time!

Page 6: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Distributed & multi-tenantA node is single Elasticsearch instanceMultiple nodes can form a clusterA cluster can manage multiple indicesA cluster is agile & self managing

continuously ensuring the distributed characteristics of allindices are maintained and that all nodes in the cluster areefficiently & effectively utilized

••••

Page 7: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

The Index

Page 8: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

What’s in an index?An identified collection of documentsBuilt & designed for small & large scales

data volumesdata can be split and distributed between shards

loads & HAeach shard can have zero or more replicas

••

Page 9: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

starting a node

node_1

Page 10: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

creating our first index

node_1

curl -XPUT 'localhost:9200/companies' -d '{ "settings" : { "index" : { "number_of_shards" : 2, "number_of_replicas" : 1 } }}'

Page 11: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

the two shards are allocated

node_1

0 1

curl -XPUT 'localhost:9200/companies' -d '{ "settings" : { "index" : { "number_of_shards" : 2, "number_of_replicas" : 1 } }}'

Page 12: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

starting a second node

node_1 node_2

0 1

curl -XPUT 'localhost:9200/companies' -d '{ "settings" : { "index" : { "number_of_shards" : 2, "number_of_replicas" : 1 } }}'

Page 13: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

shards are relocating

node_1 node_2

0 1

curl -XPUT 'localhost:9200/companies' -d '{ "settings" : { "index" : { "number_of_shards" : 2, "number_of_replicas" : 1 } }}'

Page 14: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

replicas are allocated

node_1 node_2

0 11 0

curl -XPUT 'localhost:9200/companies' -d '{ "settings" : { "index" : { "number_of_shards" : 2, "number_of_replicas" : 1 } }}'

Page 15: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Indexing Data

Page 16: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

the dataDocuments are typically JSON formatted•

curl -XPUT 'localhost:9200/companies/company/1' -d '{ "id" : "elasticsearch", "name" : "elasticsearch", "website" : "http://www.elasticsearch.com", "category" : "software", "overview" : "distributed search & analytics engine", "founded_year" : 2012, "location" : { "city" : "Amsterdam", "country_code" : "NL", "geo" : { "lat" : 52.370176, "lon" : 4.895008 } }}'

Page 17: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

sending req. to one of the nodes

node_3node_1 node_2

0 11 010

client

Page 18: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

sending req. to one of the nodes

node_3node_1 node_2

0 11 010

client

resolve the target shard

Page 19: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

resolve shard & index to primary

node_3node_1 node_2

0 11 010

client

Page 20: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

replicate to replicas

node_3node_1 node_2

0 11 010

client

Page 21: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Searching

Page 22: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

unstructured searchUsing an extensive & powerful QueryDSL•

curl -XGET 'localhost:9200/companies/_search' -d '{ "query" : {, "match" : { "overview" : "search" } }}'

Page 23: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

unstructured searchUsing an extensive & powerful QueryDSL•

curl -XGET 'localhost:9200/companies/_search' -d '{ "query" : {, "match" : { "overview" : "search" } }}'

search for the term “search” in the “overview”field

Page 24: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

structured searchnarrows the “searchable” document space•

curl -XGET 'localhost:9200/companies/company/_search' -d '{ "query" : {, "filtered" : { "query" : { "match" : { "overview" : "search" } }, "filter" : { "range" : { "founded_year" : { "gte" : 2000 } } } } }}'

Page 25: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

structured searchnarrows the “searchable” document space•

curl -XGET 'localhost:9200/companies/company/_search' -d '{ "query" : {, "filtered" : { "query" : { "match" : { "overview" : "search" } }, "filter" : { "range" : { "founded_year" : { "gte" : 2000 } } } } }}'

only search companies that were founded since year 2000

Page 26: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

returned hits{ ... "hits": [ { "_index": "companies", "_type": "company", "_id": "1", "_score": 0.13424811, "_source": { "id": "elasticsearch", "name": "elasticsearch", "website": "http://www.elasticsearch.com", "category": "software", "founded_year": 2012, "overview": "distributed search & analytics engine", "location": { "city": "Amsterdam", "country_code": "NL", "geo": { "lat": 52.370176, "lon": 4.895008 } } } } ] }}

Page 27: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

returned hits{ ... "hits": [ { "_index": "companies", "_type": "company", "_id": "1", "_score": 0.13424811, "_source": { "id": "elasticsearch", "name": "elasticsearch", "website": "http://www.elasticsearch.com", "category": "software", "founded_year": 2012, "overview": "distributed search & analytics engine", "location": { "city": "Amsterdam", "country_code": "NL", "geo": { "lat": 52.370176, "lon": 4.895008 } } } } ] }}

Page 28: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

returned hits{ ... "hits": [ { "_index": "companies", "_type": "company", "_id": "1", "_score": 0.13424811, "_source": { "id": "elasticsearch", "name": "elasticsearch", "website": "http://www.elasticsearch.com", "category": "software", "founded_year": 2012, "overview": "distributed search & analytics engine", "location": { "city": "Amsterdam", "country_code": "NL", "geo": { "lat": 52.370176, "lon": 4.895008 } } } } ] }}

Page 29: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

returned hits{ ... "hits": [ { "_index": "companies", "_type": "company", "_id": "1", "_score": 0.13424811, "_source": { "id": "elasticsearch", "name": "elasticsearch", "website": "http://www.elasticsearch.com", "category": "software", "founded_year": 2012, "overview": "distributed search & analytics engine", "location": { "city": "Amsterdam", "country_code": "NL", "geo": { "lat": 52.370176, "lon": 4.895008 } } } } ] }}

Page 30: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Query DSLQueries (unstructured)

term queries

boolean queries

phrase (proximity) queries

fuzzy/prefix/regexp/wildcards

more...

Filters (structured)term (exact match)

range

boolean

geo_* (e.g. geo_distance)

Page 31: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Analytics(a.k.a facets)

Page 32: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Analytics (facets)Slice & dice your dataCompute aggregations over field valuesAcross any index field/sAll in (near) realtime

••••

Page 33: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

used as navigation aid

Page 34: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

or analytics dashboards

Page 35: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Elasticsearch is often usedpurely for analytics

(without incorporating free text search)

Page 36: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

ExampleFind the average revenue of all companies

since 2000•

curl -XGET 'localhost:9200/companies/revenues/_search' -d '{ "query" : { "match_all" : {} }, "facets" : { "revenue_stats" : { "date_histogram" : { "key_field" : "year", "value_field" : "value", "interval" : "month" } } }}'

Page 37: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

ExampleFind the average revenue of all companies

since 2000•

curl -XGET 'localhost:9200/companies/revenues/_search' -d '{ "query" : { "match_all" : {} }, "facets" : { "revenue_stats" : { "date_histogram" : { "key_field" : "year", "value_field" : "value", "interval" : "month" } } }}'

return a yearly breakdown of stats over companies revenues

Page 38: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

response"facets": { "revenue_stats": { "_type": "date_histogram", "entries": [ { "time": 956448895664, "mean": 23.0 }, { "time": 987984922557, "mean": 267.1034482758621 }, { "time": 1019520942098, "mean": 195.51724137931035 } ... ] } }

Page 39: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

response"facets": { "revenue_stats": { "_type": "date_histogram", "entries": [ { "time": 956448895664, "mean": 23.0 }, { "time": 987984922557, "mean": 267.1034482758621 }, { "time": 1019520942098, "mean": 195.51724137931035 } ... ] } }

year 2000

avg revenue

Page 40: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Types of analyticsterms

unique value counts

rangestatistics of specific field for a set of range groups ofanother field

statisticalstats over a specific field

terms_statsstats over a specific fields for every unique field value

date_/histograma breakdown of statistics of a specific field over a

Page 41: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

There’s much moreFine control of how documents are treated

indexed, stored, text analysis, relations

Additional featureshighlighting

suggest API (type ahead, auto-completion)

percolator (reverse search)

support of document relations (parent/child)

extensive geo-location search & analytics

more....

•------

Page 42: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Introduc)on*to*Couchbase*

Page 43: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Couchbase*Server*NoSQL*Document*Database*

Page 44: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Couchbase*Open*Source*Project*

•  Leading(NoSQL(database(project(focused(on(distributed(database(technology(and(surrounding(ecosystem(

•  Supports(both(key;value(and(document;oriented(use(cases(

•  All(components(are(available(under(the(Apache*2.0*Public*License*

•  Obtained(as(packaged(so?ware(in(both(enterprise(and(community(ediAons.(

Couchbase Open Source Project

Page 45: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Easy*Scalability*

Consistent*High*Performance*

Always*On*24x365*

Grow(cluster(without(applicaAon(changes,(without(downAme(with(a(single(click(

Consistent(sub;millisecond((read(and(write(response(Ames((with(consistent(high(throughput(

No(downAme(for(so?ware(upgrades,(hardware(maintenance,(etc.(

JSONJSONJSON

JSONJSON

PERFORMANCE

Flexible*Data*Model*

JSON(document(model(with(no(fixed(schema.(

Couchbase*Server*

Page 46: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Features*in*Couchbase*Server*2.0*

JSON*support* Indexing*and*Querying*

Cross*data*center*replica)on*Incremental*Map*Reduce*

JSONJSONJSON

JSONJSON

Page 47: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Addi)onal*Features*

Built;in(clustering(–(All(nodes(equal((Data(replicaAon(with(auto;failover((Zero;downAme(maintenance(((Built;in(managed(cached((

((

Append;only(storage(layer((Online(compacAon((Monitoring(and(admin(API(&(UI((SDK(for(a(variety(of(languages(

Page 48: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Couchbase*Server*2.0*Architecture*

Heartbe

at(

Process(mon

itor(

Global(singleton

(sup

ervisor(

Confi

guraAon

(manager(

on(each(node(

Rebalance(orchestrator(

Nod

e(he

alth(m

onitor(

one(per(cluster(

vBucket(state(and(replicaA

on(m

anager(

hQp*RE

ST*m

anagem

ent*A

PI/W

eb*UI*

HTTP(8091*

Erlang(port(mapper(4369*

Distributed(Erlang(21100*Y*21199*

Erlang/OTP*

storage(interface(

Couchbase*EP*Engine*

11210*Memcapable((2.0(

Moxi*

11211*Memcapable((1.0(

Memcached*

New*Persistence*Layer*

8092*Query(API(

Que

ry*Engine*

Data*Manager* Cluster*Manager*

Page 49: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

3(3( 2(

Cross*data*center*replica)on*–*Data*flow*2(

Managed(Cache(

Disk(Que

ue(

Disk(

ReplicaAon(Queue(

App(Server(

Couchbase(Server(Node(

Doc*1*Doc*1*

Doc*1*

To(other(node(

XDCR(Queue(

Doc*1*

To(other(cluster(

Page 50: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Cross*Datacenter*Replica)on*(XDCR)*

Page 51: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Couchbase*plugYin*for*Elas)csearch*

Page 52: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

How*does*it*work?*

Elas)cSearch*

UnidirecAonal(Cross(Data(Center(ReplicaAon(

Page 53: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

ElasAcsearch(IntegraAon((via(XDCR)(

RAM(CACHE(

Doc(1(

Doc(2(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

SERVER(1(

Doc(6(

DISK(

RAM(CACHE(

Doc(1(

Doc(2(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

SERVER(2(

Doc(6(

DISK(

RAM(CACHE(

Doc(1(

Doc(2(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

Doc(

SERVER(3(

Doc(6(

DISK(

Couchbase(Cluster(West(Coast(Data(Center(

ES(SERVER(1(

ElasAc(Search(Cluster(

ES(SERVER(2( ES(SERVER(3(

Couchbase(Transport(Plugin(

Couchbase(Transport(Plugin(

Couchbase(Transport(Plugin(

Page 54: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Install*the*Couchbase*PlugYIn*•  PreYrequisite*­  ExisAng(Couchbase(and(ElasAcSearch(Clusters(

•  Install*the*Elas)cSearch*Couchbase*Transport*PlugYin*­  bin/plugin(;install((

((((((((((((couchbaselabs/elasAcsearch;transport;couchbase/1.0.0;dp(

•  Configure*the*PlugYin*­  Set(a(password(­  Install(the(Couchbase(Index(Template(

•  Restart*Elas)cSearch*•  Create*an*Elas)cSearch*index*for*your*documents*

Page 55: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Configure*Couchbase*XDCR*(step*1)*

Page 56: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Configure*Couchbase*XDCR*(step*2)*

Page 57: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Documents*are*now*indexed*in*Elas)csearch*

Document(Count(Increasing(

Page 58: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Reference*Architecture*

Page 59: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Recommended*Usage*PaQern*

Elas)cSearch*

1.((ElasAcSearch(Query(

2.(ElasAcSearch(Result(

3.(Couchbase(MulA;GET(

4.(Couchbase(Result(

Page 60: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Common*Couchbase*Use*Cases*Social*Gaming*

*•  Couchbase(stores(player(and(game(data((

•  Examples(customers(include:(Zynga(

•  Tapjoy,(Ubiso?,(Tencent(

*

*Mobile*Apps*

*•  Couchbase(stores(user(info(and(app(content(

•  Examples(customers(include:(Kobo,(PlayAka((

*

*

Ad*Targe)ng**

•  Couchbase(stores(user(informaAon(for(fast(access(

•  Examples(customers(include:(AOL,(Mediamind,(Convertro((

*

Session*store**

•  Couchbase(Server(as(a(key;value(store(

•  Examples(customers(include:(Concur,(Sabre(

*

User*Profile*Store**

•  Couchbase(Server(as(a(key;value(store(

•  Examples(customers(include:(Tunewiki(

*High*availability*cache**

•  Couchbase(Server(used(as(a(cache(Aer(replacement(

•  Examples(customers(include:(Orbitz(

*

Content*&*Metadata*Store*

•  Couchbase(document(store(with(ElasAc(Search(

•  Examples(customers(include:(McGraw(Hill(

*

*3rd*party*data**aggrega)on**

*•  Couchbase(stores(social(media(and(data(feeds(

•  Examples(customers(include:(Sambacloud(

*

Page 61: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Social*Gaming**

•  Couchbase(stores(player(and(game(data((

•  Examples(customers(include:(Zynga(

•  Tapjoy,(Ubiso?,(Tencent(

*

*Mobile*Apps*

*•  Couchbase(stores(user(info(and(app(content(

•  Examples(customers(include:(Kobo,(PlayAka((

*

*

Ad*Targe)ng**

•  Couchbase(stores(user(informaAon(for(fast(access(

•  Examples(customers(include:(AOL,(Mediamind,(Convertro((

*

Session*store**

•  Couchbase(Server(as(a(key;value(store(

•  Examples(customers(include:(Concur,(Sabre(

*

User*Profile*Store**

•  Couchbase(Server(as(a(key;value(store(

•  Examples(customers(include:(Tunewiki(

*High*availability*cache**

•  Couchbase(Server(used(as(a(cache(Aer(replacement(

•  Examples(customers(include:(Orbitz(

*

Content*&*Metadata*Store*

•  Couchbase(document(store(with(ElasAc(Search(

•  Examples(customers(include:(McGraw(Hill(

*

*3rd*party*data**aggrega)on**

*•  Couchbase(stores(social(media(and(data(feeds(

•  Examples(customers(include:(Sambacloud(

*

Page 62: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

RealYworld*example*Couchbase*+*Elas)csearch*

Page 63: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

• Content*metadata*• Content:*Ar)cles,*text**• Landing*pages*for*website*• Digital*content:*eBooks,*magazine,*research*material**

Content*and*Metadata*Store*

Use*Case:*Content*and*Metadata*Store*

•  Flexibility*to*store*any*kind*of*content*•  Fast*access*to*content*metadata*(most*accessed*objects)*and*content**•  FullYtext*Search*across*data*set*•  Scales*horizontally*as*more*content*gets*added*to*the*system*

• Fast*access*to*metadata*and*content*via*objectYmanaged*cache*•  JSON*provides*schema*flexibility*to*store*all*types*of*content*and*metadata*•  Indexing*and*querying*provides*realY)me*analy)cs*capabili)es*across*dataset**•  Integra)on*with*Elas)cSearch*for*fullYtext*search*• Ease*of*scalability*ensures*that*the*data*cluster*can*be*grown*seamlessly*as*the*amount*of*user*and*ad*data*grows*

Types*of*Data* Applica)on*Requirements*

Why*NoSQL*and*Couchbase**

Page 64: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

McGraw*Hill*Educa)on*Labs**Learning*portal*

*

Page 65: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Use*Case:*Content*and*metadata*store*

Building(a(self;adapAng,(interacAve(learning(portal(with(Couchbase(and(ElasAcsearch(

Page 66: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

As learning move online in great numbers

Growing need to build interactive learning environments that

Scale!!Scale(to(millions(of(learners(

Serve(MHE(as(well(as(third;party(content(

Including(open(content(

Support(learning(apps(

010100100111010101010101001010101010(

Self;adapt(via(usage(data(

The Problem*

Page 67: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

• Allow(for(elasAc(scaling(under(spike(periods(

• Ability(to(catalog(&(deliver(content(from(many*sources*

• Consistent(lowYlatency*for(metadata(and(stats(access(

• Require(fullYtext*search*support(for(content(discovery(

• Offer(tunable(content(ranking(&(recommendaAon(

funcAons((

Backend is an Interactive Content Delivery Cloud that must:

XML(Databases(

SQL/MR(Engines(

In;memory(Data(Grids(

Enterprise(Search(Servers(

Experimented with a combination of:

The Challenge*

Page 68: Using Elasticsearch and Couchbase Together to Build Large Scale Applications
Page 69: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

•  Document(Modeling(

•  Metadata(&(Content(Storage(

•  View(Querying(to(support(Content(Browsing(•  ElasAcsearch(IntegraAon((;  Content(Updated(in(near(Real;Time(

;  Search(Content(Summaries(

;  Relevancy(boosted(based(on(User(Preferences(•  Real;Time(Content(Updates(

•  Event(Logging(for(offline(analysis(

Techniques*Used*

Page 70: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Couchbase*2.0*****+******Elas)csearch*

Store(full-text articles(as(well(as(document metadata(for(image,(video(and(text(content(in(Couchbase(

Combine(user(preferences(staAsAcs(with(custom relevancy scoring(to(provide(personalized search results

Logs(user behavior(to(calculate(user(preference(staAsAcs((e.g.(video(>(text)(

1(

2( 4(

ConAnuously(accept updates from(Couchbase(with(new(content(&(stats(

3(

Page 71: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Data(Model(

Content Metadata Bucket

User Profiles Bucket

Content Stats Bucket

•  Stores content metadata for media objects and content for articles

•  Includes tags, contributors, type information

•  Includes pointer to the media

•  Stores user view details per type •  Updated every time a user views

a doc with running count •  To be used for customizing ES

search results per user preference

•  Stores content view details •  Updated for every time a

document is viewed •  To be used for boosting ES

search results based on popularity

Page 72: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Couchbase Views

Top Contributors & Tagsdriven by Incremental MapReduce Views!

Calcula)ng*sta)s)cs*via*Couchbase*

Page 73: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Tuning(content(ranking(via(

ElasticSearch

ElasticSearch-driven based on settings below!

Content popularity boost!

User preference boost!

Page 74: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

{ "_id": �4ae5be2df3122f06ba45b70753001841�,

�_rev�: �1-0013b349ffc3afc700000000068000000�, �$flags�: 0, �#expiration�: 0, �type�: �access�, �user�: �[email protected]�, �resource�: �379823�, �timestamp�: �2012-09-02T22:46:07Z�

}

{ "_id": �4ae5be2df3122f06ba45b70753001842�,

�_rev�: �1-0013b349ffc3afc700000000068000000�, �$flags�: 0, �#expiration�: 0, �type�: �create�, �user�: �[email protected]�, �resource�: �948177�, �timestamp�: �2012-09-02T22:48:32Z�

}

What?!

Who?!

Which?!

When?!

Analy)cs*and*Event*Logging*

•  Store*full*event*log*for*offline*analysis*

•  Stored*on*a*separate*analy)cs*cluster**

•  Limit*impact*on*OLTP*

•  Tuned*differently*

•  Keep*an*upperYbound*on*data*size*via*TTL*(24*hrs)*

Page 75: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

{ "filter": { "term": { "type": "video� } }, "boost": USER_VIDEO_PREFERENCE * PREFERENCE_SLIDER }

User*Preference*Boost*

•  Use*Elas)csearch*filter*boos)ng*

Page 76: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

"script": "_score * (((doc['popularity'].value + 1) / AVG_POPULARITY ) * POPULARITY_SLIDER)"

Document Popularity Boost*

•  Use*Elas)csearch*custom*script*to*score*documents*

Page 77: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

"filters": [ { "filter": { "term": { "type": "video" } }, "boost": USER_VIDEO_PREFERENCE * PREFERENCE_SLIDER }, … image and texts filters omitted … ], "score_mode": "total" } }, "script": "_score * (((doc['popularity'].value + 1) / AVG_POPULARITY ) * POPULARITY_SLIDER)" }

Combined Algorithm in a Query*

Page 78: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

The Learning Portal*

•  Designed and built as a collaboration between MHE Labs and Couchbase

•  Serves as proof-of-concept and testing harness for Couchbase + Elasticsearch integration

•  Available for download and further development as open source code

h"ps://github.com/couchbaselabs/learningportal5

Page 79: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Q*&*A*

Page 80: Using Elasticsearch and Couchbase Together to Build Large Scale Applications

Thank*you*

*******

dip)@couchbase.com*uri.boness@elas)csearch.com*

****