Upload
redblacktree
View
636
Download
2
Embed Size (px)
Citation preview
Couchbase & BigData-Analytics
Kadhambari AnbalaganSoftware Architect, RedBlackTree Technologies Pvt. Ltd.
©2014 Couchbase Inc.
Couchbase + Big Data
New Data Stream Merged View
All DataPrecompute
Views (Map Reduce)
Process Stream
Incremental Views
Partial Aggregat
e
Partial Aggregat
e
Partial Aggregat
e
Real-Time Data
BatchRecompute
Batch Views
Real-Time Views
Real-TimeIncrement
Merge
Batch Layer
Serving Layer
Speed Layer
Couchbase HadoopConnector
©2014 Couchbase Inc.
Use Case – Cookie Store
©2014 Couchbase Inc.
Cookie Store - The Problem
©2014 Couchbase Inc.
A Central Cookie Repo – The Solution
©2014 Couchbase Inc.
Infrastructure Requirements
Had to operate at high volume Caching and Persistence Low latency – 3 to 5 ms Flexible data structure High availability Scalability Fault tolerant Disaster Recovery
©2014 Couchbase Inc.
Migration Strategy
Analyze
Regression
Dual-ModeMigrate
Monitor
©2014 Couchbase Inc.
Functional View
CookieService
Couchbase DC A Couchbase DC B
ApplicationCookie Libraries
Couchbase Client
Customers
Front Tier
Mid Tier
Data Tier
©2014 Couchbase Inc.
Deployment Model
CookieService
CookieService
CookieService
XDCR
Active
WriteRead
Birdirectional Unidirectional
Active Passive
9
©2014 Couchbase Inc.
Cookie Analytics
©2014 Couchbase Inc.
Use Case – User Activity Tracking
12©2014 Couchbase Inc.
User activity tracking and real-time analytics
Objectives & ChallengesProvide business users with real time reports and visualizations of user interaction data Collect web and mobile clickstream in real time Integrate with other big data technologies (Hadoop and
Storm) Provide views of data across multiple dimensions (e.g., time,
location, browser and device types)
130m+ active accounts, in 190+ countries, 25 currencies
10TB data 1B documents
SolutionDeploy Couchbase Server to capture, store, and process real time web data Ingests data (via Storm) from multiple inputs, including
mobile, web, and other services, storing data as JSON documents
Integrates with Hadoop to pass data for additional offline analytics
Generates views for business users in under 1 minute, based on 10-minute data collection intervals
The Couchbase AdvantageReal time performance, easy integration with Storm and Hadoop
©2014 Couchbase Inc.
User activity tracking and real time analytics
©2014 Couchbase Inc.
User activity tracking and real-time analytics
14
Couchbase Solution Couchbase Server deployed to capture, store, and
process real time web data Ingests data (via Storm) from multiple inputs, including
mobile, web, and other services, storing data as JSON documents
Integrates with Hadoop to pass data for additional offline analytics
Results Consistent low latency (sub 10-msec response) High availability enabled by distributed caching and
XDCR Views for business users are generated in under 1
minute, based on 10-minute data collection intervals
Couchbase & Fraud Detection
16©2014 Couchbase Inc.
Fraud Detection with FICOObjective & ChallengesCapture transactions, store card / account profiles, customer profiles & user defined variables with sub-msec latency and high throughput Growing number of accounts, cards and customers means
more data needs to be tracked and faster latencies are required
Relational systems unable to scale to the required throughput HA / DR solutions not streamlined – need custom
development
Falcon #1 Fraud Detection
platform in the world
65% of worlds credit / debit cards scored by Falcon Solution
Use Couchbase as the “profiling store” and replace relational database Each Falcon customer has 100’s of millions of card and / or
account profiles that can easily be stored and updated based on consumer’s real time activity
Neural networking algorithms run on Couchbase and access data as key-value pairs. Memory-first architecture allows <1ms responses.
Complete HA / DR solution delivers 24x365 application uptime
The Couchbase AdvantageMemory-first architecture means high throughput, all with click-button scalability
©2014 Couchbase Inc.
Fraud Detection with Couchbase at Wells Fargo
17
©2014 Couchbase Inc.
Fraud Detection with Couchbase at Wells Fargo
18
Couchbase Solution Use Couchbase as the “profiling store” and replace
relational database Each Falcon customer has 100’s of millions of card and /
or account profiles that can easily be stored and updated based on consumer’s real time activity
Results Complete HA / DR solution delivers 24x365 application
uptime Memory-first architecture allows <1ms responses. Neural networking algorithms run on Couchbase and
access data as key-value pairs
IOT with Couchbase
©2014 Couchbase Inc.
Internet of Things @ Verizon
20
Objective & ChallengesEnable new service offering for Verizon enterprise customers to manage all devices connected to the company’s network Collect and store data in real time from 10K’s-100K’s of
devices on a single customer’s network Analyze data for usage statistics and patterns Provide near real-time insights and reports into device
usage
New enterprise offering
Enable enterprises to manage all non-cellphone devices on their network
Provide near real time insights and views on devices
SolutionDeploy Couchbase Server to store data and serve reports on connected devices Couchbase Server ingests data at high speed, from any kind
of connected device: alarms, locking systems, modems, solar panels, cash registers
Stream-based indexing enables fast views and reports JSON data model easily handles any data type, new data
types
The Couchbase AdvantageMassive speed and scale that’s easy to manage
©2014 Couchbase Inc.
Verizon - Architecture
©2014 Couchbase Inc. 22
Thank You