Embed Size (px)
DESCRIPTION
Speaker: Mark Johnson Real-time analytics is the process of monitoring and often reacting to web based events or machine-to-machine communication and then quickly deliver the appropriate response to an end-user or machine process. The challenge though is how can we modify the analytics algorithms and reference data without shutting down the data feeds. In this session we will demonstrate architectural and implementation patterns utilizing Spring Integration Framework, HBase, and of course Groovy to illustrate how it is possible to dynamically change your enterprise’s real-time algorithms without a system restart.
Citation preview
© Hortonworks Inc. 2013
Utilizing Groovy based closures for flexible real-time analytics
Mark Johnson [email protected]
Page 1
© Hortonworks Inc. 2013
Who Am I?
• Director Consulting Hortonworks • Distributed Compute fan • Lazy programmer who prefers to reuse rather than re-implement code • President New England Java Users Group
Page 2 Architecting the Future of Big Data
© Hortonworks Inc. 2013 Page 3 Architecting the Future of Big Data
Real – Time Analytics
Discovery Analytics
© Hortonworks Inc. 2013
Real-Time Analytics
Real-time analytics is the use of, or the capacity to use, all available enterprise data and resources when they are needed. It consists of dynamic analysis and reporting, based on data entered into a system less than one minute before the actual time of use.
Page 4 Architecting the Future of Big Data
© Hortonworks Inc. 2013 Page 5 Architecting the Future of Big Data
Real-Time Analytic
Challenges
Lost Bits
Changing Analytic models
too much time to execute
© Hortonworks Inc. 2013
Architectural Tradeoffs
Page 6 Architecting the Future of Big Data
Changeability Requirements: • 24 x 7 processing • Models change • JVM class reload issues
Performance Requirements: • Required reaction time • Resource consumption
© Hortonworks Inc. 2013
Groovy Closures
A Groovy Closure is a code block which is usable via a variable instance for later execution.
Page 7 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Groovy Closure: Example
Page 8 Architecting the Future of Big Data
© Hortonworks Inc. 2013
GroovyShell
Page 9 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Groovy closures and Hotspot
Page 10 Architecting the Future of Big Data
• Groovy script improves with hotspot execution
0
20
40
60
80
100
120
140
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
© Hortonworks Inc. 2013
Architecture
Page 11 Architecting the Future of Big Data
© Hortonworks Inc. 2013 Page 12 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Solution: Dynamic Code deployment
• Embed GroovyShell() into service activator • Reference an externally defined Groovy script from a service
activator base method implementation.
• Groovy script may be located most anywhere: – Property file – Database – External text file
Page 13 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Service Activator Example
Page 14 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Solution: Load Management
Page 15 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Solution: Fast / High Capacity Archival Storage
• Archival data necessary to back test real-time analytics • Sensor Data samples can lead to erroneous conclusions
Page 16 Architecting the Future of Big Data
Real – Time Analytics
Discovery Analytics
© Hortonworks Inc. 2013
Demo
Architecting the Future of Big Data Page 17
