Embed Size (px)
Citation preview
Resources
• Confluence page: https://confluence.oceanobservatories.org/display/CIDev/ANF-SQLstream+Integration
• ANF Detection Network code: https://github.com/daf/anf-detection-network
Introduction / Goals
• Using elastic processing (cloud computing) to operate on seismic data
• Drive development and test ION (https://confluence.oceanobservatories.org/display/CIDev/Home)
• Data from Array Network Facility (http://anf.ucsd.edu/)
• Uses SQLstream (http://sqlstream.com/) for processing for real time processing
(con’t)
• Continuation of POC/Prototype (Aug 2009, https://confluence.oceanobservatories.org/display/CIDev/SQLstream+Prototype)
Architecture
(con’t)
• ANF Application Controller– Part of four op units in a deployment– Listens for instrument announcements, partitions
those instruments into AMQP queues, requests SQLstream worker op units be created
• ANF Application Agent– Runs on SQLstream worker op unit– Manages SQLstream instances, communicates
status with Application Controller
AppAgent
• Installs/configures SQLstream on demand from AppController– Allows us to use common base image– More than one SQLstream instance (listening to
different queues) can be run on a worker op unit– Downsides: time spent transferring installer from
S3, time spent installing SQLstream
SQLstream
• Listens to AMQP queue assigned to it by App Controller
• Transmits detection events over AMQP topic “anf.detections”
Deployment
• Uses CEI developed tools cloudinit.d, epumgmt– cloudinit.d:
https://github.com/nimbusproject/cloudinit.d– epumgmt:
https://github.com/nimbusproject/epumgmt• Actual startup commands:
– cloudinitd boot launch-plans/anf/main.conf -n anf -v -v -f /tmp/anf.log– epumgmt.sh -a find-workers -n anf --wholerun– epumgmt.sh -a logfetch -n anf --wholerun
Demo
• Live system– 30 stations
• Captured Data– 10 stations, showing events
Metrics
Cloudinit.d• (6:30-9:30)
Worker Booted• (~ 2:52)
SQLstream running• (~ 4:58)
Instrument Announce
~ 7:20 startup
(con’t)
Install Ready Defined Running
36s 12.5s 44.47s 175.40s (2:55)
4:58
SQLstream running
Future Goals
• Data Preservation– Detections, possibly input events
• Service Hardening– Distributed, stateless processes
• Replacement for instrument announcement• True load balancing– Need more metrics with live data
(con’t)
• Monitoring– Graphical dashboard showing SQLstream op units,
queues/bindings, consumption rates
