Embed Size (px)
DESCRIPTION
Big Data: State of the Data Universe - What the best are doing and why your data is your future Business is being driven by data at an ever increasing rate. If you aren't gathering and making sense of more data than your competition then you are behind. Smart businesses are developing mechanisms to generate more detailed data. Customers are starting to expect more tailored products, increased support, and an overall better experience. How does one keep up? Kenny Gorman will speak to the state of the data universe. What are businesses doing? What challenges are they facing and how are they tackling these challenges? What technologies do they use? What technologies are still hype? Attendees will leave armed with the ability to tell fact from fiction, and understand how companies that are successfully using data to gain advantage are doing it.
Citation preview
State of the Data UniverseWhat the best are doing and why your data is your
future
Kenny GormanChief Technologist; Data at Rackspace
Co-Founder, ObjectRocket
@rackspace @kennygorman
Big DataWTF?
Big DataIt’s just data. It’s really important, and there
is probably lots of it, so maybe we call it:
‘really important data’
Focus on using data to be competitive
Forget how big it is or isn’t
Data is your competitive advantage
We're entering a new world in which data may be more important than software.
- Tim O'Reilly
The changing landscape
Today
DSS
BI
Big Data ????Analytics
????NoSQL
RDBMS NewSQL
● Operational Stores(PostgreSQL,MongoDB)
● Big Data (Hadoop)
● Streaming (Kafka)
Level Set
Value of the data?
1. Business Intelligence
2. Product Improvement
3. Operationalization
System Types
● Beyond traditional BI
● Capture important data, instrument everything important
● Pick your systems wisely, the right tool for the job
● Build automation & product based on data.
● Augment and extend existing systems.
Succeeding
Examples
Rush Hour Rewards is a service that helps you earn money back from your energy company by using less energy when everyone else is using more.
Using data to be more efficient with finite product availability, optimizing overall cost of goods, and thus increasing profitability
Nest
How it works(I think)
Thermostat -> API -> DB <- API <- Austin Energy
Austin Energy -> API -> Thermostat
{ "hvac_ac_state": "False",
"heat_pump_comp_threshold": "-31.5", "fan_cooling_enabled": "True", "leaf_away_high": "26.111",
"compressor_lockout_timeout": "0", "gear_threshold_low": "0.0",
"lower_safety_temp_enabled": "True", "postal_code": "78730", "learning_mode": "True", "country_code": "US",
"heat_x3_source": "lp", "fan_timer_duration": "900",
"backplate_serial_number": "02BA03AC031406Y4", "hvac_wires": "Heat,Cool,Fan,Rc,Star",
"humidifier_type": "unknown", "target_change_pending": "False",
"sunlight_correction_active": "False", "logging_priority": "informational",
"temperature_lock": "False", "dual_fuel_breakpoint_override": "none",
"has_x3_heat": "False", "alt_heat_x2_delivery": "forced-air",
"maint_band_lower": "0.56", "auto_away_learning": "ready",
"device_locale": "en_US", "learning_time": "1794",
"timestamp": "1405831101750", "time_to_target_training": "ready",
"has_fan": "True", "auto_dehum_state": "False",
"star_type": "unknown", "backplate_model": "Backplate-2.8",
"heat_x2_source": "lp", "aux_heat_source": "electric", "filter_changed_date": "0",
"equipment_type": "electric", "dehumidifier_orientation_selected": "unknown",
"forced_air": "True", "name": "Master",
"can_cool": "True", "aux_lockout_leaf": "10.0",
"filter_reminder_level": "0", "humidifier_state": "False",
"error_code": "", "leaf_threshold_cool": "23.888",
"has_x2_cool": "False", "hvac_pins": "W1,Y1,Rc,G,Star",
"creation_time": "1400446808068", "heat_pump_comp_threshold_enabled": "False",
"pin_star_description": "none", "compressor_lockout_enabled": "False", "learning_days_completed_cool": "60",
"away_temperature_low_enabled": "True", "note_codes": "[]",
"leaf_threshold_heat": "1000.0", "y2_type": "unknown",
"cooling_source": "electric", "leaf": "False",
"auto_dehum_enabled": "False", "alt_heat_x2_source": "gas",
"hvac_aux_heater_state": "False", "learning_days_completed_heat": "2",
"has_humidifier": "False", "gear_threshold_high": "0.0",
"current_schedule_mode": "COOL", "target_temperature_type": "cool",
"backplate_bsl_info": "BSL", "version": "-978861903",
"fan_cooling_readiness": "ready", "battery_level": "3.864",
"temperature_lock_high_temp": "22.222", "humidity_control_lockout_end_time": "0",
"target_temperature_high": "24.0", "ob_persistence": "True",
"hvac_safety_shutoff_active": "False", "schedule_learning_reset": "False",
"pin_y2_description": "none", "is_on_stand": "False",
"emer_heat_source": "electric", "filter_reminder_enabled": "False", "compressor_lockout_leaf": "-17.8", "aux_heat_delivery": "forced-air", "away_temperature_high": "24.444",
"fan_duty_start_time": "0", "learning_days_completed_range": "0", "target_humidity_enabled": "False",
"switch_system_off": "False", "sunlight_correction_ready": "True",
"sunlight_correction_enabled": "True", "has_emer_heat": "False",
"safety_temp_activating_hvac": "False", "has_dual_fuel": "False",
"heatpump_setback_active": "False", "has_heat_pump": "False",
"fan_control_state": "False", "model_version": "Display-2.8",
"has_aux_heat": "False", "current_version": "4.2.4",
"away_temperature_high_enabled": "False", "can_heat": "True",
"alt_heat_delivery": "forced-air", "current_humidity": "41", "target_humidity": "35.0",
"upper_safety_temp": "35.0", "heater_delivery": "forced-air",
"where_id": "00000000-0000-0000-0000-000100000006", "backplate_mono_version": "4.0.21",
"has_fossil_fuel": "True", "mac_address": "18b4302ae97f",
"serial_number": "02AA01AC041402HT", "type": "TBD",
"lower_safety_temp": "4.444", "hvac_heater_state": "False",
"humidity_control_lockout_start_time": "0", "fan_mode": "auto",
"filter_changed_set_date": "0", "range_enable": "True",
"heatpump_savings": "off", "radiant_control_enabled": "False", "temperature_lock_low_temp": "20.0",
"pin_ob_description": "none", "auto_away_reset": "False",
"humidity_control_lockout_enabled": "False", "fan_duty_cycle": "3600", "heatpump_ready": "False",
"preconditioning_enabled": "False", "hvac_fan_state": "True",
"preconditioning_ready": "True", "target_time_confidence": "0.0",
"local_ip": "192.168.1.144", "pin_w1_description": "heat",
"current_temperature": "22.43", "has_air_filter": "True",
"cooling_x2_source": "electric", "hvac_alt_heat_state": "False",
"heat_pump_aux_threshold": "10.0", "rssi": "58.0",
"fan_timer_timeout": "0", "has_alt_heat": "False",
"leaf_schedule_delta": "1.11", "backplate_bsl_version": "2.1", "user_brightness": "medium",
"preconditioning_active": "False", "pin_w2aux_description": "none", "pin_rc_description": "power", "has_dehumidifier": "False", "maint_band_upper": "0.56",
"target_temperature": "22.778", "leaf_learning": "ready",
"emer_heat_delivery": "forced-air", "pin_y1_description": "cool", "capability_level": "4.0",
"pin_rh_description": "none", "available_locales": "en_US,fr_CA,es_US,en_GB",
"dehumidifier_state": "False", "hvac_emer_heat_state": "False", "dehumidifier_type": "unknown",
"nlclient_state": "", "hvac_heat_x2_state": "False",
"upper_safety_temp_enabled": "False", "learning_state": "slow",
"hvac_heat_x3_state": "False", "hvac_cool_x2_state": "False", "fan_cooling_state": "True", "fan_duty_end_time": "0",
"auto_away": "0", "alt_heat_source": "gas",
"heat_link_connection": "0", "temperature_lock_pin_hash": "", "cooling_delivery": "unknown",
"heat_pump_aux_threshold_enabled": "True", "leaf_away_low": "16.67",
"heat_x3_delivery": "forced-air", "ob_orientation": "O", "touched_by": "{}",
"temperature_scale": "F", "emer_heat_enable": "False",
"backplate_mono_info": "TFE (BP_D2) 4.0.21 (root@bamboo) 2014-05-02 16:54:17", "auto_away_enable": "True", "pin_g_description": "fan",
"click_sound": "on", "hvac_alt_heat_x2_state": "False", "target_temperature_low": "20.0",
"has_x2_heat": "False", "away_temperature_low": "10.0",
"time_to_target": "0", "heat_x2_delivery": "forced-air", "cooling_x2_delivery": "unknown", "dual_fuel_breakpoint": "-1.0",
"_id": "ObjectId(53cb494e2239c261ac83dfe0)", "heater_source": "lp",
"pin_c_description": "none", "has_x2_alt_heat": "False"
}
Results
Why this is the future● It’s may be big data, but maybe not.
● Multiple entities, communicating via API
● Multiple layers of data analytics
● Very competitive
Rackspace Cloud Insights
How do I cut down the number of false positives in my monitoring solution?
Rackspace Cloud Insights● Pattern & anomaly detection algorithms
● Variation of Bollinger Bands algorithm
● First order differencing
● Belief network & Vector Similarity
Rackspace Cloud Insights
Rackspace Cloud Insights
metrics store -> analysis componentry -> alerting engine
https://developer.rackspace.com/blog/rackspace-cloud-intelligence-insights-in-monitoring/
Why this is the future● Answers a very simple, but powerful use
case
● Heterogeneous solution
● Very competitive
● Your brilliant data scientist spends all his/her time wrangling data not producing insights
● Technology choices gone wrong
● Capture lots of data, but never analyze it
● Waiting too long to start capturing data
Data Anti-Patterns
Data is a precious thing and will last longer than the systems themselves.
- Tim Berners-Lee
