View
230
Download
4
Category
Preview:
Citation preview
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
NIKITAIVANOVGridGainFounder&CTO
ApacheIgnitePMC
Apache2.0-TowardsConvergedDataPla>ormFastDataMeetsOpenSource
hIp://ignite.apache.org @apacheignite
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
Agenda
• FastDatavsBigData– In-MemoryDatabases– In-MemoryDataGrids– Hadoop&Spark
• ConvergedDataPla>orm• BigData+FastData
• WhatisApacheIgnite– BigBankUseCase– In-MemoryDataFabric– SharedMemoryLayer• ShareSparkRDDs• In-MemoryFileSystem
• Q&A
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• VeryAc<veCommunity• GreatWaytoLearnDistributedCompu<ng• HowToContribute:
– hIps://ignite.apache.org/community/
contribute.html#contribute
– hIps://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
ApacheIgnite:JoinUs!
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• BigData– OLAPmostly– LargerHistoricalDataSet– Read-Mostly– ThroughputNotImportant– LowQueryLatencies– Good-enoughforinterac<ve
analy<cs
FastDatavsBigData
• FastData– OLTPmostly– SmallerOpera<onalDataSet– HighThroughput(ops/sec)– LowLatencies– ConsistentandTransac<onal
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• BigData– Hadoop• MapReduce• HDFS• HBase
– Spark• MachineLearning• GraphProcessing• SQL
– Warehouse/DBVendors
FastDatavsBigData
• FastData– Streaming• Flink• Ka^a• Apex
– In-MemoryDataGrid• Ignite• Geode(incuba<ng)
– In-MemoryDatabase• MemSQL• VoltDB
– NoSQL• MongoDB• Cassandra
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• In-MemoryDatabases– MemSQL• ClosedSource• FreeLimitedCommunityEdi<on
– VoltDB• OpenSourceCommunityEdi<on(AGPL)• ClosedSourceEnterpriseEdi<on
• MainFeatures– High-Throughput– LowLatencies– FullSQLSupport• However,SQListheonlyAPI
– DiskPersistence• Diskisjustacopyofmemory
– Completereplacementofexis<ngdatabases
FastData:In-MemoryDatabases
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• In-MemoryDataGrids– ApacheIgnite–In-MemoryDataFabric– ApacheGeode(incuba<ng)– Hazelcast
• MainFeatures– Highthroughput– Lowlatencies– Key-valuestore– Transac<ons– Extensivedataqueryingcapability– Diskpersistence• Read&write-throughtodatabases• Keepyourexis<ngdatabase
FastData:In-MemoryDataGrids
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• ApacheHadoop&ApacheSpark– BigFamilyofProducts– BatchProcessing– In-MemoryProcessing(Spark)
• MainFeatures– Disk-basedstorage– Interac<veAnaly<cs– NoTransac*ons– Read-OnlyDataSets– StrongQueryingCapabili<es– Rela<velyLowLatencies• Goodenoughforhumaneye
BigDataEcosystem
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• BigData+ AddSharedMemoryStore+ AddTransac<ons
HowToBridgeTheGap?
• FastData+ AddDisk-FirstDataSets+ AddDisk-FirstProcessing
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• FastData+BigData• DistributedandScalable• RealTimeData-To-Ac<on• HybridTransac<onalandAnaly<calProcessing(HTAP)– FastDatainMemory– BigDataonDisk– CombineRAM,NAND,HDD– NoETL– Queryhistoricalandanaly<caldata– Transac<onsonhistoricalandanaly<caldata
WhatisaConvergedDataPla>orm?
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
ApacheIgniteTMIn-MemoryDataFabric:StrategicApproachtoIMC
• Supports Applications of various types and languages
• Open Source – Apache 2.0 • Simple Java APIs • 1 JAR Dependency • High Performance & Scale • Automatic Fault Tolerance • Management/Monitoring • Runs on Commodity Hardware
• Supports existing & new data sources • No need to rip & replace
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
ApacheIgniteIn-MemoryDataFabric
©2014GridGainSystems,Inc.
UseCase:Largest bank in Russia and Eastern Europe, and the third largest in Europe
• SberbankRequirements– Migratetodatagridarchitecture– MinimizedependencyonOracle– Movetoopensource
• WhyApacheIgnite– MorethanaDataGrid– Bestperformance
• 10+compe<torsevaluated– Demonstratedbest
• Faulttolerance&scalability• ANSI-99SQLSupport• Transac<onalconsistency
• JointlyDeveloping• Disk-OnlyDataSets• QueryDisk&MemoryTogether
130
Mill
ion
Cust
omer
s
Deposit Withdrawal Statement
Disk Store
Disk Store
Disk Store
1000+ Servers
GridGain Security
Deposit Withdrawal Statement
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• BasedonJCache(JSR107)– In-MemoryKey-ValueStore– BasicCacheOpera<ons– ConcurrentMapAPIs– CollocatedProcessing(EntryProcessor)– EventsandMetrics– PluggablePersistence
• IgniteDataGrid– ACIDTransac<ons– SQLQueries(ANSI99)– In-MemoryIndexes– On-Heap&Off-HeapMemory– Automa<cRDBMSIntegra<on
ApacheIgniteDataGrid
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
DataGrid:DistributedCaching
Par<<onedCache ReplicatedCache
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• ANSI-99SQL• AlwaysConsistent• FaultTolerant• In-MemoryIndexes(On-HeapandOff-Heap)• Automa<cGroupBy,Aggrega<ons,Sor<ng• Cross-CacheJoins,Unions,etc.• Ad-HocSQLSupport
DataGrid:Ad-HocSQL(ANSI99)
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
SQLCross-CacheGROUPBYExample
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• IgniteRDDDeploymentModes– ShareRDDacrosstasksonthehost– ShareRDDacrosstasksintheapplica<on– ShareRDDglobally– EmbeddedvsExternalDeployments
• FasterSQL– In-MemoryIndexes– SQLontopofSharedRDD
ShareRDDsAcrossSparkJobs
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• IgniteIn-MemoryFileSystem(IGFS)– Hadoop-compliant– EasytoInstall– On-HeapandOff-Heap– CachingLayerforHDFS– Write-throughandRead-throughHDFS– PerformanceBoost
IgniteIn-MemoryFileSystem
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
IgniteIn-MemoryMapReduce• In-MemoryNa<ve
Performance• ZeroCodeChange• Useexis<ngMRcode• Useexis<ngHivequeries• NoNameNode• NoNetworkNoise• In-ProcessDataColoca<on• EagerPushScheduling
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
• MoreSQL– Non-collocatedJoins– 100%DataModifica<onLanguage(DML)– 100%DataDefini<onLanguage(DDL)
• MoreDisk– ATMM-AdvancedTiered-MemoryModel:• Disk-firstdatasets• AnyDRAM/NAND/HDDmix
– SeamlessqueryingacrossATMM
ProposedApacheIgnite2.0RoadmapConvergedDataPla>orm
Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.
ANYQUESTIONS?Thankyouforjoiningus.Followtheconversa<on.
hIp://www.ignite.apache.org
@apacheignite
Recommended