Upload
everett-hines
View
219
Download
0
Embed Size (px)
Citation preview
Highly available web sites with Tomcat and Clustered JDBC
Emmanuel Cecchet
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 2 - 15/11/2004
Internet
Database tier should be scalable highly available without modifying the client application database vendor independent on commodity hardware
Motivations
JDBC
JDBCJDBC
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 3 - 15/11/2004
Scaling the database tier – Alternative 1 (master-slave)
Internet
Web frontend
App. server
Consfailover time on master failurescalability
Master
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 4 - 15/11/2004
Internet
Consatomic broadcast scalabilityno client side load balancingheavy modifications of the database engine
Scaling the database tier – Alternative 2 (atomic broadcast)
Atomicbroadcast
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 5 - 15/11/2004
Scaling the database tier – Alternative 3 (SMP)
Internet
Web frontend
App. server
Well-known database
vendor here
Database
Well-known hardware +database vendors here
Cons Cost Scalability limit
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 6 - 15/11/2004
Scaling the database tier – Alternative 4 (shared disks)
Internet
Web frontend
App. server Database Disks
Another well-knowndatabase vendor
Consstill expensive hardwareavailability
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 7 - 15/11/2004
Outline
RAIDbC-JDBCTomcat and C-JDBCScalabilityHigh availability
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 8 - 15/11/2004
RAIDb concept
Redundant Array of Inexpensive DatabasesRAIDb controller
gives the view of a single database to the clientbalance the load on the database backends
RAIDb levels offers various tradeoff of performance and fault tolerance
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 9 - 15/11/2004
RAIDb levels
RAIDb-0partitioningno duplication and no fault toleranceat least 2 nodes
table 2 & 3 table ...
RAIDb controller
table n-1table 1 table n
SQL requests
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 10 - 15/11/2004
RAIDb levels
RAIDb-1mirroringperformance bounded by write broadcastat least 2 nodes
Full DB Full DB
RAIDb controller
Full DBFull DB Full DB
SQL requests
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 11 - 15/11/2004
RAIDb levels
RAIDb-2partial replicationat least 2 copies of each table for fault toleranceat least 3 nodes
table x table y
RAIDb controller
table x & yFull DB table z
SQL requests
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 13 - 15/11/2004
Outline
RAIDbC-JDBCTomcat and C-JDBCScalabilityHigh availability
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 14 - 15/11/2004
C-JDBC overview
Middleware implementing RAIDb100% Java implementationopen source (LGPL)
Two componentsgeneric JDBC driver (C-JDBC driver)C-JDBC Controller
Read-one, Write all approachprovides eager (strong) consistency
Supports heterogeneous databases
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 15 - 15/11/2004
architectural overview
JVM
Applicationserver
C-JDBC JDBC driver
C-JDBCcontroller
JVM
MySQL JDBC driver
MySQLdatabase
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 16 - 15/11/2004
Inside the C-JDBC Controller
XML configuration file
C-JDBC Controller
MySQL
C-JDBC driver
Java client program
(Servlet, EJB, ...)
XML engine
MySQL
Virtual database
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Request Cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Configuration &
administrationAdministration console
RMIRMI
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
Virtual database
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Request Cache
Scheduler
Load balancer
Recovery Log
Authentication Manager
Oracle
Oracle JDBC driver
Oracle JDBC driver
Oracle
SocketsSockets
JMX
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 17 - 15/11/2004
Virtual Database
MySQL
C-JDBC driver
Java client program
(Servlet, EJB, ...)
MySQL
Virtual database
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Request Cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
gives the view of a single database
establishes the mapping between the database name used by the application and the backend specific settings
backends can be added and removed dynamically
configured using an XML configuration file
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 18 - 15/11/2004
Authentication Manager
MySQL
C-JDBC driver
Java client program
(Servlet, EJB, ...)
MySQL
Virtual database
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Request Cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
Matches real login/password used by the application with backend specific login/ password
Administrator login to manage the virtual database
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 19 - 15/11/2004
Scheduler
MySQL
C-JDBC driver
Java client program
(Servlet, EJB, ...)
MySQL
Virtual database
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Request Cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
Manages concurrency controlSpecific implementations for
RAIDb 0, 1 and 2Pass-throughOptimistic and pessimistic
transaction leveluses the database schema that is
automatically fetched from backends
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 20 - 15/11/2004
Request cache
MySQL
C-JDBC driver
Java client program
(Servlet, EJB, ...)
MySQL
Virtual database
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Request Cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
3 optional cachestunable sizes
parsing cacheparse request skeleton only onceINSERT INTO t VALUES (?,?,?)
metadata cachecolumn metadatafields of a request
result cachecaches results from SQL requeststunable consistencyfine grain invalidationsoptimizations for findByPk requests
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 21 - 15/11/2004
Load balancer 1/2
MySQL
C-JDBC driver
Java client program
(Servlet, EJB, ...)
MySQL
Virtual database
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Request Cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
RAIDb-0query directed to the backend having the
needed tables
RAIDb-1read executed by current threadwrite executed in parallel by a dedicated
thread per backendresult returned if one, majority or all commitif one node fails but others succeed, failing
node is disabled
RAIDb-2same as RAIDb-1 except that writes are
sent only to nodes owning the updated table
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 22 - 15/11/2004
Load balancer 2/2
MySQL
C-JDBC driver
Java client program
(Servlet, EJB, ...)
MySQL
Virtual database
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Request Cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
Static load balancing policiesRound-Robin (RR)Weighted Round-Robin (WRR)
Least Pending Requests First (LPRF)
request sent to the node that has the shortest pending request queue
efficient even if backends do not have homogeneous performance
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 23 - 15/11/2004
Connection Manager
MySQL
C-JDBC driver
Java client program
(Servlet, EJB, ...)
MySQL
Virtual database
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Request Cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
C-JDBC driver provides transparent connection pooling
Connection pooling for a backend
no poolingblocking poolnon-blocking pooldynamic pool
Connection pools defined on a per login basis
resource management per logindedicated connections for admin
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 24 - 15/11/2004
Recovery Log
MySQL
C-JDBC driver
Java client program
(Servlet, EJB, ...)
MySQL
Virtual database
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Request Cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
Checkpoints are associated with database dumps
Record all updates and transaction markers since a checkpoint
Used to resynchronize a database from a checkpoint
JDBCRecoveryLogstore log information in a databasecan be re-injected in a C-JDBC
cluster for fault tolerance
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 25 - 15/11/2004
C-JDBC Controller
MySQL
C-JDBC driver
MySQL
Virtual database 1
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Query result cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
C-JDBC driver
Client application (Servlet, EJB, ...)
Client application (Servlet, EJB, ...)
Functional overview (read)
connect myDBconnect login, passwordexecute SELECT * FROM t
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 26 - 15/11/2004
C-JDBC Controller
MySQL
C-JDBC driver
MySQL
Virtual database 1
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Query result cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
C-JDBC driver
Client application (Servlet, EJB, ...)
Client application (Servlet, EJB, ...)
Functional overview (write)
execute INSERT INTO t …
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 27 - 15/11/2004
C-JDBC Controller
MySQL
C-JDBC driver
MySQL
Virtual database 1
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Query result cache
Scheduler
Load balancer
MySQL JDBC driver
MySQL JDBC driver
Recovery Log
Authentication Manager
MySQL
Database Backend
Connection Manager
MySQL JDBC driver
C-JDBC driver
Client application (Servlet, EJB, ...)
Client application (Servlet, EJB, ...)
Failures
No 2 phase-commitparallel
transactionsfailed nodes are
automatically disabled
execute INSERT INTO t …
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 28 - 15/11/2004
Outline
RAIDbC-JDBCTomcat and C-JDBCScalabilityHigh availability
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 29 - 15/11/2004
Internet
Apache clustering L4 switch, RR-DNS, One-IP techniques, LVS, …
Tomcat clustering mod_jk (T4), mod_proxy/mod_rewrite (T5), session replication
Database clustering C-JDBC
Highly available web site
mod-jkRR-DNS
Parsing cacheResult cache
Metadata cache
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 30 - 15/11/2004
Result cache
Cache contains a list of SQL→ResultSetPolicy defined by queryPattern→Policy3 policies
EagerCaching: variable granularities for invalidationsRelaxedCaching: invalidations based on timeoutNoCaching: never cached
RUBiS bidding mixwith 450 clients
No cache Coherentcache
Relaxedcache
Throughput (rq/min) 3892 4184 4215
Avg response time 801 ms 284 ms 134 ms
Database CPU load 100% 85% 20%
C-JDBC CPU load - 15% 7%
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 31 - 15/11/2004
Internet
Multiple databaseschoosing RAIDb levelrecovery log for
adding nodes dynamically recovering from failures
Highly available web site
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 32 - 15/11/2004
Configuring Tomcat 5
Same setting for all Tomcat nodes
Copy c-jdbc-driver.jar to $CATALINA_HOME/common/lib
Without connection pooling1. load driver
Class.forName("org.objectweb.cjdbc.driver.Driver").newInstance();
2. get a connectionConnection conn = DriverManager.
getConnection("jdbc:cjdbc://host/myDB","user","password");
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 33 - 15/11/2004
Configuring Tomcat 5
With connection pooling
<ResourceParams name="jdbc/TestDB"> <parameter> <name>factory</name> <value>org.apache.commons.dbcp.BasicDataSourceFactory</value> </parameter>
<!-- Class name for the db driver --> <parameter> <name>driverClassName</name> <value>org.objectweb.cjdbc.driver.Driver</value> </parameter> <!-- The JDBC connection url for connecting to your db. --> <parameter> <name>url</name> <value>jdbc:cjdbc://localhost/myDB</value> </parameter>
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 34 - 15/11/2004
Configuring C-JDBC (1/3)
Virtual database configuration file
<?xml version="1.0" encoding="UTF8"?><!DOCTYPE C-JDBC PUBLIC "-//ObjectWeb//DTD C-JDBC 1.0.5//EN"
"http://c-jdbc.objectweb.org/dtds/c-jdbc-1.0.5.dtd">
<VirtualDatabase name=“myDB"> <AuthenticationManager> <Admin…> <VirtualUsers> <VirtualLogin
vLogin="user“ vPassword=“password"/> </VirtualUsers> </AuthenticationManager>
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 35 - 15/11/2004
Configuring C-JDBC (2/3)
<DatabaseBackend name="node1“ driver="org.gjt.mm.mysql.Driver“ url="jdbc:mysql://host1/backend1“ connectionTestStatement="select 1">
<ConnectionManager vLogin=“user" rLogin=“dbuser" rPassword=“dbpass">
<VariablePoolConnectionManager initPoolSize=“10" minPoolSize=“5" maxPoolSize="50“ idleTimeout="30" waitTimeout="10"/>
</ConnectionManager> </DatabaseBackend>
<DatabaseBackend name="node2" driver="org.gjt.mm.mysql.Driver" url="jdbc:mysql://host2/backend2" connectionTestStatement="select 1">
<ConnectionManager …> </DatabaseBackend>
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 36 - 15/11/2004
Configuring C-JDBC (3/3) <RequestManager> <RequestScheduler> <RAIDb-1Scheduler level="optimisticTransaction"/> </RequestScheduler>
<RequestCache> <MetadataCache/> <ParsingCache/>
<ResultCache granularity="table" /> </RequestCache>
<LoadBalancer> <RAIDb-1> <RAIDb-1-LeastPendingRequestFirst/> </RAIDb-1> </LoadBalancer>
<RecoveryLog> <JDBCRecoveryLog
driver="org.gjt.mm.mysql.Driver" url="jdbc:mysql://host/recovery" login="user" password="">
</JDBCRecoveryLog> </RecoveryLog>
</RequestManager> </VirtualDatabase>
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 37 - 15/11/2004
Internet
<VirtualDatabase name="myDB"><Distribution> <BackendRecoveryPolicy
backendName=“node1" recoveryPolicy="on"/> <ControllerName name=“controller2”/>
<BackendRecoveryPolicy/></Distribution>…
Controller replication
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 38 - 15/11/2004
Outline
RAIDbC-JDBCTomcat and C-JDBCScalabilityHigh availability
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 39 - 15/11/2004
C-JDBC vertical scalability
allows nested RAIDb levelsallows tree architecture for scalable write
broadcastnecessary with large number of backendsC-JDBC driver re-injected in C-JDBC controller
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 41 - 15/11/2004
DB 4DB 3
DB native JDBC driver
DB 5
C-JDBC driver
DB 1 DB 2
DB native JDBC driver
DB 6
DB native JDBC driver
DB7
C-JDBC controller Partial replication
C-JDBC controller Full replication
C-JDBC controller Full replication
C-JDBC controller Full replication
C-JDBC driverJVM
Client program C-JDBC
driver
JVM
Client program
C-JDBC driver
JVM
Client program
Vertical scalability
Addresses JVM scalability issues
Distributing large number of connections on many backends
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 42 - 15/11/2004
TPC-W benchmark(Amazon.com)
Nearly linear speedups with the shopping mix
0
200
400
600
800
1000
1200
1400
1600
0 1 2 3 4 5 6
Number of nodes
Th
rou
gh
pu
t in
req
ues
ts p
er m
inu
te
Single Database
Full replication
Partial replication
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 43 - 15/11/2004
Outline
RAIDbC-JDBCTomcat and C-JDBCScalabilityHigh availability
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 44 - 15/11/2004
Controller replication
Prevent the controller from being a single point of failure
Group communication for controller synchronization
C-JDBC driver supports multiple controllers with automatic failover
MySQL PostgreSQL
C-JDBC Controller
MySQL JDBC driver
PostgreSQL JDBC driver
C-JDBC driver
JVM
Java client program
C-JDBC driver
JVM
Java client program
C-JDBC driver
JVM
Java client program
C-JDBC driver
JVM
Java client program
C-JDBC Controller
JGroups
MySQLPostgreSQL
MySQL JDBC driver
PostgreSQL JDBC driver
jdbc:c-jdbc://node1:25322,node2:12345/myDB
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 45 - 15/11/2004
Controller replication
C-JDBC Controller
Distributed Request Manager
C-JDBC Controller
Distributed Request Manager
MySQL
C-JDBC driver
MySQL
Virtual database 1
Database Backend
Connection Manager
Database Backend
Connection Manager
MySQL JDBC driver
MySQL JDBC driver
Virtual database 2
Database Backend
Connection Manager
Database Backend
Connection Manager
Request Manager
Query result cache
Scheduler
Load balancer
Authentication Manager
Oracle JDBC driver
OracleMySQL
MySQL JDBC driver
C-JDBC driver
Client application (Servlet, EJB, ...)
C-JDBC driver
Client application (Servlet, EJB, ...)
Client application (Servlet, EJB, ...)
Request Manager
Query result cache
Scheduler
Load balancer
Authentication Manager
jdbc:cjdbc://node1,node2/myDB
Total order reliable multicast
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 46 - 15/11/2004
Mixing horizontal & vertical scalability
DB 6DB 5
DB native JDBC driver
DB 7
C-JDBC driver
DB 1 DB 2
DB native JDBC driver
DB 3
DB native JDBC driver
DB 4
C-JDBC controller Full replication
C-JDBC controller Full replication
C-JDBC controller Full replication
C-JDBC controller Full replication
C-JDBC driverJVM
Client program
C-JDBC driver
JVM
Client program
C-JDBC driver
JVM
Client program
C-JDBC driver
C-JDBC driver
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 47 - 15/11/2004
Building initial checkpoint
Octopus is an ETL tool Use Octopus to store a dump of
the initial database state Currently done by the user using
the database specific dump tool
PostgreSQL
C-JDBC Controller
PostgreSQL JDBC driver
C-JDBC driver
JVM
EJB Container JOnAS, WebLogic,
JBoss, WebSphere, ...
disabled
dump for initial
checkpoint
Recovery LogOctopus
disabled
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 48 - 15/11/2004
Logging
Backend is enabled All database updates are logged
(SQL statement, user, transaction, …)
PostgreSQL
C-JDBC Controller
PostgreSQL JDBC driver
C-JDBC driver
JVM
EJB Container JOnAS, WebLogic,
JBoss, WebSphere, ...
enabled
dump for initial
checkpoint
Recovery LogOctopus
JDBC Recovery Log
enabled
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 49 - 15/11/2004
Adding new backends 1/3
Add new backends while system online
Restore dump corresponding to initial checkpoint with Octopus
PostgreSQL
C-JDBC Controller
PostgreSQL JDBC driver
C-JDBC driver
JVM
EJB Container JOnAS, WebLogic,
JBoss, WebSphere, ...
dump for initial
checkpoint
Recovery Log
Octopus
PostgreSQL PostgreSQLdisabled disabledenabled
enabled
JDBC Recovery Log
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 50 - 15/11/2004
Adding new backends 2/3
Replay updates from the log
PostgreSQL
C-JDBC Controller
PostgreSQL JDBC driver
C-JDBC driver
JVM
EJB Container JOnAS, WebLogic,
JBoss, WebSphere, ...
dump for initial
checkpoint
Recovery Log
Octopus
PostgreSQL PostgreSQLenabled
enabled
JDBC Recovery Log
disableddisabled
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 51 - 15/11/2004
Adding new backends 3/3
Enable backends when done
PostgreSQL
C-JDBC Controller
PostgreSQL JDBC driver
C-JDBC driver
JVM
EJB Container JOnAS, WebLogic,
JBoss, WebSphere, ...
dump for initial
checkpoint
Recovery Log
Octopus
PostgreSQL PostgreSQLenabled
enabled
JDBC Recovery Log
enabledenabled
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 52 - 15/11/2004
Making new checkpoints (1/3)
Disable one backend to have a coherent snapshot
Mark the new checkpoint entry in the log
Use Octopus to store the dump
...
dump for last
checkpoint
PostgreSQL
C-JDBC Controller
PostgreSQL JDBC driver
C-JDBC driver
JVM
EJB Container JOnAS, WebLogic,
JBoss, WebSphere, ...
dump for last
checkpoint
Recovery Log
Octopus
PostgreSQL PostgreSQLenabled
enabled
JDBC Recovery Log
enableddisabled
dump for initial
checkpoint
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 53 - 15/11/2004
Making new checkpoints (2/3)
Replay missing updates from log
dump for
checkpoint
Octopus...
dump for last
checkpoint
dump for last
checkpoint
dump for initial
checkpoint
PostgreSQL
C-JDBC Controller
PostgreSQL JDBC driver
C-JDBC driver
JVM
EJB Container JOnAS, WebLogic,
JBoss, WebSphere, ...
Recovery Log
PostgreSQL PostgreSQLenabled
enabled
JDBC Recovery Log
disabled enabled
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 54 - 15/11/2004
Making new checkpoints (3/3)
Re-enable backend when done
PostgreSQL
C-JDBC Controller
PostgreSQL JDBC driver
C-JDBC driver
JVM
EJB Container JOnAS, WebLogic,
JBoss, WebSphere, ...
dump for
checkpoint
Recovery Log
Octopus
PostgreSQL PostgreSQLenabled
enabled
JDBC Recovery Log
enabledenabled
...
dump for last
checkpoint
dump for last
checkpoint
dump for initial
checkpoint
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 59 - 15/11/2004
Fault tolerant recovery log
C-JDBC Controller
MySQL MySQL
Application virtual database
Database Backend
Database Backend
Request Manager
Request Cache
Scheduler
Load balancer
Recovery Log
Authentication Manager
MySQL
Database Backend
Recovery virtual database
Request Manager
SchedulerLoad balancer
Authentication Manager
C-JDBC driver
Client application (Servlet, EJB, ...)
Checkpointing service
Database Backend
Database Backend
C-JDBC driver
MySQL
UPDATE statement
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 60 - 15/11/2004
Demo – xPetstore/HSQL
backend1
HypersonicSQLin-memorydatabase
xPetstore Servlet
http://xpetstore.sourceforge.net/open source implementation of PetstoreServlet solution: Velocity, WebWork, Sitemesh, POJO
and Hibernate.
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 61 - 15/11/2004
Demo 2 – xPetstore/C-JDBC/HSQL
backend1
backend2
HypersonicSQLin-memorydatabases
JMX
Recovery log
RAIDb-1
xpetstore Servlet
C-JDBC administration console
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 62 - 15/11/2004
C-JDBC today
Web site~200.000 hits/month~ 26.000 downloadsEU (18 countries) 36%, US 28%, Japan 12%, China
5%, Canada 4%, Australia 4%, India 3%, Brazil 2%, …
Community27 committers both industrial & [email protected]: >200 subscribers, 200-300
msgs/monthtranslation in japanese, italian, chinese, turkish, french
RPM on JPackage.org
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 63 - 15/11/2004
What’s next?
Tribe (.objectweb.org)replacement for JGroupsuniform total order broadcast optimized for clusters
LeWYS (.objectweb.org)hardware and software monitoringmonitoring repository
Distributed query executionX509 certificatesOptimized support for edge-side servers and
interconnected clusters
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 64 - 15/11/2004
Conclusion
RAIDbRAID-like scheme for databases
C-JDBCopen source middleware for database replicationperformance scalabilityhigh availability
Tomcat & C-JDBCno application modification requiredRDBMS vendor independent
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 65 - 15/11/2004
Q&A_________
Thanks to all users and contributors ...
http://c-jdbc.objectweb.org
Bonus slides
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 67 - 15/11/2004
Current limitations
JDBC onlyDistributed joinsUpdatable ResultSetsXA support through XAPool onlytransparent controller failover not supported
when using horizontal scalability with JGroupsnetwork partition/reconciliation not supported
HORIZONTAL SCALABILITY
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 69 - 15/11/2004
Horizontal scalability
JGroups for controller synchronizationGroups messages for writes only
DB1 DB2 DB3 DB4
Controller1
DB1 DB2 DB3 DB4
Controller1 Controller2
Client 1 Client... Client n Client 1 Client... Client n
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 70 - 15/11/2004
Horizontal scalability
Centralized write approach issues
Issues with transactions assigned to connections
DB1 DB2 DB3 DB4
Controller1 Controller2
commit
DB1 DB2 DB3 DB4
Controller1 Controller2
commit
commit commit
DB1 DB2 DB3 DB4
Controller2
? ? ? ?
DB1 DB2
Controller1 Controller2
DB1 DB2
Controller1 Controller2
T1(Read)
T1(R)
DB1 DB2
Controller1 Controller2
T1(Write)
T1(R)T1(W) T2(W)
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 71 - 15/11/2004
Horizontal scalability
General case for a write query3 multicast + 2n unicast
DistributedVirtualDatabase@controller2
DistributedRequestManager
DistributedVirtualDatabase@controller3
DistributedVirtualDatabase@controller1
DistributedRequestManager
DistributedRequestManager
execWriteRequest(AbstractWriteRequest)
Scheduler
Write query handler
Load Balancer
Scheduler
Load Balancer
Distributed scheduling handler
Wait for schedulers replies
Send request to all controllers
Send execute request order to all controllers
Wait for load balancer replies
Distributed load balancer handler
Scheduler
Load Balancer
Distributed scheduling handler
Distributed load balancer handler
Notify controllers of completion success or not
Distributed scheduling handler
Distributed load balancer handler
Notify scheduler of completion
Return result
return result
Notify scheduler of completion
Notify scheduler of completion
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 72 - 15/11/2004
Horizontal scalability
Solution: No backend sharing1 multicast + n unicast [+ 1 multicast]
DB1 DB2 DB3 DB4 DB5 DB6
Controller1 Controller2 Controller3
DistributedVirtualDatabase@controller2
DistributedVirtualDatabase@controller3
DistributedVirtualDatabase@controller1
DistributedRequestManager
DistributedRequestManager
execWriteRequest(AbstractWriteRequest)DistributedRequestManager
Scheduler
Write query handler
Load Balancer
Scheduler
Load Balancer
Distributed scheduling handler
Send request to all controllers
Wait for load balancer replies
Distributed load balancer handler
Scheduler
Load Balancer
Distributed scheduling handler
Distributed load balancer handler
Notify controller(s) that have failed if other have succeeded
Distributed scheduling handler
Distributed load balancer handler
Wait if all backends failed
Return result
return result
Wait if all backends failed
Wait if all backends failed
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 73 - 15/11/2004
Horizontal scalability
Issues with JGroupsresources needed by a channelinstability of throughput with UDPperformance scalability
TCP better than UDP butunable to disable reliability on top of TCPunable to disable garbage collectionordering implementation is sub-optimal
Need for a new group communication layer optimized for cluster
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 74 - 15/11/2004
Horizontal scalability
JGroups performance on UDP/FastEthernet
USE CASES
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 76 - 15/11/2004
Budget High Availability
High availability infrastructure “on a budget”
Typical eCommercesetup
http://www.budget-ha.com
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 77 - 15/11/2004
OpenUSS: University Support System
eLearningHigh availabilityPortability
Linux, HP-UX, Windows
InterBase, Firebird, PostgreSQL, HypersonicSQL
http://openuss.sourceforge.net
Firebird
C-JDBC Controller Full replication
Firebird JDBC driver
Firebird
C-JDBC driver
JVM
C-JDBC driver
JVM
C-JDBC Controller Full replication
Firebird JDBC driver
C-JDBC driver
JVM
JOnAS J2EE server
C-JDBC driver
JVM
Enhydra server
OpenUSS OpenUSS OpenUSS OpenUSS
Enhydra Director
Apache
JOnAS J2EE server
Enhydra server
JOnAS J2EE server
Enhydra server
JOnAS J2EE server
Enhydra server
Enhydra Director
Apache
Enhydra Director
Apache
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 78 - 15/11/2004
Flood alert system
Disaster recovery Independent nodes synchronized with C-JDBC VPN for security issues
http://floodalert.org
MySQL
C-JDBC Controller Full replication
MySQL JDBC driver
C-JDBC driver
JBoss J2EE server
Flood alert system
MySQL
C-JDBC Controller Full replication
MySQL JDBC driver
C-JDBC driver
JBoss J2EE server
Flood alert system
MySQL
C-JDBC Controller Full replication
MySQL JDBC driver
C-JDBC driver
JBoss J2EE server
Flood alert system
> 1000 miles
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 79 - 15/11/2004
J2EE benchmarking
Large scaleJ2EE clusters
http://jmob.objectweb.org
Internet
emulated users
PERFORMANCE
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 81 - 15/11/2004
TPC-W
Browsing mix performance
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 82 - 15/11/2004
TPC-W
Shopping mix performance
http://c-jdbc.objectweb.org/ - [email protected] ApacheCon’04 - 83 - 15/11/2004
TPC-W
Ordering mix performance