View
514
Download
0
Category
Tags:
Preview:
DESCRIPTION
Christian Brensing, Senior Developer, State of Bavaria The Bavarian government runs a document template application (RTF or ODF with Groovy, Python, Ruby or Tcl as scripting language) serving different government offices. Having complex and hierarchical data structures to organize the templates, MongoDB was selected to replace the Oracle-based persistence layer. In this talk you will hear about the improvements we have achieved with the migration to MongoDB, problems we had to solve underway and unit testing of the persistence layer in order to keep our quality level.
Citation preview
MongoDB@BayernMigration from RDBMS, Problems, Unit-Testing
Christian Brensing
Bayerisches Landesamt für Statistik und Datenverarbeitung
IuK / Rechenzentrum Süd
• Template-Processor (~ JSP)
• ODF and RTF
• PDF / PostScript postprocessing
• Groovy, Python, Ruby, Tcl
• In Production using RDBMS since 2008
• Central HR system (SAP) uses BayText to create output
Client for developing the templates (BayText-IDE)
ODF-Template using Ruby
Generated document
2 MillionGenerated documents per year
Tiny dataset50,000 records and 10 GB BLOB
a small system after all
So why migrate?
Because we can
developers.pop()Smaller teams
Easy to learnRead one book and you're done
Simple mapping
nodes
groups
templates
document_usages
folder_configurations
codes
permissions
requests
documents
folders
structures template_tags
roles
roles_roles
bundles
componentsdynafields
scripts
nodes
roles
properties
GridFS
{! "_id" : ObjectId(),! "_type" : "Document",! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Document",! "description" : "",! "language" : "ruby",! "format" : "odf",! "autoload_uplevel" : -1,! "requests" : [! {! "name" : "Test",! "description" : "foo", ! "xml" : zlib("<?xml ...?>")! }, ! ... ! ],! "structure" : [! { ! "alias" : "some alias",! "component_path" : "foo.bar.Document$Document",! "parent", -1 ! },! ... ! ], ! } ! "tags" : ["foo", "bar", "baz"]!}
public class RoleReadConverter implements Converter<DBObject, Role> {! @Override! public Role convert(DBObject source) {! Role role = new Role();! role.setName((String) source.get("name"));! ...! return role;! }!}!!!public class RoleWriteConverter implements Converter<Role, DBObject> {! @Override! public DBObject convert(Role source) {! return new BasicDBObject()! .append("name", role.getName())! .append(...);! }!}!!!// Convert a DBObject to an Entity!conversionService.convert(collection.findOne(...));
Easy mapping using Spring-ConversionService
Performanceup to 10x
ORM no moreThe MongoDB driver is all you need
Obstacles?
Mentality
Referential IntegrityMulti-Document-Update on denormalized schema
{! "_id" : ObjectId(),! "_type" : "Folder", ! "updated_at" : ISODate("2007-12-06T10:04:00.543Z"),! "updated_by" : "maggie",! "version" : 1,! "path" : "foo.bar.Folder",! "description" : "bla",! "bundles" : [! ... ! ]! "document_usages" : [! {! "name" : "foo",! "description" : "bla", ! "exec_order" : 5, ! "print_copies" : 2,! "type": "FAIR_COPY",! "document_path" : "a.b.Document",! "bundle" : "bar"! },! ...!}
Linking documents via a path instead of ObjectId
But paths are mutable!All references must be updated
Approach
• Pseudo-Transaction with an update locking mechanism in a custom oplog collection
• Periodical repair jobs finishing failed operations • Extended interpretation of eventual consistency
Unit-TestsContinuous Integration
F.I.R.S.T.
IsolatedTests are using a unique DB per JVM
@Configuration!@Profile("test")!public class TestSpringConfiguration extends SpringConfiguration {! @Override! public void init() {! setTestProperties();! ! super.init();! ! // Delete test db! db().dropDatabase();! }! ! private void setTestProperties() {! String dbname = env.getProperty("db.name", "baytext_test_" + getSimpleUsername());! String host = env.getProperty("db.host", "mongodb-dev.db.rz-sued.bybn.de");! String port = env.getProperty("db.port", "27016");! ! cmProperties.setProperty("servers", String.format("%s:%s", host, port));! cmProperties.setProperty("name", dbname);! }!! // OS-Username without prefix (e.g. maggie instead of lfstad-maggie)! private static String getSimpleUsername() {! String username = SystemUtils.USER_NAME;! int indexOfDash = username.indexOf('-');! return indexOfDash != -1 ? username.substring(indexOfDash + 1) : username;! }!}
Test-ApplicationContext
RepeatableCollections are dropped before each test method
public class MongoTestExcecutionListener extends AbstractTestExecutionListener {! @Override! public void beforeTestMethod(TestContext testContext) throws Exception {! purgeCollections();! }! ! private void purgeCollections() {! DB db = MongoDBHolder.getDB();! for (String collectionName : db.getCollectionNames()) {! if (collectionName.startsWith("fs.") || collectionName.startsWith("system.")) {! continue;! }! DBCollection collection = db.getCollection(collectionName);! if (!collection.isCapped()) {! collection.drop();! }! }! }!}
Spring-TestExecutionListener
@RunWith(SpringJUnit4ClassRunner.class)!@ActiveProfiles({"test"})!@ContextConfiguration(classes = TestSpringConfiguration.class)!@TestExecutionListeners({! DependencyInjectionTestExecutionListener.class,! DirtiesContextTestExecutionListener.class,! MongoTestExcecutionListener.class!})!public abstract class MongoTestSupport {!}!!!!public class DocumentRepositoryTest extends MongoTestSupport {!}!
Test base class
95% Coverage
Summary
• Developing the data layer is fun again • Reduced complexity • Flat learning curve compared to SQL/ORM • use_mongo() unless transactions_required
Thank you!
Recommended