Migrating from drupal to plone with transmogrifier

Clayton Parker, Senior Developer

Migrating From Drupal to Plone with Transmogrifier

PLONE SYMPOSIUM EAST 2011

PLONE SYMPOSIUM EAST 2011Who Am I?

• claytron

• Python dev since 2003

• Plone Core Committer

• Foundation Member

PLONE SYMPOSIUM EAST 2011What will we learn?

• Transmogrifier basics

• Migration Planning Process

• Creating a pipeline

PLONE SYMPOSIUM EAST 2011Migrations

• One off scripts

• In multiple places

• Little to no re-usability

PLONE SYMPOSIUM EAST 2011Transmogrifier

• A framework for migrations

• Re-usable parts

PLONE SYMPOSIUM EAST 2011Basics

• Pipeline

• Blueprints

• Sources

• Section

PLONE SYMPOSIUM EAST 2011Sources

• A blueprint

• First item in your pipeline


<html><body><h3>Code Sample</h3><p>Replace this text!</p></body></html>

Example Pipeline[transmogrifier]pipeline = csv_file constructor schemaupdater

[csv_file]blueprint = collective.transmogrifier.sections.csvsourcefilename = my.migration.import:my_items.csv

[constructor]blueprint = collective.transmogrifier.sections.constructor

[schemaupdater]blueprint = plone.app.transmogrifier.atschemaupdater

PLONE SYMPOSIUM EAST 2011my_items.csv

_path , _type , title , description/folder1 , Folder , First Folder , This is folder One/folder2 , Folder , Second Folder , This is folder Two/folder1/foo , Document , One Foo , A document named foo/folder2/foo , Document , Two Foo , Another doc named foo

PLONE SYMPOSIUM EAST 2011The Result

PLONE SYMPOSIUM EAST 2011Items

• Each item is a mapping

• Keys are fields

• Keys with a leading underscore are controllers

PLONE SYMPOSIUM EAST 2011Example Item{'_id': 'a-stronger-connection-to-out-customers', '_path': 'content/stronger-connection-out-customers', '_status': 1L, '_text_mimetype': 'text/html', '_transitions': 'publish', '_type': 'Document', 'allowDiscussion': False, 'creation_date': '2011/05/14 9:20:50', 'effectiveDate': '2011/05/14 9:20:50', 'modification_date': '2011/05/18 9:22:22', 'subject': 'better\ninteresting\nstronger', 'text': '<p>this is some text</p>\r\n', 'title': 'A stronger connection to out customers'}

PLONE SYMPOSIUM EAST 2011Migration Strategy

• Investigate the source

• Prepare the destination

• Find Transmogrifier blueprints

• Write the pipeline

PLONE SYMPOSIUM EAST 2011Write your own

• Missing blueprint

• Write one

• Contribute it back

PLONE SYMPOSIUM EAST 2011GenericSetup

• Make migration part of your release

• Ability to package migrations

PLONE SYMPOSIUM EAST 2011My Migration

• Drupal backed by MySQL

• transmogrify.sqlalchemy

• Plone 4.0.5

• collective.blog.star

• plone.app.discussion

PLONE SYMPOSIUM EAST 2011Package Layoutmy.migration!"" my# !"" __init__.py# %"" migration# !"" __init__.py# !"" config# # !"" articles.cfg# # !"" base.cfg# # !"" blogs.cfg# # !"" comments.cfg# # %"" pages.cfg# !"" configure.zcml# %"" profiles# %"" default# !"" metadata.xml# %"" transmogrifier.txt!"" setup.cfg%"" setup.py

PLONE SYMPOSIUM EAST 2011Registering Configs

<transmogrifier:registerConfig name="my.migration.base" title="My migration base config" description="Base settings for all transmogrifier imports" configuration="config/base.cfg" />


<transmogrifier:registerConfig name="my.migration.pages" title="Drupal pages" description="Import pages from Drupal into Plone" configuration="config/pages.cfg" />


<transmogrifier:registerConfig name="my.migration.articles" title="Drupal articles" description="Import articles from Drupal into Plone" configuration="config/articles.cfg" />


<transmogrifier:registerConfig name="my.migration.blogs" title="Drupal blog entries" description="Import blog entries from Drupal into Plone" configuration="config/blogs.cfg" />


<transmogrifier:registerConfig name="my.migration.comments" title="Drupal comments" description="Import comments from Drupal into Plone" configuration="config/comments.cfg" />

PLONE SYMPOSIUM EAST 2011transmogrifier.txt

my.migration.pagesmy.migration.articlesmy.migration.blogsmy.migration.comments

PLONE SYMPOSIUM EAST 2011Package Layoutmy.migration!"" my# !"" __init__.py# %"" migration# !"" __init__.py# !"" config# # !"" articles.cfg# # !"" base.cfg# # !"" blogs.cfg# # !"" comments.cfg# # %"" pages.cfg# !"" configure.zcml# %"" profiles# %"" default# !"" metadata.xml# %"" transmogrifier.txt!"" setup.cfg%"" setup.py

PLONE SYMPOSIUM EAST 2011base.cfg pipeline[transmogrifier]pipeline = drupal portal_type url_normalizer path publication_state text_mimetype mimetype_encapsulator folders constructor commenting comments schema_update workflow reindex_object

[settings]# Path to use if there isn’t one givenbase_path = other-content# Have to escape python string formatting for when # this gets passed into sqlalchemydate_format = %%Y/%%m/%%d %%k:%%i:%%s

PLONE SYMPOSIUM EAST 2011drupal section

[drupal]blueprint = transmogrify.sqlalchemydsn = mysql://user:password@localhost/drupal-transmog

PLONE SYMPOSIUM EAST 2011articles.cfg[transmogrifier]include = my.migration.base

[drupal]query = SELECT node.title, node.status AS status, GROUP_CONCAT(tag_data.name SEPARATOR '\n') AS subject, FROM_UNIXTIME(node.created, "${settings:date_format}") AS creation_date, FROM_UNIXTIME(node.created, "${settings:date_format}") AS effectiveDate, FROM_UNIXTIME(node.changed, "${settings:date_format}") AS modification_date, body_data.body_value AS text url_alias.alias AS _path FROM node INNER JOIN field_data_field_tags AS tag_field ON tag_field.entity_id = node.nid INNER JOIN taxonomy_term_data AS tag_data ON tag_data.tid = tag_field.field_tags_tid INNER JOIN field_data_body AS body_data ON body_data.entity_id = node.nid INNER JOIN url_alias ON url_alias.source = CONCAT("node/", node.nid) GROUP BY node.title, node.status, node.created, node.changed, body_data.body_value, url_alias.alias;

[portal_type]value = string:Document

PLONE SYMPOSIUM EAST 2011pages.cfg[transmogrifier]include = my.migration.base

[drupal]query = my.migration.base SELECT node.title, node.status AS _status, FROM_UNIXTIME(node.created, "${settings:date_format}") AS creation_date, FROM_UNIXTIME(node.created, "${settings:date_format}") AS effectiveDate, FROM_UNIXTIME(node.changed, "${settings:date_format}") AS modification_date, body_data.body_value AS text, url_alias.alias AS _path FROM node INNER JOIN field_data_body AS body_data ON body_data.entity_id = node.nid INNER JOIN url_alias ON url_alias.source = CONCAT("node/", node.nid) WHERE node.type = "page" GROUP BY node.title, node.status, node.created, node.changed, body_data.body_value, url_alias.alias;

[portal_type]value = string:Document

PLONE SYMPOSIUM EAST 2011blogs.cfg[transmogrifier]include = my.migration.base

[drupal]query = SELECT node.title, node.satus AS _status, FROM_UNIXTIME(node.created, "${settings:date_format}") AS creation_date, FROM_UNIXTIME(node.created, "${settings:date_format}") AS effectiveDate, FROM_UNIXTIME(node.created, "${settings:date_format}") AS modification_date, body_data.body_value AS text, url_alias.alias AS _path FROM node INNER JOIN field_data_body AS body_data ON body_data.entity_id = node.nid INNER JOIN url_alias ON url_alias.source = CONCAT("node/", node.nid) WHERE node.type = "blog" GROUP BY node.title, node.status, node.created, node.changed, body_data.body_value, url_alias.alias;

[portal_type]value = string:BlogEntry

[commenting]value = python:True

PLONE SYMPOSIUM EAST 2011comments.cfg[transmogrifier]include = my.migration.base

[drupal]query = my.migration.base SELECT comment.subject AS title, FROM_UNIXTIME(comment.created, "${settings:date_format}") AS published, FROM_UNIXTIME(comment.changed, "${settings:date_format}") AS updated, comment.name AS author_name, body_data.comment_body_value AS text, url_alias.alias AS _parent_path FROM comment INNER JOIN field_data_comment_body AS body_data ON body_data.entity_id = comment.cid INNER JOIN node ON node.nid = comment.nid INNER JOIN url_alias ON url_alias.source = CONCAT("node/", node.nid) GROUP BY comment.subject, comment.created, comment.changed, comment.name, body_data.comment_body_value, url_alias.alias;

[portal_type]# Override the portal type to use the "comment_type"key = string:_comment_typevalue = string:plone.app.discussion

PLONE SYMPOSIUM EAST 2011base.cfg overrides[path]blueprint = collective.transmogrifier.sections.inserter# only add a path if one does not existcondition = python:'_path' not in item and not '_parent_path' in itemkey = string:_path# Add the value in the extended configurationvalue = string:${settings:base_path}/${item/_id}

[portal_type]blueprint = collective.transmogrifier.sections.inserterkey = string:_type# We will add the value in the extended config, but we need a# default set herevalue = string:

[commenting]blueprint = collective.transmogrifier.sections.inserterkey = string:allowDiscussion# default to falsevalue = python:False

PLONE SYMPOSIUM EAST 2011Content Creation

[comments]blueprint = transmogrify.comments

[folders]blueprint = collective.transmogrifier.sections.folders

[constructor]blueprint = collective.transmogrifier.sections.constructor

[schema_update]blueprint = plone.app.transmogrifier.atschemaupdater

[workflow]blueprint = plone.app.transmogrifier.workflowupdater

[reindex_object]blueprint = plone.app.transmogrifier.reindexobject

PLONE SYMPOSIUM EAST 2011Item Modification[publication_state]blueprint = collective.transmogrifier.sections.insertercondition = python:'_status' in item and item['_status'] == 1key = string:_transitionsvalue = string:publish

[text_mimetype]# This could probably be taken from the database as wellblueprint = collective.transmogrifier.sections.inserterkey = string:_text_mimetypevalue = string:text/html

[mimetype_encapsulator]blueprint = plone.app.transmogrifier.mimeencapsulatorkey = textmimetype = python:item.get('_%s_mimetype', key)field = keycondition = mimetype


DEMO

PLONE SYMPOSIUM EAST 2011Links

• collective.transmogrifier

http://pypi.python.org/pypi/collective.transmogrifier/

• plone.app.transmogrifier

http://pypi.python.org/pypi/plone.app.transmogrifier/



PLONE SYMPOSIUM EAST 2011Useful Sources and Blueprints• plone.app.transmogrifier

• transmogrify.filesystem

• transmogrify.sqlalchemy

• transmogrify.webcrawler

• quintagroup.transmogrifier

• transmogrify.comments

Check out

sixfeetup.com/demos

Technology

Migrating from drupal to plone with transmogrifier