Embed Size (px)
Citation preview
Journeys with Transmogrifier & friends
or How not to get stuck in
the Plone dark ages
Daniel JowettJowett Enterprises Ltd
About me
● Software Professional since 1997
● Background in C, Java, SQL, etc …. even P*P
● Using Plone since 2008 on starting self-employment
● Once upon a time...
● … I came to a Plone Conference in Bristol...
● … in 2010
● And everyone seemed to be cleverer than me
● And now 4 years later? ...
● I know most of them are cleverer than me
● But I have done 4 or 5 projects with Transmogrifer / Jsonmigrator
Foundational Principles
● Calvin & Hobbes
● Calvin is an over-imaginative 6 year old
● Hobbes is his toy tiger
● A cardboard box can be anything
● But in this case it's a “Transmogrifier”
Transmogrifier: ...what is it?
● Transmogrifier – turns something into anything elselike Calvin into a toad, or tiger
● Plone's Transmogrifier: a way to migrate content
– typically between Plone versions
– also from plain html sites & other CMS databases
– Aims for and achieves a decent amount of code reuse● JsonMigrator: Builds on top of Transmogrifier so you don't have to
install it in the source Plone site
Transmogrifier Variants
● Plain Transmogrifier – 2 stage process to export/import from Plone site via xml files (or csv files)
● Funnelweb - Crawls and parses static sites for import, which are then 'pushed' into Plone
● Jsonmigrator – 1 stage process that crawls Json views of an old site
● Transmogrifier blueprints also exist that can read from SQL Dbs, so for example Drupal data
We will be focussing on Jsonmigrator
When to use Jsonmigrator
● From old Plone versions, like 2.5, but even as far back as 1.0 in theory (but requires Python >= 2.2)
● Even from old Zope/CMF sites
● particularly when you have no buildout
● When changing content type framework from archetypes to dexterity
● To clean cruft out of an error prone old site
● When scared to upgrade because you don't know what might bite you later
Note: This isn't the whole story - you still have to Upgrade Your Plone products & custom code
When NOT to use Transmogrifier
● From Plone 4.x
● Probably not from 3.3.6 with buildout
● Without considering the alternatives:
– running Plone upgrade steps (plone.app.upgrade)– plone.app.contenttypes migration for AT to DX based types
Technology Stack
● Transmogrifier:
– plone.app.transmogrifier
– collective.transmogrifier● collective.jsonify
● collective.jsonmigrator
● Products containing extra pipeline sections, such as:
– transmogrify.dexterity
– quintagroup.transmogrifier
– transmogrify.print
Setting Up Jsonmigrator
* These steps are outside the scope of this presentation – hopefully you've got an idea of what to do here (though step 5 may well not be trivial)
1) * Duplicate old Plone in your staging environment
2) install collective.jsonify to this duplicate
3) Add “external methods”
4) * Create a brand new Plone 4.x instance (try 4.3.3 for starters)
5) * Install the updated or replacement products you need to the new instance
6) install collective.jsonmigrator & any special pipeline sections to new instance
Install collective.jsonify
This is the really old way to install stuff & generally bad practise, but if you are using a virtualenv (or dedicated python) it's passable. It's much easier if you're using buildout, but I'll assume that you're not!
1. Download the egg (my version in this case) to your Plone directory:
2. Unzip it
3. Add it to your python (it will also pull in simplejson)
cd collective.jsonify-0.1.8b/../bin/zopepy setup.py install…Adding collective.jsonify 0.1.8b to easy-install.pth file…… simplejson...
unzip 0.1.8b.zip
wget https://github.com/djowett/collective.jsonify/archive/0.1.8b.zip
Add “External Methods”
● Add an Extensions folder
● Add json_methods.py
● Then run your Plone instance, go to the Zope root and create the three equivalently named External Methods, ie: “get_item”, “get_children” & “get_catalog_results”, eg:
from collective.jsonify import get_itemfrom collective.jsonify import get_childrenfrom collective.jsonify import get_catalog_results
Install collective.jsonmigrator
● Add the highlighted lines to your buildout's: develop.cfg
● You might also want to add transmogrify pipelines like transmogrify.dexterity or quintagroup.transmogrifier to your eggs section
● Then run buildout & run up your instance
Note: We check out the collective.jsonmigrator egg for development so that we can tweak the pipeline configuration when we need to
[sources]...collective.jsonmigrator = git https://github.com/djowett/collective.jsonmigrator.git
[buildout]eggs +=... collective.jsonmigrator
Looking closer
● Installing Collective.jsonify & the external methods gives a (fairly) complete json view of every piece of content on your old site
● For example...
● Collective.jsonmigrator can pick up these json data dictionarys, and using transmogrifier 'massage' them and create the corresponding objects on your new site
Have a go!
● Go to: http://<your-plone>:<port>/<instance>/@@jsonmigrator
● Select one of the pre-loaded sample configurations
● Select username, password & a small folder for your first run
Demo
So what happened there? (I)
● This was our “pipeline”
[transmogrifier]pipeline = remotesource removeid strip-leading-path correct-zope-file-types fix_index_html_in_id fix_index_html_in_path print_seq_in_desc description_remove_seq tag_general_if_ccsnews_in_desc tag_service_if_ccsnews_not_in_desc description_remove_ccsnews find_path_depth repl_relative_urls_in_text get_hardcoded_urls_in_text print_hardcoded_url_post_replace get_siteImage_urls_in_text print_siteImage_urls fix_index_html_refs_in_text find_h1_title_in_text
remove_h1_title_from_text constructor schema uid datafields fix_defaultPage# print_defaultpage browser-default# workflow-history get-last-workflow-state calc-workflow-transition publish_all_folders workflowupdater properties# permissions owner local_roles mimetype commit IMPORTING
So what happened there? (II)
● Here are some pipeline section definitions:
[constructor]blueprint = collective.transmogrifier.sections.constructor
# Zope Images & Files get incorrectly reported as Folders [correct-zope-file-types]blueprint = collective.transmogrifier.sections.manipulatorkeys = _classnamedestination = python:"_type"condition = python:(item['_type'] == 'Folder' and item.has_key('_classname')
and item['_classname'] in ['Image', 'File'])[strip-leading-path]blueprint = collective.transmogrifier.sections.inserterkey = string:_pathvalue = python: item['_path'].replace('ccs/', '', 1)
[tag_general_if_ccsnews_in_desc]blueprint = collective.transmogrifier.sections.inserterkey = string:subjectvalue = python: item['subject'] + ['general']condition = python: item['_type'] == "News Item" and 'ccsnews' in item['description']
Real life isn't Blue Peter*
So, even if it did work just now, it will probably break for you first time, so make some adjustments in the pipeline (<instance>/src/collective.jsonmigrator/collective/jsonmigrator/config_plone3.cfg)
● Exclude objects from the import (particularly plone tools)
● Use print a lot (from transmogrify.print)
● Delete the import results (if you had any) & run again
* Blue Peter was the staple TV diet for 1980's Kids in the UK, featuring the oft used phrase “here's one I made earlier” when things didn't work
Benefits compared to Legacy Plone
● A fresh start
● A chance to use new Plone technologies like Diazo, Dexterity, new Collections, plone.app.contenttypes. etc. etc.
● Moving from retired products to cutting edge equivalents
● The speed of Plone 4 & blob storage
● Using buildout to control your site
Caveats
● This doesn't export/import users (though see collective.blueprint.usersandgroups)
● And it doesn't do portlets (portlets can be exported by generic setup, but that doesn't include “deep” content portlets – you might be stuck with a manual portlet setup)
● Collective.jsonify ignores Plone's security model and is a security loophole – add this to a publicly accessible site at your own risk
● The docs can be a little misleading at times
● Many folk using this are probably working off checked out code, so egg releases are few & far between – you are going to have to get your hands dirty
● “Debugging is backwards” in transmogrifier
Credits
● Transmogrifier – Martjin Pieters, Jarn
● Jsonmigrator - Rok Garbas
● RCS – letting me loose on this
● Calvin & Hobbes – Bill Watterson – creating the original Transmogrifier
Links
● http://www.jowettenterprises.com/blog/jsonmigrator
● http://collectivejsonify.readthedocs.org/
● http://collectivejsonmigrator.readthedocs.org/
