The TARDIS Framework A Federated Repository Solution For Raw Diffraction Datasets Steve Androulakis,...

The TARDIS Framework

A Federated Repository Solution For Raw Diffraction Datasets

Steve Androulakis, Monash University, Melbourne Australia

I2S2 Workshop April 2011

TARDIS’ Internal Structure

Thanks to Alistair Grant

Generic Parameter Support

Thanks to Alistair Grant

Started in late 2007

Formerly.. Public Only Access to Raw Protein Crystallography Data

Formerly.. Fedora Commons Based

Federated Storage

Formerly.. Protein Crystallography Specific Code

Australian Synchrotron VBL Services..

Thanks Michael D’Silva @ VeRSI

Australian Synchrotron VBL Services..

Thanks Michael D’Silva @ VeRSI

Australian Synchrotron VBL Services..

Thanks Michael D’Silva @ VeRSI

Data Store

myTARDIS

Data Store

myTARDIS

Data Store

myTARDIS

Data Store

myTARDIS

TARDIS.edu.au

download

download download

download

From Capture to Publication

6 Datasets, 5140 Datafiles ~90GB

Challenges..

How shall we manage the federation at the top level? (cache index vs ‘spider search’)

How much responsibility to we take over the underlying data stores and their integrity?

Exactly how ‘generic’ should the underlying structure be?

How do we encourage users to utilise the system with regularity?

Molecular Biology Auto-Processing

Thanks Grischa Meyer

SquirrelMyTARDIS For The Archival, Preservation and Eventual Publication of ALL Research Data

We’re now the ‘official solution’ for instruments at AS + ANSTO!

..so more instruments + facilities

“TARDIS/MyTARDIS system for high-end instrumentation data capture (e.g. X-ray diffraction images) has been chosen by both the Australian Synchrotron and ANSTO (Australian Nuclear Science and Technology Organisation) as the platform for metadata capture. Congratulations to the developers of TARDIS/MyTARDIS: Monash University biochemist, Associate Professor Ashley Buckle and software engineer Mr Steve Androulakis. (Support received from the Monash e-Research Centre, the Victorian eResearch Strategic Initiative, the Australian Synchrotron and the Australian National Data Service).”

RMIT – Microscopy• 9 Microscopes• New ‘Filter’ Module Functionality• RMIT eResearch Office – Heinz Schmidt / Ian Thomas

The Ian Wark Research Institute – Particle Physics• Instrument Linkage• University of SA - Ryan Green

Peter Mac Institute – Next Gen Sequencing• Instrument Linkage• Courtesy of VeRSI – Lyle Winton

APS Chicago?!

Upcoming Projects

MicroTARDISInitiative of the eResearch Office at RMIT University, for the RMIT Microscopy and Microanalysis Facility (RMMF) in the School of Applied Science.

eResearch Office:Prof Heinz Schmidt, Dr Ian Thomas, Mr Ravi Sreenivasamurthy, Mr Peter Wolynec.

RMMF: Prof. Dougal McCulloch, Mr Phil Francis.

Deployment and Customization of the myTARDIS meta-data repository for research scientists using microscopes in applied sciences

Thanks Ian Thomas (RMIT)

Key Features- Backend connection to data from Phillips XL30 SEM, FEI Nova NanoSEM and FEI Quanta

microscopes.- Staging area to allow user-driven upload of experiment data- Automatic metadata extraction from instrument image files and ingestion

Thanks Ian Thomas (RMIT)

R-2-6.tif

R-2-2.tif

Thanks Ian Thomas (RMIT)

Thanks Ian Thomas (RMIT)

TARDIS: Internals

Lightweight Architecture

TARDIS’ Modular Framework

MyTARDISMyTARDIS

SynchrotronSynchrotron

TARDIS.edu.auTARDIS.edu.au

MicroscopyMicroscopy

VBL DownloadVBL Authentication

Visual StyleParameter Schemas

Parameter ‘Filters’Visual Style

Parameter Schemas

Federated Storage SupportVisual Style and Templates

Parameter Schemas

http://code.google.com/p/mytardis/ (BSD)

MRtardisMRtardis

App for running distributed HPC jobs and storing the

results in TARDIS