Embed Size (px)
Citation preview
1
EASY-R an SDMX Based Timeseries Repository
Oliver Lorenz
Swiss National Bank
SDMX Global Conference, 2011
2
Workflow
banks
industry
external publications www.snb.ch
internal publications Intranet
internal data sources
investment funds
statistical surveys aggregates
• banking statistics
• balance of
payments
• investment fund
statistics
Primary
Statistical
System
Central
Timeseries
Repository
3
EASY-R
~ 5 x 106
timeseries units:
• Economic Analysis
• Inflation Forecasting
• Monetary Policy Analysis
• Financial Stability
• Financial Markets
200 users
• standard users (80%)
• expert users (20%)
• administrators (2%)
Excel
User Interfaces
• Data Viewer/Editor
• Structure Viewer/Editor
• Scripting Environment
• Rights Management
4
Life-Cycle Coverage
EASY-R
Analysis
Processing
Warehousing
Management
Dissemination
5
Several Generations of Home Built Systems
• SNB has a tradition of home built systems
• Tailoring “Workflows EASY”
• Competence and confidence to design
and implement the Next Generation
System EASY-R
EASY (mvs)
197x - 2000
EASY (vb)
2000 - 2008
EASY (.Net)
2008 - 2011
EASY-R
2010 – 202x
6
EASY-R Innovation Goals
User Interfaces for 3 Distinct User Groups
Process Oriented Repository
Domain Model
Uniform Technology Stack
7
Development Approach
7
EASY-R
Internal Domain Experts
Internal „timeseries systems“ Experts
In-House Developmen
t (with Contracted
Teams)
Agile Project Management
(SCRUM)
Open Source
Technology
8
Governing Principles
• Flexible Business
Domain Model defined by economists for economists
• Self-Descriptive Data with ‚proper/formal‘ semantics
• Metadata Based
Configuration of the repository, business logic and GUI
• Retrieval using the structures and semantics of the model
• Full Revision History to keep track & undo, revert, reconstruct all changes
• Workflow using configured staging to separate production phases
• Integrated Scripting for analysis, transformation and maintenance
9
EASY-R
Repository
Easy-R
Domain Model
SDMX-based
Time Series
Engine
Various UIs
Scripting
Engine
Conversion
&
Synchronization
Component Architecture
10
Flexible Business Domain Model
and Self-Descriptive Data
10
receive structure receive data &
metadata start working
meta
data/structure
based, automatic
configuration
11
Flexible Business Domain Model
and Self-Descriptive Data
SDMX based data exchange
SDMX based repository
SDMX based time series arithmetic
11
receive structure receive data &
metadata start working
12
SNB’s SDMX-compatible Extensions for a Data
Warehouse
SDMX based Data Exchange
SDMX based repository
SDMX based time series arithmetic
12
receive structure receive data &
metadata start working
• Repository
– store and manipulate
structure or series
– validate structures or series
– efficient storage model
– workflow and data history
– user and rights
management
• Time Series Engine
– calendar
– arithmetic
– dynamic series
13
Time Series View
14
Structure View
15
Time Line and Costs
2006 2007 2008 2009 2010 2011 2012 2013
Migration (Data & Processes)
Inception & Elaboration
Construction Production R1
Construction I&E Production R2
• Costs of Construction Release 1 (estimates)
• IT:
• contracted SW Architects and Engineers : ~15PY (5Persons x 3Years) ~ 3,3 Mio CHF
• internal SW Architects and Engineers: ~ 3PY
• Software: < 20 kCHF
• Business:
• Workshops (Concepts, Planning, Review): ~ 3PY (5P * 20% * 3Y)
• Costs of Migration (5 Mio TS & ~180 Processes from 10 Business Units) • IT:
• contracted Support: ~5PY (2P * 2,5Y) 1,1 Mio CHF
• internal Support: ~3PY
• Business:
• Training, Planning, Preparation, Implementation, Testing, Deployment: > 1.5 PY (30P * 15Days)
16
Conclusions today
• The innovative vision of the new system could be
implemented with the chosen technology and architecture
• SDMX helped a lot (with some overhead)
• There is a comprehensive set of high quality Open Source
products, tools and frameworks available
• internal domain experts and IT competence (consulting,
project management, time series based information systems)
was vital for the success of the project.
• The project would not have been possible with a constant
and close support/involvement of the economists (expert
users from Statistics and Economic Analysis) (countless
workshops, meetings for planning and review)
17
Lessons Learned: Migration
• shared responsibility between IT (sum and quality of functionality) and economists (correctness and timeliness of migrated business processes)
• generaly at least as expensive and complex as building the system (time, resources)
• costs and complexity significantly reduced with self built solution – close to the original algorithms and processes
– resources & know how for user training
– IT resources to prepare „initial migration“
– 1:1 migration of data with legacy keys
– transparent interface to read from the legacy system
– improvement and adaption of the system with input from migration tasks
• the longest delay in the project roadmap
18
Lessons Learned: SDMX - TCO
• cost to learn, implement, adopt
• cost to interpret and extend
• cost of training (developers, data administrators, expert
users)
• effort to follow (and influence) the development of the
standard
• SDMX is quite heavy at times
• costs to extend a standard meant for data transmission
for use in a repository context (structured identifiers,
revisions and stages, process-information, database-
mapping, CRUD-performance)
19
Lessons Learned: SDMX - ROI
• a standard -> common vocabulary
• an evolved information model -> head start for the project
• data comes with describing metadata -> self configuring system !! – no need to restructure or transform external data
– a feasible way to describe our own data
• one format for all systems (internal, domestic format) timeseries, graphic system, statistical system
• standard for data exchange with other institutions – guaranteed correctly formatted SDMX
– SDMX sent from outside is checked during import (with some tolerance for inconsistency)
• extensible through annotations
• being a good SDMX citizen
