ITSIP

Case Study : Ireland

METIS Workshop, 4-6 July 2007

Data Management System (DMS)

2

ITSIPPresentation Agenda

Introduction and Overview

Statistical Metadata Systems and the Statistical Cycle

Statistical Metadata in each phase of the cycle

Systems and Design Issues

Organisation and Cultural Issues

3

ITSIP

Introduction & Overview: Project Governance

Project BoardCSO Directorate, CMOD Representative

Project Management Team

Margaret McLoughlin

Quality Assurance

Bob Heaton

Test Team

Edward LambeCathy Kelly

Rory NaughtonPaul O’Connor

Karl Stone

Business Analyst Team

Dave JenningsJohn Hayes

Marian McCannMairead Coughlan

Jennifer Banim

Executive Support

Cathy Kelly

Project Steering Committee

IT Director; Margaret McLoughlin, Dave Jennings; Cognizant Personnel

Project ManagerMargaret McLoughlin

Programme ManagerDave Jennings

4

ITSIP

Introduction & Overview:

Overall Strategy

Main drivers

EU requirement to move to Open Systems

Storage of all CSO data in a RDBMS

DMS to be business led with metadata driven processes

DMS to require use of common classifications (CARS)

DMS to require use of common dissemination database

CSO produced

IT Strategy for 1999-2002 & beyond (April 1999)

Data Warehouse / Data Management Strategy (November 1999)

CGEY (10 week contract) produced an implementation plan for CSO’s IT & Data Management Strategies (first quarter 2001)

5

ITSIP

Introduction & Overview: Project Objectives

ITSIP - Information Technology Strategic Implementation Programme

Deliver set of applications to meet the survey processing and dissemination needs of CSO

Migrate existing DEC Alpha-based applications to client server environment

Implement the new applications within the CSO Corporate Data Model

Interface these applications with the existing client server and Sybase systems

6

ITSIP

Introduction & Overview:Project Goals Obtained

Legacy Situation

Stove type approach to survey processing

150 systems written & maintained centrally

250 end-user applications written & maintained locally

SAS V6.12 & PC SAS V8.02, Excel, Access

Data Management System (DMS)

Consolidates legacy processes into a suite of survey processing system

Nine corporate applications reside on a corporate database storing all data and metadata required in the survey-processing lifecycle.

Promotes consistency and reuse across the various survey areas

7

ITSIP

Introduction & Overview:

Project Background

Stage A contract (6 Months) awarded to Accenture who compiled the Requirements Specification & High Level Architectural Design (April 2003)

Stage B contract (30 Months) awarded to Cognizant Technology Solutions Ltd., Chennai, India.

Project currently at performance testing phase

CSO first Irish Government office to use onshore/offshore outsourcing model

Cognizant staff onsite have ranged from 2-17 depending on need

Offshore team has ranged from 20-50 depending on Project phase

CSO ITSIP team ~ 15 staff

All above contracts were fixed price contracts

8

ITSIP

Introduction & Overview: Stage A Review (Accenture)

Requirements Analysis PhaseBottom up approach through 20+ workshops with over 100 CSO business users

Production of:

51 ‘As Is’ process descriptions with 61 process maps

Data Model (Swedish Data Model for Aggregation & Dissemination)

44 ‘To Be’ process descriptions

Consolidation of existing processes into 9 survey processing applications plus a security application

Design phase

High Level Architectural Requirements

High Level Architectural Design

High Level Performance Model

High Level Interface Requirements and Design Specification

Web Enablement Specification

9

ITSIP

Introduction & Overview: Stage B Review (CTS)

Stage B involved:

Further validation of Stage A system design for baseline DMS

Building DMS

Migration of historic data and integrity metadata from legacy systems

UAT

Migration of metadata from the UAT environment to the production environment

10

ITSIP

Introduction & Overview: Stage B Review (CTS)

Original Schedule 3 Nov 2003 - 2 May 2006

Latest Schedule 3 Nov 2003 - 29 August 2007

Delay of 16 months arose because of

delay in initial increment deliveries due to new requirements

delay in CSO testing due to underestimation of time required

extra functionality in the DMS

change in design needed for better performance

change from Windows to Unix for Sybase to cope with production load

Reworking of Java code to meet QA standards

11

ITSIP

Introduction & Overview: Data Migration Approach

Business areas identified for ~ 100 surveysminimum data and integrity metadata required to support normal survey processing

all required back versions of data including all historic data required to be migrated (back to 1939 in some cases)

any additional data which should be migrated

Cognizant produced required ETL scripts using Informatica (data restructured into cube format to use classifications)

ETL scripts run to move data to UAT environment

Same scripts will move all data to Production environment (including latest processed periods)

Minimum integrity metadata migrated to all relevant databases because of application dependancy on same metadata

12

ITSIP

Introduction & Overview: Process Metadata Migration Approach

Business areas should and would only enter process metadata once (in UAT environment)

Business areas identified process metadata entered during UATFor Survey Instance specific modules (SS, SM, DC & IMP): UAT Survey instance and Production Start Survey instance

For Survey specific modules (Reg. M., Agg. & Diss.): list of Registers, Aggregate, Weight & Disseminate Tables to be available in Production

Cognizant produced required ETL scripts to move this process metadata from the UAT to the Production environment

Comparison reports of metadata residing in UAT & Production will be used to validate migration process

Ultimate check will be another parallel run in the Production environment to ensure that all migrated metadata (process & mimimal integrity metadata ) is consistent and correct

13

ITSIP

Introduction & Overview: Recommendations for others

Consider carefully the organisation’s capacity for insourcing / outsourcing development work

Consider the time scale for implementation of the solution

Manage the change process well

Understand the complexity of the solution and in procurement stage reject very low bids

Assume contractor has no knowledge of your business

Ensure adequate in house skills in IT Design so IT Partner’s assumptions can be validated

Ensure adequate in-house skills in IT Partner’s development tools and proposed application infrastructure

Don’t accept IT Partner’s project plan lightly where your office’s resources are concerned

14

ITSIP

Introduction & Overview: Recommendations for others

Don’t under estimate the resources needed to (1) manage the project and (2) keep abreast of all project documentation

Consider carefully the items that are for sign-off, review and for information by you - these will have financial implications later

QA is more important than just ticking boxes but throughout the software development lifecycle should include:

reviewing the decisions taken to obtain technical solutions

examining the underlying deliverable

adherence to agreed standards

Allocate adequate time to reviewing the test process and test cases

Managing the contract requires high-level expert resources with project management, statistical and IT skills

Organisational support and commitment from top management critical

15

ITSIP

Introduction & Overview: Future Challenges

DMS is to Go Live in Sept 2007

Six month gradual implementation

New SAS environment as we move from SAS V6, on the VAX, and

PCSAS V8.02

New IT Strategy is required

16

ITSIP

Register Management

Sample SelectionCentral Business Register

Survey Management

Data Capture SPROCeT

Aggregation

Dissemination

BoPFACTS

DMS

Respondent Management

Imputation

SeasonalAdjustment

ExternalData CaptureApplication e.g. Blaise, Scanning

CARS

Sec

uri

tyS

ecu

rity

SA

SStatistical Metadata Systems:

Process Model

17

ITSIP

Statistical Metadata Systems: DMS Applications &

Metadata

Register Management - Create Register

- Define Register Variables

- Set-up Register Coding

Sample Selection - Set-up Sample Selection Criteria

- Define Stratification Groups

Data Capture - Create Data Capture Form

- Define Variable Characteristics

- Set-up Coding Rules

- Set-up Import Details

- Set-up Edit Rules and Validations

- Version control of data

18

ITSIP

Statistical Metadata Systems: DMS Applications & Metadata

Imputation - Set-up Imputation Groups

- Set-up Imputation Rules

Aggregation - Define Groups, Data Columns, Tables

- Create Weights and Weight Tables

- Macro edits and Confidentiality Rules

Dissemination - Create disseminate tables

- Define Additional Data Column attributes

Seasonal Adjustment - Set-up Seasonal Adjustment Rules

Survey Management - Set-up Post Out details

19

ITSIP

Statistical Metadata Systems: Existing Systems

CBR Central repository for all enterprises engaged economic activity

CARS Database containing all classifications and concordances

SPROCET Re-usable survey processing template used by the Industrial surveys in the CSO

BoPFACTS Data processing and survey management system used by the Balance of Payments section

SAS SAS V6.12 and PC SAS V8.02

External Data Capture Applications - Blaise, Scanning

20

ITSIP

Statistical Metadata Systems: Mapping the DMS to the CMF Life Cycle

Register Management Survey Preparation (2)

Sample Selection Survey Plan & Design (1)

Survey Management Survey Preparation (2)

Data Capture Data Collection (3)

Input Processing (4)

Derivation (5)

Imputation Estimation (5)

Aggregation Aggregation (5)

Dissemination Dissemination (7)

Respondant Management Post Survey Evaluation (8)

The DMS is a processing and not an analysis tool, therefore CMF LifeCycle Model “(6) Analysis” cannot be linked to the DMS.

21

ITSIP

Statistical Metadata in the Statistical Cycle: Input Metadata Examples

Group Description Examples Source

Register NameLists the Register Names stored in the DMS

R_RSI: The register for the Retail Sales survey. DMS Table: Register

Survey Name Lists the survey in the DMS RSI: The Retail Sales Survey DMS Table: Survey

DMSCommonCodes

Holds a list of metadata which may be reused across surveys

Data Type: String, Int, Boolean, Varchar, Numeric.

DMS Table: DMSCommonCodes

ClassificationsClassifications used to code variables in RM. NACE Rev 1.1 CARS

Survey Periodicity Monthly 2007M06

DMS Table: SurveyInstance

Sample Method Census DMS Table: Sample

Unit of Measure Euro

DMS Table:DMSCommonCodes

PostOut Item Type Questionnaire or Reminder

DMS Table: DMSCommonCodes

22

ITSIP

Statistical Metadata in the Statistical Cycle: Output Metadata Example

Table Name Creation type

Column Name DataType Relationship Description

M_RegisterVariable Static This table contains all the Register Variables information for a Register.VariableID_PK int Primary Key Variable IDRegisterID_FK int Register ID of the Register in which the Variable is presentVariableNameText varchar(30) Variable Name LabelText varchar(40) Variable Label

FormatID int

Format type ID

0 - Default1 - Herd Number2 - PPS Number3 - MVSerNo

DataTypeID smallint

Data type ID

1 - Numeric2 - String3 - Date4 - Boolean

ApplicableInd bit Flag to indicate if the variable is applicable in the Register.SearchVarInd bit Flag to indicate if the variable is Search applicable in the Register.CommonInd bit Flag to indicate if the Variable is a generic Variable.MandatoryInd bit Flag to indicate if the Variable is Mandatory.UpdateUser varchar(20) The login name of the user who last updated the row.UpdateDate datetime This is the date and time at when the user last updated the row.

CodedInd bitFlag to indicate if the Variable is coded.

HoldsCodedValueInd bit

Flag to indicate if the Variable holds codes directly.

0 - Does not Holds the coded Values1 - Holds the coded Values

Length int Length of the VariablePrec int Precision of the VariableCapsOnInd bit Flag to indicate if the CAPS ON property is turned on for the Variable.

23

ITSIP

Statistical Metadata in the Statistical Cycle: Output Metadata Example

24

ITSIP

Systems and Design Issues: Technical Starting Point

Sybase

In-house knowledge in Sybase (ASE) Technologies

SAS access for complex analysis

(SAS did not bid for tender)

Link to Classifications and Related Standards (CARS) system

All disseminated data groups must link to a CARS classification

Windows platform

(Not possible due to performance issues identified with Sybase transactions, hence move to Solaris)

25

ITSIP

Systems and Design Issues: Technical Overview

AS

E

(fai

lov

er)

AS

E

IQ

(fai

lov

er)

IQ

Win: WebLogic Cluster

PC / ClientIE6

JRE1.4.2_05

SAS

Filestore

CARS

SSA Names3CBR

Unix: Sybase

T3 (RMI) JDBC

26

ITSIP

Systems and Design Issues: Database Layer

The CSO has now established in-house skills in both Sybase ASE & IQ Technologies

High Level Technical Architecture:

Data Capture, Imputation : Sybase ASE

Aggregation, Dissemination : Sybase IQ

Two types of table:

Core DMS Table (Survey Metadata)

Survey Specific Table (Data)

All complex numerical processing is performed within the database layer through the use of stored procedures

User of Veritas Clustering software on database layer to facilitate database failover

27

ITSIP

Systems and Design Issues: Weblogic / J2EE MidTier

J2EE Application Server (Weblogic)

Stateless Session Beans

JMS Queues

JDBC Connection to ASE / IQ Databases

Application Security

Users validated against corporate Active Directory Service

Within DMS Database validated users will have assigned roles / privileges

28

ITSIP

Systems and Design Issues: Client Layer

The DMS is a complex GUI interface

‘Fat Client’ using Java Swing technology

The client is deployed using Java Web Start Technology

Centrally managed releases

Quick deployment to client desktop

Client uses Java RMI to communicate with the J2EE server

(Currently using WebLogic T3 protocol)

29

ITSIP

Systems and Design Issues: Other Components

Filestore

Shared network drive onto which data to be Imported / Exported to the DMS resides

SAS

Required for Seasonal Adjustment

Required for Import / Export of SAS Datasets to/from DMS

CARS (Classifications) [Statistics New Zealand]

All data to be dissemintated must use a CARS classification

CBR (Central Business Register) [Statistics New Zealand]

Hierarchical database

SSA Names3

Duplicate matching / searching of registers

30

ITSIP

Organisational & Cultural Issues: Roles within the CSO

DMS Administrator (I.T.)

Highest level of access to the DMS

Supports the DMS

Manages the day to day interaction with the DMS

Survey Administrator (Statistician)

Defines the survey

Runs the survey

Assigns staff survey access and privileges

DMS Administrator

Survey AdministratorSurvey Administrator

User

User

User

User

User

User

User

User

31

ITSIP

Organisational & Cultural Issues: DMS Maintenance

In the future the DMS will be supported by:

Cognizant Technology Solutions Ltd

1 year maintenance contract

provision for a 5 year support contract

CSO Java Development Team

CSO Weblogic Team

32

ITSIPThank You for Your Attention