Systems Integration and Data Management

Presented By Greg Devine

Director, Higher Education Blackboard Consulting April 14, 2005

Welcome

• Session Objectives:– Introduce 2-3 key points

• Innovation– Insert

• Results/Outcomes– Insert

An overview of the process, tools and operationsFor Blackboard Learning System™ (Release 6.2)

Blackboard® Snapshot Integration

• What Data = Which Files• Snapshot Components• Snapshot Tool

– Modes, Operators, Syntax

• Data Sources and Data Source Keys• Field Ownership• Remote Snapshots• Snapshot Controller

IMS Entities

• Integration Entities = Snapshot Files• Enterprise Data

COLLEGECOLLEGE ENR_FA02ENR_FA02 CRS_FA02CRS_FA02

ENR_SP03ENR_SP03 CRS_SP02CRS_SP02

Memberships GroupsPeople

LAWLAW

USR ENR*STAFFORGMEM

CRSORG

Other Data Types

• Additional Enterprise Data

Portal RolesPortal Roles courselink_FA02

courselink_FA02

ORG_CATEGORYCRS_CATEGORY

categorycategory

courselink_SP03

courselink_SP03

Cou

rses

& O

rgan

izat

ions

Institution Roles

Content SystemRoles

Content SystemRoles

ORG_CATEGORY_MEMCRS_CATEGORY_MEM

Course and Organization Catalog

Use

rs

PORTAL_ROLE_MEM

File Formats• XML

– All data in one file– Reusable– Standardized (IMS+)

• Delimited (preferred)

– Easy to Manipulate – Multiple Files– Smaller

<PERSON> <SOURCEDID> <SOURCE>Blackboard University</SOURCE> <ID>39450210223</ID> </SOURCEDID> <USERID>swang</USERID> <NAME> <FN>Mr. Stanley Wang Jr.</FN> <SORT>Wang,Stanley</SORT> <NICKNAME>Wang,Stanley</NICKNAME> <N> <FAMILY>Wang</FAMILY> <GIVEN>Stanley</GIVEN> <OTHER>Franklin</OTHER> <PREFIX>Mr. </PREFIX> <SUFFIX>Jr.</SUFFIX> </N> </NAME> <DEMOGRAPHICS> <GENDER>2</GENDER> <BDAY>1959-01-01</BDAY> </DEMOGRAPHICS> <EMAIL>Swang5@Blackboard_university.com</EMAIL> <TEL teltype="1">3104591276</TEL> <TEL teltype="2">3104591200</TEL> <ADR> <STREET>Twin Oaks Valley Rd</STREET> <STREET>attn: S. Wang</STREET> <LOCALITY>San Marcos</LOCALITY> <REGION>CA</REGION> <PCODE>92096-0001</PCODE> <COUNTRY>US</COUNTRY> </ADR> <DATASOURCE>Blackboard University</DATASOURCE> <EXTENSION> <X_BB_SYSTEMROLE>0</X_BB_SYSTEMROLE> <X_BB_INSTITUTION_ROLE>0</X_BB_INSTITUTION_ROLE> <X_BB_STUDENTID>144532</X_BB_STUDENTID> <X_BB_PASSWORD>rpeterson</X_BB_PASSWORD> </EXTENSION></PERSON>

<PERSON> <SOURCEDID> <SOURCE>Blackboard University</SOURCE> <ID>39450210223</ID> </SOURCEDID> <USERID>swang</USERID> <NAME> <FN>Mr. Stanley Wang Jr.</FN> <SORT>Wang,Stanley</SORT> <NICKNAME>Wang,Stanley</NICKNAME> <N> <FAMILY>Wang</FAMILY> <GIVEN>Stanley</GIVEN> <OTHER>Franklin</OTHER> <PREFIX>Mr. </PREFIX> <SUFFIX>Jr.</SUFFIX> </N> </NAME> <DEMOGRAPHICS> <GENDER>2</GENDER> <BDAY>1959-01-01</BDAY> </DEMOGRAPHICS> <EMAIL>Swang5@Blackboard_university.com</EMAIL> <TEL teltype="1">3104591276</TEL> <TEL teltype="2">3104591200</TEL> <ADR> <STREET>Twin Oaks Valley Rd</STREET> <STREET>attn: S. Wang</STREET> <LOCALITY>San Marcos</LOCALITY> <REGION>CA</REGION> <PCODE>92096-0001</PCODE> <COUNTRY>US</COUNTRY> </ADR> <DATASOURCE>Blackboard University</DATASOURCE> <EXTENSION> <X_BB_SYSTEMROLE>0</X_BB_SYSTEMROLE> <X_BB_INSTITUTION_ROLE>0</X_BB_INSTITUTION_ROLE> <X_BB_STUDENTID>144532</X_BB_STUDENTID> <X_BB_PASSWORD>rpeterson</X_BB_PASSWORD> </EXTENSION></PERSON>

EXTERNAL_PERSON_KEY|USER_ID|SYSTEM_ROLE|INSTITUTION_ROLE|FIRSTNAME|LASTNAME|EMAIL0142270|sait.tarhan|none|role_11|Sait|Tarhan|sait.tarhan@college.edu0001129|marcha.hunley|none|role_11|Marcha|Hunley|marcha.hunley@college.edu0001613|steven.yelton|none|role_11|Steven|Yelton|steven.yelton@college.edu0003970|dorothy.mann|none|role_11|Dorothy|Mann|dorothy.mann@college.edu0099821|sherry.hutton|none|role_11|Sherry|Hutton|sherry.hutton@college.edu0000311|gary.webster|none|role_11|Gary|Webster|gary.webster@college.edu0173103|carla.gesell-streeter|none|role_11|Carla|Gesell|carla@college.edu0000612|olivia.watts|none|role_11|Olivia|Watts|olivia.watts@college.edu0000381|robert.eveslage|none|role_11|Robert|Eveslage|robert.eveslage@college.edu0000987|debra.lierl|none|role_11|Debra|Lierl|debra.lierl@college.edu0225494|robert.nields|none|role_11|Robert|Nields|robert.nields@college.edu0000982|charalee.allen|none|role_11|Charalee|Allen|charalee.allen@college.edu0295293|david.hoctor|none|role_11|David|Hoctor|david.hoctor@college.edu0001393|colleen.meyer|none|role_11|Colleen|Meyer|colleen.meyer@college.edu00294456|clark.stull|none|role_11|Clark|Stull|clark.stull@college.edu0000577|julie.webster|none|role_11|Julie|Webster|julie.webster@college.edu0074937|george.armstrong|none|role_11|George|Armstrong|george.armstrong@college.edu

EXTERNAL_PERSON_KEY|USER_ID|SYSTEM_ROLE|INSTITUTION_ROLE|FIRSTNAME|LASTNAME|EMAIL0142270|sait.tarhan|none|role_11|Sait|Tarhan|sait.tarhan@college.edu0001129|marcha.hunley|none|role_11|Marcha|Hunley|marcha.hunley@college.edu0001613|steven.yelton|none|role_11|Steven|Yelton|steven.yelton@college.edu0003970|dorothy.mann|none|role_11|Dorothy|Mann|dorothy.mann@college.edu0099821|sherry.hutton|none|role_11|Sherry|Hutton|sherry.hutton@college.edu0000311|gary.webster|none|role_11|Gary|Webster|gary.webster@college.edu0173103|carla.gesell-streeter|none|role_11|Carla|Gesell|carla@college.edu0000612|olivia.watts|none|role_11|Olivia|Watts|olivia.watts@college.edu0000381|robert.eveslage|none|role_11|Robert|Eveslage|robert.eveslage@college.edu0000987|debra.lierl|none|role_11|Debra|Lierl|debra.lierl@college.edu0225494|robert.nields|none|role_11|Robert|Nields|robert.nields@college.edu0000982|charalee.allen|none|role_11|Charalee|Allen|charalee.allen@college.edu0295293|david.hoctor|none|role_11|David|Hoctor|david.hoctor@college.edu0001393|colleen.meyer|none|role_11|Colleen|Meyer|colleen.meyer@college.edu00294456|clark.stull|none|role_11|Clark|Stull|clark.stull@college.edu0000577|julie.webster|none|role_11|Julie|Webster|julie.webster@college.edu0074937|george.armstrong|none|role_11|George|Armstrong|george.armstrong@college.edu

Snapshot Integration Components

• 3 Primary Software Components– Generator (client developed) – Tool (Blackboard Learning System core)

– Controller (Blackboard Consulting deliverable)

Snapshot Controller

SISSISBlackboardBlackboard

SnapshotFiles

SnapshotGenerator

SnapshotToolSISSIS

SISSIS

Default Snapshot Configuration

Administrative Systems

SnapshotFiles

Blackboard Academic Suite™

Database

FTP

Snapshot Controller

SnapshotGenerator

SISSIS

SIS

SnapshotTool

JDBC

Remote Snapshot Operations

Campus System(s)

SnapshotData Files

Blackboard Client Blackboard Learning System

On Campus Remote (ASP)

FTP

Application

Snapshot Controller

SnapshotGenerator

SISSIS

SISSnapshot

Tool

SOAP Database

JDBC

Snapshot Security

Campus System(s)

SnapshotData Files

Blackboard Client Blackboard Learning System

On Campus Remote (ASP)

Database

Application

Snapshot Controller

SnapshotGenerator

SISSIS

SIS

SnapshotTool

VPN

HTTPS

SFTP, SCP

• Available/Unavailable vs. Enabled/Disabled– Availability (Y/N) is set via the GUI, Snapshot or

Event– Row Status (enabled/disabled) is set via

snapshot only.

Data States

Available_Ind Row_Status

User 1 Y Enabled

User 2 N Enabled

User 3 Y Disabled

User 4 N Disabled

See Integration Manual or Data Planning Workbook for detailed behavior of users, enrollments and courses.

• Manual (MANUAL)– Simplest mode, adds and updates records– Used to manage partial data sets.

• Snapshot (SNPSHT)– Adds, updates and disables records– Used to manage full data sets.

• Remove (REMOVE)– Removes records

• Copy Into* (COPYINTO)– Copies content between courses or organizations

Snapshot Tool - Modes

• Adding Records

Snapshot Tool – Modes

BbBbUser_ID Available_Ind Row_Status User_ID Available_Ind Row_Status

User1 Y enabled

User1 Y enabled

User1 Y enabled

ADDUser1 Y enabled

User2 Y enabled

User3 Y enabled

Adding records is the same in both Snapshot and Manual modes

Blackboard DatabaseSnapshot File

• Updating Records

Snapshot Tool – Modes

BbBb

Updating records is the same in both Snapshot and Manual modes

User_ID Available_Ind Row_Status User_ID Available_Ind Row_Status

User1 Y enabled

User2 Y disabled

User3 N enabled

User4 Y enabled

User1 Y enabled

User2 Y enabled

User3 Y enabledN

No Change

update

User4 Y enabled

disabledupdate

No Change

Blackboard DatabaseSnapshot File

BbBbBlackboard Database

Snapshot File

• Missing Records

Snapshot Tool – ModesThe difference between Manual and Snapshot is seen when records are no longer found in the file.

User_ID Available_Ind Row_Status User_ID Available_Ind Row_Status

User1 Y enabled

User2 Y enabled

User4 Y enabled

User1 Y enabled

User2 Y enabled

User3 Y enabledignore

User4 Y enabled

Manual mode just adds and updates.

disable

Snapshot mode disables missing records.

disabled

• ..\blackboard\apps\snapshot– \bin

• DSM – data source manager

• SNAPSHOT – snapshot tool

• SNAPSHOT_OVERRIDE – snapshot tool

– \config• ENV - connection info, class path and connection type

– \data• SNAPSHOT.PROPERTIES – primary snapshot configuration

• IMS-EP01.dtd – DTD for XML snapshots

Snapshot Tools

snapshot –V bbls.abc.edu –f CRS_SNAPSHOT –t ..\data\SIS_CRS_SP2003 –C ..\data\snapshot.properties

snapshot –V bbls.abc.com –f USR_MANUAL –t ..\data\SIS_USR

-C ..\data\snapshot.properties

snapshot –V bbls.abc.org –f STAFF_REMOVE –t ..\data\SIS_MEM_FA2003 -C..\data\snapshot.properties

snapshot –V bbls.abc.gov –f CRS_COPYINTO –t ..\data\SIS_CRS_SU2002 –C ..\data\snapshot.properties

snapshot -V <server name> -f <operation> -t <data file> -C <properties file>

Snapshot Tools – Command Lines

snapshot_override “-Ddata.source.key=DL_CRS” –V bbls.abc.edu –f CRS_SNAPSHOT –t ..\data\SIS_CRS_SP2003 –C ..\data\snapshot.properties

Snapshot_override “-Dmax.error.count=0” –V bbls.abc.com –f USR_MANUAL –t ..\data\SIS_USR

-C ..\data\snapshot.properties

Snapshot Tools – Snapshot Override

– Snapshot override allows you to override settings from the properties file.

# Maximum errors to process.# If you want all valid records to be processed, use -1 which indicates unlimited# If you want an all or none transaction, use 0# Otherwise, pick some reasonable number of errorsmax.error.count=-1

# Snapshot process settings

# delimteter used for parsing snapshot filesdata.delimeter=|

# Used to denote datasource to use for operation.data.source.key=SIS

# Determines if password field should be encryptedencrypt.password=Y

# Denotes string used to mark the beginning of errorserror.delimeter=(!)

# Determines character used to escape delimeterescape.character=/

# Designates flag to handle header validation for configuration property fileheader.validation=Y

# Determines if logged information should be sent to console.log.stdout=N

# Designates flag to establish handling of enumerated values.parse.allow.default=N

# Designates maximum size of database transactionsnapshot.batch.size=300

# Used to configure pause between persistence actionswait.length=-1

# Snapshot controlled field settings for Blackboard override of data

# fields for category entitiescategory.bb.controlled.fields=

# Maximum errors to process.# If you want all valid records to be processed, use -1 which indicates unlimited# If you want an all or none transaction, use 0# Otherwise, pick some reasonable number of errorsmax.error.count=-1

# Snapshot process settings

# delimteter used for parsing snapshot filesdata.delimeter=|

# Used to denote datasource to use for operation.data.source.key=SIS

# Determines if password field should be encryptedencrypt.password=Y

# Denotes string used to mark the beginning of errorserror.delimeter=(!)

# Determines character used to escape delimeterescape.character=/

# Designates flag to handle header validation for configuration property fileheader.validation=Y

# Determines if logged information should be sent to console.log.stdout=N

# Designates flag to establish handling of enumerated values.parse.allow.default=N

# Designates maximum size of database transactionsnapshot.batch.size=300

# Used to configure pause between persistence actionswait.length=-1

# Snapshot controlled field settings for Blackboard override of data

# fields for category entitiescategory.bb.controlled.fields=

• Operations Properties– delimiter, batch size, error

count– data source key

• Ownership– bb.controlled fields

• Course Copy Operations– content areas, settings,

memberships

Snapshot Properties

# Snapshot controlled field settings for Blackboard override of data

# fields for category entitiescategory.bb.controlled.fields=AVAILABLE_IND

# fields for link entitiescategorymembership.bb.controlled.fields=

# fields for group entitiesgroup.bb.controlled.fields=AVAILABLE_IND,COURSE_NAME

# fields for user entitiesperson.bb.controlled.fields=INSTITUTION_ROLE

# fields for membership entitiesmembership.bb.controlled.fields=ROLE

# Snapshot controlled field settings for Blackboard override of data

# fields for category entitiescategory.bb.controlled.fields=AVAILABLE_IND

# fields for link entitiescategorymembership.bb.controlled.fields=

# fields for group entitiesgroup.bb.controlled.fields=AVAILABLE_IND,COURSE_NAME

# fields for user entitiesperson.bb.controlled.fields=INSTITUTION_ROLE

# fields for membership entitiesmembership.bb.controlled.fields=ROLE

• Ownership = Control– Fields “owned” by Blackboard are not updated by

the snapshot process.– Fields not owned by Blackboard may be

overwritten by the snapshot.

• Ownership properties are set and stored in snapshot.properties– Fields owned by Blackboard

listed under “bb.controlled”– Fields are Specific

for Each Entity

Snapshot Properties - Ownership

Bb

User_ID FirstName LastName

User1 Joseph Smith

User2 Angela Freed

User3 Jennie Brown

User4 Frank Jones

User_ID FirstName LastName

User1 Joseph Smith

User2 Angela Freed

User3 Jennie Brown

User4 Frank Jones

1. Snapshot creates user records.

Smithe

Jones-Smith

2. Data is updated via the UI.

3. Data is updated in SIS.

4. Snapshot runs again and last names are updated, first names are left alone.

Freedman

Francis

User1 Joseph Smith

User2 Angela Freedman

User3 Jennie Brown

User4 Francis Jones

Joe

Angie

Jen

Frank

Ownership - Example

– FirstName is owned by Blackboard– LastName is owned by SIS

Blackboard DatabaseSnapshot File

• Data Source Keys can be used to separate and identify data from different sources.

SISSIS

AlumniAlumni

HRHR Blackboard

Data Sources and Data Source Keys

• Data Source Keys can also be used to separate and identify data from different logical groups.

FallFall

SummerSummer

SpringSpring

SISSIS

Data Sources and Data Source Keys

Data Sources and Data Source Keys

– Blackboard database records are mapped to a particular data source key.

– Anything entered via the GUI is mapped to the “SYSTEM” DSK.

course_ID courseName ext_crs_key dataSourceKey

Course1 Biology Bio01_02F2005 fa_crs_2005

Course2 History Hist03_06F2005 fa_crs_2005

Course3 French Fre01_05P2005 sp_crs_2005

Course4 French2 Fre02_01P2005 sp_crs_2005

Course5 Spanish Course5* SYSTEM

* Note: Course_ID & Ext_Crs_Key will be identical for GUI entered records.

Bb

DSK = “Law”

DSK = “Medical”

111-11-1111222-22-2222888-88-8888666-66-6666

555-55-5555444-44-4444777-77-7777333-33-3333

111-11-1111222-22-2222

888-88-8888

666-66-6666555-55-5555444-44-4444

777-77-7777

333-33-3333

MedicalMedicalLawLawLawMedicalLawMedical

external_person_key DSK

111-11-1111222-22-2222888-88-8888666-66-6666

The DSK ensures that only the “right”records are updated. Without DSK the “Law” records would be disabled when the “Medical”snapshot is processed.

555-55-5555444-44-4444

777-77-7777

333-33-3333 LawLawLaw

Law

Using Data Source Keys

dsm –V bbls.abc.edu –f CREATE –b BBCRS_FA2005 –d Fa05_Courses

dsm –V bbls.abc.com –f DISABLE_PERSON –b BBUSERS –t 20050305

dsm –V bbls.abc.org –f PURGE_MEMBERSHIP –b BBSTAFF_FA2005

dsm –V bbls.abc.gov –f MODIFY –b BBCAT –r BBCAT_FA05

DSM -V <server name> -f <operation] -b <data source key>

Only disabled records will be removed using the PURGE operation.

Snapshot Tools – Data Source Manager

Closing Slide

• Innovating Together in ‘05:– Recap top 2-3 points

• Resources Available:– Any links, resources available for attendees to learn more

• Follow up Contact(s):– Insert

• IF YOU ONLY REMEMBER 1 THING:– Insert