Trainer evaluation project

BI Project• Background Information In business since 1997, SetFocus is the global leader in selecting, training, placing, and supporting

Microsoft professionals worldwide. Through strategic relationships, SetFocus develops programs that directly source skilled professionals to meet the immediate hiring needs of its partners.

• Business Problem After each course week, an administrator uses a windows application to generate evaluations that are

sent to the students via email. Students fill out the evaluations and submit them using a web based application. The data generated by the evaluations is tracked and used to tune and improve performance in an array of areas. The main categories that are evaluated are lecture week, project week, room rental, and career development. Some of the areas that are evaluated within the main categories are instructor performance, course materials, curriculum, infrastructure, and support. A grading scale is used and a cumulative grade is given for each specific area per class session.

The primary categories currently used are:•Lecture Week•Lab Week•Final Evaluation

And the subcategories include:•Trainer•Support•Materials•Curriculum•Infrastructure/grid•Career development

The end user will be able to select a grade for each individual question on the evaluation. The scale is based on the numbers 1 though 10 with 1 being the lowest grade and 10 being the highest. N/A = 11Currently, the training and support department uses a set of reports to measure and manage performance. As part of the performance improvement initiative, the training and support department would like to build various reports and dashboards.

Terminology

Reporting period / quarterRefers to calendar quarter. All courses with evaluation date sent falling within the calendar quarter will be reported.

DatabasesTwo primary databases that will be used for the source data are:NewSFCoursesSurveyEvaluation

Source Database: SurveyEvaluation

Source Database: NewSFCourses

Identify Facts/Measures

Based on the report mockups the primary facts from the source database were identified as:

ScoreEvaluations Sent/Evaluations ReceivedComments

Identify Dimensions/Hierarchies

We first identified the major dimension hierarchy:

TrackTimeOfDaySessionCourseQuestion

Identify Dimensions/Hierarchies

Date Dimension: We developed a Date Dimension to join to the Session Dates. We only wanted the dates down to the Day level.

Stored Proc: Populate DimDate

Data Warehouse – Data Diagram

Set Focus Evaluation Datawarehouse ETL Utilizing SSIS

Master Package

Dim Session Course Package

- This dimension will be the most active dimension.- This dimension will consolidate the SF Courses database- Several lookups to the data dimension are referenced and the course end date is the primary date reference in the database.

Fact Evaluation

The fact evaluation table will be used as a counter for completed vs. uncompleted evaluations.A validation was done on SessionID which creates an error data file.

Dim Question

Fact Score

SSAS Overview• No Stress Datasource– Uniform design– Referential Integrity

• Three Dimensions– Dim Session Course– Dim Question– Dim Date

• Two Facts– Fact Evaluation– Fact Score

DSV Diagram

Dim Session Course

Dim Date

Dimension Usage

Calculations

Solution Explorer

Summary

• Perform bulk of calcs/member in Cube– Success

• Consolidated DW = Simpler Cube– Worked well

• Phase II changes will depend on DW and Reporting needs– Phase I stable and usable as is

Reports

• Summary Drill Down Scores by Track, Time of Day, Session.

• Primary dashboard Latest week and quarter-to-date scores by track & sub-category with KPIs.

• Trainer vs. Overall Scores for a specific instructor/trainer for the current quarter compared to the overall Scores for all instructors in the same period.

• Trainer Scores for the past iterations (up to 10) for that course for that trainer .

• Trainer Date Range Evaluation scores within a date range .

• Trainer Detail Detailed report of evaluation scores on last day/night class taught.

* User should be able to access Trainer Detail report from Trainer and Trainer Date Range reports.

Summary Drill Down Report

• Developed using SSRS.• Shows [Average Score] by

track, time of day (day and evening) and session.

• Can drill down on Track. • Can collapse on Time of Day to

hide sessions.

Summary Drill Down Report

• Developed using SSRS.• Shows [Average Score] by

track, time of day (day and evening) and session.

• Can drill down on Track. • Can collapse on Time of Day to

hide sessions.

Measures].[Score] / [Measures].[Valid Scores]

[Valid Scores] is a derived fact based on the count of records with scores not null. case when [Score] is null then 0 else 1end

[Average Score] is a calculated member.

Summary Drill Down Report

• Developed using SSRS.• Shows [Average Score] by

track, time of day (day and evening) and session.

• Can drill down on Track. • Can collapse on Time of Day to

hide sessions.

Measures].[Score] / [Measures].[Valid Scores]

[Valid Scores] is a derived fact based on the count of records with scores not null. case when [Score] is null then 0 else 1end

[Average Score] is a calculated member.

Primary Dashboard

• Developed in SSRS.• Shows [Average Score] “Value”

for the latest week and quarter-to-date “QTD Value” for Track & Sub-Category “Indicator”.

• The goal is 9.2. • >= 9.2 is green

>= 9 and < 9.2 is yellow< 9 is red.

• % Responded reflects percentage of evaluations completed for the latest week and quarter-to-date.

• >= 75% is green >= 50% and < 75% is yellow < 50% is red

Trainer vs. Overall

Trainer vs. Overall

• Developed using PPS.• Tracks the [Average Score] for

a specific instructor/trainer over the current quarter compared to the overall [Average Score] for all instructors over the same period.

• Instructor is a drop-down single-select parameter.

• Current quarter is defined as the most recent quarter with data.

Trainer vs. Overall

• Developed using PPS.• Tracks the [Average Score] for

a specific instructor/trainer over the current quarter compared to the overall [Average Score] for all instructors over the same period.

• Instructor is a drop-down single-select parameter.

• Current quarter is defined as the most recent quarter with data.

Trainer Report (Sharepoint View) , sample data for Instructor AA

Objective of this report is for a trainer to see his or her past scores for a course (up to 10 in a year).

Parameters: Instructor Course

DimSessionCourseCourse & DimSessionCourseInstructor

WITH MEMBER [Measures].[ParameterCaption] AS [Dim Session Course].[CourseName].CURRENTMEMBER.MEMBER_CAPTION

MEMBER [Measures].[ParameterValue] AS [Dim Session Course].[CourseName].CURRENTMEMBER.UNIQUENAME

MEMBER [Measures].[ParameterLevel] AS [Dim Session Course].[CourseName].CURRENTMEMBER.LEVEL.ORDINAL

SELECT {[Measures].[ParameterCaption], [Measures].[ParameterValue], [Measures].[ParameterLevel]} ON COLUMNS ,

filter([Dim Session Course].[CourseName].children, sum( [LastYear], [Measures].[Average Score] ) > 0) ON ROWS

FROM ( SELECT ( STRTOSET(@DimSessionCourseInstructor, CONSTRAINED) ) ON COLUMNS FROM [SF Course Eval DW])

dsTrainer

SELECT NON EMPTY { [Measures].[Average Score] } ON COLUMNS,

NON EMPTY { tail(filter([Course End Date].[Date].[Date].ALLMEMBERS,[Measures].[Average Score] >0 ),10) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME ON ROWS

FROM ( SELECT ( STRTOSET(@DimSessionCourseCourse, CONSTRAINED) ) ON COLUMNS FROM ( SELECT ( STRTOSET(@DimSessionCourseInstructor, CONSTRAINED) ) ON COLUMNS FROM [SF Course Eval DW])) WHERE ( IIF( STRTOSET(@DimSessionCourseInstructor, CONSTRAINED).Count = 1, STRTOSET(@DimSessionCourseInstructor, CONSTRAINED), [Dim Session Course].[Instructor].currentmember ), IIF( STRTOSET(@DimSessionCourseCourse, CONSTRAINED).Count = 1, STRTOSET(@DimSessionCourseCourse, CONSTRAINED), [Dim Session Course].[Course].currentmember ) ) CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS

Action Tab of Series Properties allows us to link to other SSRS reports.

In this case we allow user to see the TrainerDetail Report by selecting any of the Markers on the chart.

Trainer Date Range Report (Sharepoint View) , sample data for Instructor AA

Parameters And Datasets

Parameters: Instructor Date Range (From : To)

FromCourseEndDateDate & ToCourseEndDateDate

WITH MEMBER [Measures].[ParameterCaption] AS [Course End Date].[Date].CURRENTMEMBER.MEMBER_CAPTION

MEMBER [Measures].[ParameterValue] AS [Course End Date].[Date].CURRENTMEMBER.UNIQUENAME

MEMBER [Measures].[ParameterLevel] AS [Course End Date].[Date].CURRENTMEMBER.LEVEL.ORDINAL

SELECT {[Measures].[ParameterCaption], [Measures].[ParameterValue], [Measures].[ParameterLevel]} ON COLUMNS ,

filter([Course End Date].[Date].children,[Measures].[Average Score] > 0) ON ROWS

FROM ( SELECT ( STRTOSET(@DimSessionCourseInstructor, CONSTRAINED) ) ON COLUMNS FROM [SF Course Eval DW])

Query Designer for Dataset dsTrainerRange

Here we are able to setup the parameters as a range .

Trainer Detail Report

This report shows trainer’s evaluation scores for last class, day or night. This Report Defaults to most recent date where there was an score.

=Fields!Session.Value & " " & Fields!Course.Value & " " & Fields!Date.Value

="Instructor: " & Parameters!Instructor.Label & " “ & Sum(Fields!CompletedFlagInt.Value, "dsEvals") & " Evals/" & Sum(Fields!FactEvaluationCount.Value, "dsEvals") & " Students"

=Sum(Fields!Score.Value)/Sum(Fields!Valid_Scores.Value)

SELECT NON EMPTY { [Measures].[Average Score], [Measures].[Score] , [Measures].[Valid Scores] } ON COLUMNS,

NON EMPTY { ( [Dim Session Course].[Course].[Course].ALLMEMBERS * [Dim Session Course].[Session].[Session].ALLMEMBERS * Tail( Filter([Course End Date].[Date].[Date].ALLMEMBERS, [Measures].[Average Score] >0 ), 1) * [Dim Session Course].[Track].[Track] * [Dim Question].[Question].[Question].ALLMEMBERS * [Dim Question].[Sub Category].children ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME ON ROWS

FROM [SF Course Eval DW] Where (STRTOMEMBER(@Instructor) ,STRTOMEMBER(@TimeOfDay)) CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS

SELECT NON EMPTY { [Measures].[Fact Evaluation Count], [Measures].[Completed Flag Int] } ON COLUMNS,

NON EMPTY { ( [Dim Session Course].[Course].[Course].ALLMEMBERS * [Dim Session Course].[Session].[Session].ALLMEMBERS * Tail( Filter([Course End Date].[Date].[Date].ALLMEMBERS, [Measures].[Average Score] >0 ), 1) * [Dim Session Course].[Track].[Track] )} DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME ON ROWS

FROM [SF Course Eval DW] Where (STRTOMEMBER(@Instructor) ,STRTOMEMBER(@TimeOfDay)) CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS

SELECT DimQuestion.Question, FactScore.CommentsFROM FactScore INNER JOIN DimSessionCourse ON FactScore.SessionCourseID = DimSessionCourse.SessionCourseID INNER JOIN DimQuestion ON FactScore.QuestionID = DimQuestion.QuestionID INNER JOIN DimDate ON DimSessionCourse.CourseEndDateID = DimDate.DateIDWHERE (FactScore.Comments IS NOT NULL) AND (DimSessionCourse.InstructorName = @Instructor) AND (DimSessionCourse.TimeOfDay = @DayEvening) AND (DimDate.Date = (SELECT MAX(DD.Date) AS Expr1 FROM DimDate AS DD INNER JOIN DimSessionCourse AS DSC ON DD.DateID = DSC.CourseEndDateID WHERE (DSC.InstructorName = DimSessionCourse.InstructorName)

Created a datasource to access DW relational database to extract Comments

="[Dim Session Course].[Day-Evening].[Day]"

Additional Attribute [Day-Evening] added to DimSessionCourse to declare Time of day filter .

The End