T/TAC I

Final PresentationGeorge Mason UniversityGraduate School of EducationImmersion ProgramMay 11, 2004

T/TAC I & II

T/TAC I

ONLINE TRAINING

SCHOOL IMPROVEMENT

T/TAC I & II

T/TAC I

SITE LAUNCH:Special Education CouncilAPRIL 20, 2004

T/TAC I & II

T/TAC I

T/TAC I - Online Training

T/TAC II - School Improvement

T/TAC I & II -Overview of T/TAC Online Site

T/TAC I & II

T/TAC I

SEMESTER II

Tasks to Address from Last Semester

• Functionality

– www.Workshops

• Documentation

– Workflow Process

– Quick Start Guide

– E-Learning Content Guidelines

– Accessibility Guidelines

– Online Tutorial

T/TAC I Goal

To improve the overall site usability and usefulness by providing best practices along with support mechanisms and guidance in delivering good meaningful online instruction.

Project Plan

• System Usability

• Content Development

System Usability

• Determine if T/TAC staff can easily upload their training content to the T/TAC Online site and uncover any potential difficulties with the upload process.

• Determine the best practices for the inclusion of multi-media by establishing which content formats will best meet the needs of the staff and the site's audience.

• Assess the effectiveness of a pre-production workflow process designed to assist the T/TAC staff in the preparation of their content for uploading

Content Development

• T/TAC staff will feel confident about their ability to develop quality online content in any of a variety of formats

• Content posted to T/TAC Online Training site will be developed based on best practices for online learning

• Identify practices used to develop and delivery good online training.

Formative Evaluation

Using formative evaluation to achieve the team goal by seeking all information necessary to help improve the system and its content.

Formative Evaluation

Types of evaluations conducted were:– One-to-One

– Expert Reviews

– Usability Testing

T/TAC I

USABILITY EVALUATION

Usability evaluation

• Experts’ perspective– Found several issues – Changed “Courses” to “www. workshops”

• Users’ perspective– Usability tests

Usability Test

A structured interview focused on specific features in a user interface.

Usability Testing Goals

• Conduct a Kirkpatrick Level 3 (performance) evaluation on the site’s Online Training section

• Determine how to improve the site

• Gather data for constructing tutorials

• Validate tutorials

Usability Testing Procedure

• Select representative users T/TAC staff at GMU

Three without the tutorial Two with the tutorial

• Select representative tasksView, create, edit, publish for

webshops & workshops.• Develop a testing script

to ensure consistent testing• Observe users performing the tasks

Gathering Usability Results

• Users were asked to think out loud during the testing

• Observers noted thoughts & users’ errors while performing tasks

• Observers noted areas of user confusion that could be addressed in tutorials

• Observers noted areas of the user interface that could be improved with some redesign

• Observers tracked the time of completing the tasks

Usability Testing Results

• Found several design issues that when addressed improved user performance

• Users completed tasks faster with fewer errors after completing the tutorials than without the tutorials

• Users’ ratings of ease of use & time required to perform tasks were higher for those who had used the tutorials first

T/TAC I

TUTORIALS

Staff Education

• Earlier Performance Analysis indicated staff desire for tutorials

• Usability testing confirmed the need for tutorials

• Two aspects of staff education were considered– System training– Content creation and conversion

Training Deliverables

• The evaluation plan that was then developed called for the following training deliverables:

– Quick Start Guide – Preproduction Workflow– Creating Online Workshops that Work:

Introduction – Getting Started: A Framework for Designing

Online Training– Removing the Teacher without Losing the

Student– Implementing Multimedia Elements

System Tutorials

• We started with two tutorials– How to build a Webshop– How to build a www.Workshop

• Our goal: – Measure the effect of having a tutorial as

opposed to not having a tutorial as part of our formative evaluation process.

– Collect feedback on the tutorial itself

• The tutorial was only given to half of the usability test subjects.

Creating The Tutorials

• We used feedback from the initial usability tests as guidelines to create the initial tutorials.

• We concentrated on making the instruction clearly understandable

Creating The Tutorials

Shown here are some of the visual devices we used

• Easy to see and locate step numbers

• Arrows pointing to buttons mentioned in the text

Creating The Tutorials

Another visual device we used was subtle shading and a cutout window to help focus the user on the area discussed in the tutorial.

The Feedback

We gathered a number of good suggestions. The feedback included but was not limited to items like:

• Breaking original tutorials in smaller sections and combining the sections into a Workshop

• Modifications to screen language and instructions for clarity

• Additional Visuals for clarity

• Suggestions for sequencing various parts of the tutorial

• Items to be defined for the end user

Preproduction Workflow

• We have also prepared a preproduction workflow checklist for content authors

• It will be designed as a job aid for planning and construction of Webshops and Workshops

• It presents ideas and things to think about in the areas of:– Project planning– Visual asset preparation and organization– Narrative creation– Copyright issues

Quick Guide

• A quick guide can now be accessed when a Webshop or a www.Workshop

• The Quick guide provides basic, textual instructions for creating content

T/TAC I

CREATING ONLINE WORKSHOPSTHAT WORK

Development Process

Phase One: • Analysis of Existing Workshop

Materials• Recommendations for Training

Focus

Phase Two:• Incorporate Findings into • Webshops and www.Workshops

Phase One – Expert Review

1.Identify

Content Types

2.Identify

Goal Types

3.Associate Goals with Techniques

4.Identify & Apply

e-learning Principles

1. Identify Content Types

Fact

Specific and unique data or instanceYES

Concept

Category with multiple examplesYES

Process

Flow of events or activitiesYES

Procedure

Task with step-by-step actionsNO

Principle

Task performed by adapting guidelinesYES

2. Identify Workshop Goals

Inform

Communicate informationYES

Perform – Procedure

Build procedural skillsNO

Perform – Principles

Build principle-based skillsYES

3. Link Teaching Techniques with Goals

Receptive – Inform Goals

Lots of information

Limited practice opportunities

YES

Directive – Perform Procedure Goals

Require frequent responses

Provide immediate feedback

NO

Guided Discovery – Perform Principle Goals

Job-realistic problems

Supporting resources

YES

4. E-Learning Principles

Multimedia Contiguous

Coherent

Multiple Modes

Personalization

Practice

Examples

Phase One: Summary of Results

Good Presentation Techniques• Well-organized• Capture attention• Effective use of case studies

and examples

Training Focus on Web Environment• Ground in theory• Personalization• Multi-media

T/TAC I

PHASE TWO: Incorporate Findings into

Webshops and www.Workshops

Phase Two

Create www.Workshop

Conduct One-to-one Evaluation

Conduct Level One Evaluation

Create www.Workshop

Creating Online Workshops that Work

Webshops:• Getting Started: A Framework for

Designing Online Training• Removing the Teacher without Losing

the Student• Incorporating Multi-media Elements

Getting Started: A Framework for Design

This Webshop provides a structure you can use when developing online training. It discusses what is the same and what is different and how your previous experience will help you.

Removing the Teacher without Losing the Student

This Webshop introduces six techniques you can use to help keep students involved in a dialogue with the training when you are not there.

Incorporating Multimedia Elements Effectively

This Webshop discusses how you can use multimedia elements to reinforce information transfer based on the theory of human cognition

Conduct One-to-one Evaluation

One-to-one Evaluation– Teachers review raw content– Logs used for recording: page being

reviewed, comments or suggestions and changes incorporated by the designer of Webshop

– Revisions are made and content is transferred to T/TAC Online in the form of Webshops

Conduct Level One Evaluation

Level One Evaluation– Purpose: To collect reactionary data– Surveys emailed to T/TAC staff from all

regions– Feedback:

• T/TAC staff were pleased with the content of the www.Workshop

• T/TAC staff members feel more confident in their ability to create Online training

T/TAC I

RECOMMENDATIONS

Tracking Site Issues

• Web-based quality assurance system for

tracking user experience issues

• Issues filtered by page, category, date

opened, status

• Issues sorted by severity, page, category

• Screen shots included to assist with

communicating the issues

Types of Quality Assurance Issues

• Accessibility – Violations of Section 508 or WC3 WCAG

• Anomaly - Unexpected behavior or bug

• Data Entry – Using text fields, check boxes, radio buttons, etc.

• Information Architecture – Structure of information & labeling

• Management – Issues for the site manager, not programmer

• Navigation - Intuitive navigation menus and buttons

• Printing – Acquiring print outs in the most useful form

• Search - Easy options for filtering and sorting data

• User Guidance - Clarity of instructions, messages & online help

• Visual Design - Professional appearance and layout

Future Recommendations

• Determine where the Webshops and the www.Workshops with information for T/TAC staff will be located on the system.

• Determine how staff will access Webshops and www.Workshops with staff-specific information.

• Develop additional tutorial webshops on using T/TAC Online as users indicate their needs

• Review and address the issues in the Quality Assurance Tracking System

T/TAC I

SITE LAUNCH:Special Education Council

APRIL 20, 2004

T/TAC I & II

Site Launch Activities

Audience AnalysisScope, Sequence, Strategy

Review SiteConsistencyOrientation Video

SurveyBrochureNew Tab – SOL Lessons

T/TAC I

VA DEPTMENT OF EDUCATION SPECIAL EDUCATION COUNCIL PRESENTATION

T/TAC I & II

Introduction to T/TAC Online

• A community linking people and resources to help children and youth with disabilities

• An online extension of T/TAC Services– Usage tracked the same way

• Categories and subcategories• Same forms to collect data• Track by region and category

Accessing the Site

• www.ttaconline.org

• Map of Virginia– Region– Drop-down county lists– Out-of-state

• No login required to visit most of the site - only training is tracked by user

T/TAC I

Q&A

T/TAC I & II

T/TAC I

Thank You!

T/TAC I & II