1. INTRODUCTION

Assessment Master is one of the most comprehensive and user-friendly large scale online testing systems currently available on the national and international market. It has been developed based on SoNET’s extensive experience over the last 8 years in testing large numbers of students in geographically remote areas with diverse test conditions. A unique feature of the system is that it can deliver highly complex tests with simulations, multimedia and other rich features with minimal requirements on existing computer infrastructure in a secure and cost efficient manner. Assessment Master is a functionally complete system suitable for large volume testing with centralised management ideally positioned for SaaS (Software as a Service) operations, government operations or large educational institutions and associations. Most importantly Assessment Master is a well-rounded and mature software product that has been successfully used and proven in operation nationally in Australia and internationally in over 20 different countries. This overview briefly describes the process of creation, management and delivery of complex or simple electronic tests with a specific emphasis on delivery technology. The second part of the overview provides a brief functional description of the Assessment Master modules and the ICT environment in which the system operates.

2

2. ASSESSMENT MASTER: THE PROCESS

The Assessment Master platform is structured as a number of web based applications or modules that communicate via a single database. The process of test creation, management and delivery is depicted on the diagram below showing various system modules and interactions between them.

2.1. Administration Project and User Setup

The first step in the process is setting up and configuring the project structure, initial administrative accounts and users. Assessment Master has central administration module that caters for multiple organisations, departments, schools, test venues and scheduling of test sessions through a tree based folder structure. The system supports role based user management and security at any level in the hierarchy allowing for segmentation of data between projects, schools, organisational units or students while allowing aggregation of information at higher levels. Bulk import options from CSV are possible to speed up the initial process of loading in the information. SoNET will work with your organisation to plan the most effective structure that minimises the administration while maintaining security.

2.2. Test Creation and Internationalisation

Once the initial project structure has been setup it is time to begin developing the content. This functionality is provided through the Assessment Builder module which supports the creation of standard test instruments (e.g. multiple choice, open text entry, etc.) as well as special test instruments through customisation (integrated applications or simulations).

3

The item and test instrument builder module allows creation or import of question stimulus and on-screen instructions for single or multiple items. Stimuli can be provided in a text, web, .pdf, flash, video and audio format or in any combination of these formats. All test items are stored in a central repository that keeps track of all versions, including a user workspace where items can be modified and “played” with until they are determined suitable for being put back into the shared library. The Builder supports review committees or individuals and provides a simple interface to make comments on the content and work through a predetermined workflow. Section 3.1 contains the full list of item types supported. For complex interactive tests SoNET works together with you to support the creation of the content to your specifications. The typical workflow involved is as follows:

1. The client provides us with story board mock-ups (simple sketches) in word or power point along with a short informal description of the functional parts of each item.

2. SoNET will hold a conference call with the client to walk through the mock-ups and ensure the requirements are clear.

3. SoNET will then

a. build the items based on the story boards and as each module is completed it will be published it in a preview mode where it can be reviewed or

b. If the item type is such that can be added to the item authoring tool then it will be made available for the client to configure or edit the simple elements (such as text) as required.

4. Access to our issue tracking system will be supplied so item authors can log any changes required for each item in the event they are complex changes that cannot be supported through the builder.

5. We will address and release changes (frequency to be determined by the number of issues and complexity) and iterate through steps 3 & 4 until the items are functioning as appearing as the item author indented.

Once the items have been completed auxiliary pages such as warning, information, login, logoff and demographic collection pages can are developed and added. Every screen presented to the student including the log in screen can be designed and linked together through the builder. Item Authors can create adaptive tests through a drag and drop interface, where the answers and particular conditions can trigger another set of test instruments as well as looping and branching in the test. The following diagram shows the most common structure of a test. It is important to note that items are built up of assets and components allowing sharing between items. A common example of this is a stimulus that may have 5 or 6 multiple choice questions linked to it, each displayed on a new screen.

4

2.3. Administration Test and Student Management

Once a test is created it is published from the Assessment Builder module to the Administration module and can be made available to the test administrator who can then schedule test sessions, define the venue, time and any additional data through custom fields. An inbuilt wizard (see image below) is provided for downloading a template to bulk upload student information and assign the students to the test. The process contains validation steps and the ability to add additional supporting data that might be useful for reporting.

5

Tests can be open and closed at various levels in the hierarchy to prevent unwanted access and access can be provided to the school, university etc. for management of student passwords including the option to printout login lists. Prior to the delivery of the test a System Check page can be configured to capture information on the infrastructure at the testing venue including bandwidth, screen resolution, browser, OS etc. It is possible to design the page to act as a “survey” to collect any additional information you wish.

2.4. Assessment Delivery

Once the administration setup and sampling is complete focus moves over to the delivery of the tests. The Assessment Master delivery engine supports multiple methods of tests delivery. From practical experience and through SoNET’s experience with a number of projects it has been proven that one method does not fit all conditions and requirements. Due to a lack of control of external factors (network bandwidth, computer configurations, security) at the testing venues as well as level of complexity of assessments (multimedia, applications integration) different methods can be used to successfully deliver tests and provide uniform experience to all test takers wherever they are. Assessment Master supports four main test delivery methods:

1. Web test delivery via Internet using standard browser on a local computer or tablet device.

2. Offline test delivery using a USB memory stick that does not require any network connectivity and can run independently on a local computer with minimal interaction with local OS. All results are securely saved on the USB memory stick and automatically uploaded to the central database once a network connection is available.

3. Test delivery using thin client method (Citrix, Terminal Services, 2X) allows for highly secure on-line test runs with full lock down of the environment.

6

4. Appliance test delivery provides a highly controlled delivery method where the test administrator brings the test appliance to the local computer lab and connects it to the local network. Tests from the appliance can be delivered via web browser or thin client.

2.5. Marking and Reporting

This Assessment Scorer supports manual and automatic marking by allowing markers to review student results and artefacts and mark them. A blind peer review marking process is fully supported as well as preconfigured control scripts and back marking. Apart from the standard reports and data extraction processes Assessment Master supports customised reports that can be built using Crystal Reports or Stimulsoft reporting tools. Once created, the report can be easily published in the reporting module.

7

3. ASSESSMENT MASTER: FEATURES & BENEFITS

On the bloc diagram below is presented functional structure of the Assessment Master. Each building bloc of the system is described in more details in the sections that follow.

3.1. Assessment Builder

Assessment Builder provides the ability to create computer based assessments and training materials for use in the Assessment Delivery application. Assessments created using Assessment Builder are QTI and SCORM standards compatible, allowing for high level of interoperability and compliance within those international standards. The system uses universal WEB services to communicate with other components of the Assessment Master suite. This also provides for cross-compatibility and communication with other Learning Management Systems if required. The full graphical user interface (WYSWYG) requires minimal to no technical knowledge to operate. The design of this product is focused on intuitive usability including easy to use toolbars, drag and drop functions and user friendly actions such as undo, redo, real time preview using Delivery Engine and more. Help is available across the module, allowing also for editing and commenting its content subject to user permission levels. The following item types are supported:

• Text Response • Extended response • Open Response (Short) • Open Response (Long) - formatting, word count etc. • Multiple Choice (Single Select) • Multiple Choice (Multi Select) • Likert • Inline Choice • Image Hotspot • Word Spot • Drag and Drop / Matching Pairs • Gap Match

8

• Bubble Spot • Image Match • Audio • Video • Images • Image Editor • Mini Simulated Web Browser • Mini Simulated Tablet • Slide Show Maker • Web Site Maker • Animation Maker • Sortable lists/filterable content • Calculator • Ruler • Protractor • Timeline/Slider • Map Widget • Book Widget (book like content presentation with page navigation) • MS Word simulation widget • Any standard web based control • Trace a path through a Maze • Ordering (Drag and Drop sorting) • Connecting images/objects together

3.2. Assessment Admin

Security of the module is integrated across the Assessment Master Suite via the Assessment Admin, while independent logins can be also used if required. Rich administration functions are provided to fulfil all user requirements. Multilingual support is provided at the level of the package and its menus, and separately for the content. Rich translation functions and workflows are provided through Assessment Translator. The library of resources includes images, multimedia, templates and other reusable components – which can be modified as required, subject to access rights. The library is searchable, uses extended metadata and synchronises across the user groups, subject to their access rights. The Resource Builder allows for creating and modifying library resources for use within the assessment content. Templates are used to provide a common design across the assessment materials. Many ready to use Widgets are available. These can be simply dragged into the content, allowing for instant creation of high impact progress bars, clocks, choices, calculators, matches, text entries, hot spots, graphics orders, object positions, sliders and more. This provides for speedy and simple creation of powerful items.

3.3. Assessment Translator

Assessment Master supports full multilingual operations and creation and presentation of assessment items and stimuli in any language. The Assessment Translation module allows for easy management of translation of assessment items and related training materials between different languages. The module does not replace translation applications that translators are already comfortable with using. It is intended to manage the process of translating and reviewing contents and where no external translation tools are available, providing an interface to perform the translation of contents.

9

The system uses universal WEB services to communicate with other Assessment Master modules and integrated security, while independent logins can be also be used if required. The easy to understand and operate dashboard provides information on the progress of translation processes. The module makes the use of defined blocks of text, which can be reused after translation in order to maintain a high consistency of translated materials. A unique auto-population function is built in to make the process most effective. It displays any known translations that exist for each block of text, allowing translators to make their final choice including application of required changes. The translation process is supported by commenting functions including marking/unmarking comments as read. The translation workflow covers the process from the Request through to Translation in Progress, Verification Request and Verification Completed to Translation Completed. Full version control is provided through all stages. Translated assessment items can be previewed in fully interactive mode, to verify complete integrity of the translation. Integration of the translation processes makes Assessment Master the only known system providing not only internationalisation and multilingual capacity for items building and delivery, but also complete integrated workflow for geographically distributed translators. This results in high consistency and quality of the translating works and further cost reductions through application of the most effective translation process.

3.4. Assessment Delivery Engine

Assessment Master’s unique value lays in its variety of assessment delivery methods. Through extensive national and international experience in test delivery we believe that there is no single test delivery method that is appropriate for every testing project. Specifics of the target population, local infrastructure limitations, geographical distribution of participants, legislative requirements, funds available for testing – those are only examples of many factors influencing selection of the right delivery methodology. More importantly, even the same national tests may require more than one delivery method to meet objectives of reliability, cost efficiency, accessibility and general feasibility of testing. With this in mind, our unique Assessment Delivery Engine was designed to allow for a range of delivery methods – all of them being perfectly synchronised together and ready for seamless and parallel implementation across all types of assessment scenarios. Delivery methods include the following:

1. Secure (SSL) or non-secure WEB based test delivery through the web browser over the Internet. Most of the web browsers are supported (IE, Safari, Chrome, Opera, FireFox). This method is independent of the local operating system and provides high capacity delivery (tens of thousands of tests can be taken simultaneously). It is a high volume and scalable method delivering tests at the lowest cost.

2. The USB Memory stick delivery method does not require network/internet connectivity to deliver tests and it is suited for high stakes tests and for the areas of poor or no connectivity. Test results are stored on the memory stick in an encrypted form and can be automatically uploaded to Assessment Master database as soon as Internet connectivity is available. Multiple tests/candidates can be run from one memory stick. The Delivery Engine runs directly off the stick with no interference with a local PC operating system. This reduces IT administration overheads. This method can be compared with bringing paper tests to the classroom; in this case USB memory sticks replace the need for paper.

3. The thin client and/or virtual PC delivery method supports Microsoft RDP, Citrix ICA, 2X and other thin client technologies for the test delivery. This technology is normally used for secure

10

delivery of high stake tests with low level of concurrency (normally 50-100 concurrent runs per server). This is a relatively expensive, but highly secure method of testing.

4. The appliance method is akin to a mobile test centre where a test administrator brings Assessment Master mini server to the premises (e.g. computer lab at school or university) and connects directly to the local network. Tests can be run using web or thin client delivery method on the local LAN. No internet connection is required.

Each of these methods has distinctive features addressing specific physical, legal or environmental challenges - covering the most demanding assessment requirements.

3.5. Assessment Scorer

Assessment Scorer provides the ability to mark computer-based assessments delivered through the Assessment Master application. The design of the product is focused on intuitive usability, so no technical knowledge is required for its users (markers). On secure login, only those relevant functions are accessible depending on the users’ profiles. Assessment Scorer delivers four core functions: Training, Marking, Monitoring and Review. Rich and flexible configuration allows for setting up marking environments fulfilling virtually any marking scenarios and the most demanding requirements. This includes marking workflows with various seeding options, the ability to remark and double mark and more. Assessment Scorer can incorporate groups of markers and supervisors from different geographical locations, providing the ability to use the best available resources and to minimise the operational costs of setting up large central marking centres. The high performance and reliability of the Assessment Scorer in distributed marking environments had been proven many times in the past during mission critical, large marking assignments. The training function allows for simulating a full manual marking environment, monitoring the performance of markers and reviewing their marks against the supervisor’s marking style. This provides feedback to markers on required marking levels, resulting in a high consistency and uniformity of marking style and of marks across marking teams. The marking function allows for setting up and executing marking workflows – including various marking modes, methods of distributing items to markers, methods of serving items through the queues and more. Allowed score ranges for items are presented on the screen, ensuring the marking task is efficient, accurate and reliable. Many other functions including commenting on assessment items and marking them with flags (e.g. for further review) are also available. Apart from the manual marking method for essays, Assessment Scorer provides a unique capability for automated marking of essays using the external Artificial Intelligence service Intellimetric: www.vantagelearning.com/intellimetric. This process is very simple:

1. In the Assessment Builder an item has to be flagged to be scored using Intellimetrics and the marker/administrator enters the prompt code provided by Vantage Learning.

2. At the end of the test if there were any items that were flagged to use Intellimetrics their web service is invoked and the results returned are saved the same way as any other auto scored items.

11

The monitoring function provides the ability to monitor assigned manual markers in all aspects of their progress through the marking tasks. Ability to redistribute new responses to groups of markers and management of double marking is also provided through the monitoring interface. Reviewing scores assigned by markers is available to selected supervisors, with various levels of actions allocated as per specific requirements. Multilingual support is provided at the level of the package and its menus, as well as separately for the contents.

12

4. ASSESSMENT MASTER: TECHNICAL AND SECURITY INFORMATION

The Assessment Master platform has been designed to cater for Security, Integrity, Availability and Scalability. The following principles apply:

1. Security of the data and tests has been provided through use of a centralized database repository with role-based user access control. A double layer of firewalls (please refer to the diagram below) minimizes the risk of database penetration.

2. Internet communication is based on the SSL protocol that provides additional protection by using encrypted communication rather than plain text.

3. USB memory sticks are encrypted which further increases data protection and test confidentiality.

4. System and data integrity are ensured through the design of Assessment Master by using the best practices in database design, including three-layer application architecture (U/I, business, data).

5. In the case of a test interruption caused by loss of connectivity of any other reason Assessment Master will always record last complete transaction, meaning that a maximum of only one response can be lost during the testing interruption. The system will remember the student’s progress so that when the connectivity is resumed the test taker will be able to continue from the point of interruption.

6. Assessment Master is designed as a system with high availability resilient to peak demands and is easy to scale up. Normally SoNET guarantees 99.9% availability, subject to the availability of the technology used for the underlying infrastructure (servers, network, etc.). The software has been designed for cloud computing or SaaS (Software as a Service) delivery method and all of our clients have used the software in this manner.

The diagram below depicts the high level Assessment Master cloud environment architecture. Assessment Master can scale from one web and database server up to a virtually unlimited number of load balanced web servers and a number of clustered database servers.

To ensure performance for high volumes of testing we use the architecture shown on the diagram below. Multiple servers deliver: failover, performance and scalability required to service tens of thousands of concurrent users. Clustered database servers provide full failover and high performance in data integrity. The threshold load decides the number of required servers and the cloud management system will automatically create a new server in a matter of couple of minutes when load reaches defined threshold. Assessment Master primary cloud platform is in Amazon Web Services (AWS) cloud in Sydney and DR and failover platform is SoNET cloud in Melbourne and UK.

13

Assessment Master has been built in the html, PHP and SQL programming languages with very light web pages. We do NOT use Flash. This ensures high speed and low bandwidth requirements at the test taker’s end. The recommended server platform is MS Windows Server 2012 R2. We deliver Assessment Master from multiple sites (SoNET Australia cloud, AWS, SoNET UK cloud) where each of sites is potentially a disaster recovery (DR) site and ensures 99.99% availability of the system. Finally it is important to mention that it should always be possible to recover the database up to the point of the last transaction before the highly unlikely event of a database crash. Using cluster technology and active-passive failover the database crash is highly unlikely to happen. SoNET uses real time database replication and multiple instances of read databases between the production site and disaster recovery site to mitigate the risk of such an event.

14

5. APPENDIX A - ITEM EXAMPLES

1. Example of a point and click browser simulation. The mock browser is interactive with expandable

menus, new tabs, and clickable links. A set of correct inputs/actions is configured for automatic scoring.

2. Example of a poster/invitation creator simulation. Students can browse a series of webpages

using the interactive browser widget and then build an invitation by dragging images, setting backgrounds and adding/formatting text. The final artifact that is produced by the students is saved and scored manually through the Assessment Scorer module.

15

3. On screen calculator widget that can be used in mathematical test. It can be configured to be always visible as part of the item or can have its visibility toggled via a button on the interface.