BTL Surpass Performance · PDF file3 Surpass Performance Report Introduction Surpass is BTL’s comprehensive e-Assessment solution and is used by a number of leading UK Awarding Bodies

Surpass Performance Report 1

Contact Details

Technical Contact Details

Andy McAnulla - Solutions ArchitectBTL Group LtdSalts WharfAshley LaneShipleyWest Yorkshire BD17 7DB

T: +44 (0)1274 203265E: [email protected]: www.btl.com

Disclaimer

The information contained within this document is related to Surpass release 7. Performance and architecture information may change over future releases. Performance information may also vary depending on the precise test conditions and configuration used.


Contents

Contact Details

Contents

Introduction

Surpass Background

Performance Test Infrastructure & Environment

Performance Test Scenarios and Results

Conclusion

Appendix 1: Assessment Delivery Server Analysis

Appendix 2: Surpass Architectural Overview

1

2

3

4

5

7

11

12

13


Introduction

Surpass is BTL’s comprehensive e-Assessment solution and is used by a number of leading UK Awarding Bodies to develop and deliver their on-screen testing programmes. BTL is proud to share the test results from its latest performance test undertaken on the Surpass platform. As part of our continuous improvement strategy and quality assurance processes, the Surpass performance testing is designed to determine how well the system handles both the current and future performance requirements of large Awarding Bodies.

The latest performance test was designed to focus on the core processes surrounding the production and delivery of e-Assessment, these being:

• Question production • Assessment creation • Assessment scheduling • Assessment delivery • Examiner marking • Centre reporting

In order to assess the performance of these core processes, BTL created programmatic simulations of real world business scenarios which incorporated realistic pre-loaded operational data. Each of the Surpass products (ContentProducer, ItemBank and SecureAssess) was then repeatedly and rigorously tested against these scenarios.

Specifically, the system was pre-loaded with:

• 400,000 rich question items • 3,000 exams • 1,000 qualifications • 10,000 centres • 20,000 centre users • 6 million candidates.

BTL’s approach for providing a high stakes e-Assessment service to Awarding Bodies is to deliver e-Assessments offline via a centre’s local area network (LAN). BTL has been using this approach for many years and it forms a core part of the Surpass architecture. However, with the increasing availability, capacity and stability of the internet within centres and the gradual market shift towards the Web 2.0 approach, BTL has seen a growing number of Awarding Bodies now also using the online web delivery available within Surpass. This provides the key advantage of a simplified deployment into centres and allows the Awarding Body to be even more flexible and responsive to the demands of their clients. Therefore, a key objective in this latest round of testing was to determine the Surpass platform’s ability to centrally serve an extremely high volume of concurrent assessments via a secure online delivery.

This report offers empirical evidence of the Surpass platform’s capabilities, relating to scale and performance, made relative to a ‘generic’ Awarding Body’s requirements and possible volumes. Although technical in nature, this report aims to be largely understandable by non-technical readers and therefore also provides high-level background information.


Surpass Background

The Surpass platform is designed to be a fully functioning, turnkey (on-screen assessment) solution, encompassing all of the tools an Awarding Body requires to offer on-screen assessment. The standard platform, out of the box, supports integration with centre MIS systems and provides a results export via XML. The Surpass platform is already used for high stakes assessments across the UK by organisations such as AQA, WJEC, SQA, EAL and AAT who are delivering GCSEs, A Levels, Diplomas, Higher and National Qualifications, as well as specialist qualifications in areas such as accounting.

Simple overview of the Surpass on-screen testing platform:

For further details on the Surpass architecture, and possible scaled infrastructure configuration, please refer to Appendix 2.


Performance Test Infrastructure & Environment

A dedicated server infrastructure was provisioned and used for the majority of the tests detailed in this report. Load-balanced web servers were utilised using Microsoft’s NLB service. An Active/Passive SQL 2008 database cluster was also configured to test failover for resilience. A SAN was used by both the web servers and the database servers.

A diagrammatic representation of the provisioned infrastructure is shown here:

Web Server 2

Web Server 1 NLB Service

Client TestAgent(s)

DB Server 1(Active)

DB Server 2(Passive)

SAN

2e

DB Server 2(Passive)

1

DB Server 1(Active) (

ResilientInstance of

MS SQL

g

r WNLB S1 ervice 2e

Due to the security of the SecureAssess delivery process, and in order to test the Surpass platform at the volumes and level of detail required, it was necessary for BTL to write our own performance test application. This application was written in .NET and offered the ability to script ‘actions’ to perform against the Surpass platform, using multiple threads and recording the results (similar to JMeter or LoadRunner).

The performance testing was focused on the server side application performance of the Surpass platform. Therefore all testing was undertaken using a 100Mb connection to the server to exclude the impact of end user bandwidth.”


Hardware Infrastructure Detail:

• x2 Network Load-balanced Web Servers each of the following specification:

- Windows Server 2008 Enterprise Edition 64-bit - HP DL360 G5 Xeon 2.5Ghz Quad Core - 4GB DDR2 memory - 1x HP 72GB SAS Hard Disk

• x1 Active Database Server of the following specification:

- Windows Server 2008 Enterprise Edition 64-bit - 2x HP DL360 G5 Xeon 2.5Ghz Quad Core - 32GB DDR2 memory - 1x HP 146GB 15000rpm SAS Hard Disk - HP Fibre Channel Host Adaptor

• x1 Passive Database Server of the following specification:

- Windows Server 2008 Enterprise Edition 64-bit - 1x HP DL360 G5 Xeon 2.5Ghz Quad Core - 16GB DDR2 memory - 1x HP 146GB 15000rpm SAS Hard Disk - HP Fibre Channel Host Adaptor

• x1 SAN of the following specification:

- Single fibre channel host controller - RAID 5 Array – 5 x 146GB 15000rpm SAS Hard disks

• Variable number of desktop computer test agents of the following minimum specification:

- 2.0 GB RAM - 2.0 GHz CPU - Windows XP or Vista - 100Mb/s NIC


Performance Test Scenarios and Results

Each ‘real world’ business scenario was programmatically broken down into a series of activities and corresponding KPIs. A time-based “Critical Success Factor” was agreed for each KPI to determine whether the product was performing within reasonable tolerances.

Question Production (Creation of item banks and associated items)

“ContentProducer”, which is used to create and edit questions ready for use in the assembly of live exams (by publishing into “ItemBank”), was proved to remain highly responsive whilst under a load of 100 users creating and editing question items simultaneously.

The system was pre-loaded with 400,000 realistic question items, to simulate typical load under operational conditions. Under this load, creation of a new question took less than 2 seconds and ‘checking-out’ a question for editing took only 0.57 seconds.

The high-level results of this test are as follows:

Step KPI Critical Success Factor Actual Response Time (Average)1 Login < 5 seconds 0.17 seconds2 Open Project < 10 seconds 0.38 seconds3 Check-out Question Item < 3 seconds 0.57 seconds- Pause for 5 minutes4 Check-in Question Item < 3 seconds 2.39 seconds5 Create New Folder (bank) < 3 seconds 1.48 seconds6 Create New Question Item < 3 seconds 1.54 seconds

Assessment Creation (Creation of assessments from finalised items)

“ItemBank”, which is used to assemble assessments from finalised items (published from ContentProducer), was tested against a scenario of 25 users simultaneously creating, opening, editing and saving assessments.

The system was pre-loaded with 400,000 realistic question items, 1,000 qualifications and 3,000 exams. The system remained capable of saving new exams in less than 1 second under this load.


Step KPI Critical Success Factor Actual Response Time (Average)1 Login < 5 seconds 0.06 seconds2 Create Exam < 5 seconds 1.14 seconds3 Open Exam < 5 seconds 0.55 seconds- Pause for 1 minute4 Save & Close Exam < 5 seconds 0.96 seconds


e-Assessment Scheduling (Scheduling of assessments)

This performance test relates to the Surpass “SecureAssess” product. SecureAssess is used to manage the scheduling, delivery and marking of e-Assessments (from exams assembled in “ItemBank”). SecureAssess was tested against a realistic business scenario of 200 users, repeatedly and simultaneously scheduling 50 candidates to take an exam. The test revealed that, even under this load, the system was readily capable of scheduling 200,000 candidate assessments to be sat immediately.

All scheduled e-Assessments were automatically created to be unique for each candidate, using the sophisticated rules based algorithm employed by ItemBank.

The system was pre-loaded with 6,000,000 candidates; 10,000 centres, 20,000 centre users, 1,000 qualifications and 3,000 exams, to simulate typical load under operational conditions. Under this load, finalising a new scheduled e-Assessment took just over 4 seconds.


Step KPI Critical Success Factor Actual Response Time (Average)1 Login < 5 seconds 3.62 seconds2 Get Centres List < 5 seconds 3.34 seconds3 Get Qualifications List < 5 seconds 3.63 seconds4 Get Exams List < 5 seconds 3.65 seconds5 Get Cohort Tree < 5 seconds 3.45 seconds6 Get Candidates < 7 seconds 4.67 seconds7 Prepare Exam for Scheduling < 5 seconds 4.41 seconds- Pause for 1 second8 Save Scheduled Exam < 5 seconds 4.05 seconds9 View Saved Scheduled Exams < 5 seconds 4.42 seconds- Pause for 30 seconds


e-Assessment Delivery(Delivery of candidate assessments, online and offline)

SecureAssess was tested against a realistic business scenario of 18,375 candidates sitting an exam at the same time, over a 90 minute period. Four exam types were required, with an even distribution among candidates to simulate realistic delivery conditions. The four exam types used were as follows:

1. Sample multiple-choice only exam (all items included a 50KB graphic) 2. Sample multiple-choice only exam (no graphics)3. Sample mixed multiple-choice and free text (examiner marked) exam (all items included a 50KB graphic) 4. Sample mixed multiple-choice and free text (examiner marked) exam (no graphics)

It should be noted that although multiple-choice and free-text questions only represent a fraction of the question types available, they do represent the highest typical volume.

The system was pre-loaded with 6,000,000 candidates, 10,000 centres, 20,000 centre users, 1,000 qualifications and 3,000 exams to simulate typical load under operational conditions.

All 18,375 candidate exams were started and completed during the execution time of this test and the candidate experience remained very responsive. At these volumes, the performance test agents themselves reached their physical limit before any SecureAssess limits were found. Therefore, even higher volumes of simultaneous assessment delivery are thought to be possible. The testing also revealed that the act of scaling the system, through the use of load-balanced web servers and improved database server hardware, also considerably improved performance.

For further detail of the server performance under this load, please refer to Appendix 1.The high-level results of this test are as follows:

Step KPI Critical Success Factor Actual Response Time (Average)1 Login

(Enter Keycode)< 5 seconds 1.00 seconds

2 Confirm Details < 5 seconds 0.59 seconds3 Open Exam < 10 seconds 1.74 seconds4 Check Exam Status

(60 second timer)< 3 seconds 1.38 seconds

5 Move Between Questions– at 10 min peak full concurrency(x40 items with 90 second pause)

< 7 seconds GET: 3.66 seconds

SET: 2.44 seconds

6 Open Help(480KB)

< 5 seconds 3.11 seconds

7 Submit & Close Exam < 5 seconds 0.61 seconds

N.B. Due to the architecture of the offline version of SecureAssess (“SecureAssess Local”) being the same as the online version, it is theoretically capable of delivering the same volumes of exams if using similar or equivalent hardware. Therefore, it was not necessary to test the offline delivery capabilities.


Examiner Marking (Examiner marking of candidate responses)

The performance test relating to examiner marking showed that when 100 examiners were simultaneously marking candidate scripts in SecureAssess, the system was capable of fetching questions within 0.01 seconds and saving examiner marks in 0.03 seconds. Clearly, far higher volumes would be possible if required.

The system was pre-loaded with 6,000,000 candidates, 10,000 centres, 20,000 centre users, 1,000 qualifications and 3,000 exams, to simulate typical load under operational conditions. A total of 3,000 questions were marked during the execution time of this test.


Step KPI Critical Success Factor Actual Response Time (Average)1 Login < 5 seconds 0.05 seconds2 Get Exams < 5 seconds 0.92 seconds3 Get Item Script < 5 seconds 0.01 seconds- Pause for 1 second4 Assign Mark to Item < 5 seconds 0.03 seconds- Repeat Step 3 & 4 for 30

questions5 Submit Exam Script < 5 seconds 0.02 seconds

Centre Reporting (Producing standard reports against the available results)

The performance test relating to assessment reporting required 100 users to simultaneously generate candidate result reports for 50 candidates at a time.

The system was pre-loaded with 6,000,000 candidates, 10,000 centres, 20,000 centre users, 1,000 qualifications and 3,000 exams, to simulate typical load under operational conditions. A total of 40,000 candidate result reports were generated by the system, within a 10 minute period, during the execution time of this test. Under this load, generating a report for 50 candidates took less than 4 seconds.


Step KPI Critical Success Factor Actual Response Time (Average)1 Login < 5 seconds 0.08 seconds2 Get Centres List < 5 seconds 0.02 seconds3 Get Qualifications List < 5 seconds 0.29 seconds4 Get Exams List < 5 seconds 0.02 seconds- Pause for 30 seconds5 GetChoosenDistinctCandidates < 5 seconds 0.98 seconds6 Get Report XML < 7 seconds 3.27 seconds- Pause for 30 seconds


Conclusion

The results of this performance testing have revealed that the Surpass platform is capable of meeting and exceeding all of an Awarding Body’s anticipated business requirements, when the service has been scaled appropriately.

The testing has shown that it is possible to deliver over 18,000 concurrent online candidate exams, which would equate to over 140,000 online exams per day. There is also evidence that, by scaling the system through the addition of more hardware, the system reacts well and can deliver more concurrent exams and is, therefore, a safe and scalable platform.

Even through an online delivery, the candidates will experience an extremely resilient exam delivery as their answers are saved every 60 seconds, or every time they move between questions. If the candidate was required to move computers during their exam, they would simply be able to log back in and continue from where they left off.

It should be noted that if we were to follow an approach of offline delivery, using the SecureAssess Local software, then there is no overall practical limit to the volume of concurrent exams that could be delivered due to the distribution of load.

By completing the testing process, BTL has demonstrated that their technology is advanced, robust and highly reliable, which means it will consistently meet and exceed even the most demanding e-Assessment requirements. BTL recognises that this technology is crucial to an organisation’s success. BTL hope that this report will reinforce confidence in the technical capacity of the Surpass platform, and that they can continue to assist Awarding Bodies as they strive to lead the market.


Appendix 1: Assessment Delivery Server Analysis

The graph below displays the performance of the database server for the full duration of the ‘Assessment Delivery’ test.

As can be seen, there are no deadlocks occurring and the lock waits/sec are minimal during the main part of the test, when over 18,000 candidates are sitting an exam (between 6.55pm and 7.05pm). The lock wait times begin to increase towards the end of the test (approximately 7.15pm) due to all the candidates completing their exams and the server marking their results (a process which can potentially be delayed until a later time when the server is under less load). However, these lock wait times are within an acceptable range.


Appendix 2: Surpass Architectural Overview

This report does not aim to detail the full architecture of the Surpass platform. However, some high-level information has been included for the sake of completeness. The Surpass platform is made up of three independent products that communicate with each other through the use of SOAP web services. A high-level diagram of the communications between the products is shown below.


Infrastructure

The Surpass platform is capable of being deployed in a variety of configurations. The diagram below details what is likely to be the most scaled up, separated configuration.

Documents

BTL Surpass Performance · PDF file3 Surpass Performance Report Introduction Surpass is BTL’s comprehensive e-Assessment solution and is used by a number of leading UK Awarding Bodies