http://nodal.ercot.com 1Lead from the front

Texas Nodal

EDS 3 Release 5: SCED

Phase 1 Testing

Aug 14, 2007

http://nodal.ercot.com 2Lead from the front

Texas Nodal

Terminology: Black Box vs. White Box Testing

• Black box testing takes an external perspective of the test system to derive test cases. These tests can be functional or non-functional, though usually functional. The test designer selects valid and invalid input and determines the correct output. There is no knowledge of the test object's internal structure. “Simulate Market Participant interaction”

• White box testing uses an internal perspective of the system to design test cases based on internal structure. It requires programming skills to identify all paths through the software. The tester chooses test case inputs to exercise paths through the code and determines the appropriate outputs. “Validate Integration is working correctly”

http://nodal.ercot.com 3Lead from the front

Texas Nodal

MMS

EDS Phase 1 Release Testing

• The EDS team will use a Black Box approach to test the EDS system simulating the Market Participant interaction.

• The set of valid and invalid test cases that will be executed during the phase 1 Support Window will be run simulating 3 QSEs to ensure system stability.

• Any critical functionality issues that would negatively affect phase 1 progress will be fixed prior to phase 1 while non-critical functionality issues will be prioritized along with internal testing variances.

The goal is to complete testing and critical issue fixes prior to 8/24.

EWSNotification

UI

Va

lida

tion

MP

http://nodal.ercot.com 4Lead from the front

Texas Nodal

ERCOT Internal Functional and Integration Testing

• The INT team will use a White Box testing approach to perform functional testing against the External Web Services infrastructure.• Internal checkpoints and database validation checkpoints

• In addition, the team will test the Integration of the Web Services with the SCED (MMS) system.• Test submission and validation rules through EWS

• This testing will use valid test cases as well as extensive set of invalid test cases to test the validation rules.

MMS

EWS

UI

ValidationEngineMP Notification

Transformation

http://nodal.ercot.com 5Lead from the front

Texas Nodal

EDS Readiness Test Cases Examples

• Positive Test Cases for EDS 3 Submission Items (3PO, COP, OS, Inc/Dec)

– Submit, Query, Update, Cancel*– Some complexity (multiple: resources, curves, intervals, statuses,

limits, capacity)*– Validate synchronous response and asynchronous notification– Validate through MMS User Interface

• Negative Test Cases– Validate QSEs can only view and submit on their own resources– Create XML submissions to fail key validation rules (valid resource,

limits, status, required items, invalid dates, …)– Validate synchronous response and asynchronous notification

provide error messages

* Some variations only apply to certain submission items

http://nodal.ercot.com 6Lead from the front

Texas Nodal

Internal Functional and Integration Testing Examples

• Business Test Cases– Positive Path Testing – white box (approx 20%)

• Same submission items for EDS 3• Extensive complexity variations (multiple: resources, curves, intervals, statuses,

limits, capacity)• Extensive variations of validation rules and optional fields• Over-writing behavior• Validate Notification Service

– Negative Path Testing for each submission item (approx 80%)• Invalid data input types (market, trading day, QSE, resource name, ownership,

start date/time, end date/time, date variations, expiration date/time)• Rejection due to overlapping offers, submission time, no energy offer curve,

missing elements, …• Invalid energy offer curve components (above/below cap, non-monotonically

increasing, resource specific floor/cap violation)• Invalid resource (not combined cycle, RMR)• Review asynchronous error notifications

http://nodal.ercot.com 7Lead from the front

Texas Nodal

Internal Functional and Integration Testing Examples

• Technical Test Cases– EWS Security Testing

• Invalid: source, User Id, DUNS number• Test: digital signatures, certificate, certificate revocation, timestamp and nonce• Replay detection (valid / invalid / null nonce, created time)

– EWS Transformation Testing• Validate each submission item (submit, update, query, cancel*)• Validate database entries

– EWS XSD validation• Validate optional fields

– General Payload• compression, format tag, verb / noun, • mRID usage for all submission items

http://nodal.ercot.com 8Lead from the front

Texas Nodal

Internal Functional and Integration Testing Examples

• Technical Test Cases (cont.)– MMS input data validation

• Validate each submission item (submit, update, query, cancel*)• Validate optional fields

– Notification Service• Data Load Failure (staging & destination tables)• Offer and Bid Set Acceptance / Errors

– System Status Web Service• Web Service valid, error

– Get mRID Web Service• Web Service valid, error• mRID usage for all submission items

http://nodal.ercot.com 9Lead from the front

Texas Nodal

Estimated Number of Test Cases & Scripts

• Business Test Cases– Test Cases

• Approx 320

– Test Scripts• Approx 700

• Technical Test Cases– Test Cases

• Approx 120

– Test Scripts• Approx 250

• Approximately 20% are planned to be executed prior to EDS 3 kickoff on 8/24

http://nodal.ercot.com 10Lead from the front

Texas Nodal

Questions ?