B. Test Results: Data Transformation Verification and ... filekaren211jan26.txt PMR-2-PR-I Verizon PA Ordering Trunks Sample Request (Data Request #20) OrdTrunksSample Request.doc,

Final Report – PMR2 Verizon Pennsylvania

Final Report as of December 22, 2000 591

Published by KPMG Consulting – CONFIDENTIAL For Verizon Pennsylvania, Inc. and Commonwealth of Pennsylvania Public Utility Commission use only

B. Test Results: Data Transformation Verification and Validation Review (PMR2) 1.0 Description The Data Transformation Verification and Validation Review (PMR2) evaluated the overall policies and practices for replicating and converting the data necessary to produce the performance metrics. The objective of this test was to determine the adequacy and completeness of procedures for replicating and converting the data necessary for the production of performance metrics. PMR2 utilized quantitative techniques to verify and validate Verizon PA’s data used in metric calculations in Pre-Ordering, Ordering, Provisioning, Maintenance & Repair, Billing, and Network Performance. 2.0 Methodology This section summarizes the test methodology. 2.1 Business Process Description Orders/troubles submitted into Verizon Pennsylvania (Verizon PA) systems are collected in several raw data systems. Verizon PA gathers raw data on the submitted transactions. This raw data are either filtered through additional systems or are used to populate Carrier-to-Carrier Reports directly. Verizon PA identifies “processed” data as the data that comes out of the last data collection point within its information systems. These data may have exclusions or other data manipulations applied to them since the time that the data entered its first collection point within Verizon PA systems. 2.2 Scenarios Scenarios were not applicable to this test. 2.3 Test Targets & Measures The test targets examined include all domains: Pre-ordering, Ordering, Provisioning, Maintenance and Repair, Billing, and Network Performance. Processes, sub-processes, evaluation measures, and associated test cross-reference numbers are summarized in Table 2-1 which follows. The last column, “Test Cross-Reference,” indicates where the particular measures are addressed in Section 3.1 “Results & Analysis.”




Table 2-1: Test Target Cross-Reference

Process Sub-Process Evaluation Measure Test Cross-Reference

Data Processing and Retention

Transfer of data from point(s) of collection

Adequacy and completeness of the data transfer process

PMR-2-1-1-A, PMR-2-1-1-B, PMR-2-1-1-C, PMR-2-1-1-D, PMR-2-1-1-E, PMR-2-1-1-F, PMR-2-1-2-A, PMR-2-1-2-B, PMR-2-1-2-C, PMR-2-1-2-D, PMR-2-1-2-E, PMR-2-1-2-F


Conversion of data from unprocessed to processed form

Adequacy and completeness of conversion policies and procedures

PMR-2-2-1-A, PMR-2-2-1-B, PMR-2-2-1-C, PMR-2-2-1-D, PMR-2-2-1-E, PMR-2-2-1-F


Internal Controls Adequacy and completeness of the internal control process

PMR-2-3-1-A, PMR-2-3-1-B, PMR-2-3-1-C, PMR-2-3-1-D, PMR-2-3-1-E, PMR-2-3-1-F




2.4 Data Sources The data collected for the test are summarized in Table 2-2 below.

Table 2-2: Data Sources for Data Transformation Verification and Validation Review

Document File Name Location in Workpapers Source

The Processed Data as provided to KPMG Consulting for the PMR5 Metrics Replication Test

Ordering domain: May 1999 and June, August, and September 2000 replication data

Ordering Trunks: July 1999 replication data

Provisioning domain: May 1999 and April 2000 replication data

M&R POTS: July 1999 and April 2000 replication data

M&R Specials & Trunks: May 1999, June and July 2000 replication data

Network Performance domain: August 1999 and April and July 2000 replication data

See PMR5. Verizon PA

Ordering Trunks Raw Data Files From the EXACT System

KPMGEXACT.doc, EXACTAUDIT.doc, KPMGEXACT0124.doc

PMR-2-OR-TR-I Verizon PA

Provisioning (State and Entity Table)

pa_npa.xls, PAENTSO.xls PMR-2-PR-I Verizon PA

Provisioning (Documentation)

SOPDOE2SORD.doc PMR-2-PR-I Verizon PA

Provisioning (RES/BUS FID Table)

karen211jan26.txt PMR-2-PR-I Verizon PA

Ordering Trunks Sample Request (Data Request #20)

OrdTrunksSample Request.doc, OrdProvTrunkRequest. doc

PMR-2-OR-TR-I KPMG Consulting





Network Performance Sample Request (Data Request #21)

NetworkPerfReqTrack Form.doc, NetworkPerfData Request.doc

PMR-2-NP-I KPMG Consulting

Data Integrity Questions About Ordering/ Provisioning Trunks (Data Request #49)

OrderingTrunksTrackFormNum49.doc, OrderingTrunksReq49. doc, OrderingTrunks Req49_BAresp.doc

PMR-2-OR-TR-I KPMG Consulting/ Verizon PA

Provisioning documentation

Soutrule.doc PMR-2-PR-II Verizon PA

Maintenance and Repair - POTS, Documentation

s97_cgi.htm, CLEC_hndbk_vol3s8_7. htm

PMR-2-MR-PT-I Verizon PA

Ordering - Documentation

PON198638A.bmp, Table of CLEC’s.txt

PMR-2-OR-OR-I Verizon PA

Ordering Trunks - Documentation

PROVISIONING SAMPL1.doc

PMR-2-OR-TR-I Verizon PA

Documentation and Diagrams Describing Verizon PA Data Flow: Billing

KPMG1PA.ppt PMR-2-BI Verizon PA

Documentation and Diagrams Describing Verizon PA Data Flow: Maintenance & Repair

mrkpmgs.ppt PMR-2-MR Verizon PA

Documentation and Diagrams Describing Verizon PA Data Flow: Ordering

Ord Metrics FCC PA.ppt PMR-2-OR Verizon PA


Order.ppt PMR-2-OR Verizon PA

Documentation and Diagrams Describing Verizon PA Data Flow: Pre-Ordering

Preord.ppt PMR-2-PO Verizon PA

Documentation and Diagrams Describing Verizon PA Data Flow: Provisioning

provision.ppt PMR-2-PR Verizon PA






so-M&R-lk.PPT PMR-2-MR Verizon PA

Documentation and Diagrams Describing Verizon PA Data Flow: Pre-Ordering, Ordering, Billing

SysFlowsSouth0299.ppt PMR-2-PO, PMR-2-OR, PMR-2-BI

Verizon PA


tnds-tk.vsd PMR-2-OR Verizon PA

Documentation and Diagrams Describing Verizon PA Data Flow: Ordering, Provisioning

trk sysflow.doc PMR-2-OR, PMR-2-PR Verizon PA


trksplmtc-ps.ppt PMR-2-MR Verizon PA

Archived WFA/DO Files Used in Calculating Inst7 and Inst30 (M&R POTS)

padoc1.doc, padoc11.doc, padoc12.doc, padoc13.doc, padoc14.doc, padoc15.doc, padoc16.doc, padoc17.doc, padoc18.doc, padoc19.doc, padoc2.doc, padoc20.doc, padoc21.doc, padoc22.doc, padoc23.doc, padoc24.doc, padoc25.doc, padoc26.doc, padoc3.doc, padoc4.doc, padoc5.doc, padoc6.doc






Ordering Sample Request - Data Integrity Ordering Re-test (Data Request #57)

OrderingSampleRetestRequest.doc, Ordering-ResampleTrackForm.doc

PMR-2-OR-OR-I KPMG Consulting

M&R POTS Sample Request - Data Integrity M&R POTS Re-test (Data Request #58)

MRPotsSampleRequest_retest.doc, MRPOTS-ResampleTrackForm.doc

PMR-2-MR-PT-I KPMG Consulting

Follow up Questions about Ordering Data Integrity Re-test (Data Request #59)

OrderingSampleRetest_ DR59.doc, Ordering-ResampleTrackForm_ DR59.doc, Ordering_resample_ discrepancies.xls


Ordering Raw Data File (Loc_Worklist Table, Sr_Log Table, Conf_Ckts Table, Confirmation Table, So_Status Table, and Requests Table) From the Request Manager System (Ordering Data Integrity Re-test)

pons_199_prod.txt PMR-2-OR-OR-I Verizon PA

Maintenance and Repair - POTS Raw Data Files (DLTH files) From the LMOS System (M&R POTS Data Integrity Re-test)

kpmg01-15.txt, kpmg1630.txt, kpmg3147.txt, kpmg4868.txt, kpmg6995.txt, kpmg96118.txt, kpmg119135.txt, kpmg136157.txt, kpmg158174.txt, kpmg175202.txt, kpmg203223.txt






Maintenance and Repair - POTS Raw Data Files (DLR files) From the LMOS System (M&R POTS Data Integrity Re-test)

dlr01-15.txt, dlr16-30.txt, dlr31-47.txt, dlr48-68.txt, dlr69-95.txt, dlr96-118.txt, dlr119-135.txt, dlr136-157.txt, dlr158-174.txt, dlr175-202.txt, dlr203-223.txt


Follow up Questions About M&R POTS Data Integrity Re-test (Data Request #63)

MRPots_retest_followup DR63.doc, MRPOTS-ResampleTrackForm_ DR63.doc, MRPOTS_retest_ discrepancies.xls, DLTHretest_TN_disc.txt, DLTHretest_missed_disc. txt, DLTHretest_dtr_ disc.txt

PMR-2-MR-PT-I KPMG Consulting

Provisioning Sample Request - Data Integrity Provisioning Re-test (Data Request #65)

ProvisioningSample Request_retest.doc, Provisioning-ResampleTrackForm_ DR65.doc

PMR-2-PR-I KPMG Consulting

Provisioning Raw Service Order Images (Provisioning Data Integrity Re-test)

KPMGAprilEAST.txt, KPMGAprilWEST.txt, KPMG_FQ_Orders.txt, order_D2GQ21510.txt, kpmg.txt, kpmg_missing_orders1.txt

PMR-2-PR-I Verizon PA

DCAS file (Provisioning Data Integrity Re-test)

PA_dcas_0400.txt PMR-2-PR-I Verizon PA

Follow-up Questions About Provisioning Data Integrity Re-test (Data Request #69)

Provisioning-ResampleTrackForm_ DR69.doc, Provisioning_retest_ followupDR69.doc, prov_retest_disc.xls






Data Request #71: Discrepancies Between KPMG Consulting CLEC Data (Production Orders) and Verizon PA Metrics data - Ordering

DI-KPMG_Production_ OrdersDR71.doc, DI-Production_OrdersTrackFormDR71.doc, Production_DI_disc1.xls



OrderingSampleRetest DR75.doc, Ordering-ResampleTrackForm DR75.doc


M&R Specials Sample Request - Data Integrity M&R Specials Re-test (Data Request #76)

MRSpecialsNEWSampleRequest.doc, MRSpecialsNEWSampleTrackFormDR76.doc

PMR-2-MR-ST-I KPMG Consulting


OrderingSampleRetestDR77.doc, Ordering-ResampleTrackFormDR77.doc


Questions About Verizon PA Provisioning Data Discrepancies (Data Request #78)

Provisioning_data_discDR78.doc, Provisioning_data_discTrackFormDR78.doc, VZN_KPMG_disc.xls


Raw Data Files From Request Manager for the Ordering Data Integrity Re-test (Data Request #75)

pons_south_aug2000.txt PMR-2-OR-OR-I Verizon PA

Raw Data Files From Request Manager for the Ordering Data Integrity Re-test (Data Request #77)

pons_south_sep2000.txt PMR-2-OR-OR-I Verizon PA

Raw Data Files From WFA/C for the M&R Specials Data Integrity Re-test (Data Request #76)

KGPMAUDITOSSCHI. doc, MP ITEMS 1-32.doc, PA ITEMS 33-74.doc, PD ITEMS 75-93.doc, or SN ITEMS 94-111.doc

PMR-2-MR-ST-I Verizon PA





Questions About Verizon PA M&R Specials Data Discrepancies (Data Request #79)

M&RSpecials_retest_questions_DR79.doc, M&RSpecials_retest_ TrackFormDR79.doc, M&RSpecials_June00_ disc.xls


M&R Trunks Request - Data Integrity M&R Trunks Re-test (Data Request #80)

MRTrunksDIRetest_ DR80.doc, MRTrunksDIRetestTrackFormDR80.doc


Raw Data Files From WFA/C for the M&R Trunks Data Integrity Re-test (Data Request #80)

kgmgosschi.doc, AC155835 - OSSTR.doc


Additional Raw Data Files From WFA/C for the M&R Specials Data Integrity Re-test

MP402411 - OSSTR.doc, MP131359 - OSSCHI.doc


Network Performance - Questions

Network Performance April 2000 Discrepancy Records.rtf, Revised April 2000 KPMG Discrepancy Records.xls

PMR-2-NP-I KPMG Consulting





Network Performance - Response

FW FW Network Performance April 2000 Discrepancy Records - Part 1 of 2.txt, FW FW Network Performance April 2000 Discrepancy Records - Part 2 of 2.txt, 40065AT&T.doc, 42743-2RHYTHMS.doc, 44190FOCAL.doc, 44244FOCAL.doc, 44281FOCAL.doc, 44304FOCAL.doc, 44696PRISM.doc, 44709PRISM.doc, 44718PRISM.doc, 45096HARVARDNET. doc, 45123HARVARDNET. doc, 45140HARVARDNET. doc, 45152HARVARDNET. doc, 45160HARVARDNET. doc, 45171HARVARDNET. doc, 45175HARVARDNET. doc, 45180HARVARDNET. doc, 45184HARVARDNET. doc, 45194HARVARDNET. doc, 45678Harvardnet. doc, 45698Harvardnet. doc, 45713HARVARDNET. doc, 45718Harvardnet. doc, 45723Harvardnet. doc, 45727Harvardnet. doc, 45762Harvardnet.doc,

PMR-2-NP-I Verizon PA





45733HARVARDNET. doc, 45740Harvardnet.doc, 45746Harvardnet.doc, 45748Harvardnet.doc, 45755Harvardnet.doc, 45756Harvardnet.doc, 45764HARVARDNET. doc, 45781Harvardnet.doc, 45800Harvardnet.doc, 45805Harvardnet.doc

Provisioning - request for explanation of discrepancies in interval calculations

E35_ProvIntervalCalcDiscTrackFormDR84.doc, appintv_cmpintv_disc.xls


2.4.1 Data Generation/Volumes This test did not rely on data generation or volume testing. 2.5 Evaluation Methods The PMR2 test used the unprocessed data collection points as identified in the Collection and Storage of Data test (PMR1). KPMG Consulting analyzed data from sub-systems that correspond to the rawest electronic form of data available. Stratified random samples of transactions of both Verizon PA retail and CLEC orders/troubles were created based upon the data used in the calculation of Metrics. These data, which KPMG Consulting refers to as “filtered,” or “processed,” have already been through Verizon PA information systems. Certain data fields that were used in the calculation of metrics were requested from the samples. The unprocessed data were compared to the data extracted from the major systems for metrics calculations for each domain. This comparison ensures that the data used in the calculation of performance metrics is both accurate and complete. Samples of May 1999 Provisioning, Ordering, and Maintenance & Repair Specials & Trunks orders/troubles, July 1999 Ordering Trunks and Maintenance & Repair POTS orders/troubles, and August 1999 Network Performance records were requested from Verizon PA. Additional samples were requested for certain domains (Ordering - June 2000, August 2000, and September 2000 data, M&R POTS - April 2000 data, Provisioning - April 2000 data, Maintenance and Repair Specials - June 2000 data, and Maintenance and Repair Trunks - July 2000 data) when existing discrepancies or the implementation of a new system created the necessity for a re-test




of the domain. Verizon PA extracted these transactions from the raw systems. These transactions, along with records/logs of the unprocessed data and descriptions of the systems utilized were delivered to KPMG Consulting. KPMG Consulting compared the unprocessed data to the processed data. Some of the data used in the calculation of Pre-Ordering, Provisioning Trunks and Billing Metrics are what KPMG Consulting considers the rawest form of data. “Raw,” or unprocessed data have not had any exclusions or any other data manipulation applied to them prior to their being used in the calculation of performance metrics. For these domains, KPMG Consulting did not develop samples. Each unprocessed log file/record was examined separately. KPMG Consulting processed the raw data into a form that more closely resembled the processed data used in calculating metrics. KPMG Consulting extracted and calculated the fields requested in the sample from the unprocessed data files based upon explanations and documentation received from Verizon PA. Once programs were written to extract and calculate all of the requested fields, KPMG Consulting was able to compare each unprocessed record to each corresponding record in the processed data. Non-matching records and other inconsistencies were noted. KPMG Consulting also compared its own records of KPMG Consulting CLEC transactions (i.e., number of records submitted, confirmation time received, etc.) to Verizon PA’s processed data. KPMG Consulting requested participation from CLECs to compare CLEC data to Verizon PA filtered data. While many CLECs expressed interest, only one volunteered to participate. This particular CLEC was unable to provide KPMG Consulting with the data required during the test time period. 2.6 Analysis Methods The Data Transformation Verification and Validation Review included a checklist of evaluation criteria developed by the Test Manager during the initial phase of the Verizon Pennsylvania OSS Evaluation. These evaluation criteria provided the framework of norms, standards, and guidelines for the Data Transformation Verification and Validation Review. The data collected were analyzed employing the evaluation criteria referenced above. 3.0 Results Summary This section identifies the evaluation criteria and test results.




3.1 Results & Analysis The results of this test are presented in Tables 2-3 through 2-8 below.

Table 2-3: PMR2 Evaluation Criteria and Results: Pre-Ordering

Test Cross-Reference Evaluation Criteria Result Comments

PMR-2-1-1-A The transfer of unprocessed records to processed records is complete.

Not Applicable

Pre-Ordering data used in the calculation of performance metrics are already in the rawest form. Data integrity was not performed on this domain.

PMR-2-1-2-A Verizon PA did not add inappropriate records to processed data.

Not Applicable

Pre-Ordering data used in the calculation of performance metrics are already in the rawest form. Data integrity was not performed on this domain.

PMR-2-2-1-A Unprocessed data were transformed accurately to processed data.

Not Applicable

Pre-Ordering data used in the calculation of metrics are already in the rawest form. Data integrity was not performed on this domain.

PMR-2-3-1-A Verizon PA has a complete and consistent internal control process to ensure that data are transferred and transformed properly.

Satisfied KPMG Consulting observed no instances in the Pre-Ordering domain of data being populated incorrectly while replicating the Pre-Ordering metrics. All instances of script/field name changes were well documented in program code and did not affect metrics.




Table 2-4: PMR2 Evaluation Criteria and Results: Ordering


PMR-2-1-1-B The transfer of unprocessed records to processed records is complete.

Satisfied KPMG Consulting analyzed samples of actual CLEC orders and observed no cases where an unprocessed record did not appear in the processed data used in calculating performance metrics.

Additional analysis was conducted for KPMG Consulting CLEC June and July 2000 orders, and KPMG Consulting found that the transfer of data from unprocessed to processed form was complete.

PMR-2-1-2-B Verizon PA did not add inappropriate records to processed data.

Satisfied KPMG Consulting analyzed samples of actual CLEC orders and observed no cases where inappropriate records were added to the processed data used in calculating metrics.

KPMG Consulting determined that the only anomalies identified within the processed data either did not affect the reported metrics or were testing artifacts.





PMR-2-2-1-B Unprocessed data were transformed accurately to processed data.

Satisfied Ordering: For the Ordering Data Integrity test for accuracy, KPMG Consulting requested a sample of 199 records and 22 fields from the rawest collection point of data for June 2000. Out of 4378 possibilities for matches (199 records X 22 fields), Verizon PA's unprocessed data matched 99% of the time with the processed data. While there is no industry standard that applies to the accuracy of performance metrics data, KPMG Consulting deems the 99% accuracy rate to be an acceptable level of performance. In order to test the service order classification data field, KPMG Consulting also requested a sample of 223 records and 1 data field from the rawest collection point of data for August and September, 2000. Out of 223 possibilities for matches (223 records X 1 data field), Verizon PA's unprocessed data matched 99.1% of the time with the processed data. While there is no industry standard that applies to the accuracy of performance metrics data, KPMG Consulting deems the 99.1% accuracy rate to be an acceptable level of performance. Ordering Trunks: For the Ordering Trunks Data Integrity investigation for accuracy, KPMG Consulting requested 25 records and 11 fields from the July 1999 rawest collection point of data. Verizon PA provided KPMG Consulting with screen prints from the EXACT system to verify each record against the processed data. Out of 275 possibilities for matches (25 records X 11 fields), Verizon PA's unprocessed data matched 99.64% of the time with the processed data.





While there is no industry standard that applies to the accuracy of performance metrics data, KPMG Consulting deems the 99.64% accuracy rate to be an acceptable level of performance.

Additional analysis was conducted for KPMG Consulting CLEC data records from June and July 2000. KPMG Consulting found that the transfer of data from unprocessed to processed form was 100% accurate.

PMR-2-3-1-B Verizon PA has a complete and consistent internal control process to ensure that data are transferred and transformed properly.

Satisfied Verizon PA follows satisfactory control procedures; most changes are well documented and do not affect metrics data. Exceptions are noted below:

Ordering Trunks: The month in which an ASR (Access Service Request) is included in the metrics calculation is dependent upon when the confirmation is issued. However, when an ASR’s confirmation date is changed, it goes into the new confirmation date’s month for metrics calculations. In other words, if an ASR previously had a July confirmation date, and was updated to show an August confirmation date, then this record would go into the calculation of August metrics for Ordering Trunks, not July. This practice could possibly result in some double counting of records by Verizon. If an ASR had a July confirmation date when the July metrics were calculated, and then was later updated to an August confirmation date, this same ASR would also factor into August metrics. Based upon additional analysis conducted on July, August, and September 1999 Ordering Trunks data files, KPMG Consulting found that less than 1% of data records were duplicates. Therefore, this would have a negligible impact on performance metrics.




Table 2-5: PMR2 Evaluation Criteria and Results: Provisioning


PMR-2-1-1-C The transfer of unprocessed records to processed records is complete.

Satisfied KPMG Consulting analyzed a sample of actual CLEC and Verizon PA retail customers’ orders from April 2000 and observed no cases where an unprocessed record did not appear in the processed data used in calculating performance metrics.

Additional analysis was conducted for June 2000 KPMG Consulting CLEC orders, and KPMG Consulting found that the transfer of data from unprocessed to processed form was complete.

PMR-2-1-2-C Verizon PA did not add inappropriate records to processed data.

Satisfied KPMG Consulting analyzed a sample of actual CLEC and Verizon PA retail customers’ orders from April 2000 and observed no cases where inappropriate records were added to the processed data used in calculating metrics.

Additional analysis was conducted for June 2000 KPMG Consulting CLEC orders. KPMG Consulting compared KPMG Consulting CLEC data to Verizon PA’s metrics data, and there were no instances of Verizon PA adding inappropriate records to processed data.





PMR-2-2-1-C Unprocessed data were transformed accurately to processed data.

Satisfied For the Provisioning test for the Data Integrity investigation, KPMG Consulting requested a sample of 250 records and 27 fields from the rawest collection point of data for April 2000. In the test, 26 fields were analyzed since Verizon PA could not provide the raw data behind the CRIS date data field. The CRIS date field is not used in the actual calculation of metrics; however, it is used in determining the cut of records used in the monthly metrics data. For example, all orders CRIS completed in May would fall into the calculation of performance metrics for May. The CRIS date field in the North is different from the CRIS date field that is used in the South. In the North, the CRIS date is the date that the service order was billed; however, in the South, the CRIS date can represent a number of different dates. It could be the date that the order was sent to the CRIS system for billing, or it could be the date that the order was received into the SOP system. It is not the date that the order was CRIS completed, as it is in Verizon North. Verizon PA cannot provide this field, as it exists only in the background of Verizon PA systems. There could be days, or even months between the CRIS date and the field completion date; therefore, it would be difficult for a CLEC to determine which month orders would fall into for metrics calculations. Upon further investigation, it was found that this issue has a minimal impact on the actual metrics calculations.





Out of 6500 possibilities for matches (250 records X 26 fields), Verizon PA's unprocessed data matched 99% of the time with the processed data. While there is no industry standard that applies to the accuracy of performance metrics data, KPMG Consulting deems the 99% accuracy rate to be an acceptable level of performance.

Additional analysis was conducted for June 2000 KPMG Consulting CLEC orders. KPMG Consulting found one instance out of 125 records analyzed where the application date field populated by Verizon PA was significantly different from Verizon PA's confirmation date field as well as the KPMG Consulting CLEC's LSR sent date field. Since this was only one discrepancy out of all of the records analyzed, this discrepancy would have a negligible impact on metrics calculations. Other than this one occurrence, KPMG Consulting found that the transfer of data from unprocessed to processed form was accurate.





PMR-2-3-1-C Verizon PA has a complete and consistent internal control process to ensure that data are transferred and transformed properly.

Satisfied KPMG Consulting observed cases where the processed data did not match the unprocessed data. In each of these cases, a data field value had been manually altered, but upon questioning Verizon PA about the discrepancies, KPMG Consulting received a detailed explanation about the changes that were made to the data, the representative who made the change, and the reason for the change. The entire process was well documented and easy to track.

KPMG Consulting also observed irregularities in the ‘application date’ field. KPMG Consulting analyzed Provisioning metrics data from October 2000 and found 18 cases out of 3677 where the application date occurred prior to the due date and/or completion date. This small percentage (0.49%) of irregularities had a negligible impact on the Provisioning metrics.




Table 2-6: PMR2 Evaluation Criteria and Results: Maintenance & Repair


PMR-2-1-1-D The transfer of unprocessed records to processed records is complete.

Satisfied KPMG Consulting analyzed samples of actual CLEC and Verizon PA retail customers’ trouble reports and observed no cases where an unprocessed record did not appear in the processed data used in calculating performance metrics.

KPMG Consulting also analyzed KPMG Consulting CLEC data from October 1999 and found no instances of records being inappropriately deleted from metrics calculations.

PMR-2-1-2-D Verizon PA did not add inappropriate records to processed data.

Satisfied KPMG Consulting analyzed samples of actual CLEC and Verizon PA retail customers’ trouble reports and observed no cases where inappropriate records were added to the processed data used in calculating metrics.

KPMG Consulting also analyzed KPMG Consulting CLEC data from October 1999 to test the completeness of Verizon PA’s Maintenance and Repair data. KPMG Consulting found no cases where inappropriate records were added to the processed data used for metrics calculations.





PMR-2-2-1-D Unprocessed data were transformed accurately to processed data.

Satisfied M&R POTS: For the Maintenance & Repair – POTS Data Integrity test, KPMG Consulting requested a sample of 223 records and 17 fields from the April 2000 rawest collection point of data. Since the same telephone number and trouble clear date combination can apply to more than one trouble report, the total number of records received and analyzed was 251 (for example, there could be two troubles cleared for telephone number 123-456-7890 on April 15, 2000). Out of 4267 possibilities for matches (251 records X 17 fields), Verizon PA's unprocessed data matched 98.2% of the time with the processed data.

While there is no industry standard that applies to the accuracy of performance metrics data, KPMG Consulting deems the 98.2% accuracy rate to be an acceptable level of performance.

M&R Specials: For the Maintenance & Repair – Specials Data Integrity investigation, KPMG Consulting used a sample of 111 records and 12 fields from the June 2000 rawest collection point of data. Out of 1332 possibilities for matches (111 records X 12 fields), Verizon PA's unprocessed data matched 99.7% of the time with the processed data. While there is no industry standard that applies to the accuracy of performance metrics data, KPMG Consulting deems the 99.7% accuracy rate to be an acceptable level of performance.





M&R Trunks: For the Maintenance & Repair - Trunks Data Integrity investigation, KPMG Consulting requested the 33 Trunks records and 12 fields that were used in the calculation of performance metrics for July 2000 from the rawest collection point of data. Out of 396 possibilities for matches (12 records X 33 fields), Verizon PA's unprocessed data matched 100% of the time with the processed data. KPMG Consulting deems this accuracy rate to be an acceptable level of performance.

M&R RETAS: System log data matches processed data for selected fields.

Additional analysis was conducted for October 1999 KPMG Consulting CLEC data records, and KPMG Consulting found that the transfer of data from unprocessed to processed form was 100% accurate.

PMR-2-3-1-D Verizon PA has a complete and consistent internal control process to ensure that data are transferred and transformed properly.

Satisfied For the Maintenance and Repair domain, KPMG Consulting observed that with one minor exception data was transformed correctly and completely between systems. Any changes made to the data were well documented and easy to track.




Table 2-7: PMR2 Evaluation Criteria and Results: Billing


PMR-2-1-1-E The transfer of unprocessed records to processed records is complete.

Satisfied Out of eight Billing metrics, KPMG Consulting analyzed BI-2 (Timeliness of Carrier Bill) data for the Data Integrity investigation. For the remaining Billing metrics, Verizon PA Billing data is already in its rawest possible form.

For the BI-2 analysis, KPMG Consulting analyzed Verizon PA’s UNE Paper bills data from August 2000 and compared it to the UNE Paper bills received by the KPMG Consulting CLEC. KPMG Consulting found that there were no UNE Paper bills received by the KPMG Consulting CLEC that were erroneously deleted from Verizon PA’s data used for metrics calculations.

PMR-2-1-2-E Verizon PA did not add inappropriate records to processed data.

Satisfied Out of eight Billing metrics, KPMG Consulting analyzed BI-2 (Timeliness of Carrier Bill) data for the Data Integrity investigation. For the remaining Billing metrics, Verizon PA Billing data is already in its rawest possible form.

For the BI-2 analysis, KPMG Consulting analyzed Verizon PA’s UNE Paper bills data from August 2000 and compared it to the UNE Paper bills received by the KPMG Consulting CLEC. KPMG Consulting found no cases where inappropriate records were added to Verizon PA’s data used for metrics calculations.





PMR-2-2-1-E Unprocessed data were transformed accurately to processed data.

Satisfied For the analysis on BI-2 data, KPMG Consulting compared the August 2000 bills received by the KPMG Consulting CLEC to Verizon PA’s BI-2 data. Verizon PA’s data showed that all bills were sent within the required 10-day timeframe. Analysis of the bills received by the KPMG Consulting CLEC confirmed that Verizon PA’s data was accurate.

The data for the remaining Billing metrics is already in its rawest form. Therefore, the evaluation of the accuracy of data was not performed on the data for the remaining metrics in this domain.

PMR-2-3-1-E Verizon PA has a complete and consistent internal control process to ensure that data are transferred and transformed properly.

Satisfied During the analysis of Verizon PA’s BI-2 data, KPMG Consulting found that Verizon PA’s data was transformed properly.

Table 2-8: PMR2 Evaluation Criteria and Results: Network Performance


PMR-2-1-1-F The transfer of unprocessed records to processed records is complete.

Satisfied See criteria PMR-2-1-2-F. The same comment applies because the problem identified below can also result in records being erroneously deleted from metrics calculations.

PMR-2-1-2-F Verizon PA did not add inappropriate records to processed data.

Satisfied KPMG Consulting analyzed samples of actual CLEC transactions and observed one case where an unprocessed record was added to the processed data used in calculating metrics.

KPMG Consulting found that Verizon PA may update the completion date field after the Carrier-to-Carrier Reports have been published. This practice could result in potential double-counting of records.

KPMG Consulting analyzed two versions of Network Performance data:





one version that went into calculating performance metrics for July 2000, and a second version that was extracted 2 months after the original extraction from the database. The second extraction of the data contained 146 records, whereas the original Network Performance data that was used in metrics calculations contained 147 records. One application had been classified as "complete" by Verizon PA and later was cancelled by the CLEC, so this record had been removed from the completions data. This one-record difference caused a negligible impact on the metrics calculations (a difference of less than 0.1%).

To remedy this problem, Verizon PA has indicated that it has implemented a change control on 12/10/00. This change eliminates any additions or deletions of records being made to Network Performance data after Carrier-to-Carrier results have been reported.

PMR-2-2-1-F Unprocessed data were transformed accurately to processed data.

Satisfied In the Network Performance test conducted on July 1999 data, KPMG Consulting compared Verizon PA’s Network Performance (NP-2: Collocation Performance) metrics data (both Initial Response and Completions metric data) to hard copy Collocation applications and found some discrepancies in data values for certain date fields. However, the date discrepancies did not impact the outcome of the records as being on-time or late for NP-2 metrics. Therefore, these discrepancies had no impact on metrics calculations.

KPMG Consulting also reviewed April 2000 Network Performance records and verified that selected data fields used in NP-2 metrics calculations were accurate.





PMR-2-3-1-F Verizon PA has a complete and consistent internal control process to ensure that data are transferred and transformed properly.

Satisfied For the Network Performance domain, KPMG Consulting observed that any changes made to the data were well documented and easy to track. The Customer Network Engineering (CNE) database that contains Network Performance Collocation data includes a Remarks screen that captures any changes made by users/employees and creates a history of the IDs of those who make changes to records. Errors and changes/fixes are tracked by the Remarks screen. KPMG Consulting reviewed samples of Remarks screens for several records and verified the existence of these audit trails.




This page is intentionally blank.

Documents

B. Test Results: Data Transformation Verification and ... filekaren211jan26.txt PMR-2-PR-I Verizon PA Ordering Trunks Sample Request (Data Request #20) OrdTrunksSample Request.doc,