7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 1/31
CROS NT srl
Contract Research Organisation
Clinical Data Management
Statistics
Dr. Paolo Morelli, CEO
Dr. Luca Girardello, SAS programmer
Best Practice in SAS
programs validation. A CaseStudy
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 2/31
AGENDA AGENDA
Introduction
Program Verification: a Business Approach
Program Verification: some case studies
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 3/31
FACTSFACTS aboutabout CROS NTCROS NT• Headquarters in Verona (Italy)
• Founded in 1993
• Offices in Milan and Munich
• 40 employees
• Data Management, Statistical, PhV and
hosting services• Services to Pharma, Biotech and CROs
• Cooperation with Universities of Padua, Bologna, Milan
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 4/31
IntroductionIntroduction
• Topic of the presentation: how to maximize the quality of
programming while minimizing the time to verify program.
• In the first part of the presentation we will discuss about the
business part:
What is program verification?
Why program verification is necessary?
When is program verification done?
Who performs program verification?
How does the verification process work?
• In the second part of the presentation we will discuss about a
case study
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 5/31
What is program verificationWhat is program verification
• Making certain that the program does what it is
supposed to do, producing a documented evidence
of this
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 6/31
Why program verification is necessaryWhy program verification is necessary
• The aim of SAS validation in pharmaceutical research area
is that end-users will produce high quality programs that fitthe purpose for which they are designed and provide
accurate results with a style that they promote:
•Reliabity
•Efficiency
•Portability
•Flexibility
•Ease of use
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 7/31
When is program verification doneWhen is program verification done
• Program verification should performed as soon after the
development of the SAS code, before putting the “product” in
production
• Development and production environment should be clearly
defined;
•Audit trail of program changes should be present as soon
the program is released to production
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 8/31
Who performs program verificationWho performs program verification
The SAS programmer who create the code should perform basic
testing and follow coding rules, like:
• Error log search
• Warning evaluation
• Comments on critical steps
• Comments on Macro usage
• Details of the SAS program (datetime of creation, SAS
programmer name, dataset used, datetime of verification, Name
of second SAS programmer, etc)
•It should be emphasized to perform then a program
verification by a second SAS programmer
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 9/31
How does the verification process workHow does the verification process work
Biostatistician creates specs then
Submits request
SAS developer produces TLGs Then submits verification request
Quality Control programmer verifies results
Interactive Process
Interactive Process
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 10/31
Different Verification ProceduresDifferent Verification Procedures•SOP should define different verification procedures.
üIndependent programming
üReviewing resultsüRandom review of results
üVisually verify code
•Some of them should mandatory, other optional.
•The Document Containing the programming specs (for example the
SAP) should define which approach to follow, illustrating program
verification techniques (for example using alternative SAS
programming procedures)
•The determination of the level of validation should follow a risk-based
model. The key is to determine the effect on the process if the program
does not produce the desired result.
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 11/31
Error TypesError Types
• Business strategy should identify common ‘error types’ found in:
ü Statistical tables
ü Listings
ü Graphs
ü Data analysis files
ü Header section of SAS programs
ü Bad programming specifications
• Metric report related to error type should be analyzed in order to
perform preventive action correction
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 12/31
Specific CDISC SDTM Validation specsSpecific CDISC SDTM Validation specs – –
Metadata LevelMetadata Level•Verifies that all required variables are present in the dataset
•Reports as an error any variables in the dataset that are not
defined in the domain
•Reports a warning for any expected domain variables whichare not in the dataset
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 13/31
Specific CDISC SDTM Validation specsSpecific CDISC SDTM Validation specs --
Metadata LevelMetadata Level
•Notes any permitted domain variables which are not in the
dataset
•Verifies that all domain variables are of the expected data type
and proper length
•Detects any domain variables which are assigned a controlled
terminology specification by the domain and do not have a
format assigned to them
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 14/31
SAS Programming Rules whenSAS Programming Rules when
validatingvalidating
Ø Emphasizing well commented programs.
Ø Macro in order to use programs repeatedly to verify different
programs (re-usability)
Ø Using alternative SAS programming procedures when
validating.
Ø Define a workflow if error are identified
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 15/31
How to optimize the processHow to optimize the process
Good specs & Good standards & Good training
=
Good programming results
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 16/31
A Case A Case StudyStudy
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 17/31
ExampleExample of of DerivedDerived DatasetsDatasets
ValidationValidation (1/4)(1/4)
PROC COMPARE
Compare
original derived datasets
versusvalidation derived datasets
“Second Programmer” programsall derived datasets
“First Programmer” programsall derived datasets
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 18/31
ExampleExample of of DerivedDerived DatasetsDatasets
ValidationValidation (2/4)(2/4)
The COMPARE Procedur e
Compar i son of WORK. LI STI NG wi t h WORK. VALI DATI ON
( Met hod=EXACT)
Obser vat i on Summar y
Obser vat i on Base Compare I D
Fi r st Obs 1 1 pt=121
Fi rst Unequal 79 79 pt=201
Last Unequal 79 79 pt=201
Las t Obs 89 89 pt =212
Number of Obser vat i ons i n Common: 89.
Tot al Number of Obser vat i ons Read f r om WORK. LI STI NG: 89.
Tot al Number of Obser vat i ons Read f r om WORK. VALI DATI ON: 89.
Number of Observati ons wi th Some Compared Vari abl es Unequal : 1.
Number of Observat i ons wi t h Al l Compared Var i abl es Equal : 88.
proc compare base=listing compare=validation
listbase listcomp;id pt;run;
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 19/31
Val ues Compar i son Summar y
Number of Var i abl es Compared wi t h Al l Observat i ons Equal : 3.
Number of Vari abl es Compared wi th Some Observat i ons Unequal : 1.
Tot al Number of Val ues whi ch Compare Unequal : 1.
Maxi mumDi f f erence: 1.
Vari abl es wi t h Unequal Val ues
Vari abl e Type Len Label Ndi f MaxDi f
age NUM 8 AGE ( years) 1 1. 000
Val ue Compar i son Resul t s f or Vari abl es
___ ___ __ __ __ __ __ __ ____ __ ___ __ __ __ __ __ __ ____ __ ___ ___ _ __ __ _
| | AGE ( years)
| | Base Compare
pt | | age age Di f f . % Di f f
__ ___ __ | | __ __ ____ _ __ __ __ __ _ _ __ ____ __ ____ _ __ __
| |
201 | | 41 40 - 1. 0000 - 2. 4390 _ __ ___ __ __ __ __ __ __ ____ __ ___ __ __ __ __ __ __ ____ __ ___ ___ _ __ __ _
ExampleExample of of DerivedDerived DatasetsDatasets
ValidationValidation (3/4)(3/4)
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 20/31
The COMPARE Pr ocedur e
Compar i son of WORK. LI STI NG wi t h WORK. VALI DATI ON
( Met hod=EXACT)
Obser vat i on Summar y
Obser vat i on Base Compare I D
Fi r st Obs 1 1 pt =121
Last Obs 89 89 pt =212
Number of Obser vat i ons i n Common: 89.
Tot al Number of Obser vat i ons Read f r om WORK. LI STI NG: 89.
Tot al Number of Obser vat i ons Read f r om WORK. VALI DATI ON: 89.
Number of Obser vat i ons wi t h Some Compared Vari abl es Unequal : 0.
Number of Observat i ons wi t h Al l Compar ed Var i abl es Equal : 89.
NOTE: No unequal val ues were f ound. Al l val ues compared are exactl y equal .
ExampleExample of of DerivedDerived DatasetsDatasets
ValidationValidation (4/4)(4/4)
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 21/31
ExampleExample of of TablesTables ValidationValidation (1/3)(1/3)
“First Programmer” programs
all tables applying the set of
layout specifications and
saves outputs in Word
“Second Programmer” programs
all tables avoiding to add
additional SAS code to control
output
Compare of outputs
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 22/31
ExampleExample of of TablesTables ValidationValidation (2/3)(2/3)________________________________________________________________
Tmt A Tmt B
________________________________________________________________
Age (years)
n 41 48Mean (SD) 51.44 (10.39) 52.10 (11.00)
Median 55.00 55.00
Min - Max 30.00- 66.00 27.00- 71.00
Gender
Female 14 (34.15%) 21 (43.75%)Male 27 (65.85%) 27 (56.25%)
________________________________________________________________
First Programmer -
Output in Word
Second programmer -
Output SAS
proc means data=demog n mean stddev
median min max;var age;by tmt;
run;
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 23/31
________________________________________________________________
Tmt A Tmt B________________________________________________________________
Age (years)n 41 48Mean (SD) 51.44 (10.39) 52.10 (11.00)
Median 55.00 55.00Min - Max 30.00- 66.00 27.00- 71.00
GenderFemale 14 (34.15%) 21 (43.75%)
Male 27 (65.85%) 27 (56.25%)________________________________________________________________
First Programmer -
Output in Word
Second programmer -
Output SAS
proc freq data=demog;
tables gender*tmt;
run;
ExampleExample of of TablesTables ValidationValidation (3/3)(3/3)
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 24/31
ExampleExample of of ListingsListings ValidationValidation (1/2)(1/2)
“Second Programmer” prints
derived datasets in SAS
“First Programmer” programs
all listings applying the set of layout specifications and
saves outputs in Word
Compare
listing output in Word
versus
output in SAS of derived dataset
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 25/31
ExampleExample of of ListingsListings ValidationValidation (2/2)(2/2)
Li st i ng 1 Demographi c Characteri st i cs
Subj ect I D Gender Age Race
__ __ __ __ __ __ __ _ __ ____ _ __ __ __ __ _
121 M 50 3
122 M 34 3
123 F 58 3
124 M 64 3
125 M 57 3
126 F 64 3
127 M 39 3
128 M 55 2
129 M 41 3
130 M 44 3
131 M 32 3
132 M 37 3
133 M 61 3
134 F 56 3
135 M 34 3
136 M 34 3
Listing Output in Word Print of Derived Dataset
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 26/31
ExampleExample of of RegistrationRegistration ErrorsErrors
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 27/31
Programming
41%
Specification
14%
Layout
45%
MetricsMetrics onon ProgrammingProgramming ErrorsErrors
Selection of
Variables
14%
Calculation of
variables
20%
SAS
Programming
66%
Specification
not detailed
40%
Wrong
interpretation
of
specification
60%
Output
Writing
56%
Output
Structure
30%
Display
Variables
14%
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 28/31
ExamplesExamples of of ErrorsErrors
• Layout
Writing of a note in table
Incorrect: “Percentages are calculated number of patients”
Correct: “Percentages are calculated on number of patients”
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 29/31
ExamplesExamples of of ErrorsErrors
data age;
set demog;if age<20 then age_c=1;else if 20<age<40 then age_c=2;
else if age>=40 then age_c=3;
run;
• Programming
data age;
set demog;if age<20 then age_c=1;else if 20<=age<40 then age_c=2;
else if age>=40 then age_c=3;
run;
7/28/2019 SAS Validation
http://slidepdf.com/reader/full/sas-validation 30/31
ExamplesExamples of of ErrorsErrors
• Wrong interpretation of specification
Note of a table (in SAP):
“Note 1: Only patients with all value for primary analysis are
included in the table.”
In SAS Program:
In the table, all patients are included