Missing Data? Two SAS Procedures to the Rescue

Copy right © SAS Inst i tute Inc. Al l rights reserved.Copy right © SAS Inst i tute Inc. Al l rights reserved.

Missing Data? Two SAS Procedures to the RescueHPIMPUTE and SURVEYIMPUTE

Melodie RushCustomer Success Principal Data ScientistConnect with me:LinkedIn: https://www.linkedin.com/in/melodierushTwitter: @Melodie_Rush

https://www.linkedin.com/in/melodierush

Copy right © SAS Inst i tute Inc. Al l rights reserved.

AGENDA

Introduction

Proc HPIMPUTE

Proc SURVEYIMPUTE

What, Why and How

Syntax, Imputation Options, Examples

Syntax, Imputation Options, Examples


What is Missing Data?Definition

In statistics, missing data, or missing values, occur when no data value is stored for the variable in an observation. Missing data are a common occurrence and can have a significant effect on the conclusions that can be drawn from the data. - Wikipedia

https://www.google.com/search?q=definition+for+missing+data&rlz=1C1CHBF_enUS750US751&oq=definition+for+missing+data&aqs=chrome..69i57j0.4543j1j9&sourceid=chrome&ie=UTF-8


What is Missing Data?SAS

Missing Value

• is a value that indicates that no data value is stored for the variable in the current observation. There are three kinds of missing values:

• numeric

• character

• special numeric

By default, SAS prints a missing numeric value as a single period (.) and a missing character value as a blank space. See Creating Special Missing Values for more information about special numeric missing values.

http://support.sas.com/documentation/cdl/en/lrcon/62955/HTML/default/a000992455.htm


Replace with Constant or Zero

Replace with mean or mode

Replace using an imputation method

Remove observation(s)

Wh

at s

ho

uld

yo

u d

o a

bo

ut

mis

sin

g va

lues

?


Proc HPIMPUTE


1. Syntax

2. Imputation Options

3. Other Options

Proc HPIMPUTE


Proc HPIMPUTE

The HPIMPUTE procedure executes high-performancenumeric variable imputation.

• takes only numeric variables.

• runs in either single-machine mode or distributed mode.

HPIMPUTE Procedure Documentation

http://support.sas.com/documentation/cdl/en/prochp/66409/HTML/default/viewer.htm#prochp_hpimpute_examples01.htm


Proc HPIMPUTESyntax

proc hpimpute options;

input variables;

impute variables <options>;

performance <performance options>;

id variables;

freq variables;

code <options>;<…>run;


Proc HPIMPUTE

• VALUE

– Replaces missing values with the specified value

• MEAN

– Replaces missing values with the algebraic mean of the variable

• RANDOM

– Replaces missing values with a random value that is drawn between the minimum and the maximum of the variable

• PMEDIAN

– Replaces missing values with the pseudomedian of the variable

Imputation Methods


Proc HPIMPUTE

• 6 variables

• First 4 have missing values

• Fifth is the frequency variable

• Last is an index variable

Example Data

Example Code and Documentation

https://go.documentation.sas.com/?cdcId=pgmsascdc&cdcVersion=9.4_3.5&docsetId=prochp&docsetTarget=prochp_hpimpute_gettingstarted.htm&locale=en


Proc HPIMPUTEExample Code – Value Method

Replaces missing values with the specified value


Proc HPIMPUTEExample Results – Value Method

Variable Name

Indicator Name

ImputedVariable Name

Number Imputed


Proc HPIMPUTEExample Output Data – Value Method


Proc HPIMPUTEExample Code – Mean Method

Replaces missing values with the algebraic mean of the variable


Proc HPIMPUTEExample Results – Mean Method

Variable Name

Indicator Name


Number Imputed


Proc HPIMPUTEExample Output Data – Mean Method


Proc HPIMPUTEExample Code – Random Method

Replaces missing values with a random value that is drawn between the minimum and the maximum of the variable


Proc HPIMPUTEExample Results – Random Method

Variable Name

Indicator Name


Number Imputed


Proc HPIMPUTEExample Output Data – Random Method


Proc HPIMPUTEExample Code – Pseudo Median Method

Replaces missing values with the pseudo median of the variable


Proc HPIMPUTEExample Results – Pseudo Median Method

Variable Name

Indicator Name


Number Imputed


Proc HPIMPUTEExample Output Data – Pseudo Median Method


Proc HPIMPUTEID Statement


Proc HPIMPUTEID Statement

• The optional ID statement lists one or more variables from the input data set that are transferred to the output data set.

• The ID statement accepts numeric and character variables.


Proc HPIMPUTEFREQ Statement

• The variable in the FREQ statement identifies a numeric variable in the data set that contains the frequency of occurrence for each observation.

• PROC HPIMPUTE treats each observation as if it appeared n times, where n is the value of the FREQ variable for the observation.


Proc HPIMPUTEFREQ Statement

• If the frequency value is not an integer, it is truncated to an integer.

• If the frequency value is less than 1 or missing, the observation is not used in the analysis.

• When the FREQ statement is not specified, each observation is assigned a frequency of 1.


Proc HPIMPUTEFREQ Statement Results

Results with FREQ Statement Without


Proc HPIMPUTEFREQ Statement Results

Results with FREQ Statement Without


Proc HPIMPUTESyntax - CODE Statement

proc hpimpute data=ex1 out=out1;

id id;

input a b c d;

impute a / value=0.1;

impute b / method=pmedian;

impute c / method=random;

impute d / method=mean;

code file='c:/temp/hpimpute.sas';

run;

The CODE statement generates SAS DATA step code that mimics the computations that are performed when the IMPUTE statement runs in

single-machine mode and uses a single thread.


Proc HPIMPUTEResults - CODE Statement

%let HPDM_seed=5;

if a = . then do;

M_a = 1;

IM_a = 0.1;

end;

else do;

M_a = 0;

IM_a = a;

end;

length M_a IM_a 8;

if b = . then do;

M_b = 1;

IM_b = 3;

end;

else do;

M_b = 0;

IM_b = b;

end;

length M_b IM_b 8;

A B


Proc HPIMPUTEResults - CODE Statement

HPDM_vmin = 1;

HPDM_vmax = 10;

if c = . then do;

M_c = 1;

IM_c = HPDM_vmin + (HPDM_vmax –

HPDM_vmin)*ranuni(&HPDM_seed);

end;

else do;

M_c = 0;

IM_c = c;

end;

length M_c IM_c 8;

if d = . then do;

M_d = 1;

IM_d = 5.5;

end;

else do;

M_d = 0;

IM_d = d;

end;

length M_d IM_d 8;

CD


Proc HPIMPUTESyntax - PERFORMANCE Statement


id id;

input a b c d;





performance nodes=0;

run;

• Defines performance parameters for multithreaded and distributed computing, passes variables that describe the distributed computing environment, and requests detailed results about the performance characteristics of the HPIMPUTE procedure.

• Also use the PERFORMANCE statement to control whether the HPIMPUTE procedure executes in single-machine or distributed mode.

Performance Statement Documentation

https://go.documentation.sas.com/?cdcId=pgmsascdc&cdcVersion=9.4_3.5&docsetId=prochp&docsetTarget=prochp_introcom_sect026.htm&locale=en


Proc HPIMPUTEResults – Performance Statement


Proc HPIMPUTESyntax – Performance Statement

Running in a high-performance environment

option set=GRIDHOST="&GRIDHOST";

option set=GRIDINSTALLLOC="&GRIDINSTALLLOC";


id id;

input a b c d;





performance nodes=2 details

host="&GRIDHOST" install="&GRIDINSTALLLOC";

run;


Proc HPIMPUTEResults – Performance Statement

Running in a high-performance environment


Proc SURVEYIMPUTE


SURVEY Procedures

➢SURVEYSELECT

➢SURVEYIMPUTE

➢SURVEYMEANS

➢SURVEYFREQ

➢SURVEYREG

➢SURVEYLOGISTIC

➢SURVEYPHREG

Sample selection

Imputation

Descriptive statistics

Frequency tables

Linear models

Logistic regression

Proportional hazards

SAS/Stat


1. Syntax

2. Imputation Options

3. Analyzing Results

Proc SURVEYIMPUTE


Handling Missing Values in Survey Data

• How are the data collected?

• How are the missing values imputed?

Different imputation methods require different analysis techniques

Analysis of Imputed Data


Handling Missing Values in Survey DataThe Nonresponse Problem

ID Income

1 40

2 120

3 60

4 80

5

6 370

7 210

• Prevention is the best solution for nonresponse

• Information is the best tool for imputation

Average household income = 147

Average household income = 190

450

Tax Return

42

116

55

84

410

320

230


PROC SURVEYIMPUTE

The SURVEYIMPUTE procedure imputes missing values of an item in a sample survey by replacing them with observed values from the same item.

Imputation methods include • Single and Multiple Hot-Deck Imputation• Approximate Bayesian Bootstrap (ABB) Imputation• Fully Efficient Fractional Imputation (FEFI)• Fractional Hot-deck Imputation (FHDI)

PROC SURVEYIMPUTE Documentation

http://support.sas.com/documentation/cdl/en/statug/68162/HTML/default/viewer.htm#statug_surveyimpute_overview.htm


Handling Missing Values in Survey DataPROC SURVEYIMPUTE Syntax

proc surveyimpute options;

cluster variables;

repweights variables;

strata variables;

weight variable;

cells variables;

var variables;

by variables;

class variables;

id variable;

output options;

<…>run;


Proc SURVEYIMPUTESyntax - Method=HotDeck

Imputation techniques that use observed values from the sample to impute (fill in) missing values are known as hot-deck imputation.

proc surveyimpute data=work.surveyimpute;

var income;

output out=hotdeck;

run;


Proc SURVEYIMPUTEExample Results – Method=HotDeck


Proc SURVEYIMPUTEExample Output Data – Method=HotDeck


Proc SURVEYIMPUTEHot-Deck Imputation

9

5

5

7

2

1

4

9

55

7

21

4

8 87

4

7

4

Data Imputation Cells Donors Recipients


proc surveyimpute data=work.surveyimpute

method=hotdeck(selection=SRSWOR)

ndonors=1 seed=8523;

cells cell2;

var income;

id ID;

output out=hotdeck donorid;

run;

Proc SURVEYIMPUTESyntax Method=HotDeck

The SELECTION= option modifies the donor selection

Imputation techniques that use observed values from the sample to impute (fill in) missing values are known as hot-deck imputation.


Proc SURVEYIMPUTEExample Results – Method=HotDeck


Proc SURVEYIMPUTEExample Output Data – Method=HotDeck



method=hotdeck(selection=abb)

ndonors=1 seed=8523;

cells cell2;

var income;

id ID;

output out=hotdeckb donorid;

run;

Proc SURVEYIMPUTESyntax Method=HotDeck Selection=ABB

SELECTION= option modifies the donor selection

Hot Deck that requests donor selection by using the approximate Bayesian bootstrap method. For more information, see the section Approximate Bayesian Bootstrap

https://go.documentation.sas.com/?docsetId=statug&docsetTarget=statug_surveyimpute_details27.htm&docsetVersion=15.2&locale=en#statug.surveyimpute.abbDet


Proc SURVEYIMPUTEApproximate Bayesian Bootstrap

9

55

7

21

4

85

4

5

4

5 5

9

9

8

42

2

Donor Pool Donors

SRSWR

SRSWR

SRSWR

SRSWR

Imputation Cells Recipients


Proc SURVEYIMPUTEExample Results – Method=HotDeck Selection=ABB


Proc SURVEYIMPUTEExample Output Data – Method=HotDeck Selection=ABB


Proc SURVEYIMPUTE

• Uses multiple donor units for a recipient unit.

• The number of donor units for a recipient unit is equal to the number of observed levels for the missing items.

• Each donor donates a fraction of the original weight of the recipient unit such that the sum of the fractional weights from all the donors is equal to the original weight of the recipient.

• Does not introduce additional variability that is caused by the selection of donor units.

• One disadvantage is that it can greatly increase the size of the imputed data set.

Fully Efficient Fractional Imputation (FEFI)

Fully Efficient Fractional Imputation Documentation

https://go.documentation.sas.com/?docsetId=statug&docsetTarget=statug_surveyimpute_details17.htm&docsetVersion=15.2&locale=en



method=FEFI;

cells cell2;

var income;

class income;

id ID;

output out=FEFI;

run;

Proc SURVEYIMPUTESyntax Method=FEFI

The Class Statement required for FEFI

Fully Efficient Fractional Imputation


Handling Missing Values in Survey DataFully Efficient Fractional Imputation

9

55

7

21

4

8

5 7

1

8 9

42

9

55

7

21

4

8

5 7

1

8 9

42

Imputation Cells Donors Imputed DataRecipients


Proc SURVEYIMPUTEExample Results – Method=FEFI


Proc SURVEYIMPUTEExample Output Data – Method=FEFI


Proc SURVEYIMPUTE

• Uses multiple donor units for a recipient unit. • Each donor donates a fraction of the original weight of the recipient unit such

that the sum of the fractional weights from all the donors is equal to the original weight of the recipient.

• The fraction of the recipient weight that a donor unit contributes to the recipient unit is known as the fractional weight.

• The donors are selected by using probability proportional to size (PPS) selection in which the two-stage FEFI weights are used as the size measure.

• FHDI is useful for reducing the size of the imputed data when two-stage FEFI creates many imputed rows. – FHDI follows the same imputation steps as those of two-stage FEFI, but FHDI selects

a subset of second-stage donor cells from all possible second-stage donor cells for the imputation.

Fractional Hot-Deck Imputation (FHDI)

Fractional Hot-Deck Imputation Documentation




proc surveyimpute data=work.surveyimpute2

method=FHDI ndonors=3 seed=8523;

cells cell2;

var income age (clevvar=agegroup);

class income;

id ID;

output out=FHDI;

run;

Proc SURVEYIMPUTESyntax Method=FHDI

The At least 2 missing values for each row (one continuous with a binned version)

Fractional Hot-Deck Imputation


Handling Missing Values in Survey DataData - Method=FHDI

450


Proc SurveyImputeMethod=FHDI

1

2

345678


Proc SURVEYIMPUTEExample Results – Method=FHDI


Proc SURVEYIMPUTEExample Output Data – Method=FHDI


Proc SURVEYIMPUTEExample Output Data – Method=FHDI



Ignore the imputation variance

Hot-Deck Analysis: Statements

proc surveymeans data=hotdeck3;

var income;

repweights RepWt_: /Jkcoefs=0.857;

run;


Handling Missing Values in Survey DataFEFI Analysis: Statements

Use the WEIGHT and REPWEIGHTS statements

proc surveymeans data=fefi;

var income;

weight ImpWt;

repweights ImpRepWt_: / jkcoefs=0.857;

run;


Handling Missing Values in Survey DataComparing the Estimates

Estimates for Average Income

Imputation Method

Estimate Standard Error

No Missing 190.00 61.10

No Imputation 146.70 50.97

Hot-Deck 178.57 53.60

FEFI 159.04 54.43

*FHDI 167.71 27.25

▪ Same analysis but different results

* FHDI based on different data set with 20 rows versus 7 in other methods



• PROC SURVEYIMPUTE is the tool for imputing missing values from complex surveys

• FEFI introduces no additional variability from the imputation and is the preferred method for survey data

• FHDI is the preferred method for continuous data

• The analysis technique should be tailored to both the survey design and the imputation method

Handling Nonresponse in SAS/STAT


ResourcesWhere to learn more


Where to learn more?SAS Documentation

• Working with Missing Data in SAS

• Proc HPIMPUTE Documentation

• Proc SURVEYIMPUTE Documentation

• Handling Missing Values in Survey Data((Video)

• Proc SURVEYIMPUTE References

https://go.documentation.sas.com/?docsetId=lrcon&docsetTarget=p175x77t7k6kggn1io94yedqagl3.htm&docsetVersion=9.4&locale=en

http://support.sas.com/documentation/cdl/en/prochp/67530/HTML/default/viewer.htm#prochp_hpimpute_syntax05.htm

https://go.documentation.sas.com/?docsetId=statug&docsetTarget=statug_surveyimpute_details.htm&docsetVersion=15.2&locale=en

https://www.youtube.com/watch?v=7kNGOrZgHYs

https://go.documentation.sas.com/?docsetId=statug&docsetTarget=statug_surveyimpute_references.htm&docsetVersion=15.2&locale=en


Where to learn more?Papers

• Mukhopadhyay, P. K. (2016). “Survey Data Imputation with PROC SURVEYIMPUTE” In Proceedings of the SAS Global Forum 2016 Conference. Cary, NC: SAS Institute Inc.

• Stokes, Maura (and Statistical R&D Staff). “SAS/STAT 14.1: Methods for Massive, Missing, or Multifaceted Data” In Proceedings of the SAS Global Forum 2015 Conference. Cary NC: SAS Institute Inc.

• Cutler, D. Richard. “Machine Learning and Predictive Analytics in SAS® Enterprise Miner™ and SAS/STAT® Software” In the Proceedings of the SAS Global Forum 2019 Conference. Cary NC: SAS Institute Inc.

https://support.sas.com/resources/papers/proceedings16/SAS3520-2016.pdf

https://support.sas.com/resources/papers/proceedings15/SAS1940-2015.pdf

https://www.sas.com/content/dam/SAS/support/en/sas-global-forum-proceedings/2019/3263-2019.pdf


Where to learn more?Book

Complex Survey Data Analysis with SAS

https://amzn.to/3q4wqud

FIND YOUR

USER GROUP

sas.com/usersgroups

You should do the following (if you’re not already):

◊ Tap into local resources◊ Learn from other SAS Users’

experiences◊ Connect with the local SAS

Users’ network

https://www.sas.com/en_us/connect/user-groups.html

ARE YOU AN EXPLORERWhether you’re a modeler, programmer, administrator, everyone is welcome on SAS Analytics Explorers!

More ways to:◊ Learn SAS◊ Get support◊ Connect with users across the US

Ready to become an explorer? Got questions?explorers.sas.com

?

https://explorers.sas.com/en/home.html

ASK THE EXPERTDON’T BE SHY,

Tips & tricks webinars on a variety of SAS topics plus get all your questions answered by the SAS expert, live.

sas.com/asktheexpert

https://www.sas.com/en_us/learn/ask-the-expert-webinars.html


sas.com

Thank you for your time and attention!Questions?

Connect with me:LinkedIn: https://www.linkedin.com/in/melodierushTwitter: @Melodie_Rush

https://www.linkedin.com/in/melodierush

Documents

Missing Data? Two SAS Procedures to the Rescue