Embed Size (px)
DESCRIPTION
3 Simulation Methodology Monte Carlo simulations of repeated sampling to analyze the impact of various levels of rounding on the precision of the resulting sample estimates Use SAS random number generator functions RANUNI … random numbers from a uniform distribution to generate test population means RANNOR … random numbers from a normal distribution to generate sample estimates based on a population mean and standard deviation
Citation preview
1
Impact of Sample Estimate Rounding on Accuracy
ERCOTLoad Profiling Department
May 22, 2007
2
Overview
Objective is to assess the impact of rounding on the accuracy of sample estimates and on load profile accuracy
Monte Carlo simulation of sample results
Settlement in a test environment
3
Simulation Methodology
Monte Carlo simulations of repeated sampling to analyze the impact of various levels of rounding on the precision of the resulting sample estimates
Use SAS random number generator functions RANUNI … random numbers from a uniform
distribution to generate test population means RANNOR … random numbers from a normal
distribution to generate sample estimates based on a population mean and standard deviation
4
Simulation Methodology (continued)
Simulation steps Generate a randomized population mean Simulate a sample result based on
Sampling from a population having that mean Based on a sample design with a selected statistical accuracy (at 90%
confidence level)
Round the sample estimate to the hundredths, thousandths and ten-thousandths place
Calculate the difference between the rounded estimate and the population mean
Replicate 10,000 times at each precision level
645.1MeanPrecisionStDev
5
Simulation Resultspopulation mean = 0.10 - 0.11
percent of samplesprecision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 0.1049 0.1050 0.1050 2.41% 0.53% 0.48% 83.33%3.0% 0.1050 0.1050 0.1050 2.69% 1.45% 1.43% 67.41%5.0% 0.1050 0.1051 0.1051 3.28% 2.42% 2.41% 59.83%7.5% 0.1048 0.1049 0.1049 4.30% 3.68% 3.67% 55.10%
10.0% 0.1051 0.1051 0.1051 5.40% 4.89% 4.89% 52.72%15.0% 0.1050 0.1050 0.1050 7.49% 7.17% 7.17% 49.58%20.0% 0.1050 0.1049 0.1049 10.01% 9.79% 9.79% 48.50%30.0% 0.1051 0.1051 0.1051 14.73% 14.56% 14.56% 47.23%
population mean = 0.20 - 0.21percent of samples
precision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 0.2049 0.2050 0.2050 1.30% 0.50% 0.49% 75.71%3.0% 0.2049 0.2049 0.2049 1.86% 1.45% 1.45% 58.32%5.0% 0.2050 0.2050 0.2050 2.71% 2.44% 2.44% 53.53%7.5% 0.2050 0.2050 0.2050 3.83% 3.67% 3.67% 50.17%
10.0% 0.2050 0.2049 0.2049 4.94% 4.83% 4.83% 48.25%15.0% 0.2054 0.2054 0.2054 7.38% 7.29% 7.29% 47.61%20.0% 0.2055 0.2055 0.2055 9.73% 9.66% 9.66% 46.65%30.0% 0.2046 0.2045 0.2045 14.75% 14.70% 14.69% 46.98%
population mean = 0.30 - 0.31percent of samples
precision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 0.3049 0.3050 0.3050 0.93% 0.49% 0.48% 67.84%3.0% 0.3051 0.3050 0.3050 1.61% 1.44% 1.44% 52.24%5.0% 0.3049 0.3050 0.3050 2.53% 2.41% 2.41% 50.45%7.5% 0.3052 0.3052 0.3052 3.71% 3.65% 3.65% 48.33%
10.0% 0.3048 0.3048 0.3048 4.92% 4.86% 4.85% 47.82%15.0% 0.3055 0.3055 0.3055 7.25% 7.21% 7.20% 46.73%20.0% 0.3048 0.3048 0.3048 9.68% 9.65% 9.65% 46.08%30.0% 0.3056 0.3056 0.3056 14.66% 14.64% 14.64% 45.37%
average of sample means mean absolute percent error
average of sample means mean absolute percent error
average of sample means mean absolute percent error
6
Simulation Resultspopulation mean = 0.40 - 0.41
percent of samplesprecision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 0.4050 0.4050 0.4050 0.77% 0.49% 0.48% 63.29%3.0% 0.4049 0.4049 0.4049 1.59% 1.48% 1.48% 52.17%5.0% 0.4049 0.4049 0.4049 2.51% 2.43% 2.43% 49.03%7.5% 0.4048 0.4048 0.4048 3.72% 3.67% 3.67% 48.30%
10.0% 0.4054 0.4054 0.4054 4.86% 4.84% 4.84% 46.35%15.0% 0.4050 0.4050 0.4050 7.31% 7.29% 7.29% 46.27%20.0% 0.4051 0.4051 0.4050 9.65% 9.63% 9.63% 45.98%30.0% 0.4056 0.4056 0.4056 14.54% 14.52% 14.52% 46.05%
population mean = 0.50 - 0.51percent of samples
precision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 0.5050 0.5050 0.5050 0.68% 0.49% 0.48% 60.25%3.0% 0.5051 0.5051 0.5051 1.53% 1.46% 1.46% 50.22%5.0% 0.5049 0.5049 0.5049 2.45% 2.41% 2.41% 48.04%7.5% 0.5047 0.5047 0.5047 3.69% 3.67% 3.67% 46.96%
10.0% 0.5050 0.5049 0.5049 4.89% 4.86% 4.86% 47.24%15.0% 0.5044 0.5045 0.5045 7.35% 7.34% 7.34% 45.68%20.0% 0.5054 0.5054 0.5054 9.73% 9.73% 9.73% 45.21%30.0% 0.5062 0.5063 0.5063 14.48% 14.48% 14.48% 44.81%
population mean = 0.80 - 0.81percent of samples
precision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 0.8050 0.8050 0.8050 0.56% 0.49% 0.49% 52.95%3.0% 0.8050 0.8050 0.8050 1.50% 1.48% 1.48% 48.53%5.0% 0.8050 0.8051 0.8051 2.49% 2.47% 2.47% 47.72%7.5% 0.8047 0.8048 0.8048 3.62% 3.61% 3.61% 46.18%
10.0% 0.8054 0.8054 0.8054 4.87% 4.87% 4.87% 46.66%15.0% 0.8050 0.8050 0.8050 7.27% 7.26% 7.26% 45.74%20.0% 0.8030 0.8030 0.8030 9.73% 9.72% 9.72% 45.47%30.0% 0.8041 0.8041 0.8041 14.31% 14.31% 14.31% 45.76%
average of sample means mean absolute percent error
average of sample means mean absolute percent error
average of sample means mean absolute percent error
7
Simulation Resultspopulation mean = 1.00 - 1.10
percent of samplesprecision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 1.0501 1.0501 1.0501 0.53% 0.49% 0.48% 52.90%3.0% 1.0494 1.0494 1.0494 1.48% 1.46% 1.46% 47.42%5.0% 1.0498 1.0497 1.0497 2.44% 2.43% 2.43% 46.56%7.5% 1.0494 1.0494 1.0494 3.67% 3.66% 3.66% 45.70%
10.0% 1.0497 1.0497 1.0497 4.90% 4.90% 4.90% 46.06%15.0% 1.0508 1.0508 1.0508 7.29% 7.28% 7.28% 46.06%20.0% 1.0466 1.0467 1.0467 9.77% 9.77% 9.77% 44.49%30.0% 1.0478 1.0479 1.0479 14.64% 14.64% 14.64% 44.79%
population mean = 2.00 - 2.10percent of samples
precision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 2.0505 2.0505 2.0504 0.50% 0.48% 0.48% 48.87%3.0% 2.0494 2.0494 2.0494 1.45% 1.45% 1.45% 46.07%5.0% 2.0506 2.0506 2.0506 2.41% 2.41% 2.41% 45.58%7.5% 2.0496 2.0496 2.0496 3.69% 3.69% 3.69% 45.51%
10.0% 2.0500 2.0500 2.0500 4.87% 4.86% 4.86% 45.60%15.0% 2.0477 2.0476 2.0476 7.32% 7.32% 7.32% 45.46%20.0% 2.0511 2.0511 2.0511 9.70% 9.70% 9.70% 45.28%30.0% 2.0533 2.0533 2.0533 14.42% 14.42% 14.42% 44.59%
population mean = 3.00 - 3.10percent of samples
precision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 3.0501 3.0501 3.0501 0.49% 0.49% 0.49% 47.57%3.0% 3.0490 3.0490 3.0490 1.46% 1.45% 1.45% 45.51%5.0% 3.0513 3.0513 3.0513 2.43% 2.42% 2.42% 45.26%7.5% 3.0482 3.0482 3.0482 3.61% 3.61% 3.61% 45.51%
10.0% 3.0482 3.0482 3.0482 4.80% 4.80% 4.80% 45.29%15.0% 3.0543 3.0543 3.0543 7.31% 7.30% 7.30% 45.56%20.0% 3.0472 3.0472 3.0472 9.69% 9.69% 9.69% 44.95%30.0% 3.0484 3.0484 3.0484 14.60% 14.59% 14.59% 45.91%
average of sample means mean absolute percent error
average of sample means mean absolute percent error
average of sample means mean absolute percent error
8
Simulation Resultspopulation mean = 5.00 - 5.10
percent of samplesprecision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 5.0510 5.0510 5.0510 0.49% 0.49% 0.49% 47.31%3.0% 5.0502 5.0502 5.0502 1.44% 1.44% 1.44% 46.21%5.0% 5.0509 5.0509 5.0509 2.39% 2.39% 2.39% 45.52%7.5% 5.0472 5.0472 5.0472 3.63% 3.64% 3.64% 44.32%
10.0% 5.0469 5.0470 5.0470 4.85% 4.85% 4.85% 44.69%15.0% 5.0384 5.0383 5.0383 7.28% 7.28% 7.28% 44.90%20.0% 5.0462 5.0463 5.0462 9.70% 9.70% 9.70% 45.90%30.0% 5.0422 5.0422 5.0422 14.57% 14.57% 14.57% 44.90%
population mean = 10.00 - 10.10percent of samples
precision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 10.4980 10.4979 10.4979 0.49% 0.48% 0.48% 46.01%3.0% 10.5015 10.5016 10.5016 1.47% 1.47% 1.47% 44.97%5.0% 10.4960 10.4960 10.4960 2.39% 2.39% 2.39% 44.59%7.5% 10.5006 10.5005 10.5005 3.67% 3.67% 3.67% 45.46%
10.0% 10.5025 10.5025 10.5025 4.87% 4.87% 4.87% 44.86%15.0% 10.4914 10.4913 10.4913 7.28% 7.28% 7.28% 45.27%20.0% 10.5246 10.5246 10.5246 9.70% 9.70% 9.70% 45.14%30.0% 10.5398 10.5398 10.5398 14.61% 14.61% 14.61% 45.02%
population mean = 15.00 - 15.10percent of samples
precision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 15.4993 15.4993 15.4993 0.49% 0.49% 0.49% 45.65%3.0% 15.4970 15.4970 15.4970 1.44% 1.44% 1.44% 45.66%5.0% 15.4976 15.4975 15.4975 2.41% 2.41% 2.41% 45.03%7.5% 15.4983 15.4983 15.4983 3.62% 3.62% 3.62% 45.61%
10.0% 15.4939 15.4938 15.4939 4.83% 4.83% 4.83% 45.86%15.0% 15.5118 15.5118 15.5118 7.35% 7.35% 7.35% 45.07%20.0% 15.5235 15.5235 15.5235 9.78% 9.78% 9.78% 44.90%30.0% 15.5373 15.5374 15.5374 14.67% 14.67% 14.67% 44.81%
average of sample means mean absolute percent error
average of sample means mean absolute percent error
average of sample means mean absolute percent error
9
Simulation Results
population mean = 20.00 - 20.10percent of samples
precision 2 decimals 3 decimals 4 decimals 2 decimals 3 decimals 4 decimals for which 3 decimals is better
1.0% 20.4986 20.4986 20.4986 0.49% 0.49% 0.49% 45.40%3.0% 20.5052 20.5052 20.5052 1.46% 1.46% 1.46% 45.50%5.0% 20.4936 20.4936 20.4936 2.42% 2.42% 2.42% 45.66%7.5% 20.4997 20.4997 20.4997 3.66% 3.66% 3.66% 43.82%
10.0% 20.5017 20.5017 20.5017 4.84% 4.84% 4.84% 45.19%15.0% 20.4878 20.4877 20.4877 7.28% 7.28% 7.28% 43.98%20.0% 20.5137 20.5137 20.5137 9.50% 9.50% 9.50% 44.87%30.0% 20.4519 20.4518 20.4518 14.52% 14.52% 14.52% 44.21%
average of sample means mean absolute percent error
10
Simulation Findings If the precision is ±10% or worse, and the
population mean is > 0.3, MAPEs for 2-digit rounding and 3-digit rounding are virtually identical
If the precision is better than ±10%, and the population mean is < 0.3, MAPEs for 3-digit rounding are somewhat better
If the population mean is ≥ 1, 3-digit rounding is likely to do more harm than good regardless of the sample precision
11
Test Settlement Data Aggregation ran a test settlement using
profiles with three digit rounding for January 27, 2006
Compared UFE for original 2-digit profiles with results for 3-digit profiles
Compared total aggregated residential load for the same day
12
January 27, 2006 UFE Comparison
13
January 27, 2006 Residential Load Comparison
Red = 3 decimals
Blue = 2 decimals
14
Test Settlement Result
3-digit profile rounding produced virtually no difference in settlement for the selected day
