Workshop on Price Index Compilation Issues

February 23-27, 2015

Imputation of Missing Values, Seasonal Products and Quality

ChangesGefinor Rotana Hotel, Beirut, Lebanon

Lecture Outline

IntroductionImputation TechniquesTreatment of Seasonal CommoditiesImputation Techniques for QA

Causes of missing prices

Samplesnot universal coverage

Temporarily unavailable supply shortages

Seasonally unavailable

Permanently unavailable

Current practices and their implications

Take no action:samples deteriorate bias may be introduced

Ensure the observations are comparable (matched between

periods) Adjust weights

to account for missing dataCarry forward the last price

can result in bias particularly during inflationary

periodsImpute prices for missing observations

Imputation TechniquesEstimate the price change of comparable observations in the item category Move the previous price forward

1

1( 1)

1

1 ( 1)ˆ

t njt t

i tj i j

t t t ti i i

pRel

p

p p Rel

Example 1 Imputing a Missing Month’s Value using the

Jevons method

Prices Dec 2005 Jan 2006 Feb 2006

outlet A 10 10 11 B 10 10 10 C 9 10 11 D 12 12 X Comparable averages:

- current month 10.656a a = 1

3(11 10 11)

- previous month 10b b = 1

3(10 10 10) Monthly movement 1.0656c c = a / b Imputed price 12 1.0656 =12.787

Imputation TechniquesEstimate a missing current price

from the short-term change in average prices of other

observations

Mark the estimated price as “imputed”

If price is missing for several monthschain its imputed prices until a replacement is found.

See Example 2

Example 2 Imputing Multiple Month’s Missing Prices

Prices

Dec 2005 Jan 2006 Feb 2006 Mar 2006 outlet A 10 10 11 11 B 10 10 10 10 C 9 10 11 12 D 12 12 12.787 X

Comparable averages:

- current month 10.97a a = 1

3(11 10 12)

- previous month 10.656b b = 1

3(11 10 11) Monthly movement 1.0295c c = a / b

If a price is missing for several months, an imputation must be made every month, i.e., imputed prices must be updated. Imputed price = previous month’s price x movement in comparable averages

= 12.787 x 1.0295 = 13.164

Seasonal Items and Prices

“Hard” SeasonalityItems missing from markets during part of yearAt the start of a new season • changes in quality • introduction of new products

At the end of season • discounting

“Soft” SeasonalitySeasonal price movement of year-round items

Varying seasonal patterns from year to year

Treatment of Seasonal Commodities

The index must reflect the price swingfrom: the last price (of old season) to: the first price (of new season)

Impute missing prices of out-of-season observations

prevents systematic biasParticularly when new products are introduced at the beginning of season

Other techniques for seasonal items

Variable weightsMay not give reliable resultsRequires detailed monthly data for several years on the quantity of each item sold

Rolling average: extrapolate recent 6-months

Seasonal factorsMonth-on-(previous) 12-month figure OK.

Do NOT use carry forward

Seasonal Clothing

Extreme price movementEnd of season discounts Multiple selling seasons:

fall/winterspring/summer

Different varieties for each selling season

Do not replace variety once price discounting begins

Seasonal Clothing

After selling seasons and variety is no longer available,

return the price to the last “normal” price begin imputation in the following period

When the selling season returns the next year

same product may not appear again select the most comparable replacement product (requires maintaining detailed specifications)

Quality Adjustment (QA)

Direct comparisons between old and new item

Link (splice) new item into the index

Drop observation from index calculation

Alternative QA TechniquesDirect Quality Adjustment

data collector or analyst knowledge,information from producers, orhedonic regression models

Indirect Quality Adjustmentwith Imputation

overlap price availableoverall mean imputationclass mean imputation

Example 3—Direct Quality Adjustment

Variety

Price Index Month 1

Average Price in Month 1

Average Price in Month 2

Month 2 Price Relative

Price Index Month 2

Prod 1

125.0

150

160

1.067

133.3

Prod 2

150.0

225

250

1.111

166.7

Prod 3

125.0

140

-

-

Sub 1

(160)

180 1.125 140.6

All items

132.83

1.10062

146.9

Product 3 is no longer sold; Substitute 1 is the replacement. Index would rise 15%

with no QA [1 1

3 3(160 250 180) /(150 225 140) ]=1.1517 The value of the quality difference is estimated to be 20 in the previous period. The prices used for computing the index for Month 2 would be 160 and 180. The price index reflects a pure price change of only 10% (not 15%)

[1 1

3 3(160 250 180) /(150 225 160) 1.1006. ]

Example 4—Indirect QA: Overlap Pricing

Variety Price Index Month 1

Average Price in Month 1

Average Price in Month 2

Month 2 Price Relative

Price Index Month 2

Prod 1

125.0

150

160

1.067

133.3

Prod 2

150.0

225

250

1.111

166.7

Prod 3

125.0

140

- -

140.6

Sub 1

160

180 1.125

All items

132.83

1.102

146.9

Product 3 and Substitute 1 are available in the overlapping period (Month 1). The price change for Substitute 1 is used in the index for Month 2. The quality difference of 20, observed as the difference in market prices, is excluded and the index increases by 10% (not 15%)

Example 5—Indirect QA: Overall mean imputation

Variety

Price Index Month 1

Average Price in Month 1

Average Price in Month 2

Month 2 Price Relative

Price Index Month 2

Prod 1

125.0

150

160

1.067

133.3

Prod 2

150.0

225

250

1.111

166.7

Prod 3

125.0

140

(152)

1.091

136.4

Sub 1

-

180

All items

132.83

1.089

144.74

No overlap price is available for Substitute 1 in Month 1 An estimate is made for Product 3's price in Month 2 using the average change for other

similar products between Month 1 and Month 2 1 1

2 2(160 250) /(150 225) 1.08867 . The quality difference of 28, observed as the difference between the imputed price of Prod 3 and market price of Sub 1, is excluded and the index increases by about 9% (not 15%)

Example 6—Indirect Q A: Class mean imputation

Variety

Price Index Month 1

Average Price in Month 1

Average Price in Month 2

Month 2 Price Relative

Price Index Month 2

Prod 1

125.0

150

160

1.067

133.3

Prod 2

150.0

225

250

1.111

166.7

Prod 3

125.0

(140)

(156)

1.111

138.9

Sub 1

-

180

All items

132.83

1.097

145.71

No overlap price is available for Substitute 1 in Month 1 An estimate is made for Product 3's price in Month 2 using the price trend in a closely related product (Prod 2) between Month 1 and Month 2 (11.1%) The quality difference of 24, observed as the difference between the imputed price of Prod 3 and market price of Sub 1, is excluded and the index increases by 9.7% (not 15%)

[1 1

3 3(160 250 156) /(150 225 160) 1.0971. ]

Effect of QA Imputation on Price Change

Month 1

Month 2

%change

No quality Adj. Overlap Price Overall Mean Class Mean

132.83 132.83 132.83 132.83

152.98 146.19 144.74 145.73

15.2 10.1 8.87 9.71

Recommendation For Missing Price Observations

Always impute prices of missing items

Use imputed prices in the next pricing period (as the previous period prices)

Even in periods for which there are no price observations in the item group at all.

Recommendation For Missing Price Observations (con’t.)

The “carry forward” method Use only when it is fairly certain that the price remains the same.Can generate undue stability in the index.

Recommendation For Imputing Missing Price Observations

For temporarily missing productsuse one of the imputation

methods (they are self-correcting)

For products permanently missingselect a replacement

If replacement is comparable, use directlyIf replacement is noncomparable

use overlap price or indirect quality adjustment

Thank youThank you