United Nations Statistics Division Benchmarking Training Workshop on the Compilation of Quarterly National Accounts for Economic Cooperation Organization

United Nations Statistics Division

Benchmarking

Training Workshop on the Compilation of Quarterly National Accounts for Economic Cooperation Organization Member Countries

8-11 March 2015Tehran, Islamic Republic of Iran

Outline

What is benchmarking? Purpose of benchmarking Benchmarking methods Benchmarking software for QNA Conclusions Questions

2

What is benchmarking?

Benchmarking is a statistical technique to correct inconsistencies between estimates of the same variable obtained from data collected at different frequencies to produce a consistent time series• Example: quarterly and annual value-added

estimates which are compiled using different data sources such as quarterly and annual surveys

It is usually done retrospectively as annual benchmark data are available sometime after the quarterly data

3

Difference in coverage and sample• Annual survey has broader coverage and a more

representative sample• Differences in frame and sample size may exist

Difference in definitions and variables• Output replaces value added for growth measures

Accounting methods• Different quarterly and annual accounting methods

Estimation method, non-response treatment, respondent error, sampling error...

4


Reasons for differences between quarterly and annual data sources


Aspects of benchmarking• Quarterization of annual data to construct time

series of historical QNA estimates (back series) and revise preliminary QNA estimates to align them with new annual estimates when they become available

• Extrapolation to update the series from movements in the indicator for the most current period (forward series)

5

Purpose of benchmarking

Combines the relative strengths of low-frequency data (say, annual data) and high-frequency data (say, quarterly data) while preserving as much as possible the short-term movements

Creates a coherent high-frequency data series by correcting the difference between benchmark and indicator values (indicator bias)• Ensures benchmark to indicator (BI) ratio becomes 1

Example: Adjust indicator so that

6

,q yI 4

,1

BI 1y

q yq

A

I

Purpose of benchmarking

Ensures, for forward series, that the sum of the four quarters of the current year is as close as possible to the unknown future annual data

Improves quality of national accounts data in terms of comparability and coherence

7

Benchmarking methods

Numerical methods• Pro rata distribution method• Proportional Denton method

Statistical modelling methods• ARIMA-model-based methods• General least-squares regression models

Cholette Dagum Chow-Lin

8

Pro rata distribution method

For any benchmark year, distribute annual value in direct proportion to the quarterly values

,, ,

, ,

q y yq y y q y

q y q yq q

I AX A I

I I

9


,q yX is the level of the QNA estimate for quarter q of year y

,q yI is the level of the indicator for quarter q of year y

yA is the level of the annual benchmark value for year y

,

y

q yq

A

Iis the BI ratio

Pro rata distribution method

Advantages

Easy to compute and interpret

No special software needed

Quarterly estimates can be derived each year independently

Estimates are well aligned to benchmark value and are fairly reliable when BI ratios are stable

Disadvantages

Smoothens quarterly estimates only within a year

Concentrates bias in one quarter and cause abrupt change (“step problem”)

Not recommended for longer time series

10


Example of pro rata distribution method

11

Yellow cell shows step problem caused by change in BI ratioSource: IMF QNA manual (2001)


Pro rata distribution method – step problem

Source: IMF QNA manual (2001)

12


Pro rata distribution methodBenchmark-to-indicator ratio

13



Proportional Denton method Goal: Find new estimates with minimal

deviation from original indicator series

1 4

2

1

( ,.. ., )2 1

minT

Tt t

X X Xt t t

X X

I I

1,...4 ,..,t T

2

, 1,...,T

t yt

X A y

under restriction (for flow series):

14


t is time

tX is the derived QNA estimate for quarter t

yA is the annual data for year y

is the last year for which an annual benchmark is available

tI is the level of the indicator for quarter t

T is the last quarter for which quarterly source data are available

Proportional Denton method

Avoids the step problem seen in the pro rata distribution method by implicitly constructing from the annual observed BI ratios a time series of quarterly benchmarked QNA estimates-to-indicator (quarterly BI) ratios that• Is as smooth as possible• Average to the annual BI ratios for each year for the

back series• Are kept constant and equal to the ratio for the last

quarter of the last benchmark year Ensures quarterly growth rates are adjusted by

relatively similar amounts

15


Example of proportional Denton method

16



No big jump in

period-to-period

change

Proportional Denton methodSolution to step problem

17



Proportional Denton methodBenchmark-to-indicator ratio

18




Method requires that the indicator contains positive values only

For series with zeros but no negative values• Replace zeroes with values infinitesimally close to zero

For series with both negative and positive values and are derived as the difference between two non-negative series (for example, changes in inventories)• Apply method to opening and closing inventory levels, or• Turn the indicator into positive series by

Adding a large constant to all periods Doing benchmarking Removing the large constant from the results

19



Impact on back series • QNA quarter-to-quarter growth rates differ from those in the

indicator• May introduce new turning points, or • May change timing of turning points• These changes are desirable result of incorporating

information in the annual data Impact on forward series

• Quarter-on-quarter growth rates are identical to those in the indicator

• But, annual growth rate for the first year of the forward series differs from that of the source data

20



Forward series estimates are of great interest to users

Forward series estimates can be improved to reduce size of future revisions by using enhanced version of proportional Denton method which forecast annual BI ratios by incorporating information on past systematic movements in the annual BI ratios• Example

A study of movements in annual BI ratios show that the indicator on average understates annual rate of growth by 2.0%

Adjust annual BI ratio for forward series by 2.0%

21


Example of enhanced proportional Denton method

22



Statistical modelling methods

Take into account any known information about the stochastic properties of the series being benchmarked

Were mostly actually proposed to improve survey estimates where the survey design may provide information about the stochastic (sampling) properties of the series

In QNA, information about the stochastic properties of the series is usually non-existent and non-sampling errors may be more important than sampling errors

23


Statistical modelling methods

Unlike the Denton method, these methods may over-adjust the series by interpreting as errors, and thus removing, true irregular movements that do not fit the regular patterns of the statistical model

Models may mis-specify the error terms, resulting in inaccurate estimates

24


BENCH ECOTRIM XLPBM

25

Benchmarking software for QNA

BENCH

Was developed by Statistics Canada Runs under Unix/DOS operating systems Allows users to choose among many alternative

options offered by the Cholette-Dagum regression-based model

Requires input data to be prepared in pre-formatted text files

26


ECOTRIM

Was developed by Eurostat Runs in Windows Provides interactive and batch support Allows input data to be prepared in text files or

spreadsheets Allows validation of results using graphical

tools Has option for Denton method

27


XLPBM

Was recently developed by IMF Statistics Department

Comprises a Microsoft Excel add-in function with options for the enhanced proportional Denton and Cholette-Dagum methods

Is particularly useful for QNA processing systems based on spreadsheets.

28


Benchmarking is needed to derive QNA series that are• Temporally consistent with the ANA benchmarks• Preserve as much as possible the movements in the

quarterly indicators• Provide accurate extrapolations for the current year

There are a number of methods to perform benchmarking

The pro rata distribution method is not recommended due to the step problem

The proportional Denton method is robust, simple and well suited for large-scale applications

A number of software are available to do benchmarking

29

Conclusions

What method is your country using to do benchmarking in the compilation of quarterly GDP estimates? Why?

Is the benchmarking method described in the national accounts methodological notes?

30

Questions

31

Thank you

Documents

United Nations Statistics Division Benchmarking Training Workshop on the Compilation of Quarterly National Accounts for Economic Cooperation Organization