Applying Process Indicators to Monitor the Editing Process

Purpose of Process Indicators

• Monitor and improve the editing process

• Find measurement errors and their causes and prevent them in future survey rounds

• Communicate quality to the users

International Trade in Services

Statistics on international trade in services, wages and transfers are based on a quarterly sample survey involving about 5 000 enterprises and organizations.

Report values of income and expenditure for about 100 service types.

Editing in the survey

• Selekt is used• Fatal errors are always followed-up• Suspicious errors are used to create a local score

for each service type based on potential impact and suspicion

• Local scores are aggregated to a global score for each enterprise

• A threshold determines whether the enterprise needs to be followed-up or not

Descriptive information

  2013Q1 2013Q2 2013Q3Total number of observations 4041 3876 3890Number of observations failing at least one edit rule  438 297 403Number of observations failing at least one edit rule and adjusted  258 187 168

  Expenditure     Income      2013Q1 2013Q2 2013Q3   2013Q1 2013Q2 2013Q3Weighted raw value sum 373 433 302 457 694 730

Weighted raw value sum, failing at least one edit rule

319 373 194   368 616 538

Weighted edited value sum 129 130 151   216 214 297

Overall failure rate, 2013Q1-2013Q3

2013Q1 2013Q2 2013Q30

0.02

0.04

0.06

0.08

0.1

0.12

I1 Overall failure rate. Rate of observations failing at least one edit rule.

 

 

Overall hit rate, 2013Q1-2013Q3

2013Q1 2013Q2 2013Q30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

I2 Overall hit rate. Rate of observations failing at least one edit rule and adjusted.

 

 

Weighted reject ratio, 2013Q1-2013Q3

2013Q1 2013Q2 2013Q30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Income Expenditure

I3 Weighted reject ratio.   

 

 

Failure rate and hit rate for each edit, 2013Q1-2013Q3

I4

I5

Failure rate for each edit FR.

Hit rate for each edit HR 

    

 

Edit rule 2013Q1   2013Q2   2013Q3    FR HR FR HR FR HRQ03 5.05% 54% 0.77% 60% 1.93% 91%Q07 0.12% 20% 0.08% 67% 0.08% 67%Q08 6.14% 67% 6.22% 65% 5.66% 53%Q09 5.32% 68% 5.08% 64% 4.50% 50%Q10 0.40% 63% 0.34% 23% 0.39% 60%Q11 0.89% 53% 0.88% 24% 1.05% 29%Q12 0.74% 77% 0.80% 77% 0.64% 76%Q14 2.13% 72% 2.66% 67% 2.24% 64%Q15 0.20% 100% 0.03% 100% 0.10% 100%Q16 0.64% 85% 1.86% 89% 0.80% 87%Q19 0.47% 95% 0.80% 84% 0.72% 89%Q20 0.17% 57% 0.18% 29% 0.10% 50%

Editing rate for each variable, 2013Q1-2013Q3

I6 Editing rate for each variable.

  

 

  Expenditure     Income      2013Q1 2013Q2 2013Q3 2013Q1 2013Q2 2013Q3

410 Computer services 0.8% 0.4% 0.4%

463 Other services between affiliated enterprises  0.4% 0.4% 0.3%

442 Architectural, engineering and other technical services 0.7% 0.5% 0.4%

143 Goods freight by road 0.3% 0.3% 0.3%

Problems

• Tedious to calculate• Difficult to interpret• Often small amounts of data

• Variable specific indicators depend on random fluctuations

Data availability

• Dependent on available data • Some thoughts on which indicators we want

should be considered when setting up the system• Structure of data influences difficulty to generate

indicators• A system that generates standard reports could be

beneficial

Conclusion

• Standardized indicators could be beneficial• Needs guidance on how to interpret and use them • Will take time to establish indicators in the

organization and within the management teams

- What possibilities do we have to spread them to the organization?

Questions

• Do you use indicators frequently?• Are they established in your organization?• Have you seen any improvements due to the use

of indicators? • Are standardized indicators a good idea?