Conceptual and Operational Issues in the Measurement of Internet Use *

CNNIC Symposium 2003 1

Conceptual and Operational Issues in the Measurement of Internet Use*

Jonathan ZhuCity University of Hong Kong

[email protected]

* Funded by the UGC of HKSAR (CityU1152/00H)

@

2CNNIC Symposium 2003

Background: the Diffusion of the Internet in Hong Kong, Beijing and Guangzhou

0

10

20

30

40

50

60

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

% o

f 18

-74

Pop

ula

tion

Hong Kong

Beijing

Guangzhou

Source: J. H. Zhu (2003)


Internet Penetration Rate in East Asia

50% 49% 41% 38% 37% 33%

% o

f A

du

lt P

op

ula

tio

n

Japan Hong Kong Singapore Taiwan BJ/GZ Macau


Wired Internet Use vs. Wireless Internet Use

0%

10%

20%

30%

40%

50%

60%

70%

80%

Hong Kong BJ/GZ Japan

PC Home Brandband Home Wireless Web Users


Diffusion of Cable TV, the Internet, and Mobile Phone in Hong Kong

0%

25%

50%

75%

100%

% o

f P

op

ula

tio

n

Internet UsersMobile UsersCable TV


Internet vs. Mobile Phone in Beijing and Guangzhou

0

10

20

30

40

50

60

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

% o

f 18

-74

Pop

ula

tion

BJ WebGZ WebBJ MobileGZ Mobile


Issues in Measurement of Internet Use and Users The size of “Internet users” in a society is a

function of: Definition of study population (SP) Method of sample weighting (SW) Requirement of minimal usage (MU)

The amount of “online time” by Internet users is a function of: Definition of study population (SP) Method of sampling weighting (SW) Method of data collection (DC) Treatment of extreme values (EV)


Criteria for Evaluation of Measurement

Validity: how accurate or correct is the measure as compared with the “truth”?

Reliability: how precise or stable is the measure over time and/or across space?

Practicality: how efficient or economic is the measure in data collection and analysis?


Data

Hong Kong Survey 2002: telephone interviews of 1,800 residents at 6 and above in Dec. 2002 by Jonathan Zhu and his team

AC Nielsen/Netratings 2002-03: online tracking of 1,500 Internet users from 811 households in Hong Kong in Oct. 2002 and Jan. 2003.


Definitions of Study Population

WIP-Hong Kong: 18-74 CNNIC: 6+ Another popular definition: 18+ HK Census 2002:

6-17: 16.4% 18-74: 80.0% 75+: 3.6%


Impact of Population Definitions on Internet User Size

50.1% 48.5% 46.4%

0%

10%

20%

30%

40%

50%

60%

% o

f S

tud

y P

opu

lati

on

6+ 18-74 18+Data: Hong Kong 2002


Requirements of Minimal Usage

Minimal Usage Required?

Last Usage Specified?

Yes No

Yes ? ?

NoCNNIC (1 hour/week)

WIP


Impact of Minimal Requirements on Internet User Size

50.1%45.0%

48.5%43.9%

46.4%41.9%

0%

10%

20%

30%

40%

50%

60%

% o

f S

tud

y P

opu

lati

on

6+ 18-74 18+

WIPCNNIC

Data: Hong Kong 2002


Age Distribution of the Sample before and after Weighting

0

24

6

8

1012

14

166-

9

10-1

4

15-1

9

20-2

4

25-2

9

30-3

4

35-3

9

40-4

4

45-4

9

50-5

4

55-5

9

60-6

4

65-6

9

70-7

4

75-7

9

80-8

4

% o

f S

amp

le

Unweighted WeightedData: Hong Kong 2002


Impact of Weighting Methods on Internet User Size

55.3% 50.1% 54.0% 48.5% 51.2% 46.4%

0%

10%

20%

30%

40%

50%

60%

% o

f S

tud

y P

opu

lati

on

6+ 18-74 18+



Summary: Internet Users by Population, Usage Requirement & Weighting Method

41.9%

55.3%

0%

10%

20%

30%

40%

50%

60%

WIP/UW/18-74 WIP/W/18-74 CNNIC/UW/18-74CNNIC/W/18-74 WIP/UW/18+ WIP/W/18+CNNIC/UW/18+ CNNIC/W/18+ WIP/UW/6+WIP/W/6+ CNNIC/UW/6+ CNNIC/W/6+



A Mathematical Model of “True” Internet Users (TIU)

TIU = 55.3 – 1.4SP18-74 - 3.7SP18+ - 4.5MU – 5.4SW

(Adjusted R2 = 99.6%, Standard Error = 0.3%)

Where TIU is the “Unadjusted” Internet Users (%) for HK in 2002, which should be 1.4% less for a study population of 18-74, or 3.7% less for a study population of 18+, or 4.5% less if those use the Internet less than 1 hour per week are excluded, or 5.4% less if the sample is weighted based on population census.


Impact of Population Definitions on Online Time (at Home)

424 412

050

100150

200250300350400450

Min

ute

s p

er W

eek

6+ 18-74



Impact of Weighting Methods on Online Time (at Home)

517 424 468 412

0

250

500

750

Min

ute

s p

er W

eek

6+ 18-74



Impact of Extreme Values on Online Time (at Home)

499 424 473 412

0

250

500

750

1000

Min

ute

s p

er W

eek

6+ 18-74

Raw Data EV RemovedData: Hong Kong 2002


Impact of Data Collection (DC) Methods on Online Time

424236

412239

0

250

500

Min

ute

s p

er W

eek

6+ 18-74

Phone Interview Online Tracking

Data: HKS 2002 & Netratings 2002-03


Summary: Online Time by SP, SW, DC, and EV

581

241 209

468

0

100

200

300

400

500

600

W6+/Raw W18-74/Raw UW6+/Raw UW18-74/Raw

W6+/No EV W18-74/No EV UW6+/No EV UW18-74/No EV

W6+/Raw W18-74/Raw UW6+/Raw UW18-74/Raw

W6+/No EV W18-74/No EV UW6+/No EV UW18-74/No EV



A Mathematical Model of “True” Online Time (TOT)

TOT = 532 + 16SP18-74 – 22SW – 49EV - 249DC

(Adjusted R2 = 93.5%, Standard Error = 34.3)

Where TOT is the “Unadjusted” Online Time (min.) for HK users in 2002, which should be 16 min. more for a study population of 18-74, 22 min. less if the user sample is weighted, 49 min. less if extreme values are removed, or 249 min. less if data are collected through online tracking method.


Caution: Different Definitions of “Online” Activities

Telephone interview data include: Online time at both

home (68%) and elsewhere (32%);

Non-HTTP based activities such as using POP3 Email (=136 min./week) and other protocols;

Online tracking data include: Online time only at

home; Only HTTP=based

activities protocols).

It is estimated that tracking data may measure only 51%

of the total online time..


Estimated Distribution of Online Time by Location and Protocol of Usage

Usage Location

Online Activities

HTTP based Non-HTTP Total

HomeOnline

Tracking (51%)

17% 68%

Elsewhere 24% 8% 32%

Total 75% 25% 100%


Conclusion: How Many Internet Users Are There? The size of “Internet Users” is significantly affected by the

definition of study population (SP), the requirement of minimal usage (MU) and the method of sample weighting (SW).

SP (e.g., general population vs. adults) may produce a difference of 1-4% and MU (e.g., no requirement vs. 1 hour per week) up to 5%. While there is no “correct” definition of SP or MU, it is important to report the definition and adopt, whenever possible, multiple definitions.

SW (weighted vs. unweighted) may contribute another 5% difference. Since Internet use is highly correlated with age and sex, it seems both necessary and effective to weight the sample to ensure the accuracy of the measurement.


Conclusion: How Much Time Do They Spend Online? The amount of online time is marginally affected by SP (p =

0.3) and SW (p = 0.2) probably due to the fact the base of analysis is already restricted to users.

Online time is significantly affected by the treatment of extreme values (EV), which may inflate online time by up to 10%. It is thus necessary to control for it (i.e., removing EVs).

Online time is most significantly affected by the method of data collection (DC, e.g., interviews vs. online tracking), which may result in a difference of 2-folder. Although online tracking is generally more accurate, it is far more expensive and impractical in many societies. It is thus important to keep in mind the magnitude of inflation in self-reported data.


Ultimate Criteria for Evaluation

Validity: how accurate or correct is the measure as compared with the “truth”?

Reliability: how precise or stable is the measure over time and/or across space?

Practicality: how efficient or economic is the measure in data collection and analysis?


Consistency in Measurement of Internet Users over Time and across Space*

0%

10%

20%

30%

40%

50%

HK Beijing Guangzhou

% o

f S

am

ple

2000 2001 2002* Based onWIP definition.


Stability in Measurement of Sex Ratio among Internet Users in Hong Kong

53%

47%

54%

46%

54%

46%

0%

25%

50%

75%

100%

2000 2001 2002

Femal eMal e


62%

36%

69%

29%

62%

36%

0%

25%

50%

75%

100%

2000 2001 2002

Office

Elsewhere

Home

Stability in Measurement of Online Locations in Hong Kong


Consistency in Difference between Methods across Age Cohorts

Age

Telephone Interview

Online Tracking

Interview/Tracking

18-19 10.72 6.53 1.64

20-24 8.49 5.54 1.53

25-29 7.06 4.21 1.68

30-34 5.24 3.62 1.45

35-39 5.50 3.51 1.57

40-44 5.02 2.98 1.69

45-49 3.72 2.58 1.44

50-74 3.51 1.84 1.91

Total 6.13 4.28 1.43


Final Verdicts Measurement of Internet users and online time

based on interviews data is largely reliable over time and across space.

The interview-based measurement is generally more practical than online tracking method.

The interview-based measurement is generally weaker in validity, as compared to online tracking method. However, it could be adjusted if the departure from the “truth” is known (e.g., based on comparison with online tracking data.

Documents

Conceptual and Operational Issues in the Measurement of Internet Use *