Esomar 26 Questions - FAQ for Epinion Online Panel

WELCOME TO OUR WORLD OF RESPONDENTS

EPINION ONLINE PANEL FAQ

EPINION

2011

EPINION ONLINE PANEL FAQ

ESOMAR is the global organization for pro-fessional market research agencies.

�They have challenged the industry by for-mulating 26 questions all research buyers should ask when buying online surveys.

�Here are our answers.

�If you have further questions, don’t be afraid to challenge us.

2

sources which will provide us with the sort of panelists that may at any given time be in short supply in our pan we have created a global quarantine system that ensures that no potential panelist wherever that person (browser) is located around the world gets invited to our panel more than once pr. month.As a result, for each of our country panels, we measure and define quality in three dimen-sions:

Current size of active panelists Current monthly inflow of new panelists Current active recruitment channels (we aim at 75 or more open channels pr. coun-try)

02

03

01 What experience does your company have with providing online samples for market research?

The Epinion sample only technology has been in operation in Denmark for more than 3 years, and recently we passed our first milestone which was 2 million completed interviews on the system. We build our first online panel back in 2001 and has been refining and devel-oping our online methodologies and approaches ever since.

Please describe and explain the types of source(s) for the online sample that you provide (are these databases, actively managed panels, direct mar-keting lists, web intercept sampling, river sampling or other)?

Our strategy is to resemble as closely as possible the principle of equal sample-extraction probability known from traditional off line research. Our recruitment ensures that any online population over an extended period of time is equally exposed to invitations to join the panel (in no cases, we allow people to opt in to the panel without being invited). We do this by making sure that we have a large number of open recruitment channels at all times. Moreover, our system makes it possible for us to direct our recruitment efforts to

What do you consider to be the primary advantage of your sample over other sample sources in the marketplace?

The competitive advantages of the Epinion recruitment strategy in terms of delivering superior quality sample originate from two distinct traits of the business. First, we keep very close attention to maintain-ing a large number of different recruitment channels in each country as this is the only way to resemble the random probability sam-pling techniques that is used in good quality offline interviewing. We use the term ‘resem-bling’ off line sampling, as it is obviously not in

>3YEARS IN OPERATION

>2.0MILLION INTERVIEWS WAS COMPLETED BY THIS SYSTEM

EPINION ONLINE PANEL FAQ 3

any real sense possible to create a known, well defined, universe from which to draw respondents. However, representativeness is all about giving internet users an (approximated) equal probability to be exposed for an invitation to join the panel and later on to answer surveys. The Epinion approach adheres strictly to this principle. Secondly, the unique Epinion sample builder makes it possible to stratify sample so precisely that it is rarely necessary to post-stratify data (weigh data) according to demo-graphics. This is so, because:

Regardless of what combination of Soci-odemographics that are used to define a sample (say, women 15-29 years of age), the system ensures that all the subsegments in the underlying sample-matrix (region, educa-tion, income etc.) are representatively distrib-uted based on the latest statistics from the relevant National Bureau of Census.

This is so because the system automati-cally monitors whether the required number of interviews in each of the subsegments in the samplematrix are reached. If not, more invites are distributed to the specific subsegment in question. This repetitive process ensures incredibly representative sample.In combination, these advantages make for superior sample compared to traditional online sampling.

04 If the sample source is a panel or data-base, is the panel or database used solely for market research? If not, please explain?

The panel is only used for market research - nothing else. The sample source is a panel consisting of panelists that have actively opted in to participate in the Epinion panel.

05 How do you source groups that may be hard-to-reach on the internet?

We source these groups by contacting sites that are visited primarily by hard to reach target groups. For example, in many countries young low-educated males are a hard to reach target group. In these countries, we make a special effort in reaching out to local football-sites where we know that this group is over-represented.

06 What are people told when they are recruited?

People are first asked a series of questions mainly related to background demographics and then they are faced with the following question:Would you be interested in becoming a member of Epinion's shared opinions panel?


Yes, I would like to join No, thank you.The exact wording of the question is slightly different for each person, since we have long ago established that different types of argu-ments appeal to different socio demographic groups. Young people should be recruited using a different approach than old people.

07 If the sample comes from a panel, what is your annual panel turnover/attrition/retention rate and how is it calculated?

We do not have an exact number for panel retention. Actually, we don´t see panel reten-tion as a goal in and of itself. We do not believe in the panel retention strategy of many other online research companies. It may be good for profits and it is even used as a sign of quality to build up a narrow pool of people making good money out of answering ques-tions. But data from professional panelists does not mirror the average consumer. There-fore, we only use prize draws for small amounts of money as monetary incentives.However, as we take care to give respondents good feedback opportunities, appealing invites and not too long surveys, we have many panelists that complete multiple surveys. The average lifetime number of completed surveys pr. panelist is approximately 8 surveys.

08 Please describe the opt-in process

In order to become a member of Epinion‘s internet panel you have to be invited by Epin-ion. We do not accept voluntarily signups at our website or at other places. All respondents must complete a 4 step verification process to become members:When invited, first of all:

1. You have to accept our Terms & Condi-tions, 2. Then you have to accept to participate in our establishment survey, 3. You need to provide us with full demo-graphic background information in the estab-lishment survey, 4. Provide us with your e-mail address and then one more time, double confirm that you in deed wish to become a member of the panel.

Unless all of the above steps have been com-pleted using Epinion recruitment system (no affiliate recruitment is allowed), we do not consider the applicant to be a member of Epinion’s panel.

8.0

4 STEPS TO OPT-IN

> Accept Terms and Conditions

> Accept Survey Participation

> Provide demographic background

> Provide contact email address

INTERVIEWS ON AVERAGE OF A PANELIST LIFE TIME


09 Do you have a confirmation of identity procedure? Do you have procedures to detect fraudulent respondents at the time of registration with the panel? If so, please describe?

Since we do not pay individual monetary rewards to respondents, we can rely on e-mail identification as our main identification proce-dure. All invitations are distributed / linked to a unique e-mail address. Without access to this e-mail address, you cannot participate.As part of the survey process we monitor fraudulent respondents using a series of methods described in greater detail in ques-tion 24. Also, we select and invite panel mem-bers, they don’t select us. Respondents cannot find ways to self recruit, thus making it more difficult for fraudulent respondents to create multiple accounts.

10 What profile data is kept on panel members? For how many members is this data collected and how often is this data updated?

The Respondents are profiled with socio-demographic criteria. The exact wording of the profile questions is developed carefully and is in line with the acclaimed World Values Survey. This makes sure that categories are compat-ible across different countries (elsewise a very

difficult thing to handle when making cross-country surveys). As a result, it is possible to make exact comparisons across different countries between for example: highly edu-cated vs. low educated or between occupa-tional categories. In the early days of our panel-buildup, we profiled the panelists with lots of different variables: Car ownership, newspaper-subscription etc. However, it turned out that out of perhaps 150 variables we only made good use of a few. And often we found that the variables were obsolete when we actually needed them. In the meantime, the panelist have bought a new car, changed newspaper-subscription etc. Therefore, we now take a more dynamic approach to profiling meaning that we in ad hoc surveys reach narrow target groups by use of a sophisticated screening system. When wording screeners we take care to avoid self-selection bias. For instance by using a wording like: What newspaper do you read? (and then present a range of news-papers where one of them is the newspaper in question). Instead of asking: “Do you read newspaper A. Yes or No?”. Also, if clients want to create a specific subpanel, we profile over a short period of time a large part of the panel with the requested profile.


11 What is the size and/or the capacity of the panel, based on active panel mem-bers on a given date? Can you provide an overview of active panelists by type of source?At all times, one can see the current number of active panelists, the current inflow of new panelists and the number of active recruitment channels in all of our country panels at www.respomondo.com/panels.An active panelist is defined in a rather narrow way meaning that is a panelist that have responded to a survey invitation within the last 6 invitations.

12 Please describe your sampling process including your exclusion procedures if applicable. Can samples be deployed as batches/replicates, by time zones, geography, etc? If so, how is this con-trolled?

The sampling process is at the very heart of what makes Epinion very special in terms of online quality sampling. Our clients or we can on behalf of the client define a sample (say, women 15-29 years of age) and then the system will ensure that all the subsegments in the underlying sample-matrix (region, education, income etc.) are representatively distributed based on the latest statistics from the relevant National

13 Explain how people are invited to take part in a survey. What does a typical invitation look like?

People are invited to surveys by way of email-invitations. We work with acclaimed email newsletter-type companies and consistently work hard to maintain an industry-leading mail-reputation which is crucially important for mail deliverability among leading email-providers (Hotmail, Yahoo etc.).We operate with many different types of invitations depending on the type of survey. But under no circumstances can an invitation be sent out without active decline and panel-opt out links as well as the accept link itself.

211,000CURRENT ACTIVE PANEL MEMBERS

MEMBERS OF ONLINE PANEL (IN THOUSAND)

105

5

21

15

24

41

Denmark

Ukraine

Norway

Finland

Sweden

Vietnam


Census Bureau.The system automatically monitors whether the required number of interviews in each of the subsegments in the samplematrix are reached. If not, more invites are distributed to the specific subsegment in question. This repetitive process ensures incredibly repre-sentative sample.

15 How often are individual members contacted for online surveys within a given time period? Do you keep data on panelist participation history and are limits placed on the frequency that members are contacted and asked to participate in a survey?

Yes, we apply very stringent respondent quarantine practices. No respondent is invited to more than one survey every 12 days. They only receive one reminder per survey invitationWe save full statistics on invitations sent, responses (accept, decline), mail-server statis-tics, participation records on all respondents as long as they are members of the panel.

16 Is there a privacy policy in place? If so, what does it state? Is the panel com-pliant with all regional, national and local laws with respect to privacy, data protection and children e.g. EU Safe Harbour, and COPPA in the US? What other research industry standards do you comply with e.g. ICC/ESOMAR International Code on Market and Social Research, CASRO guidelines etc.?

We adhere to national and local laws, and beyond this, we follow ESOMAR´s International Code on Market and Social Research.

14 Please describe the nature of your incentive system(s). How does this vary by length of interview, respondent characteristics, or other factors you may consider?

Over the cause of the years, we have done quite a bit of research on what incentives work the best. As a result, we have structured our incentive/motivational system around the following:Monetary incentives: We almost always only use prize draws. We draw frequent winners but we award only small amounts of money at a time. We´ve found this works better than infrequent, bigger, awards. Even more, our findings suggests that micropayments to all participants do increase participation a little bit but first and foremost, it creates more flawed answers, more fraud, and above all a pool of professional panelists which in no meaningful way can be said to be representative for the average consumer.Feedback/Motivational incentives. After each survey, each respondent are prompted to give immediate feedback both by ranking the survey and by giving verbatim feedback. This provides the project manager with valuable feedback, but even more importantly, it gives respondents an opportunity to vent whatever feelings they have about the survey. This adds to our general “know-and-appreciate-your-respondent” policy.In some instances, we offer the respondent access to selected results from the survey and other specifically targeted incentives.


17 What data protection/security meas-ures do you have in place?

Epinion is concerned with safeguarding the information gathered through our Services. We employ a variety of measures designed to protect your information from unauthorized access. We work only with wellknown and professional database suppliers and only authorized personnel are allowed to conduct database operations.

18 Do you apply a quality management system? Please describe it

As most of our survey processes is embedded in sophisticated IT-technology much of the quality control is embodied in the IT-development process itself. In this process, a variety of techniques has been put to use in manual as well as automated testing.In terms of work-procedures between different departments and individuals and with respect to the content of deliveries designed and produced by our consultants or other staff, our processes and deliveries are internally audited every 12 months.

19 Do you conduct online surveys with children and young people? If so, please describe the process for obtain-ing permission

In line with the ESOMAR code of conduct, we do not commission surveys among children unless we have asked their parents for permission.

20 Do you supplement your samples with samples from other providers? How do you select these partners? Is it your policy to notify a client in advance when using a third party provider? Do you de-duplicate the sample when using multiple sample providers?

Occasionally, we use some of our partners to provide sample in countries where our own panels are too small or over-extended. We only use Partners from a preferred supplier list. That is, trusted partners with proven quality and service. As we do not split-sample between our own panel and partner-panels, we do not need to deduplicate sample between two different panels.

30-45%PARTICIPATE RATE

1-25%

10-15%

RANGE OF DROP-OUT RATE

AVERAGE DROP-OUT RATE


22 What are likely survey start rates, drop-out and participation rates in connection with a provided sample? How are these computed?

Participation rate is usually between 30-45%. This rate depends greatly on a series of parameters, like time of day the invitations are distributed, the day of the week, the target group selected, e-mail invitation text provided etc. Drop-out rates are between 1-25%, with an average around 10-15%. The dropout rates depend greatly on length of the survey, as well as the content of the survey.We continuously monitor drop-out rates, accept-rates etc. on each survey. If perfor-mance is below average, we introduce sug-

23 Do you maintain individual level data such as recent participation history, date of entry, source, etc., on your panelists? Are you able to supply your client with a per job analysis of such individual level data?

We maintain a full record of all individual participants. Since we source only respond-ents from our own in-house fully profiled panel, we are able to retrieve a full log of all actions performed by our system in regards to a single respondent, as well as a full record of all the respondent actions in relation to our survey invitations.

21 Do you have a policy regarding multi-panel membership? What efforts do you undertake to ensure that survey results are unbiased given that some individuals belong to multiple panels?

Our panel management system automatically controls that there are no duplicate emails. In addition, in each survey, by way of browser-session, IP-address and cookie-tagging, we make sure that no respondent can answer any given survey (including tracking surveys) more than once.

gestions to improve the content of the survey and we very often penalize surveys with high drop-out rates.It is our philosophy to maintain a healthy relationship with the respondents to ensure repeat business. In order to maintain that relationship, we often reject long surveys (or insist that they broken down into several independent surveys), and evaluate the language used in the surveys using the WYMAT-principle (Would Your Mother Approve This?).


24 Do you use data quality analysis and validation techniques to identify inat-tentive and fraudulent respondents? If yes, what techniques are used and at what point in the process are they applied?

We use a mix of different techniques to identify fraudulent users. We combine ‘hard’ data from the participant’s computer/browser with behavioral data analysis in order to iden-tify fraudulent respondents. Hard data consist of: Session cookies, browser cookies, IP-address validation, e-mail validation (double opt-in required, no e-mail equals no survey) + a wide range of different browser plug-ins. We utilize a wide range of information about the browser used as well as the network connection used in order to verify the participant. All these hard data are com-bined with our ability to monitor deviations in behavior when the participant completes the surveys. We analyze features like: incoherent response-patterns, average time spend devia-tions, probability of completing open-text answers etc. Together, all these information’s create a credibility-index, which allows us to monitor suspicious behavior closely.But our most effective weapon against fraudu-lent behavior is to remove the monetary incen-tive to perform fraud. Epinion almost always offers price-draws as a reward for participa-tion. No cash rewards, rebates or points are


rewarded as ‘payment’ for each completed survey. We do not require that participants fit a spe-cific profile in order to receive the reward – in this case participate in the prize-draw. If you spend time answering our surveys, you partici-pate in the prize-draws no matter if the partici-pant fits the target group or not.We have completely removed the incentive to provide false answers in order to gain an advantage by earning money, points etc.

25 Do you measure respondent satisfaction?

Yes, after every survey is completed, all respondents are motivated to give feedback. A typical feedback page looks like this (but will be modified to suit different brands and languages):


We also use the invitation e-mails to motivate the respondents to provide us with feedback, also those respondents who fail to complete the survey.Epinion measures respondent satisfaction on five comparable parameters as well as collecting open verbatim feedback from the participants of each survey. This feedback goes directly to the client project manager. The project manager can always access the respondent feedback report which is updated in real time, as the respondent feedback comes in.Critical feedback from the participants (like technical problems with the

survey or questions that are difficult to understand) that requires immedi-ate action is automatically forwarded to the project manager by e-mail. The feedback collected from the respondents is then combined with hard data about completion rates, average interview time etc. into an aggre-gated easy to read feedback report format.

26 What information do you provide to debrief your client after the project has finished?

Along with the respondent feedback report shown above, the client can at all times access the current completion report showing exactly the progression of the survey in its lifecycle.Dropout rates, participations rates, invitations rounds executed, success rates etc. are all supplied in an easy to read interface.Epinion also provides the project manager with a ‘General Statistics‘-report benchmarking the survey performance of the project manager with Epinion averages internationally. The same report evaluates not only current performance, but also past performance of the project manager’s surveys, and provides recommen-dations on how to improve future survey projects.See slide from benchmark report belowformat.


EPINION COPENHAGEN�RYESGADE 3F2200 COPENHAGENDENMARKT: +45 70 23 14 23E: [email protected]: WWW.EPINION.DK

EPINION AARHUS�SØNDERGADE 1A8000 AARHUS CDENMARKT: +45 87 30 95 00E: [email protected]: WWW.EPINION.DK

EPINION SAIGON�11TH FLOOR, DINH LE BUILDING1 DINH LE STREET, DISTRICT 4, HCMC, VIETNAMT: +84 8 38 26 89 89E: [email protected]: WWW.EPINION.VN


Documents

Esomar 26 Questions - FAQ for Epinion Online Panel