Exit Interviews Observation Methods Focus Group Discussions · A Practical Guide to Evaluating Natural and Cultural Heritage Interpretation Exit Interviews Observation Methods Focus

A Pract ica l Guide to Evaluat ing Natura l and Cul tura l Her i tage Interpretat ion

Exi t In te rv iews

Observa t ion Methods

Focus Group D iscuss ions

Gillian Savage

Jane James

2001

2 Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Gillian B Savage, BA

Gillian is a Director of the Centre for Visitor Studies which specialises in conducting audience research and evaluation for natural and cultural heritage places. Her tertiary training encompasses education, literature and the social sciences, and she has worked extensively conducting evaluation research in places such as museums, galleries, botanic gardens, zoos, cultural attractions and national parks.

With a depth of hands on experience, Gillian is active in exploring the theoretical background of research methods and in enhancing industry practice. In particular, she runs the website of the Evaluation and Visitor Research Special Interest Group (EVR SIG) of Museums Australia. This valuable resource for Australian practitioners can be found at:

www.amol.org.au/evrsig/

Version 03.1

© Gillian Savage and Jane James, 2001.

Dr Jane A James, BSc (Hons), PhD

Jane is a Senior Lecturer at Flinders University in South Australia and Coordinator of the Cultural Tourism degree programs. She has worked in tourism education for the past 11 years and has expertise in conducting training and interpretation programs for Government agencies, tourism operators and special interest groups.

She has extensive consulting experience in user demand surveys and analysis at Festivals and Events, and in Regional Tourism and at Heritage Sites in South Australia. She has conducted visitor satisfaction surveys in partnership with the South Australian Tourism Commission and Australian Major Events at a number of festivals, including Tasting Australia and Opera In the Outback.

She also has experience in interpretive planning, design and management.

Jane has worked as an international consultant, educator and researcher in the UK, China, Europe and Canada.

The Authors

Locked Bag 2116North Sydney

GPO Box 2100Adelaide

NSW 2059 SA 5001

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation 3

Foreword 4

Introduction 5

1. Interpretation 6

2. Evaluation 7

3. Exit Survey 11

4. Focus Group Discussions 17

5. Observation 23

Appendix

Contents


ACTION

This Practical Guide presents a skills-based program. In a skills program, what you learn depends on what you do.

Research shows that unless you use a new idea within 24 hours, you will probably forget what you have learned. So – make a phone call, write a note, tie a string on your finger – do something that will turn what you have learnt into action.

New skills are acquired and old ones improved by practice. So, run through it a few times, repeat it in different situations, try it out alone or with friends or colleagues.

SET GOALS

To get the most out of the exercises here, be clear on what you intend to gain.

Ask yourself:

What do I hope to change as a result of studying this Guide?

You are much more likely to follow through if you set goals now.

ACTIVELY ADAPT

Relate what you learn here to your own situation and needs. Keep your current problems and issues foremost in your mind and be thinking of ways that you can adapt the examples in this Guide.

Ask yourself:

How can I apply this to [... situation ...]?

Make these skills into a part of your armoury, don’t let them remain abstract knowledge.

NETWORK

If you are using this Guide in a Workshop situation, plan to introduce yourself to other participants and discover ways that you can help each other.

Exchange contact details so you can follow-up later.

Adapt . Amend . Ponder . Consider . Consult

Foreword

ACTSTRAIGHTAWAY

EchoRecountReview

DO ITSEVERALTIMES

RepeatRehearseReplay


This Guide is structured around action tasks. It aims to give practical advice on how to carry out Audience Evaluation using three methods:

• Exit Interviews

• Focus Group Discussions

• Observation Methods

Users will learn some foundation concepts in the field of evaluation as applied to the interpretation of natural and cultural heritage.

The material and examples in the book assume that users have experience of natural and cultural heritage interpretation, but assume no prior knowledge about evaluation methods.

The three research modules can be explored in any order. However, those who are interested in interviewing techniques will find material in both the Exit Interview section and the Focus Group Section.

We include examples from both indoor and outdoor settings and also cover guided tours as well as static displays. However, the settings we have chosen are only examples and we encourage users to apply these research methods to their own situations according to their information needs. For example, guided tours could also be evaluated by short exit interviews or by an observation study as well as focus groups.

Each module presents background information about the research technique and then walks through a typical project, step by step.

We emphasise practical usefulness over theoretical discussion.

Field researchmethods that arepractical, hands onand do-able

Introduction


People who work in the area of interpretation, subscribe to a number of formal definitions about what they do professionally, but they have one thing in common. They want to share the things that are special about their place – a museum, a park, a scenic view, an object, a gallery, a town – with others. Moreover they want to share it in such a way that passes on some of their passion to the visitor. They want the visitor to experience the same goose bumps, the same thrill, the same enthusiasm that they feel, at least in part, for their “special” place. To do this they rely on interpretation – an involvement in the “story” of the place.

Definition

Tilden in 1957 was the first person to formally define interpretation as:

An educational activity which aims to reveal meanings and relationships through the use of original objects, by first hand experience, and by illustrative media, rather than simply to communicate factual information.

Principles

He also defined six principles of interpretation:

• make the interpretation relevant to the visitor

• the core of interpretation is revelation based on information

• interpretation is an art (Ham (1992) refers to it as a craft)

• interpretation is more like provocation than instruction

• interpretation presents a whole rather than a part

• recognise the needs of different groups of people, specifically the children

Features

Much interpretative teaching and methodology has flowed under the bridge since Tilden’s initial forays into interpretation, but the basics of the definition remain essentially unchanged. The principles are sometimes simplified, as with Sam Ham (1992) who proposes that interpretation should:

• be pleasurable or fun

• be relevant

• be organised

• have a theme

Purpose

The purpose of interpretation remains the desire to share something special about a place, object or location with someone who visits it.

So, assuming that both professional and would-be professional interpreters are plying their craft (Ham), or practising their art (Tilden), how do you know if you have been successful in getting across your message, your passion, your information?

The answer lies in evaluation.

Engagingthe visitor

1. Interpretation


This book aims to outline the basics of evaluation. It will demonstrate the value of evaluation, when to do it, why to do it, how to do it and perhaps most importantly of all, what to do with the results of the evaluation. The “what to do with it” is perhaps the most empowering skill of all – it could range from self congratulations for getting it right to a need for an overhaul (major or minor).

Evaluation will provide valuable tools for planning for further interpretation, for resources, for personnel management, and for funding or sponsorship applications.

Above all, evaluation will provide irrefutable evidence of whether your interpretation is working, or whether what you are planning will work.

If it is – then that’s brilliant!! Be proud, tell your manager, tell us all!

If it doesn’t, evaluation will, most likely, indicate why not, thereby indicating, at least in part, that there is a problem. It may also suggest a way to rectify that problem.

Don’t despair, come up with a plan – an interpretative plan based on the principles of interpretation – then tell you manager you have the evidence (from the evaluation) of a problem, as well as, the beginnings of the solution.

Surely a win-win situation?

When to do it

The Business Model recommended by the ANZECC Working Group provides a benchmark for best practice. This model recommends that Evaluation be included in two ways. It should be a separate stage which sums up the effectiveness and value of the project, and it should be incorporated in the planning process as a feedback loop at every stage. These roles are summarised in the following diagram.

Fig 1: Role of Evaluation in Interpretation Planning

Source: Best Practice in Park Interpretation and Education. ANZECC Working Group

www.biodiversity.environment.gov.au/protecte/anzecc/index1.htm

The Model outlines a planning process which:

• Describes Objectives

• Defines Key Performance Indicators

• Discovers Critical Success Factors

The objectives in an Interpretation Plan can be:

• Management objectives - what you want to achieve as managers including promoting the site to attract more visitors, increasing repeat visits or increasing visitor spending at the site

• Interpretive or Communications objectives by creating specific visitor experiences or influencing visitors perceptions including:

• Learning objectives - what your visitors learn as result of your interpretation

• Emotional objectives - what your visitors feel as result of your interpretation

• Behavioural objectives - what your visitors do, where they go

Define

Deliver

DevelopEvaluate

Support

On-goingevaluation

+feedback

2. Evaluation


All of these objectives can be evaluated and you can prepare for this by the way you define your objectives.

When writing objectives, keep the SMART1 process in mind.

Planning which is SMART is more effective.

SMART = Specific, Measurable, Attainable, Results-focused and Timetabled.

1. See training session materials from the American Library Association's Annual Con-ference 2000 in Chicago, Illinois. Online at http://darkwing.uoregon.edu/~smichel/istraining/index.html.

Evaluation will be possible if your Interpretation Objectives are ‘SMART’.

Specific: you say exactly what you want to happen in concrete detail

Measurable: you know how you will measure it – numeric or descriptive

Attainable: feasible in scope and in your control

Results-focused: expressed as outputs, products or services

Timetabled: you know when you want it to happen, including interim steps

2. Evaluation


This section outlines the process of Evaluation and introduces core concepts of Visitor Research.

2.1 The process

The evaluation process checks interpretation products and services regularly and systematically for:

• Effectiveness in achieving their stated objectives;

• Degree to which Key Performance Standards are being met;

• Level to which Key Performance Indicators have been achieved;

• Continuing relevance of their objectives and design features;

• Wider anticipated or unanticipated impacts;

• Value for money.

Evaluation involves these steps:

1. Review objectives.

2. Set Key Performance Standard [e.g. the display will convey three key messages].

3. Determine a performance measure [e.g. a visitor survey].

4. Determine a Key Performance Indicator [KPI] that relates to the measure [e.g. 80% visitors know the three key messages or 100% visitors learnt one of the key messages].

5. Gather the data.

6. Apply the data and draw conclusions.

Critical Success Factors

These are the high priority factors which underpin success. KPIs are the tools used to assess how well Critical Success Factors have been achieved. Evaluation can be time consuming, so concentrate your efforts on the factors that really matter.

2.2 Visitor Research

The assessment methods which are the focus of this workbook are visitor research methods. Visitor research aims to provide data which can give direction to interpretative planning or to allow judgements to be made about the effectiveness of displays or programs.

Visitor research methods can be —

Evaluation is carried out at different stages of project development:

In the earlier stages, exploratory research can describe visitor knowledge, needs and interests to give direction to the interpretation. Concepts and plans for interpretative ideas can be shown

Direct Indirect

Interviews or questionnaires

Unobtrusive observation

Quantitative Qualitative

Numerical data in the form of counts, time spent, etc., which can be analysed statistically.

Data in the form of opinions, attitudes, perceptions, feelings or descriptions of behaviour.

Small amount of information from a large number of people.

Large amount of information from a small number of people.

Beginning Middle End

Front end Formative Summative

Exploratory Specific questions

Sums up

Focus groups, interviews, questionnaires

Observations, questionnaires

Observations, questionnaires

2. Evaluation


to visitors and their responses noted. An open ended approach is useful here because it allows the research to uncover things that were not necessarily anticipated. Qualitative methods are very appropriate.

In the middle stages, research can be used to pilot test specific aspects of the interpretation. Small scale field trials involving some observation or short interviews will be sufficient to test whether the right messages are getting across.

In the final stages, visitor research can be used to gather data about the effectiveness of the interpretation in meeting its communication aims.

2.3 Data Collection Process

So, you have reviewed the objectives of the project which are to be evaluated using visitor research. You have thought about Key Performance Standards, KPIs and Critical Success Factors. Now you want to plan a data collection study using either a survey, some focus groups or observation methods.

Here are some steps that all data collection projects follow.

1 . List your information needs and against each item write how you will use the data (see p11).

2 . Write a task list for the project.

3 . Plan your resources in terms of budget, equipment, skills and time availability.

4 . Decide what tasks to carry out in-house and what tasks to outsource. Be clear about who is responsible for the overall project.

5 . Manage the distribution of tasks and their completion.

6 . Apply the data to the question in hand and look at the wider implications of it.

2.4 Drawing conclusions

Once the data is collected, then evaluative judgements need to be made.

Where KPIs have been defined, the measures give a simple indication of whether or not the goal has been met [e.g. more than 80% of visitors know the three key messages].

In other cases, the evaluative judgements are less clearly defined. It may be that evidence from a range of sources such as exit interviews, observations and staff comments might point to a certain conclusion.

A simple process for drawing conclusions is:

1. Summarise the findings from all sources.

2. List some implications.

3. List appropriate responses.

4. Prioritise the responses in terms of need and practicality.

Checklist – Define your information needs

1. Write down all the things you want to know about the topic, including both the general and the specific.

2. Invite colleagues to think of things they would like to find out.

3. Sort through all the ideas, weed out duplicates and group similar things together.

4. For each item on the list, write down how that information would be applied.

5. Prioritise the information on the list. Three categories are useful here: must know; useful to know; nice to know. You will probably discard the ‘nice to know’ items.

6. Highlight all the items that can be answered using survey/focus groups/observation.

2. Evaluation


This module presents a small scale exit interview project as a demonstration of the process of carrying out on-site interviews with visitors. The example is carried out in a museum or visitor centre with an interpretative display, but it can be adapted to a wide range of situations.

We go through the process step-by-step.

3.1 Information needs

Firstly you need to be clear about what you want to find out and how you will use the information. It may be tempting to use this opportunity to ask all the things you have been wondering about, but the project will be much more useful, and do-able, if you keep a tight focus.

Some things you might want to know might be:

• demographic profile of visitors

• their understanding of the main message

• their understanding of specific messages

• overall rating

• suggestions for improvement

Make sure that most of the information on your list relates to the original goals and objectives for the project. Now you will see the value of writing performance indicators in ways that are measurable! Remember the SMART objectives outlined in Section 2.

It is good practice to list these information needs and to note for each one how you will use the results. It helps if others in the organisation have some input, as this prepares the ground for wide-spread acceptance and use of the information in your organisation.

For example —

3.2 Plan your resources

Resources include skills, equipment, time and money – a shortage in any of these areas may compromise the project. You will either find the necessary skills within your organisation or you will outsource some elements of the work. To some extent, you will shape the project to fit the resources that are available.

The key skills you will need for an exit interview project are:

• interviewing skills;

• clerical skills for tallying the data or doing data entry;

• writing skills to summarise the findings in a report.

These are discussed in the following paragraphs.

Information Use

Demographic profile

Know your visitors better, and shape future programs appropriately.

Main message Check that the intended message is getting across, check for unintended messages.

Specific messages Verify the effectiveness of communication.

Overall rating Benchmark against other displays and programs

Suggestions Refine and modify

3. Exit Survey


Interviewing is best carried out as a separate exercise from other visitor services. The person delivering the service is not usually regarded as an independent evaluator, so guides should not conduct interviews about their own tours. Also, interviewing requires a particular set of skills.

Interviews can be carried out by:

• staff who are a little separate from the service

• students on placement

• volunteers

• contracted interviewers.

Contracted interviewers are ready-trained in procedures and skills, but others will need some training. This should be factored into the project.

Depending on expertise, funds or equipment, the tallying or data entry can be done in three ways:

• pencil and paper tallying requires only time, patience and accuracy. It will give simple counts of the number who made each response and will probably be adequate for most purposes;

• spreadsheet tallying requires computer, software and a user who knows the basics of spreadsheet packages. It will give simple counts and allow some comparisons to be made. It also allows the data to be kept and used in flexible ways in the future (e.g. several surveys could be added together and seasonal differences noted).

• data entry into a statistics package requires the computer, software and a user who knows the software. This software is more specialist than a spreadsheet and unless you will be doing this regularly, it is better to outsource to a specialist agency. For a modest fee, the agency will turn your pile of questionnaires into neatly tabulated data. Like spreadsheet data, the data files can be kept and used in flexible ways and, indeed, data files are compatible with speadsheets allowing for transfer of data between software packages. Higher level statistical analysis can be carried out on the data.

The report writer needs to be able to write clearly about the purpose of the project, the steps followed and the findings. It helps if they know how to present data in charts and diagrams. They also need to have a good understanding of how the information can be applied in the organisation. The project manager is often the person who writes the report, or puts the final touches on a basic report prepared by an assistant.

3.3 Survey schedule

To give an accurate report of visitor views, interviews need to be planned systematically. If you want the survey to include all kinds of visitors, you will need to conduct interviews at different times of the day, across all days of the week, and perhaps taking care to include school holidays as well as school terms, or to cover both winter and summer.

To make the project more manageable, it is wise to restrict the interviewing to limited time periods (e.g. summer holiday only). When you do this, take care to report the findings as representing the views of summer holiday visitors and not necessarily representing the views of all visitors.

You will probably need to balance convenience with coverage. But you should be careful not to make large claims about data based on a limited sample of visitors.

Plain English

uses everyday language

uses shorter sentences

uses active rather than passive forms

avoids jargon and technical terms

uses headings, diagrams, lists and easy to read fonts

3. Exit Survey


3.4 Questionnaire design

First, you will need to decide whether to carry out face to face interviews or whether to ask visitors to fill out a self-completion survey. Some pros and cons are listed here.

If you can afford it, use interviewers to carry out interviews.

Self-completion is fine if you need to collect only a small amount of simple information. See examples of self-completion questionnaires in the Appendix.

Question types include:

• categorical (e.g. time of arrival, things seen and done. demographics, etc.)

• yes/no

• rating scales (agree/disagree or rating on excellence, etc.)

• open-ended (write in the answer).

Structure of interview

The interview is a social exchange and must conform to social rules. It should have a clear beginning, middle and conclusion.

Start the interview with a couple of really simple questions that everyone can answer, such as time of arrival and who they are here with.

Structure the main body of the interview then ask demographic questions at the end. If you really need to ask income level, make it the very last question.

It is good to include at least one open ended question in all surveys because it gives visitors a chance to express themselves in their own words.

Face to face Self-completion

More expensive Cheaper

More reliable sample of visitors

Stronger ‘volunteer’ effect in the sample

Better quality information - more specific and more detail

Simple tick box information

Introduce self and purpose of survey

Open with simple questions

Main flow of questions

Demographic details

Open ended question

Thank and close

3. Exit Survey

GOLDEN RULES

1. Try to plan the questionnaire so that as many questions as possible can be answered by ticking a box or circling a number. This reduces the workload on the visitor and the interviewer and makes it much simpler (and cheaper) to collate and analyse.

2. Try to frame questions in specific behavioural terms. Ask people to report actual behaviours rather than averages, tendencies or generalisations.


Questionnaire design process

Once you have drafted your questionnaire, test it with a couple of colleagues by asking them to play the role of the visitor. If possible, try to pilot it with some visitors.

You might regard the first day of interviewing as a pilot. In this case, you would print only enough questionnaires for the first day. At the end of the day, review the questionnaire with your interviewers, make the final changes then print. Resist the urge to keep making adjustments during the course of fieldwork.

See Appendix for example questionnaires.

3.5 Interviewer Briefing

Your interviewers may be experienced market research interviewers, staff, students or volunteers. When using inexperienced interviewers, you will need to run a training session where you explain some of the basic skills and rules of effective interviewing. See the Appendix for a set of Briefing Notes suitable for teaching some of these basic points.

All interviewers will need a Briefing Session which covers the following topics:

• purpose and scope of the survey

• interviewing schedule and quotas

• a run-through of the questionnaire, question by question

• orientation to the site (if necessary)

• logistics such as contacts, security and other matters

In the Briefing Session, you can be quite clear

about the kind of information you hope to gather, e.g. practical examples of what people actually do; or clear descriptions of what they found out.

The project will go more smoothly if you show an active interest by dropping by to see how things are going. Staying close to the project like this can also help you to nip potential problems in the bud. Collecting 200 questionnaires with a key error is very disheartening!

3.6 Data summary and analysis

Three methods of handling data were outlined in Section 3.2: tallying, spreadsheet and computer program. Whichever method is used, the questionnaires will need to be edited and coded.

Editing involves looking through each questionnaire and checking that all the questions have been answered properly. If the surveys are to be entered on computer then the editor should make sure that all answers are very visible so that data entry is smooth and swift.

Tallies are simple counts of the number of people who said different things. You might choose to divide the questionnaires into categories [e.g. male and female] and to tally within those categories before making an overall total.

Pre-coded questions can be tallied directly from the questionnaire, but open-ended questions will need to be coded first.

Coding open-ended questions This involves dividing all the comments into categories and allocating a code number to each category. The codes numbers are written on the questionnaires and then they are tallied like the other questions.

DraftTest

RefineUse

3. Exit Survey


Here is the process in more detail.

1. Draw up a coding frame. The coding frame is a list of numbers with comments beside them. All possible comments given by respondents should be fitted into one of the numbered categories. This list should be about 10-15 items long, and should include a ‘don’t know’ category and an ‘other comment’ category.

Every open ended question will have its own coding frame.

To draw up a coding frame for a question take a sample of about 20-30 questionnaires and write down all of the comments made. Where two people make the same comment, record this with a tally mark. Then look through the long list of comments and start to group them according to meaningful ideas. Look past the surface language to the underlying idea. Try to reduce the list to 10-15 items. Give a number to each item.

2. Use the coding frame to code the question. Now you go through all the questionnaires writing the code numbers against the question. Some questionnaires may be given several numbers for a particular question because the visitor said a number of things. Every questionnaire should have a code — people who said nothing would have the code for ‘don’t know’ or you may choose to add a code for ‘no comment’.

Once the data is coded, you can either tally it by hand, enter it in a spreadsheet or send it to a data processing house.

In the first two cases, you can carry out each tallying task individually. In the case of data processing, you will be asked to specify how you would like the tabulated data to look. The simplest and most powerful option is to ask to have all the questions cross analysed by a single ‘banner’. Some useful things to put in the banner:

agesexwith children or notfirst visit or notgeographic origin

See the Appendix for an example of a table showing data cross analysed using a banner.

Once you have the data in the form of tables, you can write your report. Summarising some of the data in graphs, charts or simple tables is the last step in data analysis.

3.7 Report

Plan for your report to make sense to:

• co-workers

• senior management

• somebody who picks it up in five years time

Your report should include the following sections:

• Introduction: what is the thing being evaluated, how will the information be used.

• Research Objectives: specific questions the research is trying to answer.

• Method: enough detail to allow the study to be repeated. Include the questionnaire in an Appendix.

• Executive Summary: outline of purpose and key findings.

• Findings: detailed findings.

• Implications (optional): suggestions for applying the results of the research.

If you have a project team or a work group, it is very good to get them involved before the report is written. Circulate the data to the team, then

3. Exit Survey


call a meeting to discuss what the results mean and how they can be applied. This is very effective in getting staff to support the results of the evaluation and also contributes insights that can be included in the report.

3.8 Circulate the results

Each research project adds to the knowledge of the organisation as well as adding to the knowledge of the project team. This new knowledge can be spread through the organisation in a number of ways:

• present the findings at existing staff forums such as staff meetings, planning meetings, professional development meetings, etc;

• circulate a summary of the key findings to senior staff;

• prepare a summary or abstract suitable for Board Reports or Annual Reports;

• put the report on the organisation intranet;

• represent the organisation in professional forums such as conferences by talking about the project, including the evaluation.

3. Exit Survey


4. Focus Group Discussions

Qualitative Research includes focus group discussions and indepth interviews. This Guide describes a ‘do it yourself’ focus group project, which can be adapted to a project which uses indepth interviews.

A focus group discussion is a special type of meeting process. It is not the same as a workshop, a staff meeting, a consultative meeting, a public meeting, a forum, seminar or or other kinds of gatherings and get-togethers.

Some key features of a focus group discussion:

• The group comprises about eight people with similar characteristics.

• A facilitator or moderator coordinates the meeting which is run as a free flowing discussion.

• The facilitator/moderator uses active listening techniques and does not reveal their own views.

• The results go beyond ‘what people said’ and take account of body language and the context. Results are often inferred from a range of contributing elements.

While most people are capable of convening a meeting and conducting it, focus group discussions draw on special skills. This workshop covers some of these skills at an introductory level, however expertise is only developed via practice under expert guidance.

Project stages

1. Define what information you need and say how you will use it.

2. Specify who you will talk to.

3. Organise recruitment, schedule and venue for groups.

4. Write discussion guide and prepare any materials you want to show.

5. Conduct the group discussions.

6. Collate, analyse and interpret the data.

7. Write report and apply the findings.

The following pages describe how to carry out each of these steps.

4.1 Define information needs

Follow the process outlined on p11 to define your information needs for the project.

4.2 Specify who to talk to

Focus groups work better if you have a fairly homogeneous group, that is partly why they are called ‘focus’ groups — because they focus on particular types of people and on particular topics. Generally it is better to conduct several groups amongst different kinds of people than to mix a wide range of people together.

For example, if you want to find out what visitors think of a new boardwalk that you have installed, you will find that a focus group of parents of young children will work better than if people of all ages are mixed together. People talk more freely when they are in a group where they have a lot in common.

Once you have decided who to talk to, draw up a list of specifications for each group. Here is an example for an evaluation of a new boardwalk where you are planning to add more interpretive material.

Group 1 Parents of children under 10 years who have been on the boardwalk.

Group 2 People under 25 without children who have been on the boardwalk.

Group 3 People aged 45+ who have no children under 18 and have been on the boardwalk.



You may also consider conducting a group or indepth interviews with teachers who have taken school groups and, perhaps, indepth interviews with people with disabilities.

It is rarely practical to conduct focus groups amongst tourists, and, if you want their feedback, you may decide to conduct some onsite interviews with tourists.

There are never enough resources to talk to everyone you are interested in, so some tough decisions need to be made about who to leave out.

4.3 Recruit and schedule

Ways to find the participants —

• use a specialist recruiting agency. Most large cities have agencies that specialise in doing this. They will charge about $45-$55 per person. You will need to write detailed specifications for each group.

• have a recruiter onsite to explain the project and invite people to come to one of the research sessions.

Using a professional recruiter, you can ask for 8 people and be fairly certain that they will turn up at the scheduled time. Our experience in recruiting directly onsite is that a high proportion say they will come but then don’t turn up. So you may need to confirm 12 attendees in order to be sure that 8 will turn up.

Respondents are paid a disbursement to attend. The current1 rate is around $40-$50. This payment is an important element in ensuring you get a good sample of people. Without the payment, you tend to get only those people who have plenty of spare time or who are especially interested in the subject. This will limit the

relevance of your findings because the respondents will not be representative of your likely visitor profile.

Focus groups can be conducted onsite if there is a suitable meeting room [education room, training room, etc.] however this often involves convoluted catering and security procedures. When the meeting is onsite, respondents are cued by the surroundings and less reliant on their personal memories of the experience.

Offsite meetings usually incur a fee for hire of the venue but the advantage is that there is less energy consumed in the logistics of ‘meet and greet’ and catering, etc.

Focus groups work well with 6-10 people. The more people you have in the room, the less each one gets to say.

4.4 Discussion guide and materials

To prepare for the discussion, you will need to write a Discussion Guide. This will ensure that you cover all the topics and that each of the groups is taken through a similar process.

The first items on any Discussion Guide are the welcome, housekeeping and ground rules for the discussion [see the example in the Appendix].

Plan your Discussion Guide as a series of topics [or headings] starting from broad areas and moving through to specific things.

Against each topic, write down the things you want to find out. Don’t write specific questions. A focus group discussion is not a question and answer session, it is a freely flowing discussion. If you are a beginner, you might want to ‘script’ the phrasing of some of your introductions and questions, but be prepared to ad lib rather than read from a page.

Think about different ways of getting the information you are after. Sometimes it is useful 1. January 2001. If you pay more than $50

there are GST implications.



to give respondents some activities to do as individuals or as a group. For example, you might write the names of places on cards and ask people to sort them according to some quality you are interested in, e.g. environmental responsibility. A group activity like this is very illuminating as people share their knowledge and views with each other when they complete the task as a group.

Your ‘kit’ for the day of the discussions will include:

• Discussion Guide

• Any visual material you want to show

• Materials for tasks you want to use

• Notebook for yourself

• Paper and pencils for participants

• Tape recorder and tapes

• Name tags

• Money for respondents [cash]

• Forms for participants to sign for cash

4.5 Conduct the group discussions

The researcher needs to establish rapport with respondents and encourage the group to talk amongst themselves. The discussion will start with a brief introduction and some ‘housekeeping’ information, then each respondent can be asked to say a little about themself. This ensures that each person speaks and gives everyone an idea of who the others are.

The first topic that is opened will act as a ‘warm up’ in which the group establishes a momentum of exchange, it also provides opportunities to learn general things about the activities and values of the participants.

The moderator should be clear about introducing a topic, encouraging exchange on it,

then moving on to another topic.

It is vital that the moderator accept every comment that is offered. Respondents should not feel that some views are more acceptable than others. All views are welcome. Take care to affirm and acknowledge what is said without showing approval or disapproval of the content.

Eye contact and body posture are important in creating a relaxed and open environment for discussion. If the moderator is tense, participants will find it harder to relax and open up. That said, a balance needs to be achieved so that individuals who are very socially dominant are ‘toned down’ sufficiently to leave space for others to contribute.

Reflective listening This is a great way of indicating that you have heard what someone has said without showing approval or disapproval. For example, if someone has described at some length their difficulties in getting around with a stroller. You might nod and say, “It sounds like you had problems with that,” then move the discussion along. “Uh-huh” is a good confirming line.

Open ended questions Questions which are open ended are valuable in opening discussion. An open ended question is one which can’t be answered with ‘yes’ or ‘no’, but requires some comment. For example, “What prompted you to go to the Park?” invites an explanation, whereas “Did you go because of the open day?” invites a yes-no response.

Leading questions are those which suggest what kind of response is considered desirable. Try to use phrasing which allows all kinds of answers to be acceptable.

Simple facts Asking people to recount simple facts is a good way of encouraging participation and also gives an opening for further questions.



For example, asking “What kinds of things do you do when you are at the Park?” gives a window onto their preferred activities and allows exploration of why they do those things, why they don’t do other things that are offered and what things they would like to do, if they were available.

Probing High quality information is specific rather than general and is well rounded. To get high quality information, you will need to probe.

Probe for:

• more material when only a little information has been provided. For example, if you want to get an idea how much people know about the subject of the interpretation you are evaluating, you will need to probe to find out ALL that they know. Keep enquiring gently until you are clear about the boundaries of their knowledge. Listen to them telling each other things, or asking each other. Be careful NOT to answer content questions that they may direct at you. You may want to confirm whether they know specific facts and this is a good way to check.

• specific examples when people are ‘waffling’ or speaking generalities. For instance, if someone says that they know all the National Parks in the area, you might ask which one they visited last and when that was. Some specific information goes a long way towards helping you understand how well they know the Parks. If they haven’t been to some for 10 years you will know that their knowledge is not up to date.

• to clarify when you are unsure what they mean by something. When clarifying, it is important not to rephrase what they said. You might ask,

I’m not quite clear what you mean, could you say a bit more about that?

Here are some examples of probing for more more material and for specific information when someone has made a general comment about

liking a picnic area and some walking trails —

You said that you liked the picnic area, what did you like about it?

You like the picnic area because.....? [with an enquiring look]

The picnic area... can you say a bit more about that?

What is the picnic area like?

Is the picnic area popular? Why do you think that is?

Confirming questions Sometimes it is helpful to confirm what you hear or observe the group to be saying. Don’t do this more than 2-3 times as it throws people into an analytical frame of mind which inhibits other forms of expression. Just confirm those things that you are really not sure of. Here’s an example —

It sounds like you are unsure about what happens there?

Group vs individual Try to encourage people to discuss, share, and exchange with each other and not just with you. This allows you to observe and note views and attitudes.

Some ways to encourage this,

Who else had experiences like this?

Have others experienced different things?

Is that a good idea?

What else do you know about this?

I see some of you nodding, do all of you agree with this?

Some things to watch for

On behalf of others Listen for when people talk on behalf of others. If locals start telling you what tourists think or do, you will need to ask them to tell you about themselves and what they think and do.



People often ‘project’ their own views onto others. For example, it is common for them to talk about what children need [usually hands on learning with low text loads]. When you think that people may be expressing their own needs by talking about others, you need to find a way to confirm that they, too, might like these characteristics.

Social dominance You may need to ‘damp down’ individuals who are more talkative, here are some suggestions:

• avoid eye contact with them

• select others for comment

• invite alternative views

• if necessary, say We’d like to hear the views of some of the others.

Researcher as observer If you succeed in getting the group members talking amongst themselves, it gives you a chance to observe and process the information so that new ideas can occur to you in the course of the discussion. These can be verified on the spot.

4.6 Analyse and interpret the data

What IS the data?

Analysis of qualitative data requires that we pay attention to what is said and how it is said, and also to what is avoided or left unsaid.

In the effort to understand participant meanings, we pay attention to all strands of communication, including — words spoken, the rhythm of discussion and exchange, types of discourse (e.g. debate or persuasion) as well as body language and facial expression.

We place varying degrees of importance on the material presented. For example, an articulate

lawyer may be very persuasive about the reasons for going to a museum, but if he has not been to one in the past 5 years, we have to discount his views as second hand. On the other hand, a group of teenage boys who respond to a suggestion with smiles, eye contact and shifts in body posture are indicating genuine interest even if their only verbal expression is ‘Yeah’.

Don’t rely on transcripts alone for interpreting qualitative material.

Data recording

Tape recordings or video recordings are the best ways of capturing the exchange that takes place in a focus group discussion. If the researcher is new to the process, the tapes should be transcribed verbatim if possible.

If recording is not possible then a note taker can be used. Preferably, the note taker should be in the room with the respondents and will need some training in recording in an unbiased way. One way to minimise bias is for the note taker to simply write down everything that is said, using the exact language of the respondents. They must NOT filter out material that they regard to be unimportant or less relevant.

A notetaker can be helpful when you are thinking about the meaning and implications of the findings – someone to verify perceptions and bounce ideas off.

Analysis

Analysis of qualitative material can use the following processes:

• counting and comparing — how many people used the word ‘discover’, is that higher or lower than when we studied XYZ?;

• categorising — this is a wayfinding issue, but that is a comprehension issue;

• noting patterns and themes — e.g. older people seem to emphasise this over that,



someone in every group mentioned the importance of ABC;

• seeing plausibility — it makes sense that parents would want to know that children could reach it;

• making metaphors — they are ‘all at sea’ and can’t quite grasp the central message;

• assuming particulars into general — they gave four different examples of stories they want to know about so there appears to be interest in narrative forms of presentation;

• noting relationships — the people who have been to XYZ seem to have higher expectations;

• finding coherence — what evidence ‘hangs together’ and which elements appear to be contradictory?

In this process we develop ideas and explanations and check the data for support. Some useful questions that guide our collation and analysis of qualitative data are:

• What are the broad-brush threads so far?

• What is puzzling, strange or unexpected in the data?

• What expectations are confirmed so far?

• This is interesting, but what are the implications?

• How confident am I that this is what it means? What else could it mean?

4.7 Reporting

Rather than just reporting everything people said, the written report should address issues and questions of specific interest to the project. It should report the findings of the study, and can comment on connections between the data and relevant theories from disciplines such as psychology, education, communication and

museology.

Two report styles

A fully detailed, narrative report, supported by examples and quotations from the fieldwork will provide strong evidence for the conclusions drawn from the study.

A summary action report is an effective way of communicating the main findings.

Applying the data

The following grid provides a convenient framework for applying research findings to project development.

The main findings from the research are listed under the four headings as shown in the following example.

When used as a cooperative exercise, perhaps using a whiteboard, this task can help the project team synthesise and apply the research findings.

Con

firma

tion

s

Un

ex

pecte

d

disc

overie

s

Cau

tion

s

Opp

ortu

nitie

s

Audibility is a key issue.

People are happy for the tours to teach values as well as facts.

Be careful not to be too strident about value-based messages.

Some tours could include quite a lot of material that is not directly related to plants.



5. Observational Evaluation

This module outlines the concept and methodology of observational evaluation, which uses unobtrusive methods to gain information about visitors and interpretation.

Observation includes the following activities:

• tracking

• counting and timing

• watching, listening and noting

To be effective, observational data needs to be:

• collected systematically, so it is not just a record of anecdotal information or opinions

• unobtrusive, so it records natural behaviour

Observation studies are generally small scale, i.e. you may observe 10-20 encounters rather than 100s.

Why evaluate by observation?

It is:

• quick and relatively simple

• cost effective

• useful for getting a basic understanding of physical and behavioural characteristics of visitors

• repeated over time, it can indicate trends

• shows what actually happens in terms of movement, emotions, interest levels

• distinguishes between “Streakers, Strollers and Students”, i.e. people who spend varying amounts of time at the site or exhibit

• achievable in short time frames

• able to produce reliable, quality data for analysis

• it complements other forms of evaluation, notably visitor questionnaires.

When do you do it?

• when the site is operational – to check effectiveness and plan remedies

• during development – to check that ideas work by making mock ups and drafts

Observation can show you:

• how visitors use the site/exhibition

• how the elements attract visitors’ attention

• how the elements hold visitors’ attention

• how the traffic flow pattern works

• group interaction

Before you begin

Some things to keep in mind —

• decide what you want to find out

5. Observation

What we do, or what we think we do?

When we evaluated the John Curtin exhibition at Parliament House we askedpeople how long they stayed in the exhibition. We also timed how long peoplespent there. On average, visitors’ estimates of how long they spent wereabout twice the times we recorded for actual visits.

Sometimes, visitor perceptions are quite wrong. Observation gives moreaccurate information about actual behaviour than visitors’ estimates of whatthey have done.


• do a pilot study and don’t try to discover too much – find a balance between knowing a little about a lot, and a lot about a little

• don’t just copy another study – modify it to suit your situation

• the ethical requirements of your organisation may limit the kinds of observation you do (e.g. use of video material). Some places require the display of a notice advising that observation is taking place

Observers should

• be discrete – act ‘invisible’ – keep a distance

• avoid contact with visitors, but if approached they should answer basic functional questions (e.g. where are the toilets?) then get back to their task

• not tell visitors what they are doing – if asked, they should offer a vague response like I’m just monitoring how things are working.

• carry out definite tasks of short duration (e.g. 60 minutes) as they will need to maintain high levels of attentiveness while they are observing – accuracy will be greater for three short tasks than for a single longer one

5.1 Tracking where people go

Tracking is used to track the movement of people around a site or exhibition, to see where people go and what they are interested in. It is time consuming but valuable, and can show you:

• how easily visitors orientate themselves and get to where they want to go

• how visitors choose what they will investigate based on curiosity

• visitor flow diagrams of how they move through a site

• what actually happens at a site or in a gallery, as opposed to perceptions of what ‘should’

happen

• how physical aspects of a site affect the way visitors move or use an area (e.g. lighting, width of walkways, slope etc)

• what visitors stop and look at

• what they show to each other and relationships between particular groups e.g. adult showing child and vice versa

• the level of use of a particular facility (e.g. touch screen), by whom and for how long (e.g. < 1 minute, 1-5 minutes, > 5 minutes).

How to do tracking

Tracking is most often carried out by having trained observers follow visitors discreetly plotting their path on a map (see exercise 1 in section 5.6).

Galleries and exhibitions often have security videos, and the film may be used to track visitor movement (ethical considerations may apply).

5.2 Counting and Timing

Counting is used to determine the number of people at a site or exhibition and timing is used to see how long they spend there. Counting and timing will show you a variety of things, including:

• the number of visitors in a particular location or at a specific site at particular times

• how many people/group size/age/gender

5. Observation


• what is the longest/shortest/average visit length

• who does what and with whom (e.g. adult/children)

• when (time and day) are the busy or quiet times

• whether particular groups focus on different things

• the attraction power and holding power of a site or exhibit (see formulas in section 5.4)

How to do Counting and Timing

Resources you will need:

• a stop watch or a watch with a second hand

• clip board and pen

• a form that allows you to record the relevant data (see exercise 2 in section 5.6)

• trained observers who understand the importance of being unobtrusive

5.3 Watching who does what and how

The purpose of Watching is to record the various behaviour patterns of visitors. Observers may watch for:

• who stops where

• who points to what and shows what to whom

• interaction with the displays and with other group members

• visible levels of comfort and interest

• reactions to noise, to the space

• attentiveness, especially with tour groups

• effectiveness of interpretation for people with disabilities, people with small children, people who are non-English speaking etc

• level and quality of interaction between a guide and visitors

• attracting power of a site or exhibit

• holding power of a site or exhibit

• observers may listen for –

• comments which provide clues to how visitors think or feel

• level and nature of exchange within groups (e.g. an interaction where a grandparent explains a wind up telephone to a child and, further on in the display, the child explains the working of a digital mobile phone to the grandparent)

• the content of questions visitors ask

Observation tasks should be structured to achieve one or two specific watching goals rather than multiple goals.

How do you do Watching?

As with all forms of (observational) evaluation, you need to be prepared.

• design and prepare a form on which you will record your data (see Appendix for sample form).

• use a check-off form with a section for comments

• decide what visitor behaviour you want to watch for – don’t just wander and try to record everything

• select a sample of typical subjects to observe (age, gender, group composition)

• plan to watch 10 people of a particular type (e.g. families with primary-age children), review the results then add more observations until little new material is emerging (law of diminishing return) – this way you can be confident that you have uncovered the main findings and confirmed them

• observe at various times of the day and week,

5. Observation


school holidays and non-school holidays, with and without special events, different seasons.

5.4 Attraction Power + Holding Power

Two useful indices of exhibit or site performance are derived from simple observations of visitors and from measurements of time spent.

The Attraction Power of an exhibit or site can be calculated. The number of people who stop at an exhibit or site, divided by the total number that pass it (including those who stop), expressed as a percentage, will provide a relatively reliable indication of the ‘attractiveness’ or attention grabbing power. This is a simple measure of how successfully an exhibit catches people’s attention.

e.g suppose that 60 people walk past an exhibit and 15 of them stop to look at it.

The Attraction Power is 15/60 x 100 = 25%

The Holding Power of a site or exhibit can also be calculated. The average time that people spend at an exhibit is divided by the actual time that is required to read or review it, the result is expressed as a percentage. This gives a measure of how well an exhibit holds people’s interest once they have stopped at it.

e.g. suppose you need 10 minutes to read all the text on a panel and to operate a small model attached to it.

If the average time people spend at the panel is 2 minutes,

its Holding Power is 2/10 x 100 = 20%

The higher the percentage in both Attraction and Holding Power, the better the site or exhibit.

5.5 Reporting the data

For maximum usefulness, write a brief report, covering:

1. Aim of project

2. Procedure used (methodology)

3. Main findings, for example –

• compile responses – e.g. plot all visitor movement patterns

• calculate results – e.g. length of time that people stay for, Attraction and Holding Power

• identify patterns – of behaviour, of movement, of comments

• collate observations – who does what, with whom. Collate both positive observations (e.g. popularity of particular sites or exhibits) and negative observations (e.g. some displays not seen by anyone)

• summarise new information as well as information which confirms things you already knew

4. Implications and recommendations for action

Disseminate the results to interviewers, management, front of house, guides, planners and other relevant stakeholders.

Use the results to improve things – plan changes, modify interpretation, congratulate those things/people that work well for visitors, apply for funding or sponsorship.

5. Observation


5.6 Observation Exercises

Exercise 1: Tracking, Watching and Listening

Resources you will need:

• a gallery or site with specific boundaries

• map(s) of the location with the positions of displays, exhibits and signs marked

• tracking, watching and listening form (see Appendix)

Purpose of Exercise:

• to trial observation evaluation utilising tracking, watching and listening

• to record visitor flows through a site or gallery

• to give a brief description of common behaviours

Part 1 Onsite, observers should:

• select a visitor and follow them from a distance

• record start time

• describe them on the form in space provided next to ‘Visitor A, Visitor B’ etc

• draw their path on the map as they go

• mark with numbers 1, 2, 3, 4, 5 where they stop – mark the number on the map so that it corresponds with the same number column on the form

• on the form record how long they stop at each stop

• record type of behaviour (if possible)

• record comments overheard (if possible)

• record end time

Part 2 After the observation

• note major pathways through the site, and note alternatives that were also used

• collate observations of behaviour, note major patterns (e.g. people which children tended to let the children lead)

• collate comments heard

• “interpret” these!

5. Observation


Exercise 2: Counting and Timing

This is what you need

• a gallery or site with specific boundaries

• map(s) of the location with the positions of displays, exhibits and signs marked

• counting and timing observation form (see appendix)

Purposes of Exercise:

• to measure length of time visitors spend in the gallery or at the site

• to give a demographic profile of visitors

• to give a brief description of common visitor behaviours

• to count the number of people who do things/do not do things (this is the basis for the Attraction Power and Holding Power calculations)

Part 1 Prior to survey:

• plan observation tasks, design the recording form

• plan observation positions and times

• distribute stop watches or use watches with second hands

Part 2 Onsite, observers position themselves and record the information on the form following the accompanying directions (see Counting + Timing Form in Appendix).

Part 3 After the survey

• calculate how many people visited in the time, you might arrive at a figure for ‘persons per hour on weekdays’

• using entry and exit time, calculate how long people stayed

• do a gender breakdown - how many males/

females visited

• compile a demographic profile

• browse through the tally sheets and see what patterns emerge

• e.g. do people with children stay longer

• calculate what percentage of people stayed for various timeframes (e.g. < 3 minutes, 4-10 minutes, > 10 minutes)

• calculate the average time spent

• summarise common behaviours observed

• calculate Attraction Power using the formula provided in section 5.4, p26.

• calculate Holding Power using the formula provided in section 5.4, p26

5. Observation


References

ANZECC Working Group. Best Practice in Park Interpretation and Education. www.biodiversity.environment.gov.au/protecte/anzecc/

index1.htm

Askin, M, (1999), A Revised Edition - Troy, Keskin Color Kartpostalcilik Ltd., Sti. Matbaasi, Turkey.

Borun, M., Korn, R. and Adams, R., (1999), Introduction To Museum Evaluation , American Association of Museums, Washington

Carter, J., (1997), A Sense of Place – An Interpretive Planning Handbook, Tourism and Environment Initiative, Scotland and online at www.scotinterpnet.org.uk

Diamond, Judy, (1999), Practical Evaluation Guide : Tools for Museums and Other Informal Educational Settings , Walnut Creek, California: AltaMira Press

Hall, C.M. and McArthur, S., (1996), Heritage management in Australia and New Zealand – The Human Dimension , Oxford University Press, Australia.

Ham, S. H., (1992), Environmental Interpretation – A Practical Guide for People with Big Ideas and Small Budgets , North American Press, Colorado

Kruger, R.A., (1994), Focus Groups: A Practical Guide for Applied Research, Second Edition. California: Sage Publications, Inc.

Leask, A. and Yeoman, I., (1999), Heritage Visitor Attractions – An Operations Management Perspective, Cassell, London.

Rennie, L. J. and McClafferty T., (1996) Handbook for formative evaluation of interactive exhibits. Questacon, Canberra.

Tilden, F., (1957), Interpreting Our Heritage, University of North Carolina Press.

University of Oregon, Writing Measurable Objectives. http://darkwing.uoregon.edu/~smichel/istraining/index.html

Useful websites

Evaluation and Visitor Research Special Interest Group of Museums Australia (EVR SIG). www.amol.org.au/evrsig/

Australasian Evaluation Societywww.aes.asn.au/

References


Examples and master sheets

1. Self completion questionnaire sample (p.13)

2. Interview questionnaire sample (p.14)

3. Briefing notes for interviewers (p.14)

4. Data cross analysis using a banner (p. 15)

5. Discussion guide for focus group (p.18)

6. Tracking and watching recording form (p.27)

7. Counting and timing recording form (p. 28)

Appendix

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

1. Self completion questionnaire sample

(Referred to on p.13)

This is an example questionnaire in self-completion format. It has been adapted from a questionnaire used to evaluate a number of different kinds of events at a cultural institution.

You could adapt it for your situation by omitting questions, changing items or modifying questions.

Self completion questionnaire 1

Date: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Event: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1. What was your main reason for visiting the [ place ] today? Circle ONE answer only

2. Including this visit, how many times have you visited the [ place ] in the last 2 years?

....................................................................................

3. Have you been to other events at [ place ] in the past?

Yes . . . . . . . . . . . . . . . . . . 1No . . . . . . . . . . . . . . . . . . . 2

4. What was your main reason for coming to this event today?

....................................................................................

....................................................................................

....................................................................................

5. Where have you heard about this event?

Circle as many as apply

To see the [place] in general 1

To see/go on/do [name particular event] 2

To spend time with family or friends 3

To go to the café or shop 4

To [name particular thing] 5

To [name particular thing] 6

Other (specify) 7

Information at the [ place ] 1

Talking to someone 2

Newspaper or magazine 3

Radio 4

Brochure or flyer 5

The internet 6

Can’t remember where 7

Other (specify)................................................. 8

6. Before arriving today, what did you expect this event to be like?

....................................................................................

....................................................................................

....................................................................................

....................................................................................

7. Were these expectations met?

Yes, mostly . . . . . . . . . . . .1

Yes, partially . . . . . . . . . . .2No . . . . . . . . . . . . . . . . . . .3

If ‘no’ — how was it different?

....................................................................................

....................................................................................

....................................................................................

....................................................................................

....................................................................................

8. How would you rate this event overall?

9. How would you rate each of the following aspects of the event? (Circle rating number)

Low << ============= >> High

1 2 3 4 5

Low << ======= >> High

Organisation of the event

1 2 3 4 5

Amount of information provided

1 2 3 4 5

Relevance of information given

1 2 3 4 5

Interest generated in the topic

1 2 3 4 5

Quality of the presentation 1 2 3 4 5

Suitability of the venue (room/space)

1 2 3 4 5

Self completion questionnaire 2

10. Please suggest ways to improve similar events here at [ place ]

....................................................................................

....................................................................................

....................................................................................

....................................................................................

....................................................................................

11. Would you recommend an event like this to other people?

Yes . . . . . . . . . . . . . . . 1

No. . . . . . . . . . . . . . . . 2

....................................................................................

....................................................................................

12. Do you have any suggestions for subjects for talks, tours or events at [ place ]?

....................................................................................

....................................................................................

....................................................................................

....................................................................................

13. Who did you come to the event with?

By myself . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Other family adult . . . . . . . . . . . . . . . . . . . . . 2A child/children . . . . . . . . . . . . . . . . . . . . . . .3

Non-family adult . . . . . . . . . . . . . . . . . . . . . 4Part of a tour group . . . . . . . . . . . . . . . . . . . . 5

14. Are you a member of the Friends of [ place ]?

Yes . . . . . . . . . . . . . . . 1No. . . . . . . . . . . . . . . . 2

Finally, there are some questions about you . . .

15. Were you born in –

Australia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Overseas in English-speaking country . . . . . . . . . 2Overseas in non-English speaking country. . . . . . 3

16. Which of these age groups do you come into?

Under 18. . . . . . . . . . .118-25 . . . . . . . . . . . . . 226-35 . . . . . . . . . . . . . 335-55 . . . . . . . . . . . . . 4Over 55. . . . . . . . . . . . 5

17. Gender Male. . . . . . . . . . . . . . .1

Female . . . . . . . . . . . .2

18. What is your main occupation?Skilled work/Trade/Unskilled. . . . . . . . . . . . . . . . 1

Clerical/Administration/Sales . . . . . . . . . . . . . . . 2

Professional . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Looking for work . . . . . . . . . . . . . . . . . . . . . . . . . 4

Home duties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Student . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Retired. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Other . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

19. Where do you live?

Local area. . . . . . . . . . . . . 1 Postcode . . . . . . . . . . . .Other Australia . . . . . . . . . 2 Postcode . . . . . . . . . . . Overseas. . . . . . . . . . . . . .3 Country . . . . . . . . . . . . .

20. Have you attended public events at any of the following places in the last three months? (Circle all that apply)

List some local attractions. . . . . . . . . . . . . . . . . . 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

23. Have you used any of the following facilities at the [ place ] today? How would you rate the ones you have used?

Used Not usedIntend to

use Very Poor Poor Average GoodVery good

Cafe 1 2 3 1 2 3 4 5

Shop 1 2 3 1 2 3 4 5

Public toilets 1 2 3 1 2 3 4 5

Public telephones 1 2 3 1 2 3 4 5

Carpark 1 2 3 1 2 3 4 5

Visitor Information Desk 1 2 3 1 2 3 4 5


2. Interview questionnaire sample


This is an example of a questionnaire that would be used by an interviewer. It has been adapted from a questionnaire used to evaluate an exhibition.

You could adapt it for your situation by omitting questions, changing items or modifying questions.

Onsite questionnaire 1

Hello, my name is …… and I am doing a survey of visitors to the Botanic Gardens.May I ask you a few questions? [It will take about 10 minutes.]

1. Is this your first visit to the Botanic Gardens?

Yes . . . . . . . . . . . . . . . . . . 1No . . . . . . . . . . . . . . . . . . . 2

2. What was your main reason for visiting the Gardens today?

Don’t read out. Circle ONE number only.

To see the Gardens in general . . . . . . . . . . . . . . . . . . . . .1To see a particular part of the Gardens . . . . . . . . . . . . . . 2To visit the shop/cafe . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

Just passing through . . . . . . . . . . . . . . . . . . . . . . . . . . . .4Other (specify) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3. Did you come by yourself or with other people? By self. . . . . . . . . . . . . . . .1 skip to 6

With others . . . . . . . . . . . . 2

4. Including yourself how many people are in your group today? Record number . . . . . . . . . . . . . . . . . . . . .

5. Which of the following parts of the Garden have/will you

Thinking now about [particular part of the Gardens] —

6. Which of the following statements applies to you? Circle one only

I have looked closely at most things . . . . . . . . . . . . . . . . . . . . . . . . . . .1I have looked at quite a lot, but skipped past some things . . . . . . . . . .2

I had a general look but didn’t look closely at many things . . . . . . . . . .3

7. What are the main things this part of the Gardens is trying to show? Probe for content

.............................................................................................................................................................................................

.............................................................................................................................................................................................

.............................................................................................................................................................................................

.............................................................................................................................................................................................

8. And on a scale of 1 to 10, where 1 is terrible and 10 is excellent, what score would you give it for —

Score Comments

Layout

Amount of information

Ability to hold your interest

Beauty

Onsite questionnaire 2

9. How would you rate this part of the Gardens overall? Very good . . . . . . . . . . . . .1

Would you say it was — Good. . . . . . . . . . . . . . . . . 2

Poor . . . . . . . . . . . . . . . . . 3Very poor . . . . . . . . . . . . .4DK. . . . . . . . . . . . . . . . . . . 5

10.

11. Was there anything that you LIKED in particular about this garden? Probe

.............................................................................................................................................................................................

.............................................................................................................................................................................................

.............................................................................................................................................................................................

.............................................................................................................................................................................................

12. And was there anything that you DISLIKED in particular? Probe

.............................................................................................................................................................................................

.............................................................................................................................................................................................

.............................................................................................................................................................................................

.............................................................................................................................................................................................

Now, I have a couple of questions about you —

13. What is the postcode of your home address? Postcode . . . . . . . . . . . . . . . . . . . .

Country . . . . . . . . . . . . . . . . . . . .

14. What year were you born in? . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Thank you for your help

15. Record gender Male . . . . . . . . . . . . . . . . . . . . . . . 1

Female . . . . . . . . . . . . . . . . . . . . .2

Time of interview: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Date: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Name of interviewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Agree strongly

Agree mildly

Disagree mildly

Disagree strongly

Don’t know

The display added a lot to what I know about this topic 4 3 2 1 0

The display presented information in an interesting way 4 3 2 1 0

4 3 2 1 0

The display ... 4 3 2 1 0

4 3 2 1 0


3. Briefing notes for interviewers


All inexperienced interviewers should be given a training session based on these notes.

1

Practical Guide to Evaluating Natural and Cultural Heritage

Interviewer Notes1

What is Market Research?

Market research is the business of gathering facts, ideas or opinions. Organisations are continually developing new concepts and ‘products’ which rely on effective communication to reach their markets, consumers or audiences. To be successful over long periods, organisations realise they must keep in touch with their customers, users, visitors or audiences.

Market research provides

• A means by which people providing goods or services can find out what members of the public think about the products being offered for sale, the advertising, the quality of the services offered and so on. This can apply to both the private sector (businesses) and the public sector (government).

• An opportunity for the members of the public to give their opinions and ideas back to the suppliers of these goods and services.

Survey Research

There are different types of Market Research. The project we are conducting is called Survey Research. Survey Research involves collecting facts, ideas or opinions from a large number of people.

When is survey research used? Surveys are conducted when information needed cannot be collected more accurately or less expensively by any other method. For Example: An airline wants to know how many people use their planes to fly between Melbourne and Sydney. It does not need to do a survey to find out. A simple count of tickets sold will give the answer.

If the same airline wants to know what air travellers think about in-flight service between Melbourne and Sydney then a survey is needed. We would talk to a group of people who had made the trip recently and ask for their opinions and suggestions. To record their opinions and suggestions we would use a standard form, a list of questions, the questionnaire.

The specific group of people we talk to for that survey would be referred to as the sample .

By talking to a relatively small group of people we get results that represent the opinions of the total population under study. Just as you can know what a pot of soup tastes like by sampling a few spoonfuls, so you can find out what users think by interviewing a sample of them.

1. Source: Adapted from Market Research Society of Australia Interviewer Training Materials

2


All surveys have specific purposes, for example to give answers to problems, to measure the size of a market, to measure awareness of advertising, to establish attitudes to products, advertising or social issues, or to measure the effectiveness of programs.

While survey objectives may vary, the basic methods of collecting the information are standardised.

Standard Interviewing Techniques

Today we will take you through some of the standard procedures of interviewing, and outline the skills you will develop in becoming a professionally competent interviewer.

Where do you fit in? The Interviewer is vital to the success of Market Research as a profession. Every survey is only as good as the interviewing that is done to collect the information.

This information can be coded, punched into the newest computers, run on very sophisticated computer programs and the results looked at by the best brains in the country, but if the interviewing is not of the highest standard, then the whole study is wasted.

Your central role means that the way you do each interview is vitally important.

Reliability

• You must interview the right people

• You must get complete and accurate information

• You must work to a schedule.

• Your interviews will be checked to ensure company standards are met.

The interviewing role

As interviewers, you will play three roles:

• The first role is that of the ’technician’' who applies standard techniques and uses the same questionnaire for each interview.

• The second role is that of a human being who builds up a suitable relationship with each respondent. This enables you to 'record' and act as a tape-recorder to relay information to the survey clients and so help them provide better services.

• The third role is as a caring person. You should conduct the interview in such a way that the respondents feel good about the encounter. You can help them with any queries they may have.

3


Basic Rules of Interviewing

• Use a separate questionnaire for each respondent.

• Do not interview people who you know personally, or people who have been present while someone else was interviewed.

• Approach people with a confident, smiling attitude. Create a pleasant, friendly rapport.

• If people refuse to be interviewed, simply thank them and look for someone else. Don't let a refusal affect your positive approach to the next respondent.

• Be careful not to suggest answers to respondents. We want their views, not yours. If they ask you a question, avoid answering it. At the end of the questionnaire you can have an exchange of views with the respondent if you like.

• Do not show the questionnaire to respondents, position yourself so that it is not easy for them to read it.

• Wear your identification badge while you are approaching respondents and interviewing.

The questionnaire

• As far as possible, you should read the questions exactly as written. As far as possible, all interviews should be the same, so all interviewers should follow the questionnaire in the same way. Ask every question (except when following 'skip' instructions), follow the order of the questionnaire.

• If a respondent does not understand a question, repeat it exactly as it is written on the questionnaire.

• There is no point in conducting poor quality interviews. We need high quality information, and you are the person who must be careful and thorough to get the best information.

The Approach

Approach everyone who you think has looked through the exhibition. Approach all age groups and types of people. Let them see you approach them, make eye contact, smile and use a suitable greeting, introduce yourself.

For longer questionnaires you will need to ask whether the respondent has the time to answer your questions. If they have, ask the screening questions then sit down with them and begin. Try to interview your respondent separately from their companions.

4


Types of questions

Introduction

You introduce yourself to the respondent.

Screening questions

Check you are speaking to the right person. Ask these before you sit down.

Pre-coded questions

These are questions where the answers are written on the questionnaire. You circle the code number beside the answer which the respondent gives. If you need to change an answer, cross it out clearly and circle the right number.

These questions can be either 'single response' where you circle one answer only, or 'multiple response' where you can circle more than one number.

Open-ended questions

For these questions, you need to record word for word what the respondent says, for example, “I like to go swimming every day”. Use abbreviations if necessary but be prepared to make the writing clearer after the interview.

Be sure the respondent is actually answering the question. If the question is “What kind of art do we have here?” the respondent might say “I don’t really mind what kind of art, I like all art.” While this might be interesting to know, it does not answer the question. You will need to ask the question again, using the same words. You might affirm that you have heard what the respondent has said, by saying “You like all art” and then repeat the question.

Probing All open ended questions need probing, that is getting more information without influencing the respondent in any way. It is important not to suggest answers, or to lead respondents towards an answer in any way.

Probing is the most challenging, important and interesting part of interviewing. The quality of an interview depends a great deal on your ability to probe successfully.

Have patience, listen and be interested in what the respondent has to say.

There are three types of probing:

• to help respondents enlarge on what they have said

• to help respondents clarify what they have said

• to help respondents explain the reasons behind what they have said.

5


To enlarge – “Where else do you know of?” “Do you know of any other places?” You keep asking these kinds of questions until you have gathered all the places they know of, not just the places they thought of first.

To clarify – Particularly when they give general answers, like 'souvenirs' as a response to a question like, “What things do you want to buy in Australia?” Here you would probe like this:

Q: “What kind of souvenirs would you like to buy?”

A: “Small things, Australian things.”

Q: “What do you mean 'small things'”?

A: “Well, things like scarves and ties, or maybe macadamia nuts.”

Q: “And what do you mean by 'Australian things'”?

A: “Macadamia nuts, or t-shirts, or well, anything with Australia, or Sydney written on it I suppose.”

Other probes might be “What do you mean by .............?”

This kind of questioning will take you from the general to the specific.

Specific information is high-quality information.

To explain – when we want to know why they say something, a probe might be “Why do you feel unsafe in the streets at night?” “Are there any other reasons that you feel unsafe?”

'Skip' instructions

Where there are no 'skip' instructions always go to the next question. 'Skip' instructions allow respondents to skip questions which do not apply to them.

Show cards

Show cards are sometimes used to give respondents a range of possible answers. They are presented in a book or folder which you might like to think of as an “answer book”. Of course, all the answers in this book are right, there are no wrong answers.

You may let the respondent read each page, or help them by reading out the longer ones.

6


Rating Scales

These are used to measure the strength of a respondent's reply. For example:

Agree stronglyAgree a littleNot sureDisagree a littleDisagree strongly

If a respondent says 'disagree', ask: Would that be 'disagree a little' or 'disagree strongly'? and circle the appropriate answer.

Demographic data

It is vital that these questions be answered correctly. They are used to cross-analyse the other questions on the survey to look for differences in response between various groups of people, men compared with women, for example.

Clerical Procedures

• Fill in questionnaires accurately. Write quickly in pencil while the interview is underway. At the end of the interview, quickly check through the questionnaire to make sure you have recorded everything accurately and clearly.

Your health

Prepare to be out in the sun for long periods of time. To avoid sunburn, wear a hat, long sleeves and trousers, and take a 15+ sunscreen lotion with you.

Public Relations

It is our experience that people are happy to be interviewed about the things they like to do. People in a strange place often need information. Have a city map with you, and know where the nearest toilets and information centre are. Be prepared to be helpful to the tourists you encounter (without spending too much time).

Contacts

Please feel free to call your supervisor on the following phone numbers, if you have any queries, or if you want to clarify anything.


4. Data cross analysis using a banner

(Referred to on p. 15)

This is an example of a table of data which has been cross-analysed by a standard banner. In this case, the banner includes: sex, age, area, number of adults in household and no. of children in household. It is taken from the LeisureScope survey of Environmetrics Centre for Visitor Studies.

We see that 6.5% of the total population had been to the Powerhouse Museum in the previous 6 months, compared with 9.7% of people aged 35-49 years.

A higher proportion of people with children under 12 years had been (7.3%) compared with people with teenagers (4.4%).


5. Discussion guide for focus group


This is an example of a discussion guide used to evaluate a guided tour in a botanic garden.


Example discussion guide

Introduce self and general topic.

One person at a time, quieter speak up, your views are important, no right or wrong answers. [Mention recording/mirror if applicable.]

Ask each person to introduce themselves: name, where they live, family circumstances, hobbies and interests.

Previous experience of guided walks.

Why did you decide to take this guided walk?

You went on two different walks, can you describe for us what your walk was about?

What were the main things they were telling you?

What were the highlights of the walk for you?

What kind of person would you recommend this walk for?

Task: Order of importance for guide skills.

Discuss your guide with respect to these skills.

Suggestions for improving the walk you went on.

Suggestions for other kinds of walks that they could offer.


6. Tracking and watching recording form


When doing a tracking study, you use a fresh form for each party you are following.

You record their start and finish times and trace their path on the floorplan.

There is space to record details of the group composition and observations about behaviours and activities.


7. Counting and timing recording form

(Referred to on p. 28)

When counting and timing people at a particular activity, you use this form. It allows for 20 different observations.

For each person you observe, record a brief description of them, note the time they spend at the activity and note any specific behaviours.

Documents

Exit Interviews Observation Methods Focus Group Discussions · A Practical Guide to Evaluating Natural and Cultural Heritage Interpretation Exit Interviews Observation Methods Focus