1

Evidencing Impact and Value

Alison O’Hara

Chief Executive

Audiences North East

2

Impact

Image: courtesy of Hales Gallery, London and I-20 Gallery, New York

3

Definitions

• Impact: the effect or impression of one thing against another

• Economic impact: Radich(1987), the economic impact of a given phenomenon can be defined as ‘the effect of that phenomenon on such economic factors as the economic behaviour of consumers, businesses, the market, industry, the economy as a whole, national wealth or income, employment and capital’

• Social impact: Landry et al (1993), ‘those effects that go beyond the artefacts and the enactment of the event or performance itself and have a continuing influence upon, and directly touch, people’s lives’

4

Value

Image: courtesy of BePak

5

Definitions

• Value: worth in usefulness or importance, utility or merit

• Matarasso (1996), which value systems are used to provide benchmarks against which work will be measured, and about whom defines quality, value and meaning

• Social and economic rationales for the arts, with their emphasis on the arts as a means to other ends will serve to devalue arts for its own sake eg John Tusa

6

Current drivers for measuring impact and value

• Economic downturn– Local and national governments invest public

money in cultural opportunities. How do we know if the money is well spent or wasted? Does a £1 invested by government deliver more than a £ in benefit?

• The Big Society

7

Cuts in public investment

• DCMS – 25% cut for 2011/12 to 2014/15• ACE cuts – 29% cut, 6.9% cut for majority

of RFOs for 2011/12 and reduction of 14.9% to overall budget available for RFOs for 4-year settlement period

• National museums – 15% cut and remain free to enter

• English Heritage – 32% cut• Visit Britain – 34% cut

8

The Big Society

• www.cabinetoffice.gov.uk Building the Big Society

• Five priorities:– Give communities more power– Encourage people to take an active role in their

communities– Transfer power from central to local government– Support co-ops, mutuals, charities and social

enterprises– Publish government data

9

Measuring Intrinsic Value

• Measuring Intrinsic Value: How to stop worrying and love economics (2009) by Hasan Bakhshi, Alan Freeman and Graham Hitchen

Image: Jaipur Kawa Brass Band, Durham BRASS Fest

10

Measuring Intrinsic Value

• Recommends cultural sector embracing the need to measure economic impact

• Cultural world will back itself into a corner if it claims immunity from measuring economic impact when money could be spent on other things

• Even though it’s difficult to measure, it’s not really optional – governments choose between alternative expenditures

11

Intrinsic v instrumental

• Intrinsic value (own merits) v instrumental value (non artistic side effects eg social inclusion, crime prevention and learning)

• Proper assessment of intrinsic value• Public’s own valuation of the arts – contingent

value (CV) and willingness to pay (WTP)• Ask the public what they would be prepared to

pay, faced with a choice of spending the money on something else

12

Culture and Sport Evidence Programme (CASE)

• Vision is to influence culture and sports’ policy development and policy agenda through the development of strategic, policy-relevant, high-quality, cross-cutting, social and economic evidence base for culture and sport sectors– Current research– Research needs to provide– Tools to collect research

13

CASE

• Publication– Understanding the drivers,

impact and value of engagement in culture and sport, July 2010 www.culture.gov.uk

• Database– www.impact.arts.gla.ac.uk/

c5,800 individual studies or reviews on the drivers, impact and value of engagement in culture and sport, advanced search terms, links to reports

Image: The Alnwick Garden

14

Benefits of engaging in culture and sportIndividual engager

Achievement

Continuity with the past

Creativity

Diversion

Enjoyment

Escape

Expression

Health

Income

Inspiration

Knowledge of culture

Self-esteem

Self-identify

Skills/competency

Solace/consolation

Community

Bequest value

Community cohesion

Community identity

Creativity

Employment

Existence value

Innovation

Option to use

Productivity

Reduced crime

Shared experience

Social capital

National

Citizenship

International reputation

National pride

15

CASE publications

• Measuring short-term private benefit of engagement– Subjective well-being measures, two-step approach

• First survey data is used to estimate how a person’s SWB changes when they engage in culture and sport

• This change in SWB is valued monetarily using the ‘income compensation approach’ ie the analysis estimates the increase in SWB generated by an increase in income

– Deriving economic values from SWB• The income compensation approach can be used to convert

estimates of the SWB effect of policy outcomes such as engagement in culture and sport into estimates of monetary value of these policy outcomes.

16

CASE publications

• Measuring long-term public benefit of engagement– Due to data and evidence constraints work

was focused on the health gains associated with doing sport

• However, CASE regional insights have been developed to provide partners with data and evidence on the position and role of culture and sport within key local and regional agendas

17

CASE database

18

Impact and value frameworks

• Arts Council England - self evaluation framework

• MLA - Inspiring Learning • Local Government Improvement and

Development (formerly IDeA)• Audiences London – festivals and outdoor

events• New Economics Foundation – theatre• Film Council – impact of local cinemas

19

Arts Council England – self evaluation framework

• Online flexible development tool to support arts organisations in evaluating their own performance and to help them inform their future planning, July 2010

• www.artscouncil.org.uk/selfevaluation/background-self-evaluation/

Image: Constellation by Kiki Smith, National Glass Centre

20

Arts Council England – self evaluation framework

• Six key areas:– Vision– External environment– Artistic aspirations and programme– Participation and engagement– Organisational capacity and capability– Business model

• Each area has three strands:– Topic – a breakdown of the key area into smaller areas of focus– What success looks like – a high-level description of an organisation

that is delivering each topic effectively– Questions you might ask yourselves – types of questions you might ask

to evaluate your strengths and weaknesses relating to that topic– Resources – website links

21

MLA - Inspiring Learning

• A self-help improvement framework for museums, libraries and archives, 2008

• Inspiring Learning supports organisations to:– Assess their strengths and plan improvements – Provide evidence of the impact of their activities

through the generic learning and generic social outcomes

– Improve their strategic and operational performance 

• www.inspiringlearningforall.gov.uk/

22

MLA - generic learning outcomes

• The generic learning outcomes are underpinned by a broad definition of learning which identifies benefits that people gain from interacting with museums, libraries and archives

• www.inspiringlearningforall.gov.uk/toolstemplates/genericlearning/

Image: Killhope

23

MLA - generic learning outcomes

• GLO checklist• Recording and

analysing qualitative and quantitative data

• GLO coding• Templates

24

MLA - generic social outcomes

• Case studies, guidance and tools to support museums, libraries and archives in planning how they deliver their services and in measuring their contribution to social outcomes

• Step-by-step approach to using the guidance, sample questionnaires and case studies

• www.inspiringlearningforall.gov.uk/toolstemplates/genericsocial/index.html

Image: Sunderland Museum and Winter Gardens

25

MLA - generic social outcomes• Three outcome areas that

relate to government policy priorities

• Look at organisation’s high level aims

• Identify which audiences are priorities to target

• Consider how your project/service contribute to one or more of the GSOs

• Decide what evidence you need to gather

26

Local Government Improvement and Development

• Formerly IDeA new web resource on how to create a local outcomes framework for culture and sport, September 2010

• Local outcomes framework will help you measure and evidence the contribution culture and sport provision makes to better outcomes for your area

• www.idea.gov.uk/idk/core/page.do?pageId=21649171

27

Local outcomes framework - process

28

Local outcomes framework - resources

• Guidance on what a local culture and sport outcomes framework can help you do

• What should an outcomes framework include:– Examples of outcomes triangles: children and young

people, economy, environment, health and well-being, older people, safer communities, stronger communities

– Logic model templates– Evidence template– Performance indicators template

29

Audiences London – festivals and outdoor events

• Information sheets available to download, September 2010

• Key points to consider when trying to measure and evidence success or impact of a festival

• www.audienceslondon.org/1891/our-resources/advice-for-audience-research-at-festivals-and-outdoor-events.htmlImage: Stockton International

Riverside Festival

30

Audiences London – festivals and outdoor events

• Information sheets:– Evidencing success– Methods– Using questionnaires– Sampling– Working with volunteers– Research guidelines and data protection– Measuring economic impact

31

New Economics Foundation - theatre

• Capturing the audience experience – a handbook for the theatre

• Presents a new model for describing the audience experience with standard survey templates and guidance on how to use them

• Value – explores what it is about a good theatre experience that makes it ‘worth coming out for’

• www.itc-arts.org/uploaded/documents/Theatre%20handbook.pdf Image: Theatre Royal

32

The Audience Experience Framework

33

Film Council - impact of local cinemas

• Impact of local cinema – five case studies, 2005

• Suggested tools for use in further cinema impact studies

• Methodological notes• www.ukfilmcouncil.org.uk/

10022?page=1&step=10&viewby=category&value=17004

Image: Tyneside Cinema

34

Film Council - case studies• Desk research

– No. of screens and seats– Type of programme– Annual ticket sales– Revenue from food/drink/merchandising, advertising, project funding,

other income– Annual expenditure– Type of location and town– Population of town

• Site visit• Depth interviews with cinema manager and staff• Focus group with cinema audience• Phone interviews with cinema’s local suppliers, community groups

and educational organisations, local council and local press

35

Film Council - case studies report

• The social, cultural and environmental impact of local cinema

• The impact of local cinema on the local economy

• Appendix – tables which analyse information (quantitative and qualitative) by impact eg economic, social, cultural, environmental

36

Film Council - suggested tools for further research

• Pack of research materials:– Templates for data collection– Topic guides for interviews with stakeholders– Briefing notes for cinema staff– Monitoring form– Draft letters– Forms for measuring spending

37

Impact and value frameworks – the highlights

• Arts Council England - self evaluation framework

• MLA - Inspiring Learning • Local Government Improvement and

Development (formerly IDeA)• Audiences London – festivals and outdoor

events• New Economics Foundation – theatre• Film Council – impact of local cinemas